Basicintroduction
Thekernelisthecoreofanoperatingsystem.Itisthefirst-layersoftwareexpansionbasedonhardware,whichprovidesthemostbasicfunctionsoftheoperatingsystem.Itisthebasisfortheworkoftheoperatingsystem.Itisresponsibleformanagingthesystem’sprocesses,memory,devicedrivers,filesandnetworksystems,anddeterminestheperformanceandperformanceofthesystem.stability.
Inthemodernoperatingsystemdesign,inordertoreducetheoverheadofthesystemitself,itisoftencloselyrelatedtothehardware(suchasinterrupthandlers,devicedrivers,etc.),basic,common,andhigheroperatingfrequency.Themodules(suchasclockmanagement,processscheduling,etc.)andkeydatastructuresareseparatedtomakethempermanentinmemoryandprotectthem.Usuallythispartiscalledthekerneloftheoperatingsystem.
Programscanbedirectlytransferredtothecomputerforexecution.Thisdesignshowsthatthedesignerdoesnotwanttoprovideanyhardwareabstractionandoperatingsystemsupport.Itiscommoninthedesignofearlycomputersystems.Finally,someauxiliaryprograms,suchasprogramloaderanddebugger,aredesignedintothecoreofthemachine,orsolidifiedinread-onlymemory.Whenthesechangesoccurred,theconceptoftheoperatingsystemkernelgraduallybecameclear.
(Summarypicturesource:)
Historicaldevelopment
ThefirstpublicversionofLinuxwasversion0.02inOctober1991.Twomonthslater,InDecember1991,Linuxreleasedversion0.11,whichwasthefirstindependentkernelthatcouldbeusedwithoutrelyingonMinix.
Onemonthafterthereleaseofversion0.12,inMarch,theversionnumberjumpedto0.95,reflectingthatthesystemwasbecomingmature.Notonlythat,itwasamilestoneuntiltwoyearslater,inMarch1994.1.0.0wascompleted.
Fromthenon,westartedtousethetwo-waynumberingmethodtomarkthedevelopmentofkernels.Even-numberedkernels(suchas1.0,2.2,2.4,2.6)arestable,andthe"product"model,atthesametime,Theodd-numberedkernelversions(1.1,2.3)arecutting-edgeor"developing"kernels.Afewmonthsafterthereleaseofastablekernel,thedevelopmentofanewkernelwillbegin.However,thedevelopmentof2.5didnotstartuntildozensofmonthsafterthecompletionof2.4.
Mostofthediscussioncontentofthepost-halloweendocumentisthemainchangesthatusersneedtopayattentionto,andthesystemtoolsthatneedtobeupdated(inordertotakeadvantageofthem).ThosewhocareaboutthisinformationaremainlyLinuxdistributorswhowanttoknowinadvancewhatisinthe2.6kernel,aswellasendusers,whichallowsthemtodeterminewhetherthereareprogramsthatneedtobeupgradedinordertotakeadvantageofthenewcomponents.
TheKernelJanitorsprojectmaintainsalistofminordefectsthatneedtobefixedandsolutions.Mostofthesedefectsolutionsarecausedbytheneedtochangemanypartsofthecodewhenapplyinglargerpatchestothekernel,forexample,someplaceswillaffectthedevicedriver.Thosewhoarenewlyengagedinkerneldevelopmentcanchoosetheitemsinthelistatthebeginningoftheirwork,sothattheycanlearnhowtowritekernelcodethroughsmallprojectsandhavetheopportunitytocontributetothecommunity.
Also,inanotherpre-releaseproject,JohnCherrytrackedtheerrorsandwarningsfoundwhencompilingeachreleasedkernelversion.Thesecompilationstatisticshavebeendecliningovertime,andpublishingtheseresultsinasystematicformmakestheprogressachievedataglance.Inmanycases,someofthesewarningsanderrormessagescanbeusedliketheKernelJanitorslist,becausecompilationerrorsareusuallycausedbysmalldefectsandrequiresomeefforttofix.
Finally,thereisAndrewMorton's"must-fix"list.Sincehehasbeenselectedasthemaintainerofthe2.6kernelaftertherelease,heusedhisprivilegestooutlinetheissuesthathebelievesmosturgentlyneedtoberesolvedbeforethefinal2.6kernelisreleased.Themust-fixlistcontainsthedefectsinthekernelBugzillasystem,thepartsthatneedtobecompleted,andotherknownissues.Iftheseissuesarenotresolved,itwillhinderthereleaseof2.6.Thisinformationcanhelpindicatewhatstepsareneededbeforethenewkernelisreleased;itcanalsoprovidevaluableinformationforthosewhoareconcernedaboutwhenthemuch-anticipated2.6kernelreleasewillbecompleted.
Kernelclassification
Singlekernel
Monolithickernelisaverybigprocess.Itsinteriorcanbedividedintoseveralmodules(orhierarchicalorother).Butwhenitisrunning,itisasinglebigbinaryimage.Thecommunicationbetweenthemodulesisachievedbydirectlycallingfunctionsinothermodules,ratherthanmessagetransmission.
Thesingle-corestructuredefinesahigh-levelabstractinterfaceontopofthehardware,andusesasetofprimitives(orsystemcalls)toimplementthefunctionsoftheoperatingsystem,suchasprocessmanagement,filesystem,andstorageManagementandsoon,thesefunctionsarecompletedbymultiplemodulesrunninginthecorestate.
Althougheachmoduleservestheseoperationsseparately,thekernelcodeishighlyintegratedanddifficulttowritecorrectly.Becauseallmodulesarerunningonthesamekernelspace,asmallbugcancrashtheentiresystem.However,ifthedevelopmentgoeswell,thesingle-corestructurecanbenefitfromoperatingefficiency.
Manymodernsingle-kernelstructurekernels,suchasLinuxandFreeBSDkernels,cancallmodulesintoexecutionatruntime,whichcanmakeiteasiertoexpandthefunctionsofthekernel,anditcanalsomakethecoreofthekernel.Partbecomesmoreconcise.
Single-corestructureisaveryattractivedesign,duetothecomplexityofthesystemcontrolcodethatimplementsalllow-leveloperationsinthesameaddressspace,andtheefficiencywillbehigherthanthatindifferentaddressspacessome.Thesingle-corestructureistendingtobeeasytodesigncorrectly,soitsdevelopmentwillbefasterthanthemicro-kernelstructure.
Anexampleofasingle-kernelstructure:thetraditionalUNIXkernel-suchastheversionreleasedbytheUniversityofBerkeley,theLinuxkernel.
Microkernel
Themicrokernel(Microkernelkernel)structureconsistsofaverysimplehardwareabstractionlayerandasetofkeyprimitivesorsystemcalls.TheseprimitivesonlyincludetheestablishmentofThenecessarypartsofasystem,suchasthreadmanagement,addressspace,andinter-processcommunication.
Thegoalofmicrokernelistoseparatetherealizationofsystemservicesfromthebasicoperatingrulesofthesystem.Forexample,theinput/outputlockingserviceoftheprocesscanbeprovidedbyaservicecomponentrunningoutsidethemicrokernel.Theseverymodularuser-modeserversareusedtocompletemoreadvancedoperationsintheoperatingsystem.Thisdesignmakesthedesignofthecorepartofthekerneleasier.Thefailureofaservicecomponentwillnotcausetheentiresystemtocrash.Whatthekernelneedstodoistorestartthiscomponentwithoutaffectingotherparts.
ThemicrokernelputsmanyOSservicesintoseparateprocesses,Suchasfilesystem,devicedriver,andtheprocesscallsOSservicesthroughmessagepassing.Themicro-kernelstructuremustbemulti-threaded.Thefirst-generationmicro-kernelprovidesmoreservicesinthecore,soitiscalledthe"fatmicro-kernel",anditstypicalrepresentativeisMACH.ItisbothGNUHURDandthecoreofAPPLESERVEROS.Itcanbesaidtobethriving.ThesecondgenerationprovidesonlythemostbasicOSservicesforthemicrokernel.ThetypicalOSisQNX.QNXiswell-knowninthetheoreticalworldandisconsideredanadvancedOS..
Themicrokernelonlyprovidesasmallpartofthehardwareabstraction,andmostofthefunctionsareperformedbyaspecialuser-modeprogram:theserver.Micronucleiareoftenusedintheembeddeddesignofrobotsandmedicaldevices,becausethekeypartsoftheirsystemsareinseparate,protectedstoragespaces.Thisisimpossibleforasingle-coredesign,evenifitusesthewayofloadingmodulesatruntime.
Examplesofmicrokernels:AIX,BeOS,L4microkernelseries,.MachisusedinGNUHurdandMacOSX,Minix,MorphOS,QNX,RadioOS,VSTa.
Hybridkernel
Thehybridkernelisverysimilartothemicro-kernelstructure,exceptthatitscomponentsrunmoreinthecorestatetoobtainfasterexecutionspeed.
Thehybridkernelisessentiallyamicrokernel,butitallowssomemicrokernelstructuretorunintheuserspacecodetoruninthekernelspace,whichmakesthekernelmoreefficient.Thisisacompromise.Thedesignerreferstothetheorythatthemicrokernelstructureofthesystemrunspoorly.However,laterexperimentsprovedthatapuremicrokernelsystemcanactuallybehighlyefficient.Mostmodernoperatingsystemsfollowthisdesigncategory,andtheWindowsoperatingsystemdevelopedbyMicrosoftisagoodexample.ThereisalsoXNU,thekernelrunningonApple'sMacOSX,whichisalsoahybridkernel.
Examplesofmixedkernels:BeOSkernel,DragonFlyBSD,ReactOSkernel
WindowsNT,Windows2000,WindowsXP,WindowsServer2003andWindowsVistaandotheroperatingsystemsbasedonNTtechnology.
Outercore
Outercoresystem,alsoknownasverticalstructureoperatingsystem,isamoreextremedesignmethod.
Theexternalkerneldoesnotprovideanyhardwareabstractoperations,butallowsadditionalruntimelibrariestobeaddedtothekernel,throughwhichapplicationscandirectlyornearlydirectlyoperateonthehardware.
Itsdesignconceptistoletthedesigneroftheuserprogramdecidethedesignofthehardwareinterface.Theouterkernelitselfisverysmall,anditisusuallyonlyresponsibleforservicesrelatedtosystemprotectionandsystemresourcereuse.
Traditionalkerneldesigns(includingsingle-coreandmicro-core)haveabstractedthehardware,hidinghardwareresourcesordevicedriversunderthehardwareabstractionlayer.Forexample,inthesesystems,ifapieceofphysicalstorageisallocated,theapplicationdoesnotknowitsactuallocation.
Thegoaloftheexternalcoreistoallowtheapplicationtodirectlyrequestaspecificphysicalspace,aspecificdiskblock,andsoon.Thesystemitselfonlyguaranteesthattherequestedresourceiscurrentlyfree,andtheapplicationallowsdirectaccesstoit.Sincetheexternalcoresystemonlyprovidesrelativelylow-levelhardwareoperations,anddoesnotprovidehigh-levelhardwareabstractionlikeothersystems,thenadditionalruntimelibrarysupportisneeded.Theseruntimelibrariesrunontheoutercoreandprovidecompletefunctionsforuserprograms.
Intheory,thisdesignallowsvariousoperatingsystemstorunonanexternalcore,suchasWindowsandUnix.Andthedesignercanadjustthefunctionsofeachpartofthesystemaccordingtotheoperatingefficiency.
Theoutercoredesignisstillintheresearchstage,andnocommercialsystemhasadoptedthisdesign.Severalconceptualoperatingsystemsarebeingdeveloped,suchasNemesisfromtheUniversityofCambridge,theCitrixsystemfromtheUniversityofGlasgow,andasystemfromtheSwissAcademyofComputerSciences.MITisalsoconductingsuchresearch.
Comparisonofsingle-coreandmicro-kernel
Single-corestructureisaveryattractivedesign,becauseallcomplexlow-leveloperatingsystemcontrolsareimplementedinthesameaddressspaceTheefficiencyofthecodewillbehigherthantheimplementationinadifferentaddressspace.
Intheearly1990s,thesingle-corearchitecturewasconsideredobsolete.ThedesignofLinuxasasingle-kernelstructureinsteadofamicrokernelhascausednumerouscontroversies.
Single-corestructuretendstobelesserror-proneindesign,soitsdevelopmentwillbefasterthanmicro-kernelstructure.Therearesuccessfulcasesinbothcamps.
AlthoughMachisawell-knownmulti-purposemicrokernel,peoplehavedevelopedseveralothermicrokernels.L3isademonstrativekernel,justtoprovethatthemicro-kerneldesignisnotalwayslowspeed.ItssubsequentversionL4canevenusetheLinuxkernelasoneofitsprocesses,runninginaseparateaddressspace.
QNXisamicrokernelsystemthathasbeendesignedsincethe1980s.ItisclosertotheconceptofmicrokernelsthanMach.Itisusedinsomespecialfields;inthesecases,systemfailureduetosoftwareerrorsisnotallowed.Forexample,themanipulatoronthespaceshuttle,andthemachinethatgrindsthelensofthetelescope,alittlemistakecancausethousandsofdollarsinlosses.
ManypeoplebelievethatbecauseMachcannotsolvesomeoftheproblemsthatwereaddressedwhenthemicrokerneltheorywasputforward,themicrokerneltechnologyisuseless.FansofMachindicatethatthisisaverynarrowview,butunfortunatelyitseemsthateveryoneisbeginningtoacceptthisview.
Advantages
Abstracthiding
Thekernelprovidesahardwareabstractionmethodtocompletehardwareoperations,becausetheseoperationsareverycomplex,andhardwareabstractionhidesComplexityprovidesasetofconciseandunifiedinterfacesforapplicationsoftwareandhardware,makingprogramdesigneasier.
SourceCodeManagement
Inhistory,therehasneverbeenaformalsourcecodemanagementorrevisioncontrolsystemfortheLinuxkernel.Infact,manydevelopershaveimplementedtheirownmodifiedcontrollers,butthereisnoofficialLinuxCVSarchivethatallowsLinusTorvaldstochecktheaddedcodeandallowotherstoobtainthecode.Fixingthelackofcontrollersoftencreatesa"generationgap"betweenreleaseversions.Noonereallyknowswhatchangeshavebeenadded,whetherthesechangescanbewellintegrated,orwhatnewcontentisworthlookingforwardtointheupcomingrelease..Generally,ifmoredeveloperscanunderstandthosechangesastheyunderstandthechangestheyhavemade,certainproblemscanbeavoided.
Itisverynecessarytouseareal-time,centralizedarchivetosavethelatestupdatestotheLinuxkernel.Everychangeorpatchacceptedbythekernelistrackedasachangeset.Endusersanddeveloperscansavetheirownsourcefilearchivesandupdatethemwiththelatestsetofchangeswithasimplecommandasneeded.Fordevelopers,thismeansthatyoucanalwaysusethelatestcopyofthecode.Testerscanusetheselogicalchangesetstodeterminewhichchangescausedtheproblemandshortenthetimerequiredfordebugging.Evenuserswhowishtousethelatestkernelcandirectlyusethereal-time,centralizedarchive,becauseoncethepartsordefectfixestheyneedareaddedtothekernel,theycanupdateitimmediately.Whenthecodeisintegratedintothekernel,anyusercanprovideimmediatefeedbackanddefectreportsonthecode.
Paralleldevelopment
Kernelstructurediagram(6photos)
AstheLinuxkernelgrows,itbecomesmorecomplexandattractsmoreDevelopersfocusedtheirattentiononthespecializeddevelopmentofspecificaspectsofthekernel,andtherewasanotherinterestingchangeinthewayLinuxwasdeveloped.Duringthedevelopmentofthe2.3kernelversion,inadditiontothemainkerneltreereleasedbyLinusTorvalds,therearesomeotherkerneltrees.Duringthedevelopmentof2.5,thekerneltreeexploded.Sincetheuseofsourcecodemanagementtoolscanmaintainthesynchronizationandparalleldevelopmentofdevelopment,itispossibletoachievepartialparallelizationofdevelopment.Inorderforotherstotesttheirchangesbeforetheyareaccepted,somedevelopmentneedstobeparallelized.Thosekernelmaintainerswhomaintaintheirowntreesworkonspecificcomponentsandgoals,suchasmemorymanagement,NUMAcomponents,improvedscalability,andcodeforspecificarchitectures,andsometreescollectandtrackcorrectionsformanysmalldefects.
Theadvantageofthisparalleldevelopmentmodelisthatitallowsdeveloperswhoneedtomakemajorchanges,orthosewhomakealargenumberofsimilarchangesforaspecificgoal,tofreelydevelopinacontrolledenvironment,Anddoesnotaffectthestabilityofthekernelusedbyothers.Whenthedevelopershavefinishedtheirwork,theycanreleasepatchesforthecurrentversionoftheLinuxkerneltoimplementthechangestheyhavedonesofar.Inthisway,testersinthecommunitycaneasilytestthesechangesandprovidefeedback.Aftereachparthasbeenproventobestable,thosepartscanbeintegratedintothemainLinuxkernelindividually,orevenallatthesametime.
(Sourceofsomepicturesinthealbumofthekernelstructurediagram:)
CodeCoverageAnalysis
Thenewtoolprovidesthefunctionofcodecoverageanalysisforthekernel.Coverageanalysisshowswhichlinesofcodeinthekernelareexecutedwhenagiventestisrun.Moreimportantly,thecoverageanalysisindicateswhichpartsofthekernelhavenotbeentestedatall.Thisdataisimportantbecauseitpointsoutwhichnewtestsneedtobewrittentotestthosepartsofthekernelsothatthekernelcangetamorecompletetest.
Alotofinformation
Intheprocessofdevelopingthefuture2.6Linuxkernel,inadditiontotheseautomatedinformationmanagementmethods,differentmembersoftheopensourcecommunityalsocollectandtrackAstaggeringamountofinformation.
Forexample,astatuslistwascreatedontheKernelNewbiessitetokeeptrackofthenewcomponentsofthekernelthathavebeenproposed.Thislistcontainsentriessortedbystatus.Iftheyhavebeencompleted,theyareincludedinwhichkernel,andiftheyhavenotbeencompleted,itwilltakehowlong.LinkstomanyitemsonthelistpointtotheWebsiteofalargeproject,orwhentheitemsaresmall,thelinkpointstoacopyofthee-mailmessageexplainingthecorrespondingpart.
Testing
Applicationtesting
Testbackground
Inthepast,LinuxkerneltestingmethodsrevolvedaroundopensourcecodeThedevelopmentmodeliscarriedout.Sincethecodeisreleasedtootherdevelopersforreviewonceitisreleased,therehasneverbeenaformalverificationcyclesimilartootherformsofsoftwaredevelopment.Thetheoreticalbasisbehindthismethodistheso-called"Linuslaw"in"TheCathedralandtheBazaar"(pleaserefertoreferencesforrelevantreferences).Thecontentofthislawis"people'seyesarediscerning."Inotherwords,aintensivereviewcanidentifymostoftherealbigproblems.
However,inreality,thekernelhasmanycomplexinterconnections.Evenwithsufficientreview,manyseriousflawswillstillbemissed.Inaddition,oncethelatestkernelisreleased,enduserscan(andoftendo)downloadanduseit.When2.4.0wasreleased,manypeopleinthecommunityproposedmoreorganizedteststoensuretheintensityofspecifictestsandcodereviews.Organizedtestingincludestheuseoftestplans,repeatabilityinthetestingprocess,andsoon.Usingallthreemethodswillresultinhighercodequalitythanusingonlytwomethodsinitially.
TestProject
TheearliestcontributortotheorganizedtestingofLinuxwastheLinuxTestProject(LinuxTestProject,LTP).ThepurposeofthisprojectistoimprovethequalityofLinuxthroughmoreorganizedtestingmethods.Partofthistestprojectisthedevelopmentofautomatedtestsuites.ThemaintestsuitedevelopedbyLTPisalsocalledLinuxtestproject.Whenthe2.4.0kernelwasreleased,theLTPtestsuitehadonlyabout100tests.Withthedevelopmentandmaturityofthe2.4and2.5versionsofLinux,theLTPtestsuiteisalsodevelopingandmaturing.Currently,theLinuxtestprojectincludesmorethan2,000tests,andthisnumberisstillgrowing.
Regressiontesting
Inthe2.5developmentcycle,anotheritemusedintheLinuxtestprojectistousetheLTPtestsuitetoperformregressiontestingontheLinuxkernelformultipledays.PeopleuseBitKeepertocreateareal-time,centralizedarchivetogetasnapshotoftheLinuxkernelatanytime.WhenBitKeeperandsnapshotswerenotused,testershadtowaituntilthekernelwasreleasedbeforetheycouldstarttesting.Aslongasthekernelischanged,thetestercanperformthetest.
Anotheradvantageofusingautomatedtoolstoperformregressiontestingthatlastsformanydaysisthatthechangesaresmallcomparedtotheprevioustest.Ifanewregressiondefectisfound,itisusuallyeasytodetectwhichchangethedefectmayhavecaused.
Similarly,becauseitisthelatestchange,itstillhasadeepimpressioninthedeveloper'smind-hopethiswillmakeiteasierforthemtorememberandrevisethecorrespondingcode.PerhapstheLinusLawshouldhavesuchaconclusionthatsomedefectsareeasiertofindthanothers,becausethosearetheonesdiscoveredanddealtwithbythekernelregressionteststhatlastedformanydays.Thesetestscanbeperformeddailyduringthedevelopmentcycleandbeforetheactualrelease,whichallowstesterswhoonlyfocusonthefullreleaseversiontofocusonmoreseriousandtime-consumingdefects.
Extensibletesting
AnotherteamcalledOpenSourceDevelopmentLabs(OSDL)hasalsomadeimportantcontributionstoLinuxtesting.Soonafterthe2.4kernelwasreleased,OSDLcreatedasystemcalledScalableTestPlatform(STP).STPisanautomatedtestplatformthatallowsdevelopersandtesterstorunthetestsprovidedbythesystemonOSDLhardware.Developerscanevenusethissystemtotesttheirownkernel-specificpatches.Theextensibletestplatformsimplifiestheteststeps,becauseSTPcanbuildthekernel,setupthetest,runthetest,andcollecttheresults.Thengettheresultsforin-depthcomparison.Manypeoplecannotaccesslargesystems,suchasSMPmachineswith8processors.ThroughSTP,anyonecanruntestsonlargesystemslikethis.Anotheradvantageofthissystem(STP)liesinthis.
Trackingdefects
Sincethereleaseof2.4,oneofthebiggestimprovementstotheorganizedtestingoftheLinuxkernelisdefecttracking.Inthepast,defectsfoundintheLinuxkernelwerereportedtotheLinuxkernelmailinglist,tothemailinglistofspecificcomponentsorspecificsystems,ordirectlytotheindividualwhomaintainedthepartofthecodewherethedefectwasfound.WiththeincreaseinthenumberofpeopledevelopingandtestingLinux,theshortcomingsofthissystemwerequicklyexposed.Inthepast,unlesspeople'sreportsondefectscouldbemaintainedsurprisingly,defectswereoftenmissed,forgottenorignored.
OSDLhasinstalledadefecttrackingsystemtoreportandtrackdefectsintheLinuxkernel.Thesystemisconfiguredsothatwhenacomponent’sdefectisreported,themaintainerofthatcomponentwillbenotified.Themaintainercaneitheracceptandfixthedefect,orreassignthedefect(ifitisfinallydeterminedthatitisactuallyadefectinanotherpartofthekernel),orexcludeit(ifitisfinallydeterminedthatitisnotarealdefect,suchasamisconfiguredsystem).Thedefectsreportedtothemailinglistarealsoindangerofbeinglost,asmoreandmoree-mailsarefloodingtothatlist.However,inthedefecttrackingsystem,thereisalwaysarecordofeachdefectanditscurrentstatus.