Markov chain

History

TheproposaloftheMarkovchaincomesfromtheRussianmathematicianAndreyMarkov(АндрейАндреевичМарков).IntheresearchpublishedbyMarkovbetween1906and1907,inordertoprovethattheindependencebetweenrandomvariablesisnotanecessaryconditionfortheestablishmentoftheweaklawoflargenumbersandthecentrallimittheorem(centrallimittheorem),heconstructedabasisArandomprocessinwhichconditionalprobabilitiesdependoneachother,anditisprovedthatitconvergestoasetofvectorsundercertainconditions.ThisrandomprocessiscalledaMarkovchaininlatergenerations.

AftertheMarkovchainwasproposed,PaulEhrenfestandTatianaAfanasyevausedtheMarkovchaintoestablishtheEhrenfestmodelofdiffusionin1907.In1912,JulesHenriPoincaréstudiedMarkovchainsonfinitegroupsandobtainedPoincaréinequality.

In1931,AndreyKolmogorov(АндрейНиколаевичКолмогоров)extendedtheMarkovchaintothecontinuousexponentialsetinthestudyofthediffusionproblemandobtainedthecontinuous-timeMarkovchain,Andintroducedthecalculationformulaofitsjointdistributionfunction.IndependentofKolmogorov,in1926,SydneyChapmanalsoobtainedthecalculationformulawhenstudyingBrownianmotion,whichisthelaterChapman-Kolmogorovequation.

In1953,NicholasMetropolisetal.completedtherandomsimulationofthefluidtargetdistributionfunctionbyconstructingaMarkovchain.ThismethodwasfurtherimprovedbyWilfredK.Hastingsin1970anddevelopedintothecurrentMetreRobolis-Hastingsalgorithm(Metropolis-Hastingsalgorithm).

In1957,RichardBellmanfirstproposedMarkovDecisionProcesses(MDP)throughadiscretestochasticoptimalcontrolmodel.

1959-1962,theformerSovietmathematicianEugeneBorisovichDynkinperfectedKolmogorov’stheoryandusedtheDynkinformulatocomparethestationaryMarkovprocesswiththemartingaleprocess.connect.

BasedonMarkovchains,morecomplexMarkovmodels(suchasHiddenMarkovModelsandMarkovRandomFields)havebeenproposedoneafteranotherandobtainedfrompracticalproblemssuchaspatternrecognitionApplied.

Definition

AMarkovchainisasetofdiscreterandomvariableswithMarkovproperties.Specifically,fortherandomvariablesetwithaone-dimensionalcountablesetastheindexsetintheprobabilityspace,ifthevalues​​oftherandomvariablesareallinthecountablesetInside:,andtheconditionalprobabilityoftherandomvariablesatisfiesthefollowingrelationship:

TheniscalledaMarkovchain,Thecountablesetiscalledthestatespace,andthevalueoftheMarkovchaininthestatespaceiscalledthestate.TheMarkovchaindefinedhereisadiscrete-timeMarkovchain(Discrete-TimeMC,DTMC).Althoughithasacontinuousexponentset,itiscalledacontinuous-timeMarkovchain(Continuous-TimeMC,CTMC),butInessence,itisaMarkovprocess.Commonly,theindexsetofaMarkovchainiscalled"step"or"time-step".

TheaboveformuladefinestheMarkovpropertywhiledefiningtheMarkovchain.Thispropertyisalsocalled"memorylessness",thatis,therandomvariableatstept+1isgivenAfterstept,therandomvariableisconditionallyindependentfromtherestoftherandomvariables:.Onthisbasis,theMarkovchainhasstrongMarkovproperty,thatis,foranystoppingtime,thestateoftheMarkovchainbeforeandafterthestoppingtimeisindependentofeachother.

Explanatoryexample

AcommonexampleofMarkovchainisthesimplifiedstockfluctuationmodel:ifastockrisesinoneday,itwillbeThereisaprobabilitythatpwillstarttofalland1-pwillcontinuetorise;ifthestockfallsinoneday,thereisaprobabilitythatqwillstarttorisetomorrowand1-qwillcontinuetofall.TheriseandfallofthestockisaMarkovchain,andtheconceptsinthedefinitionhavethefollowingcorrespondenceintheexample:

• Randomvariable:thestateofthestockondayt;state;Space:"rising"and"falling";indexset:numberofdays.

• Conditionalprobabilityrelationship:Bydefinition,evenifallthehistoricalstatesofthestockareknown,itsriseorfallonacertaindayisonlyrelatedtothestateofthepreviousday.

• Nomemory:Thestock’sperformanceonthedayisonlyrelatedtothepreviousdayandhasnothingtodowithotherhistoricalstates(theconditionalprobabilityrelationshipisdefinedwhilethememorylessnessisdefined).

• Thestatebeforeandafterthestopisindependentofeachother:takeoutthestock’sriseandfallrecords,andtheninterceptasectionfromit.Wecan’tknowwhichsectionwasintercepted,becausetheinterceptionpointisthestoptime.Therecordsbeforeandaftert(t-1andt+1)havenodependencies.

n-orderMarkovchain

n-orderMarkovchainHavingn-thordermemorycanberegardedasageneralizationofMarkovchain.ByanalogytothedefinitionofMarkovchain,then-orderMarkovchainsatisfiesthefollowingconditions:

Accordingtotheaboveformula,thetraditionalMarkovchaincanbeconsideredasa1-orderMarkovchainKofuchain.AccordingtotheMarkovproperty,wecangetann-orderMarkovchainbytakingmultipleMarkovchainsascomponents.

Theoryandproperties

Transfertheory

ThechangeofthestateofarandomvariableinaMarkovchainovertimeiscalledevolutionortransfer(transition).HereweintroducetwowaystodescribethestructureofMarkovchain,namelytransitionmatrixandtransitiongraph,anddefinethepropertiesofMarkovchainintheprocessoftransition.

Transitionprobability(transitionprobability)andtransitionmatrix(transitionmatrix)

_{Mainentry:Transitionmatrix}

MarkovchainTheconditionalprobabilitybetweenrandomvariablescanbedefinedasthefollowingformsof(single-step)transitionprobabilityandn-steptransitionprobability:

wherethesubscript

Indicatesthetransferofthenthstep.AccordingtotheMarkovproperty,aftertheinitialprobabilityisgiven,thecontinuousmultiplicationofthetransitionprobabilitycanrepresentthefinite-dimensionaldistributionoftheMarkovchain:

Theintheformulaisthesamplepath,thatis,thevalueofeachstepoftheMarkovchain.Forthen-steptransitionprobability,itcanbeknownfromtheChapman–Kolmogorovequationthatitsvalueisthesumofallsampletracks:

TheaboveequationshowsthatMarlThedirectevolutionofthekoffchainbynstepsisequivalenttoitsfirstevolutionbynkstepsandthenbyksteps(kisinthestatespaceoftheMarkovchain).Theproductofthen-steptransitionprobabilityandtheinitialprobabilityiscalledtheabsoluteprobabilityofthestate.

IfthestatespaceofaMarkovchainislimited,thetransitionprobabilitiesofallstatescanbearrangedinamatrixinasingle-stepevolutiontoobtainthetransitionmatrix:

ThetransitionmatrixoftheMarkovchainistherightstochasticmatrix,andtherowofthematrixrepresentswhenistakenTheprobabilityofallpossiblestates(discretedistribution),sotheMarkovchaincompletelydeterminesthetransitionmatrix,andthetransitionmatrixalsocompletelydeterminestheMarkovchain.Accordingtothenatureoftheprobabilitydistribution,thetransitionmatrixis​​apositivedefinitematrix,andthesumoftheelementsineachrowisequalto1:Then-steptransitionmatrixcanalsobedefinedinthesameway:

,Fromthenatureofthen-steptransitionprobability(Chapman-Kolmogorovequation),then-steptransitionmatrixis​​thecontinuousmatrixmultiplicationofallprevioustransitionmatrices:.

Transitiongraph

1.Accessibleandcommunicate

TheevolutionofaMarkovchaincanberepresentedasatransitiongraphinagraphstructure,andeachedgeinthegraphisassignedatransitionprobability.Theconceptof"reachable"and"connected"canbeintroducedthroughthetransitiongraph:

IfthestateintheMarkovchainis:,thatisAlltransitionprobabilitiesonthesamplingpatharenot0,thenthestateisthereachablestateofthestate,whichisrepresentedasadirectedconnectioninthetransitiondiagram:

.Ifismutuallyreachable,thetwoareconnected,formingaclosedloopinthetransitiondiagram,whichismarkedas.Bydefinition,reachabilityandconnectivitycanbeindirect,thatis,itdoesnothavetobecompletedinasingletimestep.

Connectivityisasetofequivalencerelations,soequivalenceclassescanbeconstructed.InMarkovchains,equivalenceclassesthatcontainasmanystatesaspossiblearecalledcommunicatingclasses.).

2.Closedsetandabsorbingstate

Asubsetofagivenstatespace,suchasaMarkovchainAfterenteringthesubset,youcannotleave:,thenthesubsetisclosed,calledaclosedset,andallstatesoutsideaclosedsetarenotreachable.Ifthereisonlyonestateintheclosedset,thestateisanabsorptionstate,whichisaself-loopwithaprobabilityof1inthetransitiondiagram.Aclosedsetcanincludeoneormoreconnectedclasses.

3.Anexampleofatransitiondiagram

Here,anexampleofatransitiondiagramisusedtoillustratetheaboveconcepts:

BydefinitionItcanbeseenthatthetransitiongraphcontainsthreeconnectedclasses:,threeclosedsets:,andanabsorptionstate,state6.Notethatintheabovetransitiondiagram,theMarkovchainwilleventuallyentertheabsorbingstatefromanystate.ThistypeofMarkovchainiscalledanabsorbingMarkovchain.

Properties

HerewedefinethefourpropertiesofMarkovchains:irreducibility,recurrence,periodicityandergodicity.DifferentfromMarkovproperties,thesepropertiesarenotnecessarilythepropertiesoftheMarkovchain,butthepropertiesthatitexhibitstoitsstateduringthetransferprocess.Theabovepropertiesareallexclusive,thatis,Markovchainsthatarenotreduciblearenecessarilyirreducible,andsoon.

irreducibility

IfthestatespaceofaMarkovchainhasonlyoneconnectedclass,thatis,allthemembersofthestatespace,thenTheMarkovchainisirreducible,otherwisetheMarkovchainhasreducibility(reducibility).TheirreducibilityoftheMarkovchainmeansthatduringitsevolution,randomvariablescanbetransferredbetweenanystates.

Recurrence

IftheMarkovchainreachesastate,itcanreturntothatstaterepeatedlyduringevolution,thenThestateisarecurrentstate,ortheMarkovchainhasa(local)recurrence,otherwiseithasatransient(transience).Formally,forastateinthestatespace,thereturntimeofaMarkovchainforagivenstateistheinfimumofallpossiblereturntimes:

If,thereisnotransientorrecurrenceinthisstate;if,thecriteriaforjudgingthetransientandrecurrenceofthisstateareasfollows:

Whenthetimesteptendstoinfinity,thereturnprobabilityoftherecurrentstate(returnprobability),thatis,theexpectationofthetotalnumberofvisitsalsotendstoinfinity:

Inaddition,ifthestateisrecurring,themeanrecurrencetimecanbecalculated:

Iftheaveragereturntimeis,thestatusis"positiverecurrent",otherwiseitis"nullrecurrent".Ifastateiszero-returning,itmeansthattheexpectationofthetimeintervalbetweentwovisitsoftheMarkovchaintothestateispositiveinfinity.

Fromtheabovedefinitionoftransientnessandrecurrence,thefollowinginferencescanbemade:

1. Inference:forafinitenumberofstatesofMarrTheKofuchainhasatleastonerecurrentstate,andallrecurrentstatesarenormal.

2. Inference:IfaMarkovchainwithafinitenumberofstatesisirreducible,allofitsstatesreturnnormally.

3. Inference:IfstateAisrecursiveandstateBisreachablestateofA,thenAandBareconnected,AndBisafrequentreturn.

4. Inference:IfstateBisthereachablestateofA,andstateBistheabsorbingstate,thenBistherecurrentstate,andAistheinstantaneousstate.Changestate.

5. Inference:Thesetcomposedofthenormalreturnstateisaclosedset,butthestateintheclosedsetmaynotbearecurrentstate.

Periodicity

AMarkovchainthatreturnsnormallymayhaveperiodicity,thatis,initsDuringtheevolution,theMarkovchaincanoftenreturntoitsstatewithaperiodgreaterthan1.Formally,givenanormalreturnstate,itsreturnperiodiscalculatedasfollows:

wheremeansTakethegreatestcommondivisorofthesetelements.Forexample,ifinthetransitiondiagram,thenumberofstepsrequiredforaMarkovchaintoreturntoacertainstateis,thenitscycleis3,whichistheminimumnumberofstepsrequiredtoreturntothisstate.Ifiscalculatedaccordingtotheaboveformula,thestateisperiodic.If,thestateisaperiodicity.Fromthedefinitionofperiodicity,thefollowinginferencescanbemade:

1. Inference:Theabsorptionstateisanon-periodicstate.

2. Inference:IfstateAandstateBareconnected,thenAandBhavethesamecycle.

3. Inference:IfanirreducibleMarkovchainhasaperiodicstateA,thenallstatesoftheMarkovchainareperiodicstates.

ergodicity

IfastateoftheMarkovchainisnormallyreturnedandaperiodic,Thestateisergodic.IfaMarkovchainisnotreducible,andacertainstateisergodic,thenallstatesoftheMarkovchainareergodic,whichiscalledtheergodicchain.Fromtheabovedefinition,theergodicityhasthefollowinginferences:

1. Inference:IfstateAisanabsorbingstate,andAisthereachablestateofstateB,thenAistraversed,Bisnottraversed.

2. Inference:IfaMarkovchainofmultiplestatescontainsabsorbingstates,thentheMarkovchainisnotanergodicchain.

3. Inference:IfaMarkovchainofmultiplestatesformsadirectedacyclicgraph,orasingleclosedloop,thentheMarkovchainisnotTraversethechain.

Theergodicchainisanon-periodicstableMarkovchainwithsteady-statebehavioronalong-termscale,soitisaMarkovchainthathasbeenwidelystudiedandapplied.

Steady-stateanalysis

Hereweintroducethedescriptionofthelong-timescalebehaviorofMarkovchain,namelystationarydistributionandlimitdistribution,anddefinestationaryMarkovchain.

Stationarydistribution(stationarydistribution)

GivenaMarkovchain,ifthereisaprobabilitydistributioninitsstatespaceandthedistributionmeetsthefollowingconditions:

ThenisthestationarydistributionoftheMarkovchain.Whereisthetransitionmatrixandtransitionprobability.Thelinearequationsontherightsideoftheequivalentsymbolarecalledbalanceequations.Further,ifastationarydistributionoftheMarkovchainexists,anditsinitialdistributionisastationarydistribution,thentheMarkovchainisinasteadystate.Fromageometricpointofview,considerthestationarydistributionofMarkovchainsas,sothesupportsetofthisdistributionisastandardsimplex.

ForanirreducibleMarkovchain,ifandonlyifithasauniquestationarydistribution,thatis,whenthebalanceequationhasauniquesolutiononthepositivesimplex,theMarkovchainThechainisreturnednormally,andthestationarydistributionisexpressedasfollows:

Theaboveconclusioniscalledthestationarydistributioncriterion.ForirreducibleandrecurrentMarkovchains,solvingthebalanceequationcangettheonlyeigenvectorexceptthescale,thatis,theinvariantmeasure.IftheMarkovchainisnormallyrecursive,itsstationarydistributionisTheeigenvectorwhentheeigenvalueis1isobtainedwhensolvingthebalanceequation,thatis,theinvariantmeasureafternormalization.Therefore,thenecessaryandsufficientconditionfortheMarkovchaintohaveastabledistributionisthatithasanormalreturnstate.Inaddition,itcanbeknownbyexamplethatifaMarkovchaincontainsmultipleconnectedclassescomposedofnormalreturnstates(thepropertyshowsthattheyareallclosedsets,sotheMarkovchainisnotnormalreturn),theneachconnectedclasshasHaveastationarydistribution,andthesteadystateoftheevolutiondependsontheinitialdistribution.

Limitingdistribution(limitingdistribution)

IfthereisaprobabilitydistributioninthestatespaceofaMarkovchain

andsatisfythefollowingrelationship:ThenthedistributionisthelimitdistributionoftheMarkovchain.Notethatthedefinitionofthelimitdistributionhasnothingtodowiththeinitialdistribution,thatis,foranyinitialdistribution,whenthetimesteptendstoinfinity,theprobabilitydistributionoftherandomvariabletendstothelimitdistribution.Bydefinition,thelimitdistributionmustbeastationarydistribution,buttheoppositeisnottrue.Forexample,aperiodicMarkovchainmayhaveastationarydistribution,buttheperiodicMarkovchaindoesnotconvergetoanydistribution,anditsstationarydistributionisnotalimitdistribution.

1.Limitingtheorem

Twoindependentnon-cyclicstationaryMarkovchains,thatis,ifthetraversalchainhasthesametransitionmatrix,Thenwhenthetimesteptendstoinfinity,thedifferencebetweenthetwolimitdistributionstendstozero.Accordingtothecouplingtheoryinrandomprocesses,theconclusionisexpressedas:forthesametraversalchaininthestatespace,givenanyinitialdistribution,thereis:

Wheremeanssupremum.Consideringthenatureofthestationarydistribution,thisconclusionhasacorollary:forthetraversalchain,whenthetimesteptendstoinfinity,itslimitdistributiontendstobeastationarydistribution:

section>ThisconclusionissometimesreferredtoasthelimittheoremofMarkovchain(limittheoremofMarkovchain),indicatingthatiftheMarkovchainisergodic,itslimitdistributionisastationarydistribution.Foranirreducibleandnon-periodicMarkovchain,theergodicchainisequivalenttotheexistenceofitslimitdistribution,anditisalsoequivalenttotheexistenceofitsstabledistribution.

2.Ergodictheorem(ergodictheorem)

IfaMarkovchainisanergodicchain,thenbytheergodictheorem,theTheratioofthenumberofvisitstothetimestepapproachesthereciprocaloftheaveragereturntimewhenthetimestepapproachesinfinity,thatis,thestationaryorlimitingdistributionofthestate:

TraversaltheoremTheproofofreliesontheStrongLawofLargeNumbers(SLLN),whichshowsthatregardlessoftheinitialdistributionofthetraversalchain,afterasufficientlylongevolution,multipleobservations(limittheorem)ofoneoftherandomvariablesandmultipleOneobservationofrandomvariables(leftsideoftheaboveequation)cangetanapproximationofthelimitdistribution.Sincetheergodicchainsatisfiesthelimittheoremandtheergodictheorem,MCMCbuildstheergodicchaintoensurethatitconvergestoastabledistributionduringiteration.

StationaryMarkovchain

IfaMarkovchainhasauniquestationarydistributionandthelimitdistributionconvergestoastationarydistribution,Bydefinition,itisequivalenttothattheMarkovchainisastationaryMarkovchain.AstationaryMarkovchainisastrictlystationaryrandomprocess,anditsevolutionhasnothingtodowiththetimesequence:

ItcanbeknownfromthelimittheoremthatthetraversalchainisastationaryMarkovchain.Inaddition,fromtheabovedefinition,thetransitionmatrixofastationaryMarkovchainisaconstantmatrix,andthen-steptransitionmatrixis​​then-thpoweroftheconstantmatrix.AstationaryMarkovchainisalsocalledatime-homogeneousMarkovchain.Correspondingtoit,iftheMarkovchaindoesnotmeettheaboveconditions,itiscalledanon-stationaryMarkovchain(non-stationaryMarkvochain)ornon-homogeneousMarkovchain(time-inhomogeneousMarkovchain).

IfastationaryMarkovchainsatisfiesthedetailedbalanceconditionforanytwostates,ithasreversibilityandiscalledareversibleMarkovchain(reversibleMarkovchain):

ThereversibilityofaMarkovchainisastricterirreducibility,thatis,itcannotonlytransferbetweenanystates,butalsotransfertoeachstateTheprobabilitiesareequal,sothereversibleMarkovchainisasufficientnon-essentialconditionforastableMarkovchain.InMarkvochainMonteCarlo(MCMC),constructingareversibleMarkovchainthatsatisfiesthefinebalanceconditionisamethodtoensurethatthesamplingdistributionisastabledistribution.

Specialcase

Bernoulliprocess(Bernoulliprocess)

_{Mainentry：Bernoulliprocess}

TheBernoulliprocessisalsocalledBinomialMarkovchain.Itsestablishmentprocessisasfollows:GivenaseriesIndependent"identifications",eachidentificationisbinomial,andtheprobabilityistakenaspositiveandnegative.Letthepositiverandomprocessrepresenttheprobabilityofkpositiveidentifiersinnidentifiers,thenitisaBernoulliprocessinwhichtherandomvariablesobeythebinomialdistribution:

Fromtheestablishmentprocess,itcanbeseenthattheprobabilityofpositivesignsintheaddednewsignshasnothingtodowiththenumberofpreviouspositivesigns,andhasMarkovproperties,sotheBernoulliprocessisaMarkovchain.

Thegambler'sruinproblem(gambler'sruin)

_{See:Gambler'sruin}

Assumingthatthegamblerholdsalimitednumberofchipstobetinthecasino,eachbetwillwinorloseonechipwiththeprobabilityof.Ifthegamblerkeepsbetting,thetotalnumberofchipsheholdsisamarkIthasthefollowingtransfermatrix:

Thegambler’slightbetistheabsorptionstate,whichcanbeknownfromone-stepanalysis.When

Atthetime,theMarkovchainwillinevitablyentertheabsorptionstate,thatis,nomatterhowmanychipsthegamblerholds,itwilleventuallyloseoutasthebetprogresses.

Randomwalk(randomwalk)

_{Mainentry:RandomWandering}

Defineaseriesofindependentidenticallydistributed(iid)integerrandomvariables,anddefinethefollowingrandomprocess:

Therandomprocessisarandomwalkintheintegerset,andisthesteplength.Sincethesteplengthisiid,thecurrentstepandthepreviousstepareindependentofeachother,andthisrandomprocessisaMarkovchain.BoththeBernoulliprocessandthebankruptcyproblemofgamblersarespecialcasesofrandomwalk.

Fromtheaboveexampleofrandomwalk,wecanseethatMarkovchainshavegeneralconstructionmethods.Specifically,iftherandomprocessinthestatespaceis

Therearesatisfyingforms:

Amongthem,andareiidrandomvariablesofspaceandIndependentof,therandomprocessisaMarkovchain,anditsone-steptransitionprobabilityis:.ThisconclusionshowsthattheMarkovchaincanbenumericallysimulatedbyrandomvariables(randomnumbers)thatfollowtheuniformdistributionofiidintheintervalof.

Promotion

Markovprocess

_{Mainentry:Markovprocess}

MarkovprocessisalsocalledcontinuoustimeMarkovchain,itisMarkovchainordiscretetimeMarkovchainInthegeneralizationof,itsstatespaceisacountableset,buttheone-dimensionalindexsetnolongerhasthelimitofacountablesetandcanrepresentcontinuoustime.ThepropertiesofaMarkovprocessandaMarkovchaincanbecompared,anditsMarkovpropertyisusuallyexpressedasfollows:

SincethestatespaceoftheMarkovprocessisForasetofnumbers,thesampletrackincontinuoustimeisalmostnecessarily(as)arightcontinuoussegmentfunction,sotheMarkovprocesscanbeexpressedasajumpprocessandrelatedtotheMarkovchain:

whereisthesojourntimeofacertainstate,andisthesequentialindexsetmember(timesegment).TheMarkovchainandtheresidencetimesatisfyingtheaboverelationshiparethejumpingprocesslocallyembeddedprocessunderalimitedtimesegment..

Markovmodel(Markovmodel)

_{Mainarticle:Markovmodel}

MarkovchainorMarkovprocessisnottheonlyrandomprocessbasedonMarkovproperty.Infact,hiddenMarkovmodel,Markovdecisionprocess,MarkovrandomprocessStochasticprocesses/randommodelssuchasairportshaveMarkovpropertiesandarecollectivelyreferredtoasMarkovmodels.HereisabriefintroductiontoothermembersoftheMarkovmodel:

1.HiddenMarkovModel(HiddenMarkovModel,HMM)

HMMisAstatespaceisnotcompletelyvisible,thatis,aMarkovchaincontaininghiddenstatus.ThevisiblepartoftheHMMiscalledtheemissionstate,whichisrelatedtothehiddenstate,butitisnotenoughtoformacompletecorrespondence.Takespeechrecognitionasanexample.Thesentencethatneedstoberecognizedisinaninvisiblehiddenstate,andthereceivedvoiceoraudioistheoutputstaterelatedtothesentence.Atthistime,thecommonapplicationofHMMisbasedontheMarkovnatureandderivedfromthevoiceinput.Thecorrespondingsentenceistoreversethehiddenstatefromtheoutputstate.

2.Markovdecisionprocess(Markovdecisionprocess,MDP):

MDPintroduces"actions"onthebasisofstatespaceTheMarkovchain,thatis,thetransitionprobabilityoftheMarkovchainisnotonlyrelatedtothecurrentstate,butalsorelatedtothecurrentaction.MDPincludesasetofinteractiveobjects,namelyagentandenvironment,anddefines5modelelements:state,action,policy,rewardandreturn.AmongthemStrategyisthemappingfromstatetoaction,andrewardisthediscountoraccumulationofrewardovertime.IntheevolutionofMDP,theagentperceivestheinitialstateoftheenvironment,implementsactionsaccordingtothestrategy,andtheenvironmententersanewstateundertheinfluenceoftheactionandfeedsbacktotheagentareward.Theagentreceivesthe"reward"andadoptsnewstrategiestocontinuouslyinteractwiththeenvironment.MDPisoneofthemathematicalmodelsofreinforcementlearning(reinforcementlearning),whichisusedtosimulatetherandomnessstrategiesandrewardsthatcanbeachievedbyagents.OneofthepopularizationsofMDPisthepartiallyobservableMarkovdecisionprocess(POMDP),whichconsidersthehiddenstateandoutputstateoftheMDPintheHMM.

3.MarkovRandomField(MarkovRandomField,MRF)

MRFisaMarkovchainfromone-dimensionalindexsettohigh-dimensionalspacePromotion.TheMarkovpropertyofMRFshowsthatthestateofanyrandomvariableisonlydeterminedbythestateofallitsadjacentrandomvariables.Analogoustothefinite-dimensionaldistributioninaMarkovchain,thejointprobabilitydistributionofarandomvariableinMRFistheproductofallcliquescontainingtherandomvariable.ThemostcommonexampleofMRFistheIsingmodel.

Harrischain

HarrischainisthegeneralizationofMarkovchainfromcountablestatespacetocontinuousstatespace,givenThestationaryMarkovchainonthemeasurablespace,ifanysubsetofthemeasurablespaceandthereturntimeofthesubset

,theMarkovchainsatisfies:

thentheMarkovchainisaHarrischain,whereisameasurablespaceΣ-finitemeasure(σ-finitemeasure).

Application

MCMC

BuildingaMarkovchainwithsamplingdistributionasthelimitdistributionisMarkovChainMonteCarlo(MarkovChainMonteCarlo,MCMC)ThecorestepofMCMCistocontinuouslyiteratethetimestepsontheMarkovchaintoobtainrandomnumbersthatapproximatelyobeythesamplingdistribution,andusetheserandomnumberstoapproximatethemathematicalexpectationofthetargetforthesamplingdistribution:

ThelimitdistributionnatureofMarkovchaindeterminesthatMCMCisunbiasedestimation,thatis,whenthenumberofsamplestendstoinfinity,thetruevalueofthemathematicalexpectationofsolvingthetargetwillbeobtained.ThiscombinesMCMCanditsalternativemethod,Forexample,variationalBayesianinference(variationalBayesianinference)isdistinguished,thelatterisusuallylesscomputationallyexpensivethanMCMC,butitcannotguaranteeanunbiasedestimate.

Others

Inphysicsandchemistry,MarkovchainsandMarkovprocessesareusedtomodeldynamicsystems,formingMarkovdynamics(Markovdynamics).dynamics).Inthequeueingtheory,theMarkovchainisthebasicmodelofthequeuingprocess.Intermsofsignalprocessing,Markovchainsaresomesequentialdatacompressionalgorithms,suchasthemathematicalmodelofZiv-Lempelcoding.Inthefinancialfield,theMarkovchainmodelisusedtopredictthemarketshareofenterpriseproducts.