Date  Topics covered  
Oct 9  Introduction, Discrete Time Markov Chains (DTMCs)  definitions, some properties, Bertrand's paradox. 
Oct 11  Probability spaces, paths of a DTMC, cylinder sets.

 Homework Assignment

Oct 18  DTMCs: Probabilistic reachability, computing reachibility probabilities as a fixed point of a linear equation system, iterative method, least fixed point characterization, bounded reachability.

Oct 23  DTMCs: Transient state probabilities, long run behaviours/steadystate distributions, computation and characterization, qualitative and quantitative properties.

Oct 25  Markov reward models: DTMCs with costs and rewards, expected reward for reachability, costbounded reachability, constrained reachability, long run properties.

 Quiz (Nov 11)

Nov 6  Markov decision processes (MDPs), capturing nondeterminism and probabilities, definitions, examples, schedulers.

Nov 8  MDPs: reachability properties, qualitative reachability, optimality equations, optimal schedulers.

Nov 13  MDPs: computing reachability probability over optimal schedulers, value iteration algorithm, linear program formulation, policy iteration.

Nov 15  PRISM Tool for probabilistic systems  demo and presentation by Ganesh Narwane.

Nov 16  Tutorial

Nov 26  Exam
