| Date | Topics covered | |
|---|
| Oct 9 | Introduction, Discrete Time Markov Chains (DTMCs) - definitions, some properties, Bertrand's paradox. |
| Oct 11 | Probability spaces, paths of a DTMC, cylinder sets.
|
| | Homework Assignment
|
| Oct 18 | DTMCs: Probabilistic reachability, computing reachibility probabilities as a fixed point of a linear equation system, iterative method, least fixed point characterization, bounded reachability.
|
| Oct 23 | DTMCs: Transient state probabilities, long run behaviours/steady-state distributions, computation and characterization, qualitative and quantitative properties.
|
| Oct 25 | Markov reward models: DTMCs with costs and rewards, expected reward for reachability, cost-bounded reachability, constrained reachability, long run properties.
|
| | Quiz (Nov 11)
|
| Nov 6 | Markov decision processes (MDPs), capturing nondeterminism and probabilities, definitions, examples, schedulers.
|
| Nov 8 | MDPs: reachability properties, qualitative reachability, optimality equations, optimal schedulers.
|
| Nov 13 | MDPs: computing reachability probability over optimal schedulers, value iteration algorithm, linear program formulation, policy iteration.
|
| Nov 15 | PRISM Tool for probabilistic systems - demo and presentation by Ganesh Narwane.
|
| Nov 16 | Tutorial
|
| Nov 26 | Exam
|
co-taught by S. Krishna.