Note: Chapter 3 in Kalyanakrishnan's dissertation (2011) serves as a
more up-to-date version of this article, incorporating minor revisions
and additions.

Erratum: The update rule for Expected Sarsa in Section 3.1 is
incorrect. Kalyanakrishnan's dissertation (2011, see Section 3.2.1,
page 53) provides the correct update rule.

Erratum: In the first paragraph of Section 2.1, the following
statement is incorrect.

``On taking N (E), the agent moves north (east) with probability p and
it moves east (north) with probability 1 - p.''

The statement should instead read as follows.

``On taking N (E), the agent moves north (east) with probability 1 - p
and it moves east (north) with probability p.''

The authors thank Ruohan Zhang for pointing out this error.