Erratum: The equation in Section 2 with $Q^{\pi}(s, a)$ in the LHS is
incorrect. The equation should be replaced by
``$Q^{\pi}(s, a) = R(s, a) + \gamma \sum_{s^{\prime} in S} T(s, a,
s^{\prime}) V^{\pi}(s^{\prime})$.''
Erratum: The definition of the Bellman operator in the proof of
Theorem 2 is incorrect. The equation with $B^{\pi}(X)(s)$ in the LHS
should be replaced by
``$B^{\pi}(X)(s)$ \eqdef R(s, \pi(s)) + \gamma \sum_{s^{\prime in S}}
T(s, \pi(s), s^{\prime}) X(s^{\prime})$.''