Erratum: The equation in Section 2 with $Q^{\pi}(s, a)$ in the LHS is incorrect. The equation should be replaced by ``$Q^{\pi}(s, a) = R(s, a) + \gamma \sum_{s^{\prime} in S} T(s, a, s^{\prime}) V^{\pi}(s^{\prime})$.'' Erratum: The definition of the Bellman operator in the proof of Theorem 2 is incorrect. The equation with $B^{\pi}(X)(s)$ in the LHS should be replaced by ``$B^{\pi}(X)(s)$ \eqdef R(s, \pi(s)) + \gamma \sum_{s^{\prime in S}} T(s, \pi(s), s^{\prime}) X(s^{\prime})$.''