The Bellman Optimality Operator Is a Contraction
A proof that the Bellman optimality operator is a gamma-contraction under the infinity norm.
The Bellman optimality operator $T^*$ is a $\gamma$-contraction under the infinity norm:
$$ \left\lVert T^*V_1-T^*V_2 \right\rVert_\infty \le \gamma \left\lVert V_1-V_2 \right\rVert_\infty. $$
The following handwritten derivation proves this result by bounding the difference between two maximized action values and then applying the properties of transition probabilities and the infinity norm.
