Intrinsic Practical Stability in Discrete-Time RL

Deep reinforcement learning (RL) has achieved impressive empirical results in complex continuous control tasks—such as dynamic legged locomotion, high-speed autonomous drone racing, and dexterous manipulation. Fundamentally, these tasks are set-stabilization problems: rather than driving a system to a single static point, the agent must steer the state $x_k$ from any initial condition $x_0$ toward—and indefinitely keep it within—a desired target set $\mathcal{A}$ (such as a repeating walking gait limit cycle or a spatial racing line).

To safely deploy learned policies in the real world, a large body of literature modifies the RL algorithm to enforce formal stability guarantees—for example, by jointly training neural Lyapunov functions, embedding Lyapunov decrease constraints into policy updates, or applying external safety filters. While these techniques are valuable for training-time safety during exploration, they alter the core RL objective, which can restrict natural exploration and complicate training.

In this work, we offer a complementary perspective:

Modifying the core RL objective is often unnecessary to guarantee the stability of the deployed policy. We prove that, under mild conditions, near-optimal policies trained on the unmodified discounted RL objective intrinsically guarantee Uniform Practical Asymptotic Stability (UPAS).

1. Why Steady-State Error in RL is a Feature of Discounting, Not a Bug

In classical control theory, asymptotic stability requires a trajectory starting from $x_0$ to converge all the way into the exact target set $\mathcal{A}$ over an infinite horizon (left panel below). In contrast, practical asymptotic stability guarantees that the trajectory converges into and indefinitely remains within a small, bounded neighborhood of radius $\rho > 0$ around the target set $\mathcal{A}$ (right panel below).

Why do standard RL agents naturally exhibit practical asymptotic stability—settling into a small steady-state offset near the target rather than converging to exact zero error? Two fundamental factors govern this behavior:

1. Discounting Creates a Myopic Agent ($\gamma \in (0, 1)$): Rather than optimizing an infinite-horizon undiscounted sum ($\gamma = 1$), RL maximizes an expected discounted return weighted by $\gamma^k$ at each future time step $k$. As shown below, discounting exponentially diminishes the weight of distant rewards $\gamma^k R$, effectively truncating the optimization horizon and causing the agent to prioritize near-term rewards over distant ones. Once the state gets “close enough” to the target $\mathcal{A}$, the immediate control effort required to push the error smaller outweighs the discounted future benefit. Thus, native steady-state error in RL is not an algorithmic failure—it is the direct consequence of optimizing a discounted objective.

1. Neural Network Approximation Error ($\delta > 0$): In practice, deep RL searches over parameterized feedforward neural networks rather than all possible continuous functions. Because finite-capacity networks produce near-optimal policies with some small suboptimality $\delta > 0$, this approximation error slightly enlarges the steady-state neighborhood around the target.

2. How the Theoretical Guarantee Works

To formally prove that unmodified deep RL guarantees closed-loop practical stability, we adapt the discrete-time Input-to-State Stability (ISS) framework and extend tools from discounted Model Predictive Control (MPC) to neural network policies:

• Modeling Suboptimality as Bounded Disturbances: We analyze the optimal value function as a Lyapunov-like energy candidate for the closed-loop system. Using the Bellman equation, we show that both the horizon truncation from discounting ($\gamma \in (0, 1)$) and the suboptimality of the learned neural network ($\delta > 0$) act as bounded disturbances to the ideal closed-loop dynamics.
• Bridging Expected Return to Worst-Case Pointwise Guarantees: A key theoretical challenge is that RL algorithms maximize an expected return averaged across an initial state distribution, whereas closed-loop stability requires a guarantee that holds pointwise at every individual state. By formulating the policy search over a compact space of continuous feedback laws (such as feedforward neural networks with continuous activations and bounded weights) and a strictly positive initial state distribution, we establish two crucial links:
  1. Exact RL (Lemma 3.8 & Theorem 3.9): A policy that maximizes the expected return over the initial state distribution is automatically pointwise optimal at every individual state, steering trajectories into a discount-dependent neighborhood of radius $\nu(\gamma)$ around $\mathcal{A}$ (left figure below).
  2. Approximate RL (Lemma 4.4 & Theorem 4.5): Bounding the expected suboptimality of a neural network policy by $\delta > 0$ automatically guarantees a uniform worst-case bound on its value error across the entire state space, steering trajectories into an expanded neighborhood of radius $\tilde{\nu}(\gamma, \delta)$ around $\mathcal{A}$ (right figure below).

Crucially, the size of the steady-state neighborhood $\tilde{\nu}(\gamma, \delta)$ shrinks to zero as the approximation error $\delta$ approaches $0$ and, subsequently, the discount factor $\gamma$ approaches $1$.

3. Interactive Numerical Validation: How Discounting ($\gamma$) and Network Error ($\delta$) Shape Steady-State Behavior

In Examples 3 and 4 of the paper, we illustrate these theoretical bounds on a discrete-time system $x^+ = x + 0.1u$ with state and action spaces $[-1, 1]$, stabilizing the target origin $\mathcal{A} = \lbrace 0 \rbrace$. Here, $\omega(x_k) = \lvert x_k \rvert$ measures the distance to the target at discrete time step $k$, and the reward function is:

Because the linear control effort penalty ($0.5\lvert u \rvert$) dominates the quadratic state error penalty ($x^2$) near the origin, the optimal policy stops applying control ($u^\ast = 0$) once the state enters the analytical deadzone $\nu(\gamma) = \frac{c(1-\gamma)}{2\gamma\Delta t}$. Furthermore, a 2-neuron ReLU network $\hat{u}(x) = \text{clip}\big(-\frac{1}{\Delta t}\max(w x - \nu(\gamma), 0) + \frac{1}{\Delta t}\max(-w x - \nu(\gamma), 0), -1, 1\big)$ with input weight scaling $w \in [0.6, 1.0]$ induces a uniform value approximation error $\delta = \max_x \lvert V^\ast(x) - \hat{V}(x) \rvert$ that enlarges this steady-state neighborhood to $\tilde{\nu}(\gamma, \delta)$.

Use the interactive simulation below (powered directly by the exact closed-loop dynamics and value-function evaluator from the paper) to explore how $\gamma$, $\delta$, and the initial state $x_0$ shape the closed-loop trajectories:

• Effect of the Discount Factor $\gamma$ (Left Plot — Example 3): Under the exact optimal policy, the steady-state error boundary is given analytically by $\nu(\gamma) = \frac{5(1-\gamma)}{2\gamma}$. As $\gamma$ increases from $0.85$ to $0.99$, the agent becomes more farsighted and the steady-state offset shrinks directly toward zero.
• Effect of Neural Network Approximation Error $\delta$ (Right Plot — Example 4): At any chosen discount factor $\gamma$, evaluating parameterized ReLU network policies across decreasing value approximation errors $\delta = \max_x \lvert V^\ast(x) - \hat{V}(x) \rvert$ simulates how a neural network policy improves over the course of training. Larger approximation errors act as a larger disturbance that widens the steady-state error, while reducing $\delta \to 0$ during training shrinks the trajectory bound $\tilde{\nu}(\gamma, \delta)$ directly onto the ideal discounted baseline $\nu(\gamma)$.

For the full mathematical formulation—including the continuity of the value functional in function space, the universal approximation bounds for deep narrow networks, and the complete stability proofs—please refer to our RLC 2026 paper (de Priester & Sanfelice, 2026).