Timer-Based Hybrid Supervisor for Robust Policy Switching

Across robotics and sequential decision-making for safety-critical applications, a single control policy is often not enough. Instead, an autonomous system relies on a set of specialized policies or discrete decisions, each trained with its own value function to handle a distinct maneuver, sub-goal, or operating mode.

A standard way to coordinate multiple policies at runtime is greedy value-based switching: at every step, the agent evaluates the learned value functions at the current state and selects whichever option promises the highest return. However, whenever two or more policies offer similar expected returns, critical decision boundaries ($\mathcal{M}^\ast$) emerge in the state space. Along these boundaries, small sensor perturbations or an adversary can easily flip the value ranking back and forth—causing a memoryless supervisor to rapidly oscillate between conflicting decisions (chattering) or stall in a deadlock.

Crucially, scaling up to larger AI models or richer policy banks does not remove this vulnerability: in higher-dimensional state spaces and complex reward landscapes, critical boundaries of indecision persist in far more places than expected.

Existing approaches to prevent policy chattering face a fundamental trade-off:

• Spatial Hysteresis Partitioning: Constructing overlapping spatial regions around switching boundaries works well when critical regions have a structured geometry (as in our Hysteresis-Based Reinforcement Learning framework), but becomes intractable when value-function boundaries are fragmented, non-convex, or unknown in advance.
• Fixed-Timer Switching: Holding each policy for a constant waiting time creates a rigid trade-off: a timer that is too short fails to stop chattering under perturbations, while a timer that is too long makes the agent unresponsive and traps it oscillating between decision regions.

In this work, we resolve both limitations by introducing a predictive timer-based hybrid supervisor:

Rather than relying on pre-computed spatial partitions or a fixed waiting time, our supervisor uses short-horizon state predictions and the learned value functions to adaptively determine how long to commit to the active policy. At each switch, the supervisor forecasts when a competing policy will become significantly better—locking in an adaptive dwell timer $\delta_d$ that prevents chattering and deadlocks while preserving underlying policy properties such as stability.

1. Why Greedy Supervisors Chatter—and When Switching is Actually Beneficial

Why does selecting the policy with the highest value function $Q^\ast(z)$ cause catastrophic chattering in some regions of the state space, while working harmlessly in others?

In our paper, we show that switching boundaries between competing policies fall into two distinct categories:

1. Exploitative Switching (Benign): Far from a target, two policies may temporarily trade leadership even though both push the system in the same general direction (for example, when one policy saturates the actuators more aggressively toward the goal). Switching here actually accelerates convergence, and noise near this boundary does not stall the agent.
2. Critical Areas $\mathcal{M}^\ast$ (Catastrophic Chattering): Along critical areas, two policies yield the exact same optimal value ($\hat V_1(z) = \hat V_2(z)$), but their control actions pull the system in opposing directions. When the true state $z$ approaches $\mathcal{M}^\ast$, small measurement noise of magnitude $\varepsilon$ pushes the perceived state across the boundary, activating the opposing policy and driving the state right back toward the boundary. Under alternating sensor noise, the critical boundary becomes an artificial trap that locks the system into high-frequency chattering.

Illustrative Example: Stabilizing Two Disconnected Targets

Consider a vehicle on a line tasked with stabilizing either of two disconnected target locations ($-1$ or $+1$) using two policies $\pi_1$ and $\pi_2$ with learned value functions $\hat V_1$ and $\hat V_2$:

As shown in the closed-loop trajectories below, under bounded measurement noise ($\varepsilon = 0.3$), the greedy supervisor chatters rapidly across $z_c = 0$ and fails to stabilize trajectories starting near the origin (middle plot), whereas our Timer-Based Hybrid Supervisor eliminates chattering and smoothly stabilizes all initial states (right plot):

2. How Predictive Dwell-Time Supervision Works

To prevent chattering without needing prior geometric maps of the critical areas $\mathcal{M}^\ast$, our supervisor augments the control loop with a resettable timer $\tau$, the active policy index $q$, and an adaptive dwell time $\delta_d$:

At runtime, the closed-loop hybrid system operates through a clean, two-part mechanism:

• Continuous Execution (Commitment While Timer Runs): The agent applies the locked-in policy $\pi_q$ and increments the timer $\tau$ continuously as long as either the adaptive dwell time has not yet elapsed ($\tau \le \delta_d$) or the active policy remains optimal under the value functions. This guarantees that measurement noise cannot trigger premature switching while the agent is crossing a vulnerable critical boundary.
• Predictive Dwell-Time Update at Each Switch: When the dwell timer has expired ($\tau \ge \delta_d$) and a better policy $q’$ emerges, the supervisor switches to $q’$, resets the timer ($\tau^+ = 0$), and simulates a short nominal trajectory forward in time (up to a maximum horizon $\bar\delta_d$). The new dwell time $\delta_d^+$ is set to the first predicted time at which another policy beats $q’$ by a relative value margin of at least $\mu > 0$ (or defaults to $\bar\delta_d$ if no policy beats $q’$ within the horizon).

3. Application: Multi-Target Navigation in a Rugged Reward Landscape

To demonstrate the impact of predictive dwell-time supervision in environments where spatial partitioning is impractical, we evaluate a 2D autonomous agent navigating toward four target setpoints across a rugged, spatially varying reward landscape:

Because the periodic peaks and valleys in the reward landscape make straight-line paths suboptimal, the greedy policy regions $Q^\ast(z)$ fracture from clean geometric quadrants (left figure below) into highly fragmented, non-convex decision islands (right figure below):

When deployed on this rugged landscape under adversarial measurement noise ($\varepsilon = 0.1$), the three supervisory strategies exhibit dramatically different behavior:

• Greedy Supervision Stalls (Left Plot): Without temporal memory, sensor noise along the fragmented boundaries triggers rapid chattering among the four policies, trapping several trajectories far from the targets.
• Fixed-Timer Supervision Gets Trapped (Middle Plot): Holding every policy for a fixed duration ($\delta_d = 0.5$) prevents high-frequency chattering, but because a fixed timer ignores where the trajectory is heading, agents starting near $(-0.25, \pm 0.25)$ repeatedly overshoot into neighboring regions and switch back—getting trapped in a persistent oscillation.
• Predictive Timer Supervision Succeeds (Right Plot): By forecasting when a competing policy will offer at least a $\mu = 15\%$ relative value gain, our supervisor switches at the right moments and drives every evaluated initial condition smoothly to the target set.

For the complete mathematical formulation—including outer semicontinuity of the predictive jump map, non-Zeno hybrid time domains, and the hybrid Lyapunov stability proofs—please refer to our RLC 2025 paper (de Priester & Sanfelice, 2025).