Hysteresis-Based Reinforcement Learning for Chatter-Free Autonomy

📄 Research Overview: HyRL (ACC 2022) • MultiHyRL (RLC 2024) • L-MultiHyRL (2026)

This article provides a unified overview of our line of research on Hysteresis-Based Reinforcement Learning—spanning our published foundational algorithms in HyRL (ACC 2022) (de Priester et al., 2022) and MultiHyRL (RLC 2024) (de Priester et al., 2024), and highlighting simulation and physical quadrotor flight results from our latest 3D extension, L-MultiHyRL (de Priester et al., 2026) (manuscript in preparation; full algorithmic details to appear upon publication): [ACC 2022 PDF →] [RLC 2024 PDF →]

In sequential decision-making for safety-critical applications, an autonomous agent must frequently choose between multiple equally viable options—whether selecting a trajectory around an obstacle, choosing a rotation direction, or committing to one of several competing tasks. Because standard continuous and discrete reinforcement learning (RL) policies are memoryless mappings from observations to actions, these choices create critical regions of indecision ($\mathcal{M}^\ast$) in the state space where opposing decisions diverge.

Near these critical boundaries, small sensor noise or an adversary can easily perturb the agent’s observations to make an alternative option appear momentarily better—causing the policy to rapidly oscillate back and forth (chattering), stall in a deadlock, or crash. Crucially, scaling up to larger neural networks and more advanced AI models does not eliminate this vulnerability: in higher-dimensional state spaces, critical decision boundaries persist across the state space in far more places than expected.

Autonomous obstacle avoidance and attitude control provide a clear, visual setting where this structural failure mode appears directly. Whenever a vehicle approaches an obstacle on its way to a target, it must commit to a bypass route (passing above or below in 2D, or maneuvering in 3D space). Along the critical boundary $\mathcal{M}^\ast$ ahead of the obstacle, small adversarial observation shifts move the perceived obstacle into the vehicle’s active passage direction—making the opposing route appear shorter and causing a memoryless agent to chatter until it collides head-on.

Animation 1: Why Standard Memoryless RL Policies Chatter & Crash ● Baseline Failure Case

Topological Chattering Under Adversarial Obstacle Shifts: Phase 1 (Vehicle Tries Upper Route $\rightarrow$ Obstacle Moves Up): As the vehicle steers to pass above the obstacle (blue dashed intended route), an adversarial observation perturbation shifts the perceived obstacle upward into its passage direction. Phase 2 (Vehicle Switches to Lower Route $\rightarrow$ Obstacle Moves Down): Blocked above—and seeing a shorter path below—the memoryless policy reverses to the lower route (green dashed intended route), where an opposing downward shift blocks it again. Phase 3 (Repeated Route Chattering & Crash): The vehicle and perturbed obstacle alternate directions across five rapid route reversals until the vehicle crashes head-on into the obstacle.

Overview of the Bird's-Eye View Obstacle Avoidance Setting (RLC 2024)

Topological Chattering in Obstacle Avoidance (Figure 1 of (de Priester et al., 2024)): Left: A vehicle (orange triangle) observes a local bird's-eye view window (yellow square) while navigating past an obstacle toward a target. Above or below the red dashed decision boundary, the RL policy steers along smooth bypass trajectories (blue and green). Near the boundary, small measurement noise induces rapid switching (red chattering trajectory) and collision. Right: Vehicle orientation $\xi$ and relative target tracking errors $e_p$ and $e_\theta$.

Across HyRL (de Priester et al., 2022), MultiHyRL (de Priester et al., 2024), and L-MultiHyRL (de Priester et al., 2026), our core insight is simple:

Continuous neural networks cannot robustly break symmetry around obstacles without discrete memory. By partitioning the state space into overlapping regions and embedding specialized RL policies inside a closed-loop hybrid system with hysteresis switching, Hysteresis-Based Reinforcement Learning eliminates decision chattering and guarantees decisive, collision-free obstacle avoidance under worst-case observation noise.

1. How Hysteresis-Based RL Works: Overlapping Partitions & Hysteresis Logic

How do we systematically eliminate topological chattering without hand-crafting avoidance controllers? Beginning with our ACC 2022 (de Priester et al., 2022) and RLC 2024 (de Priester et al., 2024) papers, we established a general hysteresis-based reinforcement learning framework built on three steps:

Empirical Critical Point Identification: Starting from a single baseline RL policy, we probe the environment with small opposing state perturbations $\pm \nu$. States from which perturbed rollouts split around opposite sides of an obstacle reveal the baseline policy’s vulnerable critical region $\mathcal{M}^\ast$ (panel a below).
Overlapping State-Space Partitioning: We bisect the critical region $\mathcal{M}^\ast$ into two domains $\mathcal M_1$ (e.g., an Upper Partition) and $\mathcal M_2$ (e.g., a Lower Partition) and expand each across the centerline boundary by a margin $\rho$ to create an overlapping hysteresis region ($\mathcal M_1^{\text{exp}} \cap \mathcal M_2^{\text{exp}}$) whose width exceeds the worst-case sensor or adversarial perturbation (panels b and c below).
Dedicated Policies & Hysteresis Switching Logic: Two specialized RL policies are trained on the overlapping domains—one committed to passing each side of the obstacle. A discrete logic state locks in the initialized policy inside the overlap and ignores perturbation-induced shortcuts, only permitting a switch if the vehicle moves completely outside the active policy’s expanded partition.

Critical Point Identification and Overlapping Hysteresis Partitions

Constructing Overlapping Hysteresis Regions: (a) Small opposing perturbations $z_0 \pm \nu$ cause baseline trajectories $\phi_1$ and $\phi_2$ to diverge around the obstacle, identifying critical points. (b) The empirical critical points $\mathcal{M}^\ast$ (red dots) are bisected into $\mathcal M_1$ and $\mathcal M_2$ and expanded by $\rho_1$ to create an overlap buffer. (c) Inside the overlap between $\mathcal M_1^{\text{exp}}$ and $\mathcal M_2^{\text{exp}}$, both specialized policies safely route the vehicle around opposite sides of the obstacle without chattering.

Animation 2: How Overlapping Partitions & Hysteresis Lock In the Active Route ● Hysteresis Switching Logic

Eliminating Route Chattering via Overlapping Spatial Partitions: Phase 1 (Overlapping Spatial Partitions Create Hysteresis): The Upper Partition (blue) and Lower Partition (green) extend across the centerline to form a central Overlapping Region where both bypass routes are valid. Phase 2 (Initialized Upper Route Commits Despite Upward Shift): When initialized to the Upper Route policy (blue), the obstacle shifts upward—making the lower path geometrically shorter (red dashed curve). Because the vehicle remains inside the Upper Partition, it ignores the shorter route and commits to the upper trajectory with 0 route reversals. Phase 3 (Initialized Lower Route Commits Despite Downward Shift): Symmetrically, when initialized to the Lower Route policy (green) and confronted with a downward obstacle shift, the vehicle stays inside the Lower Partition and commits smoothly beneath the obstacle with 0 route reversals.

Scaling from Single-Obstacle State Feedback to Multi-Obstacle Visual Perception

HyRL (ACC 2022) (de Priester et al., 2022) introduced this hysteresis-based RL architecture for single-obstacle avoidance using state coordinates (below left), proving that overlapping training sets prevent adversarial chattering along $\mathcal{M}^\ast$.
MultiHyRL (RLC 2024) (de Priester et al., 2024) generalized the framework to cluttered multi-obstacle 2D arenas and local bird’s-eye-view visual observations (below right), combining a CNN-MLP perception architecture with directional hysteresis and obstacle-focus switching across relaxed Voronoi cells.

HyRL Single-Obstacle Avoidance Trajectories (ACC 2022, Fig. 5)

MultiHyRL Closed-Loop Control Architecture (RLC 2024, Fig. 2)

Left (HyRL — ACC 2022, Figure 5 (de Priester et al., 2022)): Under measurement noise ($\epsilon = 0.1$), a standard DQN policy (red dashed) chatters along $y = 0$ and crashes into the obstacle $\mathcal{O}$, whereas HyRL (blue solid) commits to passing above ($q_0 = 0$) or below ($q_0 = 1$). Right (MultiHyRL — RLC 2024, Figure 2 (de Priester et al., 2024)): Closed-loop control architecture combining local bird's-eye view CNN features with relative target error MLP features.

Multi-Obstacle Relaxed Voronoi Partitions (RLC 2024, Fig. 5)

Multi-Obstacle State-Space Partitioning (MultiHyRL — RLC 2024, Figure 5 (de Priester et al., 2024)): Relaxed Voronoi partitioning across four obstacles ($1, 2, 3, 4$), where blue and green regions denote clockwise and counterclockwise bypass domains and the red region denotes their hysteresis overlap around each obstacle and the target $p^\ast$.

2D Multi-Obstacle Avoidance Under Adversarial Perturbations

2D Cumulative Success Rate Across Perturbation Magnitudes

Left (2D Multi-Obstacle Trajectories Under Adversarial Noise, $\varepsilon_p = 0.20$): Standard continuous PPO (red dashed) chatters and collides with the obstacles (gray circles; black circles mark terminal states), while hybrid hysteresis switching (blue and green trajectories) routes all initial conditions safely to the target (gold star). Right (2D Cumulative Success Rate Across 1,000 Runs): As perturbation magnitude increases to $\varepsilon_p = 0.20$, Standard PPO (solid lines) drops below 25% success, whereas our hybrid hysteresis framework (dashed lines) achieves 100% collision-free convergence.

2. Extending to 3D Flight & Attitude Tracking: L-MultiHyRL Results (2026)

In our latest work on Latent Multi-Policy Hysteresis-Based Reinforcement Learning (L-MultiHyRL) (de Priester et al., 2026), we extend hysteresis-based reinforcement learning from 2D planar navigation to full 3D aerial flight in $\text{SE}(3)$ and rotational attitude tracking in $\text{SO}(3)$ using local onboard visual observations.

In 3D flight, a vehicle faces two simultaneous topological obstructions:

3D Spatial Obstacle Avoidance: Breaking symmetry around 3D obstacles in cluttered environments under position observation noise $\varepsilon_p$.
3D Attitude Tracking on $\text{SO}(3)$: Because antipodal unit quaternions $\pm q_e$ represent the same physical 3D orientation via opposite rotation directions, continuous policies encounter a critical switching boundary at a $180^\circ$ rotation error ($\eta_e = 0$), where attitude noise $\varepsilon_q$ induces violent angular velocity chattering.

(Because the L-MultiHyRL manuscript is currently under review, we omit the internal algorithmic and architectural details here and highlight the 3D simulation and physical hardware validation results below.)

Across 1,000 randomized 3D initial conditions under worst-case adversarial observation shifts ($\varepsilon_p = 0.20\text{ m}$) and attitude disturbances ($\varepsilon_q = 0.4$), Standard PPO suffers from severe chattering—colliding with obstacles in 3D space and stalling at unaligned orientations near $\eta_e = 0$. In contrast, L-MultiHyRL achieves 100.0% collision-free convergence across every tested noise level:

Standard RL Policy Attitude Chattering at the Equator

L-MultiHyRL Chatter-Free Attitude Convergence

3D Attitude Tracking Under Adversarial Noise ($\varepsilon_q = 0.4$): Left (Standard PPO): Initial orientations near the antipodal boundary ($\eta_e(0)$ near $0$) are trapped at $\eta_e = 0$ as angular velocity commands $u_y$ and $u_z$ chatter at maximum frequency. Right (L-MultiHyRL): Hybrid hysteresis eliminates control chattering and smoothly stabilizes all initial orientations.

Multi-Obstacle 3D Simulation Trajectories Under Adversarial Noise

3D Multi-Obstacle Navigation Under Adversarial Observation Perturbations ($\varepsilon_p = 0.20\text{ m}$): Top-down ($p_x$-$p_y$, left) and side ($p_x$-$p_z$, right) projections from initial conditions (black crosses, with terminal states marked by black circles). Standard PPO (red dashed) chatters and collides with the spherical obstacles (gray spheres), while L-MultiHyRL smoothly routes all trajectories to the target (gold star) via committed bypass modes (blue and green).

3D Cumulative Success Rate Across Perturbation Magnitudes

3D Cumulative Success Rate Over Time (1,000 Runs per Condition): As the adversarial perturbation magnitude increases from $\varepsilon_p = 0.00$ to $\varepsilon_p = 0.20$, Standard PPO (solid curves) degrades to 60.4% success in the single-obstacle arena and 68.6% in the multi-obstacle arena. L-MultiHyRL (dashed curves) maintains 100.0% collision-free convergence across all noise levels.

3. Physical Flight Validation on a ModalAI Sentinel Quadrotor

To verify that these robustness gains transfer to real-world aerial hardware under aerodynamic disturbances and closed-loop tracking delays, we deployed the simulation-derived closed-loop hybrid system—consisting of the trained local policies, latent partition classifier, and discrete hysteresis logic—directly onboard a physical ModalAI Sentinel quadrotor (VOXL 2 companion computer running at 25 Hz over ROS 2 / PX4) inside a motion-capture flight arena without retraining or hardware-specific fine-tuning.

Hardware Flight Trajectories on the ModalAI Sentinel Quadrotor

Physical Quadrotor Multi-Obstacle Flights Under Perturbations ($\varepsilon_p = 0.25\text{ m}$): Top-down ($p_x$-$p_y$, left) and side ($p_x$-$p_z$, right) flight paths recorded on the ModalAI Sentinel quadrotor (black crosses mark initial states; black circles mark terminal states). Standard PPO (red) chatters between passage routes and crashes into the obstacles (gray spheres) for a subset of initial conditions, whereas L-MultiHyRL (blue and green) commits to a single passage route and reaches the target across all evaluated initial conditions.

Hardware Attitude Chattering Under Standard PPO

Hardware Chatter-Free Attitude Convergence Under L-MultiHyRL

Physical Quadrotor Attitude Tracking Under Perturbations ($\varepsilon_q = 0.4$): Left (Standard PPO): Initial orientations near the antipodal boundary trigger severe hardware angular rate chattering and stall near $\eta_e = 0$. Right (L-MultiHyRL): Hybrid hysteresis eliminates chattering on the physical quadrotor and drives all trajectories smoothly to the desired orientation.

🛠️ Broader Impact: A Foundation for Robustness in Modern AI & Robotics

Critical regions of indecision are not limited to obstacle avoidance—they arise whenever a continuous or discrete policy must choose between equally viable options. Left unaddressed, these boundaries can be easily exploited by small perturbations or an adversary to trigger deadlocks or crashes. Crucially, scaling up to larger neural networks and more advanced AI models does not eliminate this vulnerability: in higher-dimensional state spaces, critical decision boundaries persist in far more places than expected. From 2D state-based HyRL (ACC 2022) (de Priester et al., 2022) and 2D vision-based MultiHyRL (RLC 2024) (de Priester et al., 2024) to 3D L-MultiHyRL (2026) (de Priester et al., 2026), our methods provide a foundational layer of robustness against these structural failure modes across robotics and sequential decision-making for safety-critical applications.

Deep reinforcement learning (RL) enables agile autonomous navigation from onboard sensory data, but standard continuous and memoryless RL-based policies lack robustness in topologically obstructed spaces. Specifically, when bypassing obstacles in SE(3) or tracking attitudes in SO(3), topological obstructions create decision boundaries where perturbations can induce rapid switching (chattering) between passage routes or rotation directions, trapping the vehicle at unaligned attitudes or causing collisions. To resolve these vulnerabilities, we propose Latent Multi-Policy Hysteresis-Based RL (L-MultiHyRL) for robust, perception-driven obstacle avoidance in SE(3). L-MultiHyRL identifies the critical points of a baseline RL policy, where trajectories locally diverge around an obstacle, and partitions the state space into overlapping, contractible partitions. It then predicts whether the vehicle is inside a partition directly from onboard sensory data and trains a specialized local policy for each partition. Similarly, to resolve the obstruction in SO(3) attitude tracking, it defines overlapping tracking domains for the antipodal quaternion set-points. Finally, a discretized closed-loop hybrid system orchestrates the switching between these local policies and attitude set-points using hysteresis logic to achieve robustness against chattering. Simulations and physical quadrotor experiments demonstrate that L-MultiHyRL is robust against adversarial perturbations that cause a standard continuous RL-based policy to get trapped at unaligned attitudes and crash in single- and multi-obstacle environments.

Standard continuous reinforcement learning policies lack formal robustness guarantees against adversarial perturbations in the observation space during obstacle avoidance, where topological obstructions create vulnerable decision boundaries. MultiHyRL identifies critical points of a baseline policy and constructs overlapping relaxed Voronoi partitions with hysteresis switching, guaranteeing robust navigation across multi-obstacle environments.

Combines hybrid control theory with reinforcement learning by embedding hysteresis-based switching logic across overlapping state-space regions, eliminating chattering vulnerabilities in obstacle avoidance tasks.

Hysteresis-Based Reinforcement Learning for Chatter-Free Autonomy

📄 Research Overview: HyRL (ACC 2022) • MultiHyRL (RLC 2024) • L-MultiHyRL (2026)

1. How Hysteresis-Based RL Works: Overlapping Partitions & Hysteresis Logic

Scaling from Single-Obstacle State Feedback to Multi-Obstacle Visual Perception

2. Extending to 3D Flight & Attitude Tracking: L-MultiHyRL Results (2026)

3D Simulation Benchmark: Single- & Multi-Obstacle Navigation

3. Physical Flight Validation on a ModalAI Sentinel Quadrotor

🛠️ Broader Impact: A Foundation for Robustness in Modern AI & Robotics

References

2026

2024

2022