⚛️ UC Santa Cruz • Hybrid Systems Lab

Jan de Priester

Ph.D. Candidate & Safe RL Researcher | UC Santa Cruz

I am a Ph.D. candidate in Electrical and Computer Engineering at the UC Santa Cruz Hybrid Systems Laboratory (expected Sept. 2027), advised by Prof. Ricardo G. Sanfelice. With 5+ years of experience in reinforcement learning and hybrid control, I develop learning-based control methods with formal safety, stability, and robustness guarantees.

My work focuses on building RL policies (Python, PyTorch, JAX) that prevent failure in adversarial edge cases—such as indecision and chattering around obstacles—and transfer reliably to physical hardware, as well as proving mathematically when standard RL policies intrinsically guarantee closed-loop stability. I am a first-author researcher (RLC 3×, ACC) and serve as a Senior Reviewer for RLC.

🎯 Seeking a 2027 Research Scientist role or internship in the SF Bay Area.

Explore Research Directions Read Research Articles Publications CV (PDF)

Ph.D. Candidate (2027)

UC Santa Cruz • ECE

Santa Cruz, CA

Research Overview

High-Level Research Directions

My research combines reinforcement learning with control and hybrid systems theory so that learned policies remain stable, decisive, and robust in the real world. Explore the three core directions below through visual animations and interactive articles.

Direction I • Hysteresis-Based RL & Hybrid Supervision HyRL • MultiHyRL • L-MultiHyRL • Timer Supervision

Hysteresis-Based Reinforcement Learning for Robust, Chatter-Free Autonomy

1. The Problem: Indecision & Chattering in Standard RL

When an autonomous agent navigates around obstacles or chooses between competing tasks, it must commit to a viable route. Because standard RL policies are continuous or memoryless discrete mappings, obstacles create decision boundaries where opposing paths diverge. Near these boundaries, small sensor noise or adversarial perturbations can make an alternative route appear momentarily better—causing the agent to rapidly switch back and forth (chattering), stall, or crash.

⚡ Why Standard & Recurrent RL Fall Short

Recurrent policies (RNNs) and frame stacking embed memory via continuous hidden states—which merely shifts the decision boundary into a higher-dimensional space rather than eliminating it. Under adversarial observation perturbations, standard RL obstacle-avoidance success drops to ~60% in 3D and below 25% in 2D.

🛠️ Our Approach (Visualized Below)

We combine RL policies with hybrid discrete memory to guarantee decisive commitment—using overlapping spatial hysteresis partitions (HyRL, MultiHyRL & L-MultiHyRL, with L-MultiHyRL predicting partition membership directly from onboard perception via learned latent features) or predictive dwell-time timers (Timer-Based Hybrid Supervisor) when switching boundaries between policies are complex or unknown in advance.

1. Baseline Failure: Chattering & Crash ● Standard Continuous RL

Why Memoryless RL Policies Fail Around Obstacles: As the vehicle steers toward the upper route (Phase 1), an adversarial perturbation shifts the obstacle upward to block its path. Blocked above, the memoryless policy switches to the lower route (Phase 2), where a downward shift blocks it again—triggering repeated route reversals (Phase 3) and a head-on collision.

2. Solution I: Spatial Hysteresis Partitions ● HyRL • MultiHyRL • L-MultiHyRL

Hysteresis-Based RL (Onboard Perception): Upper (blue) and Lower (green) partitions overlap around the centerline so both bypass routes are valid (Phase 1). When the obstacle shifts upward (Phase 2) or downward (Phase 3) to make the opposing route shorter, the agent stays inside its active partition, ignores the shortcut, and commits with 0 route reversals.

3. Solution II: Predictive Dwell-Time Timer ● Timer Supervisor (RLC '25)

Timer-Based Hybrid Supervisor (Complex Value Boundaries): When switching boundaries between multiple policies are fragmented or unknown in advance, our supervisor uses a short forward prediction (Phase 1) to forecast when an alternative policy becomes significantly better ($V_1$ exceeding $V_2$), locking in an adaptive dwell timer ($\delta_d = 5$) to safely clear each obstacle.

💡 Why It Matters & Where It Extends

Critical regions of indecision are not unique to obstacle avoidance—they arise whenever a continuous or discrete policy must choose between equally viable options. Left unaddressed, these boundaries are easily exploited by small perturbations or an adversary to trigger deadlocks or crashes. Crucially, scaling up to larger AI models does not remove this vulnerability: in higher-dimensional state spaces, critical decision boundaries persist in far more places than expected. Our methods provide a foundational layer of robustness against these structural failure modes across robotics and sequential decision-making for safety-critical applications.

📖 Deep-Dive Articles & Related Work

By embedding discrete hysteresis memory and predictive dwell timers, our methods achieve 100% obstacle avoidance success in simulation (vs. 60% for standard RL) and transfer reliably to physical hardware while preserving underlying policy performance and stability. Explore the full details in our interactive articles:

📖 Article: Hysteresis-Based RL (HyRL, MultiHyRL & L-MultiHyRL) 📖 Article: Timer-Based Hybrid Supervisor (RLC '25)

Direction II • Theoretical Foundations of RL Closed-Loop Practical Stability • Continuous Control

Intrinsic Practical Stability in Reinforcement Learning

1. The Setting & Core Question

Many continuous control tasks in robotics—such as legged locomotion, drone racing, and manipulation—are set-stabilization problems: the policy must steer the system toward and keep it within a desired target set or trajectory. Existing safe RL methods often constrain the learning algorithm with Lyapunov conditions or external safety filters, which can restrict exploration and complicate training.

2. Main Theoretical Result (RLC 2026)

We prove mathematically that, for deterministic continuous control tasks with a continuous optimal policy, standard discounted RL policies intrinsically guarantee closed-loop practical asymptotic stability without imposing constraints on the learning process. Modeling the discount factor ($\gamma < 1$) and neural network approximation error ($\delta > 0$) as bounded disturbances shows that stability scales directly with learning performance: trajectories converge into a bounded neighborhood around the target that shrinks to zero as approximation error $\delta \to 0$ and, subsequently, $\gamma \to 1$.

1. Animation: Stabilizing a Locomotion Limit Cycle ● RLC 2026

Practical Stability of a Periodic Locomotion Gait: A galloping quadruped tracks a repeating target gait (dashed loops). Phase 1: As training progresses, neural network approximation error shrinks and the guaranteed stability tube tightens. Phase 2: As the discount factor approaches $1$ (farsighted), the tube contracts to the exact target gait.

2. Theoretical Tubes & Steady-State Bounds ($\gamma, \delta$) ● Theorems 3.9 & 4.5

Theorem 3.9: Practical Stability Neighborhood under Discounting

Numerical Steady-State Bounds Across Discount Factors

Theorem 4.5: Practical Stability Neighborhood under Neural Network Approximation

Numerical Steady-State Bounds Across Approximation Errors

Analytical & Numerical Practical Stability Bounds: Top (Discounting $\gamma$): Trajectories converge into a neighborhood $\nu(\gamma)$ (blue) around $\mathcal{A}$ that shrinks toward zero as $\gamma \to 1$. Bottom (Network Error $\delta$): Approximation error expands the bound to $\tilde{\nu}(\gamma, \delta)$ (green), which contracts back to $\nu(\gamma)$ as $\delta \to 0$.

3. Why It Matters & Where It Extends

While training-time constraints remain valuable for safe exploration, this result shows that modifying the RL objective is often unnecessary to guarantee the stability of the deployed policy. It also explains why small steady-state offsets are a natural effect of discounting rather than an algorithmic bug, and guarantees that any target accuracy can be certified at a standard discount factor $\gamma < 1$.

📖 Interactive Deep-Dive Article & Paper

Explore the proof intuition and an interactive steady-state simulator where you can vary $\gamma$, $\delta$, and reward weights live in your browser:

📖 Article & Interactive Simulator: Intrinsic Stability in RL (RLC '26) 📄 RLC 2026 Paper PDF

Direction III • Physical Robotics & Sim-to-Real Validation ModalAI Sentinel UAV • 6-DoF Elfin e03 Manipulator

From Theory & Simulation to Physical Robots: Real-World Robustness Validation

1. The Goal: Scaling Theoretical Guarantees to Real Hardware

While our core methods are grounded in control theory and simulation, a central goal of this direction is showing that the robustness we design transfers reliably to physical robots. Real hardware introduces unmodeled aerodynamics, actuator delays, sensor drift, and contact dynamics—conditions where standard switching boundaries cause severe oscillations.

2. 3D Quadrotor Flight & Attitude Control (L-MultiHyRL)

We deploy the simulation-derived closed-loop hybrid system from L-MultiHyRL—comprising the trained local policies, latent partition classifier, and discrete hysteresis logic—directly onboard a physical ModalAI Sentinel quadrotor (VOXL 2 at 25 Hz over ROS 2 / PX4) without retraining. Under adversarial obstacle shifts ($\varepsilon_p = 0.25\text{ m}$) and $\text{SO}(3)$ attitude disturbances ($\varepsilon_q = 0.4$) where standard PPO chatters and crashes, the hybrid system commits decisively to a single passage route and converges smoothly.

3. Robotic Contact & Force Regulation (IEEE CCTA 2026)

On a 6-DoF Elfin e03 manipulator with an OnRobot RG2FT force/torque gripper, we formulate a two-level hybrid hysteresis controller across free-space position control (`MOVE`) and quantized gripper force modes (`TIGHTEN`, `HOLD`, `LOOSEN`). Unlike baseline switching—which suffers from rapid force oscillations—our hysteresis logic maintains stable force regulation under setpoint changes, shifting centers of gravity, changing object stiffness, and 30-minute sensor drift.

1. 3D Quadrotor Flight & Attitude Tracking ● ModalAI Sentinel UAV

Physical ModalAI Sentinel Quadrotor Multi-Obstacle Flight Trajectories

Hardware Attitude Chattering Under Standard PPO

Hardware Chatter-Free Attitude Convergence Under L-MultiHyRL

Closed-Loop Hybrid Deployment on a ModalAI Sentinel Quadrotor: Top: Physical 3D multi-obstacle flights ($\varepsilon_p = 0.25\text{ m}$)—Standard PPO (red) chatters and collides, while the L-MultiHyRL closed-loop hybrid system (blue/green) flies collision-free. Middle vs. Bottom: Hardware $\text{SO}(3)$ attitude tracking ($\varepsilon_q = 0.4$) comparing Standard PPO control chattering (middle) against smooth L-MultiHyRL convergence (bottom).

2. 6-DoF Manipulator & Contact Force Regulation ● IEEE CCTA 2026

RG2FT End Effector and 6-DoF Elfin e03 Manipulator Hardware Setup

Baseline Force Chattering Without Hysteresis

Chatter-Free Hysteresis Force Regulation

Hysteresis Contact & Force Control on a 6-DoF Manipulator: Left: Physical Elfin e03 6-DoF arm and RG2FT force/torque gripper. Right (Top): Baseline switching without hysteresis suffers from rapid state and contact-force oscillations. Right (Bottom): Our hysteresis controller smoothly regulates $3\text{ N}$ and $1\text{ N}$ contact force setpoints without chattering.

💡 Why It Matters & Where It Extends

Physical robots regularly operate across switching boundaries—both in 3D aerial obstacle avoidance and attitude tracking and in contact-rich manipulation between free-space motion and force control. Discrete hysteresis provides a lightweight, general mechanism that prevents hardware chattering and transfers reliably from simulation to the real world.

📖 Hardware Validation & Related Work

Explore the experimental flight results and physical manipulation studies in detail:

🚁 Quadrotor 3D Flight Validation (L-MultiHyRL) 🤖 6-DoF Hysteresis Contact Control (IEEE CCTA '26)

Accessible Deep Dives

Featured Research Articles

Read concise, visual explanations of our research—complete with intuitive diagrams, animations, and experimental results.

Hysteresis-Based RL • 2D & 3D ACC '22 • RLC '24 • 2026

Hysteresis-Based Reinforcement Learning for Chatter-Free Autonomy (HyRL, MultiHyRL & L-MultiHyRL)

How overlapping spatial hysteresis partitions eliminate structural indecision and adversarial chattering in safety-critical sequential decision-making—achieving 100% collision-free 2D/3D navigation and transferring reliably to a physical quadrotor.

Read Article →

RL Foundations • Practical Stability RLC 2026

Intrinsic Practical Stability in Discrete-Time Reinforcement Learning

Why modifying the core RL objective is often unnecessary for deployed stability: proving mathematically that standard discounted RL policies intrinsically guarantee closed-loop practical stability that scales with learning performance.

Read Article →

Hybrid Supervision RLC 2025

Timer-Based Hybrid Supervisor for Robust Policy Switching

Preventing indecision, chattering, and deadlocks across fragmented multi-policy value boundaries using short-horizon state predictions and an adaptive dwell-time timer while preserving underlying policy stability.

Read Article →

Updates

Recent News

Sep 15, 2026	Completed physical quadrotor hardware experiments for L-MultiHyRL: Latent Multi-Policy Hysteresis-Based RL for Robust 3D Obstacle Avoidance in $\text{SE}(3)$ and $\text{SO}(3)$ on a ModalAI Sentinel UAV! Read our interactive research article.
May 07, 2026	Our collaborative paper An Experimentally Validated Hybrid Control Strategy for Robust Contact Detection and Force Regulation has been accepted to CCTA 2026! This work demonstrates a hysteresis-based framework to eliminate end-effector chattering in physical 6-link manipulators.
May 05, 2026	Our paper Intrinsic Closed-Loop Practical Asymptotic Stability in Discrete-Time Reinforcement Learning has been accepted to RLC 2026! We prove that the unmodified discounted RL objective intrinsically guarantees practical asymptotic stability for continuous control tasks. Read the deep-dive article or PDF.
Dec 08, 2025	Officially advanced to Ph.D. Candidacy in the Department of Electrical and Computer Engineering at UC Santa Cruz!
May 09, 2025	A new paper titled A Timer-Based Hybrid Supervisor for Robust, Chatter-Free Policy Switching has been accepted to RLC 2025! This work introduces a hybrid supervisor to guarantee robust, chatter-free policy switching in RL.
May 14, 2024	A new paper titled MultiHyRL: Robust Hybrid RL for Obstacle Avoidance against Adversarial Attacks on the Observation Space has been accepted to RLC 2024! We present a novel hybrid RL algorithm featuring hysteresis-based switching for robust obstacle avoidance

Bibliography

Selected Publications

View all papers organized by research area on the full publications page →

Preprint

Latent Multi-Policy Hysteresis-Based Reinforcement Learning for Robust 3D Obstacle Avoidance

Jan de Priester, Joseph Lullo, Kyle Volle, and 2 more authors

2026

In preparation, target IEEE Transactions on Robot Learning (T-RL)

Article Abs

Deep reinforcement learning (RL) enables agile autonomous navigation from onboard sensory data, but standard continuous and memoryless RL-based policies lack robustness in topologically obstructed spaces. Specifically, when bypassing obstacles in SE(3) or tracking attitudes in SO(3), topological obstructions create decision boundaries where perturbations can induce rapid switching (chattering) between passage routes or rotation directions, trapping the vehicle at unaligned attitudes or causing collisions. To resolve these vulnerabilities, we propose Latent Multi-Policy Hysteresis-Based RL (L-MultiHyRL) for robust, perception-driven obstacle avoidance in SE(3). L-MultiHyRL identifies the critical points of a baseline RL policy, where trajectories locally diverge around an obstacle, and partitions the state space into overlapping, contractible partitions. It then predicts whether the vehicle is inside a partition directly from onboard sensory data and trains a specialized local policy for each partition. Similarly, to resolve the obstruction in SO(3) attitude tracking, it defines overlapping tracking domains for the antipodal quaternion set-points. Finally, a discretized closed-loop hybrid system orchestrates the switching between these local policies and attitude set-points using hysteresis logic to achieve robustness against chattering. Simulations and physical quadrotor experiments demonstrate that L-MultiHyRL is robust against adversarial perturbations that cause a standard continuous RL-based policy to get trapped at unaligned attitudes and crash in single- and multi-obstacle environments.
RLC

Intrinsic Closed-Loop Practical Asymptotic Stability in Discrete-Time Reinforcement Learning

Jan de Priester and Ricardo G. Sanfelice

In Reinforcement Learning Conference (RLC) / Reinforcement Learning Journal (RLJ), 2026

Article Abs PDF

Deep reinforcement learning (RL) has achieved impressive empirical success in complex control tasks, many of which are fundamentally set-stabilization problems. To safely deploy learned policies, existing methods often modify RL algorithms to enforce formal stability guarantees. However, these algorithmic modifications alter the core RL objective, which can restrict natural exploration and complicate training. This paper offers a complementary perspective: while such algorithmic modifications are valuable for training-time safety during exploration, modifying the core RL objective is often unnecessary to guarantee the stability of the deployed policy. We prove that the unmodified discounted RL objective intrinsically guarantees closed-loop practical asymptotic stability in deterministic continuous control settings. By adapting the discrete-time input-to-state stability (ISS) framework and drawing inspiration from discounted model predictive control (MPC), we model the suboptimality induced by the discount factor and the neural network approximation error as bounded disturbances. We establish that near-optimal parameterized feedback laws steer the closed-loop system into a bounded neighborhood of the target set. The size of this neighborhood vanishes in the limit as the approximation error approaches zero and, subsequently, the discount factor approaches one.
RLC

A Timer-Based Hybrid Supervisor for Robust, Chatter-Free Policy Switching

Jan de Priester and Ricardo G. Sanfelice

In Reinforcement Learning Conference (RLC) / Reinforcement Learning Journal (RLJ), 2025

Article Abs PDF

We address the challenge of switching among multiple learned policies in reinforcement learning control systems, where conventional value function-based methods can lead to chattering in the presence of small measurement noise. Our goal is to design a switching logic that assures asymptotic stability and maintains a robustness margin so that rapid switching is prevented under bounded measurement noise. To this end, we propose a timer-based hybrid supervisor that integrates a resettable timer that enforces a minimum dwell time on the active policy. This dwell time is adaptively adjusted by predicting the evolution of the state of the system, ensuring that a switch occurs only when a significantly better alternative is predicted. We derive sufficient conditions under which the hybrid supervisor is guaranteed to exhibit non-Zeno behavior and render a compact set robustly globally asymptotically stable in the presence of bounded measurement noise. Simulation results on representative decision-making problems demonstrate that our hybrid supervisor is robust under noisy conditions where a conventional switching strategy fails.
RLC

MultiHyRL: Robust Hybrid RL for Obstacle Avoidance against Adversarial Attacks on the Observation Space

Jan de Priester, Zachary Bell, Prashant Ganesh, and 1 more author

In Reinforcement Learning Conference (RLC) / Reinforcement Learning Journal (RLJ), 2024

Abs PDF

Standard continuous reinforcement learning policies lack formal robustness guarantees against adversarial perturbations in the observation space during obstacle avoidance, where topological obstructions create vulnerable decision boundaries. MultiHyRL identifies critical points of a baseline policy and constructs overlapping relaxed Voronoi partitions with hysteresis switching, guaranteeing robust navigation across multi-obstacle environments.