Parallel simulated tasks
Three manipulation tasks across randomized objects, textures, lighting, cameras, and physics. Record every seed and environment revision.
Built and measured
Current work
Each status describes the artifact that exists today, not the final system I eventually want to build.
Working local demo · deployment in progress
Tracklet: episode-centric robot annotation
Tracklet adds persistent object identity, masks, grasp points, and subtask phases to LeRobot demonstrations. Its key shortcut is physical: gripper width reveals grasp and release boundaries, so deterministic signal processing proposes the episode structure and SAM handles only the pixel segmentation.
28.5 MBQuantized SAM 2.1 tiny encoder downloaded to the browser.
6.98–7.05Holding plateau across six SO-101 episodes.
2.2 s / 65 framesOptional warm H100 propagation path, about 30 fps.
Matched 50-seed evaluation
Learned control
Diffusion policy for manipulation
A 262M-parameter PushT policy trained from scratch and evaluated beside the published reference under identical rollout seeds and confidence intervals.
Planning prototype · 40 paired seeds Motion planningGPS-denied planning, scoped honestly
Exact signed-distance fields, sphere-traced collision checks, and controlled RRT comparisons in a known map. The estimator and closed-loop flight controller remain unbuilt.
Embodied stack
Where the projects connect
The next experiment should reuse the same episodes, simulators, metrics, and deployment path instead of becoming another isolated demo.
01 · Demonstrate
Teleoperation + episodes
Observations, actions, language, proprioception, contact, and provenance.
02 · Represent
VLA + world state
Object tracks, spatial tokens, latent dynamics, action chunks, and uncertainty.
03 · Improve
Imitation + planning + RL
Behavior cloning first, then model-predictive planning and policy improvement.
04 · Prove
Held-out worlds
New objects, layouts, viewpoints, perturbations, latency, safety, and sim-to-real gap.
Build specification
World-model and VLA simulation lab
A serious next flagship needs a running simulator, artifact-backed evaluations, and a narrow real-world transfer target.
Imitation and VLA baselines
Compare a compact action-chunking or diffusion baseline with a pretrained VLA adapter under the same episodes and action representation.
Learned dynamics
Predict future latent observations and rewards, show predicted-versus-actual rollout strips, then use the model for receding-horizon selection.
Generalization, not memorization
Success heatmaps on held-out objects and layouts, calibration of predicted success, compute/success Pareto curves, and failure videos.
Promotion rule. This becomes a project card only when
the simulator and baseline run from a public repository, the evaluation
split is frozen before tuning, and both successful and failed rollouts
are visible.
Technical library
Foundations behind the builds
Long-form notes are supporting depth, not evidence that an unlinked product already exists.
How to use Isaac Sim
USD, PhysX, RTX sensors, hardware constraints, installation, and the cases where a purpose-built simulator is the better tool.
How to use Isaac Lab
GPU-parallel robot learning, environment APIs, version compatibility, training, and the Isaac Gym-to-Lab lineage.
Embodied foundation models
Behavior cloning, DAgger, diffusion policies, VLA models, world models, sim-to-real, and evaluation.
Robot generalization system design
Fleet data, interventions, mixture design, held-out worlds, and the improvement flywheel.
Teleoperation
Latency, control interfaces, observation/action recording, safety, and demonstration quality.
Decision-making under uncertainty
Belief state, planning, risk, and the boundary between learned and model-based control.
How to use Isaac Sim
USD stage, the SimulationApp bootstrap, sensors, and the RT-core requirement that decides which machines can run it at all.
How to use Isaac Lab
GPU-parallel robot learning, the 2.0 import rename that breaks most tutorials online, and why a datacenter GPU cannot run it.