Image labelers know what exists in one frame. Robot learning needs to know that the brick approached at frame 40 is the same brick held at frame 120, where contact began, and when the subtask ended. Tracklet makes the episode (not the image) the unit of annotation.
The actual workspace: episode video, persistent track identity, pixel tools, and robot signals in one timeline.
The product decision
COCO-style annotation gives an object a new identity in every
image. That is acceptable for still-image detection and lossy for
manipulation data: identity, contact, motion, and phase are
temporal. Tracklet therefore stores an episode-scoped
track_id over frame intervals, with masks and grasp
points as observations of that persistent object. It exports a
lossless native format, COCO with an explicit non-standard
track_id, and LeRobot-shaped interval events with
pixel masks in a sidecar.
Identity persists through time; export adapters make information loss visible instead of silently discarding it.
A sensor is sometimes the best model
A gripper closing on empty air reaches its mechanical stop. When a brick is between the jaws, it stops early. In the bundled episodes those states form three distinct levels: empty rest around 1.2–1.4, holding around 7.0, and fully open around 20–25. The holding plateau appears at 6.98, 7.05, 6.99, 6.98, 6.98, and 6.99 across six independent episodes.
A hysteretic three-state detector turns those levels into grasp and release proposals. A human confirms them; no VLM is needed for the boundary. SAM is reserved for the genuinely visual problem: turning a point or box into a mask.
The detector reads the physical three-level signal, not a fragile open/closed threshold.
Why the model downloads instead of the embeddings
The default path runs entirely in the browser using ONNX Runtime Web. In a headless Chromium worst case with software rendering, model download and load took 4.2 seconds, the first click on a frame took 12.6 seconds including encoding, and a second click on the cached frame took 2.8 seconds. A SAM 2 embedding is about 16 MiB per frame; a 20-second, 30 fps episode would require roughly 9.4 GiB. Shipping the 28.5 MB quantized encoder is the better exchange after only a few frames.
An optional GPU service adds 0.05-second warm point masks and 65-frame propagation in 2.2 seconds on an H100. It is capability probed and additive: if it is unavailable, the browser workflow stays intact.
What I would measure next
- Mask quality and edit time against a manually audited robot-scene set.
- Track consistency through occlusion and across cameras.
- Policy success with and without phase, mask, and grasp supervision.
- Annotator agreement and time saved per episode, not clicks alone.