Egocentric video (training data)
How it works
1) People record activities with a first-person worn camera (optionally with audio, IMU, gaze tracking). 2) The data is curated and (optionally) annotated: narrations, actions, hand-object interactions. 3) A model learns from the video โ self-supervision / video pretraining or learning from demonstration โ acquiring representations of dynamics, objects and activities. 4) The representations/policies are transferred (transfer/fine-tuning) to a robot, accounting for the embodiment gap (hand-to-gripper retargeting, fine-tuning on a small amount of robot data).
Problem solved
Robot-collected demonstrations (teleoperation) are expensive and hard to scale. Human egocentric video provides a cheap, abundant, robot-viewpoint-aligned source of data about manipulation and behavior in the world.
Components
A head-/body-worn camera capturing the scene from a first-person viewpoint (optionally with IMU, audio, gaze tracking).
Official
A curated, large-scale collection of egocentric recordings, often with annotations (narrations, actions, interactions).
Official
The process of learning from video (pretraining/SSL or imitation) and transferring representations/policies to a robot.
Official
Implementation
Human hands and body differ from a gripper/robot, hindering direct policy transfer.
Raw video lacks control commands and body state, limiting direct policy learning.
First-person recordings may contain sensitive data and carry domain bias.
Evolution
A large-scale egocentric dataset of kitchen activities popularizes egocentric video in research.
Meta and a consortium release Ego4D, a massive egocentric video dataset, establishing it as a training source for embodied AI.
Egocentric (and human) video becomes common pretraining fuel for VLA and video-action (VAM) models, alongside Ego-Exo4D.
Hyperparameters (configurable axes)
RGB, optionally audio, IMU, gaze tracking, depth.
Narrations, action labels, hand-object interactions, or none (self-supervised).
Amount of footage; key for pretraining quality.
Computational complexity
Time complexity: Zalezna od skali danych i modelu. Space complexity: O(godziny ร rozdzielczosc ร FPS).
Compute bottleneck
Processing/storing large-scale video is costly, and the difference between human and robot morphology (embodiment gap) plus the lack of explicit action/proprioceptive labels hinder direct transfer.
Execution paradigm
Concerns the training phase (learning from video).
No routing โ a training-data strategy.
Parallelism
Video pretraining is highly parallel (data and model).
Hardware requirements
Training models on large-scale video is highly compute-intensive and benefits from GPU/TPU accelerators.