Video Object Tracking with DAVIS

By the end of this page, you will understand how video object segmentation datasets map to Pixano’s data model and know how Pixano handles tracking and segmenting objects across frames.

Background: What is DAVIS?

DAVIS (Densely Annotated Video Segmentation) is a benchmark for video object segmentation, introduced in 2016 and extended through 2019. It addresses a fundamental challenge in video understanding: given the mask of an object in the first frame of a video, can a system accurately segment that object in every subsequent frame?

This is the semi-supervised setting — the model (or annotator) receives one reference annotation and must propagate it across the entire video, handling appearance changes, occlusions, camera motion, and deformation.

The DAVIS 2017 dataset contains 150 video sequences with pixel-accurate segmentation masks at every frame. Each video features one or more objects, each tracked independently. The annotation quality is exceptionally high — every frame has a manually verified, pixel-perfect mask.

DAVIS introduced evaluation metrics that are now standard in the field:

• J (Jaccard index) — measures region overlap between predicted and ground truth masks (similar to IoU)
• F (boundary F-measure) — evaluates how well the predicted boundary matches the ground truth
• J&F — the combined metric, averaging both to capture both region accuracy and boundary precision

The challenge influenced real-world applications in autonomous driving (tracking pedestrians and vehicles), video editing (rotoscoping and background removal), and surveillance (object re-identification).

From images to video: new challenges

If you worked through the Object Detection use case, you annotated objects in single images. Video introduces several new problems:

• Temporal consistency — An object must maintain the same identity across frames. A “car” in frame 1 should be the same “car” in frame 100, even if its appearance changes dramatically.
• Identity persistence through occlusion — Objects disappear behind other objects and reappear. The system needs to re-associate them with the correct identity.
• Per-frame variation — The same object can be visible, partially occluded, fully occluded, or out of frame at different times. You need to track these states separately from the geometric annotations.

To handle these challenges, Pixano introduces two concepts that don’t exist in image datasets:

• Tracklets — A tracklet represents a continuous temporal segment of an entity’s visibility. It defines when an entity appears and disappears within the video. Per-frame annotations (bounding boxes, masks) are linked to a tracklet, which is itself linked to an entity.
• EntityDynamicState — Stores per-frame attributes that change over time (e.g. visibility, occlusion level, pose) separately from geometric annotations. This decouples “what does the annotation look like?” from “what is the object’s state in this frame?”

The segmentation problem in Pixano terms

Here is how DAVIS’s structure maps to Pixano:

The key insight: in image datasets, an entity has annotations directly. In video datasets, an entity has a tracklet that spans multiple frames, and each frame’s annotation is linked to that tracklet. This extra level of indirection is what gives Pixano temporal tracking capabilities.

Dataset schema

The schema lives in a dataset.yaml file at the root of the source folder:

pixano: 2
format: pixano_jsonl
dataset:
  name: davis_2017
  workspace: video
schema:
  views:
    image: { kind: sequence_frames }
  entity:
    attrs:
      category: { type: str, default: object }
  annotations: [mask, tracklet]

Compared to the image dataset schema, several things are new:

• workspace: video — Switches the UI to video mode with a timeline, frame navigation, and track inspector.

• kind: sequence_frames instead of image — A sequence-frame view stores one row per frame with frame_index (integer position in the sequence) and timestamp (time in seconds). This is how Pixano knows the temporal ordering of frames within a video.

• tracklet — Creates a tracklet table. Each tracklet has start_timestep, end_timestep, start_timestamp, and end_timestamp fields that define when the tracked entity is visible. Per-frame masks link back to the tracklet.

• mask — Segmentation masks in compressed run-length encoding. Each mask is a per-frame annotation linked to a tracklet and entity.

You could also add per-frame entity state (e.g. entity_dynamic_state: { attrs: { occluded: { type: bool, default: false } } }) — DAVIS doesn’t need it beyond the mask itself.

Import the dataset

The Pixano Cookbook provides a script that exports DAVIS 2017 into a Pixano-compatible source folder.

Clone the cookbook and generate the data

git clone https://github.com/pixano/pixano-cookbook.git
cd pixano-cookbook

python data_importation/davis/generate_sample.py ./davis_sample /path/to/DAVIS \
    --num-samples 5 --splits train val

This produces a source folder with frames and masks organized by video:

davis_sample/
  train/
    frames/
      bear/
        00000.jpg
        00001.jpg
        ...
      bmx-trees/
        ...
    masks/
      bear/
        00000.png
        00001.png
        ...
    metadata.jsonl
  val/
    frames/...
    masks/...
    metadata.jsonl

What’s inside metadata.jsonl

Each line describes one video using glob patterns for frames and masks:

{
  "attrs": { "status": "validated" },
  "views": {
    "image": { "frame_pattern": "frames/bear/*.jpg", "fps": 24 }
  },
  "annotation_files": [
    { "kind": "mask", "view": "image", "pattern": "masks/bear/*.png", "encoding": "index_png", "entity_map": "auto" }
  ]
}

Notice the differences from image datasets:

• "views" uses an object with "frame_pattern" (a glob) and "fps" (frame rate, required — nothing is inferred) instead of a simple filename. Pixano expands the glob to discover all frames, ordered alphabetically, one sequence frame per file.
• "annotation_files" is a list of typed sidecars. "encoding": "index_png" means indexed PNGs where each pixel value is one object (0 = background); Pixano matches masks to frames by filename stem (e.g. 00000.png matches 00000.jpg). "entity_map": "auto" names the objects object_1, object_2, … — or map pixel values to names explicitly: {"1": "bear"}.
• Tracklets are derived automatically from the frames each object appears in — no manual bookkeeping.

Run the import

pixano init ./my_library
pixano data import ./my_library ./davis_sample

Pixano validates every line first and prints a plan; use --dry-run to only validate.

Launch and explore

pixano server run ./my_library

Open http://127.0.0.1:7492 and click the DAVIS dataset card. When you open a video record, the UI switches to video mode: the current frame is displayed in the center, and the Video Inspector appears at the bottom with the timeline and playback controls.

Key Pixano features for video tracking

Tracklets

A tracklet is the core concept that connects single-frame annotations into a temporal identity. Select an entity in the Entities tab to reveal its track as a horizontal bar along the timeline; you can pin up to two entities to compare them side by side. Colored segments within a bar represent tracklets — continuous periods where the entity is visible.

Keyframes (frames with real annotations) are shown as dots on the tracklet. Frames between keyframes can be interpolated automatically for bounding boxes and keypoints.

You can split, merge, and relink tracklets to correct tracking errors — for example, splitting a tracklet when an object is temporarily occluded, or merging two tracklets that belong to the same entity after an identity switch.

Bbox tracking

For bounding box workflows in video, Pixano can propagate a bounding box annotation across frames. You annotate a few keyframes, and Pixano interpolates the box positions for the frames in between. This dramatically reduces the annotation effort — instead of drawing a box on every frame, you only need to annotate the frames where the object’s position changes significantly.

Interactive segmentation in video (SAM)

SAM integration extends to video: you can use the smart segmentation tool on any individual frame to generate a pixel-level mask from point prompts. When connected to a Pixano Inference server running SAM2, you can propagate masks across frames — similar to the DAVIS semi-supervised task itself.

This is particularly powerful for video annotation: annotate a precise mask on one frame, propagate it forward, then correct on frames where the prediction drifts. The combination of tracklets (for temporal identity) and SAM (for per-frame precision) mirrors the workflow that made DAVIS such an influential benchmark.

EntityDynamicState

While tracklets and masks capture where an object is, EntityDynamicState captures what state the object is in at each frame. This is stored separately from geometric annotations because the same object can have different per-frame attributes (occluded, truncated, moving, stationary) independent of its mask or bounding box.

In the DAVIS example, DAVISEntityDynamicState is empty — but for your own datasets you can add custom fields. For example, in an autonomous driving dataset you might track:

class VehicleDynamicState(EntityDynamicState):
    is_occluded: bool = False
    is_moving: bool = True
    visibility: float = 1.0

This separation keeps your annotation tables clean and lets you query temporal attributes independently of geometry.

What you’ve learned

Let’s connect the dots:

• DAVIS is a video object segmentation benchmark where objects are tracked and segmented with pixel-accurate masks across every frame.
• Moving from images to video introduces temporal challenges: identity persistence, occlusion handling, and per-frame variation.
• Pixano handles this with SequenceFrame views (ordered frames with timestamps), Tracklets (temporal segments linking per-frame annotations to entities), and EntityDynamicState (per-frame attributes separate from geometry).
• The video schema combines workspace: video, sequence_frames views, and tracklet annotations (optionally per-frame entity_dynamic_state attrs) — which together unlock the timeline UI, track management, and temporal annotation tools.
• SAM in video combines tracklets for temporal identity with per-frame segmentation, enabling efficient annotation workflows that mirror the DAVIS semi-supervised task.

Next steps

• Object Detection with Pascal VOC — if you haven’t already, start with the image use case to understand the foundations
• Key Concepts — deeper dive into the data model
• API Reference — full Python API documentation