Multi-View Detection

Work with synchronized multi-camera datasets. This guide uses the FLIR ADAS v2 dataset (RGB + thermal) and covers: import in image or video mode, browse synchronized views side by side, and annotate across modalities.

Prerequisites

  • Pixano installed (see Installation)
  • A data directory initialized with pixano init ./my_data
  • The FLIR ADAS v2 dataset downloaded locally (video test sets)
FLIR ADAS v2 layout

This script uses the video test sets which contain time-synced RGB/thermal frame pairs documented in rgb_to_thermal_vid_map.json. The expected layout:

flir_adas_v2/
  rgb_to_thermal_vid_map.json
  video_rgb_test/
    coco.json
    data/
      video-{vid}-frame-{n}-{hash}.jpg
  video_thermal_test/
    coco.json
    data/
      video-{vid}-frame-{n}-{hash}.jpg

Import the dataset

FLIR ADAS v2 supports two import modes: image (individual frame pairs) and video (grouped by video sequence).

Image mode — static frame pairs

Generate a sample of individual RGB + thermal frame pairs:

Terminal window
python examples/flir/generate_sample.py ./flir_sample /path/to/flir_adas_v2 \
    --mode image --num-samples 50

The resulting folder:

flir_sample/
  test/
    rgb/
      000000.jpg
      000001.jpg
      ...
    thermal/
      000000.jpg
      000001.jpg
      ...
    metadata.jsonl

Each line in metadata.jsonl pairs an RGB image with its thermal counterpart and includes bounding boxes detected in the thermal view:

{
  "views": {
    "rgb": "rgb/000000.jpg",
    "thermal": "thermal/000000.jpg"
  },
  "attrs": { "status": "validated" },
  "entities": [
    {
      "attrs": { "category": "person" },
      "annotations": [
        { "kind": "bbox", "view": "thermal", "coords": [0.12, 0.25, 0.15, 0.30], "format": "xywh", "is_normalized": true }
      ]
    }
  ]
}

With more than one view declared, every annotation names its view — here the bounding box was detected in the thermal view.

The schema for image mode lives in dataset.yaml at the source root:

pixano: 2
format: pixano_jsonl
dataset:
  name: flir_static_sample
  workspace: image
schema:
  views:
    rgb: { kind: image }
    thermal: { kind: image }
  entity:
    attrs:
      category: str
  annotations: [bbox]

Note views has two entries: "rgb" and "thermal", both images. This tells Pixano to display them side by side.

Import:

Terminal window
pixano data import ./my_data ./flir_sample

Video mode — synchronized sequences

Generate a sample grouped by video:

Terminal window
python examples/flir/generate_sample.py ./flir_video_sample /path/to/flir_adas_v2 \
    --mode video --num-samples 5

The resulting folder:

flir_video_sample/
  test/
    rgb/
      video_000/*.jpg
      video_001/*.jpg
    thermal/
      video_000/*.jpg
      video_001/*.jpg
    bboxes/
      video_000/*.json
      video_001/*.json
    metadata.jsonl

Each metadata entry uses glob patterns for frame sequences and per-frame annotation files:

{
  "attrs": { "status": "validated" },
  "views": {
    "rgb": { "frame_pattern": "rgb/video_000/*.jpg", "fps": 30 },
    "thermal": { "frame_pattern": "thermal/video_000/*.jpg", "fps": 30 }
  },
  "annotation_files": [
    { "kind": "bbox", "view": "rgb", "pattern": "bboxes/video_000/*.json", "encoding": "track_json" }
  ]
}

Frames matching each frame_pattern are imported in lexicographic order at the declared fps; the per-frame track_json files are stem-matched to their frames. The video-mode schema:

pixano: 2
format: pixano_jsonl
dataset:
  name: flir_dynamic_sample
  workspace: video
schema:
  views:
    rgb: { kind: sequence_frames }
    thermal: { kind: sequence_frames }
  entity:
    attrs:
      category: str
  annotations: [bbox, tracklet]

Import:

Terminal window
pixano data import ./my_data ./flir_video_sample

Launch the server

Terminal window
pixano server run ./my_data

Explore in the UI

Multi-view image mode

When you open an item with multiple views, Pixano tiles the images side by side. You can:

  • Pan and zoom each view independently.
  • Drag views to rearrange them.
  • Double-click a view to bring it to the front.

Annotations are per-view: a bounding box on the thermal image is separate from one on the RGB image, but both can be linked to the same entity.

Multi-view video mode

In video mode, the timeline synchronizes all views. When you play or step through frames, all views update together. The Video Inspector shows tracks across all views — tracklet bars are stacked by view on each track row.

Annotation across views

You can annotate objects in both the RGB and thermal views. Create a bounding box in the thermal view, then switch to the RGB view and add another annotation for the same entity. Pixano links them through the shared entity reference.

Annotate

All annotation tools work the same as in single-view mode:

  • Bounding boxes — click and drag on any view.
  • Polygons, polylines and brush — available per view.
  • Smart segmentation — with a pixano-inference server connected and a segmentation model available, the magic wand works on any view.

Every annotation you draw is linked to an entity, so the same entity can carry boxes in several views at once.

Next steps

Esc
Type to search the documentation