Entity Linking

Link named entities between image regions and text spans. This guide uses the MEL (Multimodal Entity Linking) dataset format and covers: import, explore the image + text UI, and annotate with bounding boxes and text selection.

What is entity linking?

Entity linking connects mentions of the same real-world entity across different modalities. For example, a sentence might mention “the Eiffel Tower” while the image shows the tower itself. Entity linking creates a shared entity that ties the text span to a bounding box (or mask) on the image.

Prerequisites

• Pixano installed (see Installation)
• A data directory initialized with pixano init ./my_data

Import the dataset

Prepare the source folder

An entity-linking dataset has an image and a text document per item. Organize your source folder like this:

mel_source/
  dataset.yaml
  train/
    images/
      image_001.jpg
      image_002.jpg
    metadata.jsonl

Media paths in metadata.jsonl are relative to the split folder, so images/image_001.jpg below resolves to mel_source/train/images/image_001.jpg.

Each line in metadata.jsonl describes one item with both an image view and a text view, plus entities that link regions across both:

{
  "views": {
    "image": "images/image_001.jpg",
    "text": { "content": "The Eiffel Tower is a wrought-iron lattice tower in Paris." }
  },
  "attrs": { "status": "validated" },
  "entities": [
    {
      "attrs": { "name": "Eiffel Tower" },
      "annotations": [
        { "kind": "bbox", "view": "image", "coords": [0.3, 0.1, 0.4, 0.8], "format": "xywh", "is_normalized": true },
        { "kind": "text_span", "view": "text", "mention": "Eiffel Tower", "spans_start": [4], "spans_end": [16] }
      ]
    }
  ]
}

• A text view takes {"content": …} for inline text or {"uri": "doc.txt"} for a file — exactly one, always explicit.
• A text_span carries the mention string plus parallel spans_start/spans_end character offsets (a mention can span several ranges).
• With two views declared, every annotation names its view: the bbox lives on image, the span on text.

Review the schema

The schema lives in dataset.yaml at the source root:

pixano: 2
format: pixano_jsonl
dataset:
  name: mel_sample
  workspace: image_text_entity_linking
schema:
  views:
    image: { kind: image }
    text: { kind: text }
  entity:
    attrs:
      name: str
  annotations: [bbox, mask, text_span]

Key elements:

• workspace: image_text_entity_linking enables the image + text panel UI.
• views has two entries: "image" (an image) and "text" (a text document).
• text_span is the annotation kind that marks character ranges in the text.
• Entities can be linked to both a bbox (on the image) and a text_span (on the text).

Run the import

pixano data import ./my_data ./mel_source

Pixano validates every line first and prints a plan; use --dry-run to only validate.

Launch the server

pixano server run ./my_data

Explore in the UI

Workspace — entity linking mode

When you open a record from an entity-linking dataset, you see:

• Image in the center — with bounding boxes and masks drawn on it.
• Text panel on the side — displaying the full text with color-coded spans. Each text span is highlighted with the same color as its linked entity’s bounding box.

Text panel interactions

• Click a text span to select the corresponding entity. The entity’s bounding box highlights on the image, and its card opens in the right panel.
• Select text and click “Tag selection” to create a new text span annotation. The creation form opens so you can assign it to an existing entity or create a new one.

Entities tab

The Entities tab on the right lists every entity in the record. Expand an entity card to see:

• Attributes — the entity’s fields (name, category, and anything else your schema declares).
• Annotations — every annotation attached to it (bounding boxes, masks, text spans), with each text span showing its highlighted text.

The neighbouring Record tab holds record-level information: status, record attributes, and per-view metadata.

Annotate

Bounding boxes

Use the bounding box tool on the image to draw regions around the entity. When prompted, select the entity you want to link it to.

Text spans

Select text in the text panel and click Tag selection. Choose the same entity that has the bounding box annotation.

Masks

If your schema lists mask, you can also use the polygon tool or smart segmentation to create segmentation masks linked to entities.

Next steps

• Visual Question Answering — image + conversation Q&A
• Object Detection — image-only detection with pre-annotation
• API Reference — full Python API documentation