pixano.datasets.dataset

Initialize the dataset. Args: path: Path to the dataset.

Add or replace a constraint. Args: table: Table name. field_name: Name of the field to constrain. values: List of allowed values. restricted: True if no other values are allowed.

Add data to a table. Args: table_name: Table name. data: Data to add. ignore_integrity_checks: List of integrity checks to ignore. raise_or_warn: Whether to raise or warn on integrity errors. Returns: The added data.

Append rows to the record-embedding table (plain float vectors, no model). Args: rows: Dicts with at least ``record_id`` and ``vector``; ``id``/``view_id`` optional.

Insert rows into multiple tables in a single call. Accepts the same ``dict[str, LanceModel | list[LanceModel]]`` format yielded by :meth:`DatasetBuilder.generate_data`. Tables are flushed in dependency order (source -> record -> view -> entity -> entity_dynamic_state -> annotation -> embedding) so that foreign-key constraints are satisfied. Args: data: Mapping of table name to one or more rows to insert. ``None`` or empty-list values are silently skipped. check_integrity: Integrity-check mode. ``"raise"`` (default) aborts on the first error, ``"warn"`` emits warnings, ``"none"`` skips validation. stamp_timestamps: Overwrite ``created_at``/``updated_at`` with now (default). Importers pass ``False`` so explicitly provided timestamps (e.g. a re-imported export) survive; rows without explicit values keep their construction-time defaults, which are equally "now".

Build the vector + record_id indexes on the record-embedding table. The vector (ANN) index needs enough rows to train; when there are too few, it is skipped and search falls back to an exact brute-force scan (still correct).

Compute the view embeddings via the embedding function stored in the table metadata. Args: table_name: Table name containing the view embeddings. data: Data to compute. Dictionary representing a view embedding without the vector field.

Count rows in a table, optionally filtered by a WHERE clause. Uses LanceDB native count_rows() which avoids full table materialization. Args: table_name: Table name. where: Optional WHERE clause to filter rows. Returns: Number of matching rows.

Create a new dataset on disk. This initialises the directory structure, writes ``info.json`` and ``features_values.json``, creates the LanceDB database with empty tables for every entry in ``info.tables`` and returns the opened :class:`Dataset`. Args: path: Directory where the dataset will be stored. The directory is created (with parents) and **must not already exist**. info: Dataset information including the ``tables`` mapping. If ``info.id`` is empty, a random ID is generated automatically. Returns: The newly created dataset. Raises: FileExistsError: If ``path`` already exists. ValueError: If ``info.tables`` is empty or invalid.

Create (or replace) the record-embedding table and persist its sidecar descriptor. Args: dim: Embedding dimensionality. model_id: Identifier of the model that produced the vectors. metric: Distance metric (``cosine``/``l2``/``dot``). source_view: Logical view the embeddings were computed from. index_type: LanceDB vector index type to build once populated.

Create scalar indexes on the given columns of the given tables. Idempotent: columns that are already indexed or absent from a table's schema are skipped. Indexes make ``merge_insert``, foreign-key lookups, export paging, and filtered pagination scale past full-table scans. Args: columns: Column names to index where present. Defaults to the standard filter columns (``FILTER_INDEX_COLUMNS``). tables: Table names to index. Defaults to every dataset table.

Add a table to the dataset. Args: name: Table name. schema: Table schema (must be a LanceModel subclass). data: Table data. mode: Table mode ('create', 'overwrite'). exist_ok: If True, do not raise an error if the table already exists. on_bad_vectors: Raise an error, drop or fill bad vectors ("error", "drop", "fill"). fill_value: Value to fill bad vectors. Returns: The table created.

Delete data from a table. Args: table_name: Table name. ids: Ids to delete. Returns: The list of ids not found.

Delete records and all associated data across all tables. Cascading deletion: deletes from the record table first, then from all auxiliary tables where ``record_id`` matches the given ids. Args: ids: Record ids to delete. Returns: The list of ids not found in the record table.

Delete the record-embedding table and its sidecar descriptor. Safe on missing/corrupt tables; used by force-recompute to escape states where the sidecar advertises embeddings whose table is broken.

Find a Dataset in a directory. Args: id: Dataset ID to find. directory: Directory to search in. Returns: The found dataset.

Search ids in a table. Args: table_name: Table name. ids: Ids to find. Returns: Dictionary of ids found. Keys are the ids and values are True if found.

Generate a preview for the dataset. It samples images from the dataset, creates a mosaic and saves it to the previews directory. Only images with embedded blob data are used. Returns: The preview base64 string.

Get all the ids from a table. Args: table_name: table to look for ids. sortcol: column to order by. order: sort order (asc or desc). where: where clause to filter ids. Returns: list of the ids.

Read one conversation aggregate from a messages table. Args: table_name: The messages table name. conversation_id: The conversation ID. Returns: The conversation or None if not found.

Read paginated conversation aggregates from a messages table. Args: table_name: The messages table name. record_id: Filter by record ID. entity_id: Filter by entity ID. view_id: Filter by view ID. source_type: Filter by source type. where: Additional WHERE clause. limit: Maximum conversations to return. skip: Number of conversations to skip. Returns: Tuple of (conversations, total_count).

Read data from a table. Data can be filtered by ids, record ids, where clause, or limit and skip. Args: table_name: Table name. ids: ids to read. limit: Amount of rows to read. If not set, will default to table size. skip: The number of rows to skip. where: Where clause. record_ids: Record ids to filter by (filters on ``record_id`` column). sortcol: column to order by. order: sort order (asc or desc). force_full_scan: Force the safe full-scan-then-slice path (for a ``where`` using an operator a scalar index can't serve, e.g. ``!=`` or ``LIKE``). See `TableQueryBuilder.force_full_scan`. Returns: List of values.

Locate a record within a filtered, sorted result set and its neighbors. Powers item-to-item navigation in the explorer: given the same filter and sort the table view uses, return the ids immediately before and after ``record_id`` and its 1-based position, so previous/next stay inside the current result set instead of walking the whole dataset. The scan projects only the ``id`` column (ordered with an ``id`` tie-break for a stable order) and returns a tiny payload. For a record that is not in the filtered set, ``prev``/``next``/``position`` are ``None``. Args: record_id: The record to locate. table_name: Table to navigate (defaults to the record table). where: Where clause matching the explorer's active filter. sortcol: Column to order by (defaults to ``id``). order: Sort order, ``"asc"`` or ``"desc"`` (defaults to ``"asc"``). Returns: ``{"prev": str | None, "next": str | None, "position": int | None, "total": int}``.

Read records from the record table. Args: ids: Record ids to read. limit: Amount of records to read. skip: The number of records to skip. Returns: List of records, a single record, or None.

Get record counts grouped by split and status. Opens the record table as PyArrow, groups by ``["split", "status"]`` and aggregates counts. If the ``status`` column does not exist the table is old-schema and an empty list is returned for backward compatibility. Returns: List of SplitStatusCount sorted by split asc then status asc.

Load one ordered batch of temporal view frames with their binary payloads. Returns a list of ``(frame_index, blob_bytes, format_string)`` tuples, reading from the embedded ``raw_bytes`` column. Args: table_name: View table name containing temporal frames. record_id: The record ID to filter by. view_name: Optional logical view name to filter by. start_frame: Starting frame index. batch_size: Number of frames to load. Returns: List of (frame_index, blob_bytes, format) tuples.

Load binary content for a single view row. Returns the embedded blob data and inferred format. Args: table_name: View table name. row_id: The row ID. Returns: Tuple of (blob_bytes, format_string) or None if not found.

Whether this dataset has a computed record-embedding table.

Notify registered caches that a dataset changed on disk (e.g. after an import).

List the datasets information in directory. Args: directory: Directory to search in. Returns: List of dataset infos.

Upsert rows into multiple tables in a single call. The multi-table counterpart of :meth:`update_data` and the idempotent counterpart of :meth:`add_records`: rows whose ``id`` already exists are updated, new ids are inserted, so re-running the same payload converges instead of duplicating. Tables are processed in FK dependency order (record → view → entity → … → annotation). Args: data: Mapping of table name to one or more rows, or to an Arrow ``RecordBatch``/``Table``. Arrow payloads currently require ``check_integrity="none"`` — vectorized Arrow integrity checks land with the import engine (spec §8); missing ``created_at``/``updated_at`` columns are appended to Arrow payloads so the LanceDB schema cast cannot fail on them. check_integrity: Integrity-check mode for row payloads. Uniqueness against already-stored ids is intentionally not enforced (existing ids are updates by design); in-batch duplicates, missing ids, and foreign keys are checked. known_ids: Optional ``table → ids`` mapping (e.g. an import engine's id ledger) used to resolve foreign keys without DB lookups. Returns: Mapping of table name to number of rows upserted.

Open a dataset table with LanceDB. Hold :meth:`write_lock` when mutating the returned LanceDB handle so local imports and rollback cannot run concurrently with that mutation. Args: name: Name of the table to open. Returns: Dataset table.

Open the dataset tables with LanceDB. Args: names: Table names to open. If None, open all tables. exclude_embeddings: Whether to exclude embedding tables from the list. Returns: Dataset tables.

Cheap health check of the record-embedding storage. The sidecar alone advertises semantic search; the physical LanceDB table is only touched at query time, so a crashed job or external deletion can leave the feature advertised but broken. This inspects metadata only (no vector scan). Returns: ``{"status", "model_id", "dim", "rows", "records", "detail"}`` where status is one of ``absent`` (no sidecar), ``missing_table``, ``empty``, ``dim_mismatch``, ``corrupt``, ``partial`` (fewer vectors than records) or ``ready``.

Return the record-embedding space descriptor (model, dim, metric), or None.

Register a callable invoked with a dataset id when its caches must be dropped.

Rank records by similarity of their embedding to ``query_vector``. Args: query_vector: The query embedding (same space/dim as the stored vectors). k: Maximum number of records to return. record_id_filter: Optional record-id allowlist applied as a vector-search prefilter. Returns: A tuple of ``(records, distances)`` ordered by increasing distance.

Perform a semantic search. It searches for the closest records to the query in the table embeddings. Args: query: Text query for semantic search. table_name: Table name for embeddings. limit: Limit number of records. skip: Skip number of records. Returns: Tuple of records, distances, and full sorted list of record ids.

Update data in a table (upsert). Args: table_name: Table name. data: Data to update. return_separately: Whether to return separately added and updated data. ignore_integrity_checks: List of integrity checks to ignore. raise_or_warn: Whether to raise or warn on integrity errors. Returns: If ``return_separately`` is True, returns (updated_data, added_data). Otherwise, returns the full data list.

Protect a mutation and refresh cached handles before its integrity reads. Callers performing read/modify/write operations may hold this context around several Dataset methods; nesting in the same thread is safe.