Metadata Schema

TL;DR

A metadata schema declares typed fields (like price: f32 or created_at: i64) on a dataset so VectorAmp can evaluate range filters against typed values instead of strings. You can predefine the schema when you create a dataset, or let VectorAmp infer it from the data you ingest — and if you do both, your declarations always win.

Every vector in VectorAmp can carry metadata — key/value pairs like {"category": "electronics", "price": 42.5} — and every metadata field is filterable whether or not it appears in a schema. The schema adds types on top: it tells the SABLE index which fields are numbers and how wide they are, which upgrades range filtering from string parsing to typed column evaluation and gives each declared field its own per-list numeric statistics for query pruning.

Do you need a schema?

No — and that's the point of the design:

• Without a schema, equality and in filters work on every field, and numeric-looking values still support range filters through a shared, coarse numeric statistic per index partition.
• With a schema, each declared numeric field gets its own typed column and its own per-partition statistics. Range filters like price < 100 prune more of the index before any distance computation runs, and values are compared as real numbers rather than parsed strings.

Schemas never change what matches — only how efficiently and precisely the engine gets there. Search recall is unaffected either way.

Field types

A schema is a flat list of { "name", "type" } entries. Types are scalar only — nested objects and arrays are not schema-typeable.

Type	What it is	Range	Typical use
string	UTF-8 text, compared exactly	—	Categories, statuses, labels, tags
u32	Unsigned 32-bit integer	0 to 4,294,967,295	Counts, years, small non-negative quantities
i32	Signed 32-bit integer	−2,147,483,648 to 2,147,483,647	Signed quantities, deltas, offsets
i64	Signed 64-bit integer	±9.2 × 10¹⁸	Timestamps (epoch seconds/millis), large IDs used numerically
f32	32-bit float	~7 significant digits	Prices, scores, ratings — most float metadata
f64	64-bit float	~15–16 significant digits	High-precision measurements

Requests also accept common aliases, which are normalized to the canonical name in every response:

You send	Stored as
str, text	string
uint32	u32
int, int32	i32
int64, long, timestamp	i64
float, float32	f32
double, float64, number	f64

Field names must be 1–128 characters from letters, digits, _, -, and .; names must be unique, and a dataset holds at most 128 schema fields.

Predefined vs. dynamically inferred schema

There are two ways a field ends up in the schema, and they are designed to compose:

Predefining the schema (recommended when you know your data)

Declare fields when you create the dataset, or later with the schema update endpoint:

{
  "name": "products",
  "schema": [
    { "name": "tenant_id", "type": "u32" },
    { "name": "price", "type": "f32" },
    { "name": "created_at", "type": "i64" },
    { "name": "status", "type": "string" }
  ]
}

Predefine when:

• You know the field set up front (product catalogs, event streams, structured exports).
• A field looks numeric but must stay categorical — ZIP codes, SKUs, phone numbers, fixed-width codes like "00123". Inference deliberately refuses to type these; declaring "type": "string" settles it explicitly, and declaring a numeric type overrides the safety heuristics if you really do want ranges.
• You want typed range filtering to be precise from the very first vector, rather than after inference has observed enough data.

Fields you declare are marked "source": "user" and are authoritative: inference will never override, retype, or remove them.

Letting VectorAmp infer the schema

If you skip the schema, VectorAmp watches the metadata you insert and promotes fields into the schema automatically — conservatively, because a wrong type would hurt filter correctness more than a missing one:

• A field is considered only after ~1,000 non-null observations, and only when ≥ 99% of its values parse as numbers.
• Integer values get the narrowest fitting type (u32 → i32 → i64); anything with decimals or exponents becomes f64.
• ID-like names never auto-promote to numeric — id, *_id, uuid, zip, zipcode, postal_code, phone, account_number, sku, code, *_code, *_key and similar stay categorical, as do fields whose values carry leading zeros ("00123" parses as a number but isn't one). Declare these yourself if you genuinely want numeric semantics.
• String-only fields are not promoted at all: categorical filtering already works on every field without a schema, so a string schema entry would add nothing.
• Once promoted, a field's type never "flaps". If later data contradicts an inferred type, the field is marked "state": "conflicted" in the schema rather than being retyped, and typed structures simply skip unparseable values.

Inferred fields appear in the dataset response with provenance you can audit:

{
  "schema": [
    { "name": "price", "type": "f32", "source": "user" },
    { "name": "popularity", "type": "f64", "source": "inferred", "confidence": 0.997, "samples": 12000 }
  ],
  "schema_version": 3
}

Use inference when your metadata shape is evolving, connector-driven, or unknown — for example documents flowing in from Google Drive or Jira, where field sets differ per source. You can always inspect what was inferred and pin it down (or correct it) with an explicit declaration later; a user declaration for the same field name replaces the inferred entry.

Schema changes are cheap and non-disruptive

Adding or updating schema fields is a metadata-only operation. VectorAmp rebuilds the typed columns and per-field statistics from the metadata it already has, asynchronously, in the background — it does not retrain the vector index (centroids, quantization codebooks, and graphs are untouched), and the dataset stays fully searchable throughout. schema_version increments on every change so you can track propagation, and inferred promotions use the same mechanism.

Interaction with search filters

Search requests are unchanged by schemas — the same filters (exact match) and advanced_filters (ranges, in) work either way; see Advanced Filters. What changes under the hood:

• Range operators (gt, gte, lt, lte) on a declared numeric field evaluate against the typed column and use the field's own statistics for pruning.
• Range operators on an undeclared field fall back to string-parsed comparison and coarse shared statistics — still correct, less prunable.
• Equality and in filtering behave identically with or without schema.

Related

• Dataset API reference — schema payloads and the update endpoint
• Vector Indexes — how SABLE uses filter-native structures