Skip to main content

Datasets

Create, manage, search, and configure vector datasets. All dataset endpoints are available through the public API gateway under the unprefixed /datasets namespace. Paths below are shown relative to the API base URL (https://api.vectoramp.com).

Unprefixed paths

Use the unprefixed /datasets/... paths. The legacy /v1/datasets/... prefix is deprecated (it still routes for back-compat), and /api/v1/... is not routed (404).

info

All dataset IDs are UUIDs. Use the dataset's UUID in path parameters, not its name.

Dataset Object​

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "product-embeddings",
  "org_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "dim": 2560,
  "metric": "cosine",
  "tuning": {
    "nlist": 4096,
    "nprobe_min": 32,
    "nprobe_max": 32
  },
  "embedding": {
    "provider": "vectoramp",
    "model": "VectorAmp-Embedding-4B"
  },
  "index_type": "sable",
  "schema": [
    { "name": "price", "type": "f32", "source": "user" },
    { "name": "created_at", "type": "i64", "source": "user" },
    { "name": "popularity", "type": "f64", "source": "inferred", "confidence": 0.997, "samples": 12000 }
  ],
  "schema_version": 3,
  "created_at": "2026-04-22T15:29:39Z",
  "updated_at": "2026-04-22T15:29:39Z"
}
FieldTypeDescription
idUUIDUnique dataset identifier
namestringHuman-readable name
org_idUUIDOwning organization
dimintegerVector dimensions
metricstringDistance metric, typically cosine
tuningobjectIndex tuning parameters
embeddingobjectPublic embedding configuration
embedding.providerstringPublic embedding provider name
embedding.modelstringEmbedding model name
index_typestring | nullDataset index type
schemaarrayTyped metadata schema fields. Each entry has name, type (string | u32 | i32 | i64 | f32 | f64), and source (user for declared fields, inferred for fields VectorAmp promoted from observed data — inferred entries also carry confidence and samples). Omitted when the dataset has no schema.
schema_versionintegerMonotonic version, incremented on every schema change (including inferred promotions). Omitted when the dataset has no schema.
created_atstringISO 8601 timestamp
updated_atstringISO 8601 timestamp
info

The public API uses normalized field names. For example:

  • embedding is nested under embedding.provider and embedding.model
  • advanced filter configuration uses filters
  • index_type is public, not index_type_default
  • internal compatibility fields such as rf and embedding_normalization are not part of the public contract

List Datasets​

GET /datasets

Returns datasets for the authenticated organization.

Query Parameters​

NameTypeRequiredDescription
limitintegerMax results to return. Defaults to 50, max 200
offsetintegerPagination offset. Defaults to 0
curl "https://api.vectoramp.com/datasets?limit=50&offset=0" \
  -H "X-API-Key: <api_key>"

Create Dataset​

POST /datasets

Only name is required. When embedding is omitted, VectorAmp uses the managed default model VectorAmp-Embedding-4B (provider vectoramp) and infers dim automatically. metric defaults to cosine. The index type is always SABLE and is not configurable.

Request Body​

NameTypeRequiredDefaultDescription
namestring✅—Dataset name
dimintegerinferredVector dimensions. Inferred from the embedding model when omitted (VectorAmp-Embedding-4B → 2560, text-embedding-3-small → 1536, text-embedding-3-large → 3072). Required for custom/unknown models.
metricstringcosineDistance metric
index_typestringsableAlways sable. SDKs force this value and do not expose it.
embeddingobjectmanagedEmbedding configuration. Omit to use the managed default.
embedding.providerstringvectorampPublic embedding provider, for example vectoramp or openai
embedding.modelstringVectorAmp-Embedding-4BEmbedding model identifier. OpenAI BYOM supports text-embedding-3-small (1536 dims) and text-embedding-3-large (3072 dims).
embedding.secret_refstring—Optional reference to a stored organization provider secret (for BYOM providers). For OpenAI BYOM, create/store the org secret first with PUT /org-secrets/emb%3Aopenai%3Aapi_key and use emb:openai:api_key.
hybridbooleanfalseEnable hybrid (dense + sparse) indexing at creation time
metadataobject{}Arbitrary dataset metadata
schemaarray—Typed metadata schema: [{"name": "price", "type": "f32"}, ...]. Types: string, u32, i32, i64, f32, f64 (aliases like int, long, timestamp, float, double, number, text are accepted and normalized). metadata_schema and metadata_fields are accepted as body aliases. Omit to let VectorAmp infer the schema dynamically from ingested data.
# Name-only create: managed embedding (VectorAmp-Embedding-4B),
# dim inferred (2560), metric cosine, SABLE index.
curl -X POST https://api.vectoramp.com/datasets \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{ "name": "docs" }'
Dimension inference

You only need to send dim when using a custom or unknown embedding model. For the managed model and the two OpenAI models above, VectorAmp infers dim from a built-in table. The request field is dim — not dimension.

Dataset readiness

Dataset creation returns after the dataset record is accepted. Provisioning the backing vector engine is asynchronous and normally finishes within a few seconds. If a write or search arrives before routing is ready, clients should treat 503/dataset_not_ready or vecstack_router_failed as retryable with short backoff, then retry the same request.

Uploading Files into a Dataset​

The recommended document ingestion flow is:

  1. Create the dataset.
  2. Create a source with source_type: "file_upload" via POST /ingestion/sources.
  3. Initialize the upload with POST /ingestion/sources/{sourceId}/upload/init, passing the list of files. The response returns a job_id and a presigned upload_url per file.
  4. PUT each file's bytes directly to its presigned upload_url.
  5. Complete the upload with POST /ingestion/sources/{sourceId}/upload/complete, passing the job_id and the uploaded file_ids.
  6. The ingestion job processes the uploaded files into the dataset.

See Sources for the full presigned upload request/response shapes, and Jobs for job tracking.

Get Dataset​

GET /datasets/:datasetId

ParameterTypeInDescription
datasetIdUUIDpathDataset identifier
curl https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000 \
  -H "X-API-Key: <api_key>"

Delete Dataset​

DELETE /datasets/:datasetId

Permanently deletes a dataset and all its vectors.

ParameterTypeInDescription
datasetIdUUIDpathDataset identifier
warning

This action is irreversible. All vectors and index data will be permanently deleted.

curl -X DELETE https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000 \
  -H "X-API-Key: <api_key>"

List Dataset Documents​

GET /datasets/:datasetId/documents

Returns dataset-level source document metadata for files that were ingested through sources or file uploads. This is a logical document catalog: it does not expose internal storage buckets, object keys, payload_ref, or chunk storage details.

Use this endpoint when you need to enumerate original documents and decide which originals can be downloaded. Use search when you need chunk-level retrieval.

Query Parameters​

NameTypeRequiredDescription
datasetIdUUID✅ pathDataset identifier
limitintegerMax documents to return. Defaults to 50, max 200
cursorstringOpaque cursor from the previous response's next_cursor
statusstringOptional status filter, for example processing, ready, or failed
info

Document pagination is cursor-based. Do not use offsets or totals for this endpoint. Request the first page without cursor, then pass next_cursor until it is null or omitted. The cursor is the previous page's last document id and should be treated as an opaque string by clients.

curl "https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/documents?limit=50&status=ready" \
  -H "X-API-Key: <api_key>"

Response notes​

  • download_available: true means the retained original can be downloaded with Download Dataset Document.
  • download_available: false means the document may still have indexed chunks/search results, but the original bytes are not available from VectorAmp.
  • status tracks catalog/indexing state. ready documents are indexed successfully; processing documents are still being processed; failed documents are cataloged with an ingestion failure.
  • Internal storage coordinates are intentionally never returned. Clients should use only the public document id and download endpoint.

Download Dataset Document​

GET /datasets/:datasetId/documents/:documentId/download

Downloads the retained original document bytes for a dataset document.

ParameterTypeInDescription
datasetIdUUIDpathDataset identifier
documentIdUUID/stringpathDocument identifier returned by List Dataset Documents
curl -L "https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/documents/b9e9d6c4-9f91-4f93-9b71-4c1a4e8af3c1/download" \
  -H "X-API-Key: <api_key>" \
  -o overview.md

Download behavior​

  • The endpoint may stream bytes directly or return a redirect to a short-lived download URL. Use an HTTP client that follows redirects (curl -L, fetch default redirect handling, requests default redirect handling, etc.).
  • If download_available is false, if the raw object was not retained, or if the object can no longer be fetched, the API returns document_unavailable.
  • The redirect URL, if present, is short-lived and should not be stored as the document identity. Store the dataset id + document id instead.
  • Storage bucket names, object keys, and internal payload references are not part of the public contract.

Dataset Statistics​

GET /datasets/:datasetId/stats

Returns statistics about the dataset including vector count, storage size, and index status.

ParameterTypeInDescription
datasetIdUUIDpathDataset identifier
curl https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/stats \
  -H "X-API-Key: <api_key>"

Index Management​

Get Index Status​

GET /datasets/:datasetId/index

Returns the current index type and build status.

Set Index Type​

POST /datasets/:datasetId/index

Sets the index type for the dataset.

Rebuild Index​

POST /datasets/:datasetId/index/rebuild

Triggers a full index rebuild. This is an asynchronous operation.

tip

Index rebuilds run in the background. Use Get Index Status to monitor progress.

curl -X POST https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/index/rebuild \
  -H "X-API-Key: <api_key>"

Metadata Schema​

A dataset's typed metadata schema declares which metadata fields are numeric (and how wide) so range filters evaluate against typed columns with per-field index statistics. Declare it at creation time, update it here, or leave it out entirely and let VectorAmp infer it from ingested data — user-declared fields always take precedence over inferred ones.

Supported field types: string, u32, i32, i64, f32, f64. Requests may use aliases (text/str → string, uint32 → u32, int/int32 → i32, int64/long/timestamp → i64, float/float32 → f32, double/float64/number → f64); responses always use canonical names. Field names: 1–128 chars from letters, digits, _, -, .; unique; at most 128 fields per dataset. See the Metadata Schema concept page for type ranges and guidance.

The current schema (with source provenance and schema_version) is returned by Get Dataset — see the Dataset Object.

Update Metadata Schema​

PATCH /datasets/:datasetId/schema

Updates the dataset's typed metadata schema. The change is persisted first, then the index's typed metadata structures are rebuilt asynchronously from existing metadata — the dataset stays fully searchable throughout, and the vector index itself (centroids, quantization, graphs) is not retrained. schema_version increments on every successful update.

Requires an admin role

This endpoint requires an admin or owner role in the dataset's organization. API keys must carry the admin scope.

Request Body​

NameTypeRequiredDefaultDescription
schemaarray✅—Schema fields: [{"name": "...", "type": "..."}, ...]. metadata_schema / metadata_fields are accepted as aliases.
modestringmergemerge: add or update the given fields, keep all others (including inferred ones). replace: the given fields become the complete schema — anything not listed is removed.

A user declaration for an existing field name overrides it, including fields that were previously inferred. merge never removes fields; to drop a field, send the full desired schema with mode: "replace".

# Add/retype two fields; everything else in the schema is kept.
curl -X PATCH https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/schema \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "schema": [
      { "name": "rating", "type": "f32" },
      { "name": "created_at", "type": "timestamp" }
    ]
  }'

Note in the response how the timestamp alias was normalized to i64, and the previously inferred popularity field survived the merge.

Errors​

StatusCodeWhen
400—Empty/duplicate/invalid field name, unsupported type, more than 128 fields, invalid mode, or empty schema with mode: "merge"
403insufficient_roleCaller lacks an admin-class role in the org
404dataset_not_foundDataset missing — or owned by a different organization
Schema changes don't interrupt traffic

The typed-structure rebuild runs in the background from metadata VectorAmp already stores; searches and inserts continue uninterrupted, and results are identical during the rebuild. Use schema_version in Get Dataset to confirm the new version is active.

Hybrid Search Settings​

Get Hybrid Settings​

GET /datasets/:datasetId/hybrid

Returns hybrid search configuration for the dataset.

PUT /datasets/:datasetId/hybrid/enabled

NameTypeRequiredDescription
enabledboolean✅Enable or disable hybrid search
curl -X PUT https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/hybrid/enabled \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

Set Hybrid Alpha​

PUT /datasets/:datasetId/hybrid/alpha

Controls the balance between vector and keyword search. 0.0 = pure keyword, 1.0 = pure vector.

NameTypeRequiredDescription
alphafloat✅Hybrid search weight (0.0–1.0)
curl -X PUT https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/hybrid/alpha \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"alpha": 0.7}'

POST /datasets/:datasetId/search

Perform similarity search against the dataset. Provide either a raw vector (query) or a text string (query_text) that will be automatically embedded using the dataset's configured embedding model.

Request Body​

NameTypeRequiredDescription
queryfloat[]Raw vector for search (mutually exclusive with query_text)
query_textstringText query — auto-embedded using the dataset's embedding model
embedding_modelstringOverride the dataset's default embedding model for text queries
embedding_providerstringOverride the dataset's default embedding provider for text queries
top_kintegerNumber of results to return. Defaults to 10
filtersobjectSimple string-equality metadata filters ({"key": "value"})
advanced_filtersarrayAdvanced numeric/categorical filters (see below)
nprobe_overrideintegerOverride the number of clusters to probe
rerank_depth_overrideintegerOverride SABLE candidate-depth rerank for the vector index (0 = dataset default). This is not the optional model reranker.
rerankboolean | objectEnable the optional model reranker after initial retrieval. Omit it for default fast search; use true or { "enabled": true } for best relevance.
rerank.enabledbooleanEnable model rerank for this request. Default: false when rerank is omitted.
rerank.providerstringOptional model-rerank provider override. Omit to use the server default.
rerank.modelstringOptional model-rerank model override. Omit to use the server default.
rerank.top_nintegerOptional number of reranked results to return. Omit to use the server default, normally the request top_k.
rerank.max_documentsintegerOptional maximum initial candidates to send to the model reranker. Omit to use the server default.
rerank.fail_openbooleanOptional fail-open behavior. Omit to use the server default.
hybridbooleanEnable hybrid (vector + keyword) search
sparse_querystringKeyword query for hybrid search
alphafloatHybrid weight: 0.0 = pure keyword, 1.0 = pure vector
include_embeddingsbooleanReturn the vector embeddings in results
include_documentsbooleanReturn document content in doc_value. Defaults to false
include_metadatabooleanReturn result metadata. Defaults to true; when false, metadata is omitted

Advanced Filters​

The advanced_filters array enables numeric range queries, categorical filtering, and multi-value matching on metadata fields. Each filter object has the following structure:

NameTypeRequiredDescription
fieldstring✅Metadata field name to filter on
opstring✅Comparison operator: eq, gt, gte, lt, lte, in
valuenumber | stringSingle value for comparison (eq, gt, gte, lt, lte)
valuesarrayMultiple values for in operator (numbers or strings)

Supported operators:

OperatorDescriptionExample
eqEqual to{"field": "status", "op": "eq", "value": "active"}
gtGreater than{"field": "price", "op": "gt", "value": 10.0}
gteGreater than or equal{"field": "rating", "op": "gte", "value": 4}
ltLess than{"field": "price", "op": "lt", "value": 100.0}
lteLess than or equal{"field": "count", "op": "lte", "value": 50}
inValue in set{"field": "category", "op": "in", "values": ["electronics", "books"]}
tip

Combine filters and advanced_filters in the same request. Simple filters handle exact string matches, while advanced_filters handle numeric ranges and multi-value lookups.

Typed range filtering

Range operators work on any numeric-looking field, but fields declared in the dataset's metadata schema evaluate against typed columns with per-field index statistics — more precise pruning and true numeric comparison. Declare a schema (or let VectorAmp infer one) for fields you routinely range-filter, such as prices and timestamps.

curl -X POST https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/search \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": [0.1, 0.2, 0.3, "..."],
    "top_k": 5,
    "include_documents": true,
    "include_metadata": true
  }'
VectorAmp Rerank

Model rerank runs after the initial vector or hybrid retrieval step and reorders the retrieved candidates for relevance. It is separate from rerank_depth_override, which only changes SABLE's vector-index candidate-depth rerank behavior. Model rerank is optional per request:

  • Fast search should omit rerank for the lowest-latency path.
  • Best search should send rerank: {"enabled": true}.
  • provider, model, top_n, max_documents, and fail_open are optional. If omitted, VectorAmp uses server defaults.
  • Rerank is fail-open: if rerank fails, times out, is throttled, hits a quota/billing limit, or has a provider/IAM issue, VectorAmp returns the original search results instead of failing the search request. Response metadata indicates rerank was not applied when available.
Response Fields
  • id — Internal vector ID (integer, derived via FNV-1a hash if string IDs are used)
  • score — Distance from the query for the dataset metric. For cosine datasets, lower is better: identical vectors return 0, orthogonal vectors return about 1, and results are ordered ascending. Model rerank, when enabled, may reorder results for relevance after initial retrieval.
  • metadata — Key-value metadata stored with the vector. Omitted when include_metadata: false.
  • metadata.text and metadata.doc_value — Not returned when include_documents: false.
  • metadata.document_storage_key, metadata.document_storage_bucket, and metadata.payload_ref — Never returned; these are internal implementation details.
  • metadata.additional.caption — Not returned when it duplicates the chunk/document text.
  • embedding — The vector values (only if include_embeddings: true).
  • doc_kind — Document type: "inline" (text stored directly) or "url" (reference). null unless include_documents: true.
  • doc_value — Document content or URL (only if include_documents: true).
  • query_time_ms — Server-side query execution time in milliseconds.
  • rerank — Model-rerank status metadata when model rerank was requested.

Insert Vectors​

POST /datasets/:datasetId/insert

Insert one or more vectors into the dataset. The vector array is sent under the vectors key — not /vectors as a path.

Request Body​

NameTypeRequiredDescription
vectorsarray✅Array of vector objects

Vector Object:

NameTypeRequiredDescription
idstring | integer✅Unique vector identifier. Accepts a string or an integer; integer ids are preserved as JSON numbers and not coerced to strings.
valuesfloat[]✅Vector values (must match dataset dim)
metadataobjectArbitrary scalar key-value metadata. String, number, boolean, and null values are accepted; numbers/booleans are stored as scalar metadata values and can be used with advanced_filters where supported. Nested objects/arrays are rejected.
Numeric vector ids

The id field accepts both strings and integers. Integer ids are serialized as JSON numbers and preserved verbatim. Send the value as a JSON number ("id": 1024) — not a quoted string — to keep it numeric.

Metadata values

Vector metadata accepts scalar JSON values. For example, {"tenant_id": 999, "active": true, "source": "crm"} is valid. Avoid nested objects or arrays in vector metadata; flatten them into scalar fields before insert.

curl -X POST https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/insert \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "vectors": [
      {
        "id": "doc-001",
        "values": [0.1, 0.2, 0.3],
        "metadata": {"title": "Introduction", "source": "manual"}
      },
      {
        "id": "doc-002",
        "values": [0.4, 0.5, 0.6],
        "metadata": {"title": "Chapter 1", "source": "manual"}
      }
    ]
  }'

Delete Vectors​

DELETE /datasets/:datasetId/vectors

Delete one or more vectors from a dataset by vector id. This removes the vector from future search results; it does not delete source documents or ingestion job records.

Request Body​

NameTypeRequiredDescription
idsarray✅Vector ids to delete. String ids and integer ids use the same id format accepted by insert.
write_concernstringOptional durability target: default, one, quorum, or all.
Irreversible operation

Vector deletion is immediate for the addressed ids. If you need the vectors again, re-insert or re-ingest them.

curl -X DELETE https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/vectors \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "ids": ["doc-001", 1024],
    "write_concern": "quorum"
  }'

Embed Text​

POST /datasets/:datasetId/embed

Generate embeddings using the dataset's configured embedding model.

info

Requires embedding_provider and embedding_model to be configured on the dataset.

Request Body​

NameTypeRequiredDescription
textstringSingle text to embed
textsstring[]Multiple texts to embed (batch)
curl -X POST https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/embed \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"texts": ["Hello world", "Machine learning"]}'

Promotion Policy​

Get Promotion Policy​

GET /datasets/:datasetId/promotion-policy

Returns the current promotion policy for the dataset.

Set Promotion Policy​

PUT /datasets/:datasetId/promotion-policy

Updates the promotion policy configuration.

Default Settings​

Set Default Index Type​

POST /datasets/:datasetId/defaults/index-type

Sets the default index type for new partitions in this dataset.

Set Default Tuning​

POST /datasets/:datasetId/defaults/tuning

Sets the default tuning parameters for new partitions.

SLA Settings​

Get SLA Settings​

GET /datasets/:datasetId/sla

Returns the SLA configuration for the dataset.

Set SLA Settings​

PUT /datasets/:datasetId/sla

Configure search latency SLA targets. VectorAmp will auto-tune index parameters to meet the target.

Request Body​

NameTypeRequiredDescription
target_p99_usinteger✅Target p99 latency in microseconds
min_nprobeinteger✅Minimum nprobe value
max_nprobe_fractioninteger✅Maximum nprobe as fraction of nlist
min_beam_widthinteger✅Minimum beam width for HNSW
max_beam_widthinteger✅Maximum beam width for HNSW
curl -X PUT https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/sla \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "target_p99_us": 50000,
    "min_nprobe": 1,
    "max_nprobe_fraction": 10,
    "min_beam_width": 4,
    "max_beam_width": 64
  }'

Get Vector Payload​

GET /datasets/:datasetId/payload/:vectorId

Retrieve the stored payload (metadata) for a specific vector.

ParameterTypeInDescription
datasetIdUUIDpathDataset identifier
vectorIdstringpathVector identifier
curl https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/payload/doc-001 \
  -H "X-API-Key: <api_key>"

Ensure Engine​

POST /datasets/:datasetId/ensure-engine

Force the dataset engine to be provisioned and ready. Useful after creating a dataset or recovering from errors.

curl -X POST https://api.vectoramp.com/datasets/550e8400-e29b-41d4-a716-446655440000/ensure-engine \
  -H "X-API-Key: <api_key>"