Skip to main content

Datasets

A dataset is the retrieval boundary your application queries. It owns source content, vectors, metadata, SABLE indexing, search behavior, and the operational state needed to answer questions with evidence.

Design principle

Create datasets around access and retrieval boundaries, not around arbitrary folders. If two groups of content require different permissions, freshness rules, metadata filters, or evaluation checks, they usually deserve separate datasets.

Common dataset shapes​

Dataset shapeExample
Company knowledgePolicies, product docs, support articles, onboarding material.
Customer-scoped knowledgeOne dataset per tenant or customer when isolation is strict.
Product searchCatalog, documentation, tickets, or knowledge base content behind an app search box.
Compliance workspaceApproved policies, control evidence, security answers, and audit artifacts.
Agent knowledge toolCurated source set exposed through MCP or an internal agent workflow.

Create a dataset​

Open Datasets in VectorAmp, then choose Create Dataset.

For most first runs, you only need a name. VectorAmp defaults to a managed embedding model and SABLE indexing. Advanced teams can choose different embedding behavior or hybrid search when the workload requires it.

FieldGuidance
NameUse a human-readable name that describes the retrieval boundary, such as product-docs-prod or support-knowledge-staging.
Embedding modelUse the managed default unless you need a specific provider/model for compatibility.
MetricCosine is the common default for text embeddings.
Hybrid searchEnable when exact terms, IDs, names, or policy language matter alongside semantic similarity.
Metadata schemaDecide early which fields your app will filter on: tenant, source, document type, visibility, version, timestamp, etc.

Add content​

There are three common ingestion paths:

  1. File upload — drag files or folders into the dashboard for quick dataset creation and review.
  2. Sources and pipelines — connect source systems and let VectorAmp parse, chunk, embed, index, retry, and track jobs.
  3. API insertion — insert vectors or text from an application-controlled pipeline.

The file upload flow is:

  1. Create or select a dataset in Datasets.
  2. Create a file_upload source for that dataset.
  3. Initialize upload and receive presigned upload URLs.
  4. Upload file bytes directly to S3.
  5. Complete the upload.
  6. Let the ingestion job process files into searchable vectors.

The browser never needs AWS credentials. VectorAmp mints temporary presigned URLs and handles the backend ingestion flow.

Metadata and filters​

Metadata is the contract between ingestion and governed retrieval.

Example metadata:

{
  "tenant_id": "org_123",
  "source": "google_drive",
  "doc_type": "policy",
  "visibility": "internal",
  "version": "2026-06",
  "updated_at": "2026-06-26T00:00:00Z"
}

Use metadata filters to enforce query scope:

  • tenant or customer boundary,
  • user role or visibility class,
  • approved source set,
  • document type,
  • region,
  • version,
  • recency window.

Do not rely on the model prompt to enforce permissions. Resolve authorization in your app and pass the allowed scope as filters.

Search and ask​

Use search when your application wants ranked results and metadata:

results = dataset.search(
    "What is our refund policy?",
    top_k=10,
    filter={"doc_type": "policy", "visibility": "internal"},
)

Use ask when you want a source-grounded answer:

answer = dataset.ask(
    "Can customers request refunds after 30 days?",
    filter={"doc_type": "policy"},
)
print(answer["answer"])
print(answer.get("citations"))

Delete individual vectors​

Use vector deletion when your application needs to remove specific application-level ids without deleting the whole dataset:

curl -X DELETE https://api.vectoramp.com/datasets/<dataset_id>/vectors \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"ids": ["doc-001", "doc-002"]}'

This deletes vectors from search. It does not delete retained source documents, source records, or ingestion job history. Re-insert or re-ingest the ids if you need them back.

Dataset health checks​

Before you expose a dataset to users, confirm:

  • ingestion jobs are complete,
  • expected files/documents are present,
  • metadata fields are populated consistently,
  • search returns obvious known sources,
  • citations point to supporting evidence,
  • out-of-scope queries fail safely,
  • filters enforce the intended tenant/source/role boundary.