Skip to main content

Production readiness checklist

A VectorAmp prototype is useful when it returns the right answer once. A production retrieval system is useful when it returns the right scoped answer repeatedly, with evidence, under load, and with a clear failure mode.

Use this checklist before exposing VectorAmp-backed search, RAG, or agent workflows to customers or internal teams.

Rule of thumb

Do not ship retrieval until you have tested relevance, scope, citations, ingestion freshness, auth boundaries, and no-result behavior. The happy path is the easy part.

1. Define the retrieval boundary​

Be explicit about what a dataset represents.

QuestionProduction answer
What content belongs here?The dataset has a named purpose: product docs, support knowledge, customer files, policies, contracts, etc.
Who can query it?API keys, app users, and agent tools map to a known tenant or permission scope.
What should never be retrieved?Private sources, archived documents, unapproved drafts, unrelated tenants, or restricted roles are excluded or filtered.
How fresh must it be?You know whether ingestion must run in minutes, hourly, daily, or on manual approval.

2. Attach useful metadata​

Metadata is what turns vector search into governed retrieval. Add it at ingestion time and keep names consistent.

Recommended fields:

{
  "tenant_id": "org_123",
  "source": "google_drive",
  "source_id": "file_abc",
  "doc_type": "policy",
  "title": "Expense Policy",
  "owner": "finance",
  "visibility": "internal",
  "version": "2026-06",
  "updated_at": "2026-06-26T00:00:00Z"
}

Good filters are boring and predictable. Avoid one-off field names such as customer, customerId, and client_id all meaning the same thing.

3. Validate search quality​

Create a small evaluation set before launch:

  • 10 obvious queries that must retrieve a known source.
  • 10 ambiguous queries that should retrieve multiple plausible sources.
  • 10 negative queries that should return no result or a low-confidence response.
  • 10 scoped queries that prove tenant/role/source filters work.
  • 5 freshness checks against newly ingested or recently updated content.

Track:

  • top result correctness,
  • citation correctness,
  • whether irrelevant sources appear,
  • latency at expected concurrency,
  • user-visible failure mode when no answer is available.

4. Test citations, not just answers​

A fluent answer is not enough. Review whether citations point to the right source material.

Before launch, check that:

  • every answer has evidence when evidence is required,
  • cited snippets actually support the claim,
  • source labels are meaningful to users,
  • users can open or inspect the referenced source,
  • stale or superseded content does not appear as authoritative.

5. Exercise permission and tenant boundaries​

For SaaS and internal tools, test retrieval with real permission scenarios:

  • user from tenant A cannot retrieve tenant B content,
  • admin and non-admin roles see different source sets when expected,
  • revoked API keys fail immediately,
  • agent tools cannot bypass filters by changing query wording,
  • support workflows only retrieve the current customer context.

If the application owns authorization, enforce it before calling VectorAmp and pass the resolved scope as filters. Do not ask a model to enforce permissions in natural language.

6. Make ingestion observable​

A production user will ask: “Is the answer wrong, or is the document missing?” Make ingestion state visible.

Track:

  • source connection status,
  • job status and failures,
  • files/documents processed,
  • skipped files and reasons,
  • last successful sync time,
  • embedding/indexing completion,
  • retry policy and dead-letter handling.

7. Choose the right integration surface​

SurfaceUse when
DashboardOperators need to create datasets, monitor jobs, inspect API keys, or debug source state.
CLIYou want repeatable ingestion and smoke tests in terminals or CI.
REST APIAny backend needs direct control over datasets, jobs, sources, search, or intelligence.
SDKsApp teams want typed helpers and higher-level ergonomics.
LangChainYou are building a RAG app inside an existing LangChain workflow.
MCPAI agents need a governed tool for company knowledge access.

8. Define no-result behavior​

No-result behavior is a product decision. Decide what users should see when retrieval is weak.

Good patterns:

  • “I don’t have enough evidence to answer that.”
  • Show closest sources without generating an answer.
  • Suggest filters or sources that may be missing.
  • Route to a human workflow for policy, legal, security, or customer-facing answers.

Bad patterns:

  • Generate anyway.
  • Hide citations when evidence is weak.
  • Let the model invent source names.
  • Treat a broad semantic match as permission to answer.

9. Review API keys and environments​

Before launch:

  • Separate development, staging, and production keys.
  • Restrict keys to the application or environment that uses them.
  • Rotate any key that appeared in local logs, screenshots, or demos.
  • Confirm production clients point at production datasets and sources.
  • Confirm non-production docs, previews, or staging sites are not indexable unless intended.

10. Load and latency checks​

Run tests that match the real product shape:

  • expected top-k,
  • filter complexity,
  • concurrent users,
  • answer generation vs search-only calls,
  • large documents and long metadata values,
  • cold-start behavior after idle periods,
  • ingestion running while search traffic is active.

Launch checklist​

  • Dataset purpose and owner are clear.
  • Metadata schema is documented.
  • Filters enforce tenant/source/role boundaries.
  • Search evaluation set passes.
  • Cited answers are source-grounded.
  • Negative/no-result queries fail safely.
  • Ingestion job state is visible.
  • API keys are scoped and environment-specific.
  • Production docs and app URLs use the correct robots/sitemap behavior.
  • Rollback plan exists for bad ingestion, bad source data, or degraded retrieval.