Skip to main content

Ingest cloud documents with the CLI

The VectorAmp CLI is useful for repeatable ingestion from local machines, CI, or operational scripts.

What you will build​

A CLI-driven ingestion workflow that creates or uses a dataset, ingests files from cloud or local sources, monitors jobs, and verifies search.

Prerequisites​

  • Node.js 18+.
  • A VectorAmp API key.
  • Access to the source system.

Step 1: Install and configure the CLI​

npm install -g @vectoramp/cli
export VECTORAMP_API_KEY=vsk_...

vectoramp config set --api-key vsk_...
vectoramp datasets list

Step 2: Create or select a dataset​

vectoramp datasets create product-docs --metadata '{"owner":"docs"}'
vectoramp config use ds_123

Step 3: Ingest local files​

vectoramp --dataset ds_123 datasets ingest-files ./docs \
  --extensions md,txt,json,csv \
  --source-name "Product docs upload"

Step 4: Ingest S3 or GCS​

vectoramp --dataset ds_123 sources ingest s3 s3://company-docs/product \
  --config '{"sync_mode":"incremental"}'

vectoramp --dataset ds_123 sources ingest gcs gs://company-docs/product \
  --config '{"sync_mode":"incremental"}'

Step 5: Ingest Google Drive​

vectoramp sources gdrive google-drive-folder-id --name product-drive

If you create the source through the dashboard OAuth flow, copy the source ID and ingest it through the dataset/source flow supported by your environment. For scripted service-account setups, pass the required source configuration as JSON.

Step 6: Monitor jobs​

vectoramp jobs list --dataset-id ds_123
vectoramp jobs get job_123 --poll --interval 5000 --timeout 600000
vectoramp jobs retry job_123
vectoramp --dataset ds_123 datasets search "installation requirements" \
  --filter source=product-docs \
  --top-k 5

Production notes​

  • Use CI secrets for VECTORAMP_API_KEY.
  • Avoid committing source credentials in --config JSON.
  • Prefer incremental sync for large cloud sources.
  • Poll jobs with a timeout in automation.