Bulk ingest vectors from Parquet

Use Parquet upload when you already have embeddings and need to load hundreds of thousands or millions of vectors. VectorAmp stages each file, creates an ingestion job, and indexes the rows asynchronously. You do not need a cloud-storage account or cloud credentials.

For small, interactive writes, use the dataset insert endpoint instead. For large backfills and migrations, Parquet is more efficient and easier to resume in chunks.

Before you begin

You need:

• A VectorAmp API key
• The UUID of a dataset whose dim matches the vectors in the file
• One or more .parquet files in the format below

The bulk-upload endpoints currently use the legacy /v1 route. This is an exception to the otherwise unprefixed public API. Do not add /api before /v1.

Parquet file format

Each row represents one vector.

Column	Required	Accepted names	Type	Description
Vector	Yes	values, embedding, vector, embeddings, or vectors	list<float32> or fixed_size_list<float32>[dim]	Dense vector. Every row must have exactly the dataset's configured dimension.
ID	Recommended	id, vector_id, doc_id, or index	Integer	Stable vector ID. IDs must be unique across every file loaded into the dataset. If omitted, VectorAmp assigns zero-based row offsets within each file, so omit it only for a single-file load.
Metadata	No	Any other column name	Scalar	All columns other than the detected ID and vector columns are attached as metadata. Use strings, integers, floats, or booleans.

Use exactly one accepted vector column and no more than one accepted ID column. Vector values must be finite numbers; do not include NaN or infinity. For metadata filtering, define a metadata schema when creating the dataset or allow VectorAmp to infer it from ingested values.

This PyArrow example creates a valid file:

import pyarrow as pa
import pyarrow.parquet as pq

dimensions = 3

table = pa.table({
    "vector_id": pa.array([1001, 1002], type=pa.int64()),
    "values": pa.array(
        [[0.12, 0.34, 0.56], [0.78, 0.90, 0.11]],
        type=pa.list_(pa.float32(), dimensions),
    ),
    "category": ["books", "electronics"],
    "price": pa.array([19.99, 249.00], type=pa.float32()),
})

pq.write_table(table, "vectors.parquet", compression="zstd")

Size and batching limits

• Maximum supported file size: 5 GB per file
• Maximum files per ingestion job: 1,000
• Temporary upload URLs expire after 1 hour
• A file can contain any number of rows that fit within the file-size limit

For predictable retries and processing, target roughly 250 MB to 1 GB per file. Keep row groups reasonably sized and use Parquet compression such as Zstandard or Snappy. A practical starting point is about one million low-dimensional vectors per file, then adjust for vector dimension and metadata width.

Every vector ID must remain unique across chunks. Do not restart IDs at zero in each file.

Upload large files

The recommended large-file flow has three steps:

1. Ask VectorAmp for temporary upload URLs.
2. Upload each file to its URL.
3. Tell VectorAmp that the files are ready for ingestion.

1. Initialize the upload

Send each file's name, exact byte size, and content type:

curl -X POST \
  "https://api.vectoramp.com/v1/datasets/<dataset_id>/upload/init" \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "files": [
      {
        "name": "vectors-000.parquet",
        "size_bytes": 412345678,
        "content_type": "application/vnd.apache.parquet"
      }
    ]
  }'

VectorAmp returns a job ID, a file ID, and a temporary URL for each file:

{
  "job_id": "5aa9d43d-4fab-4eb8-a85f-c852541685f3",
  "uploads": [
    {
      "file_id": "14fdd88e-cea2-4145-a042-e3aaee25f19f",
      "upload_url": "https://<temporary-upload-url>",
      "expires_in": 3600
    }
  ]
}

Keep the job_id and every file_id for the completion request.

2. Upload each file

Upload the raw file bytes to the returned URL. Use the same content type sent during initialization:

curl -X PUT "<temporary_upload_url>" \
  -H "Content-Type: application/vnd.apache.parquet" \
  --upload-file vectors-000.parquet

Do not send the VectorAmp API key to the temporary upload URL. If the URL expires, initialize a new upload.

3. Complete the upload

After every PUT succeeds, complete the job with the IDs returned during initialization:

curl -X POST \
  "https://api.vectoramp.com/v1/datasets/<dataset_id>/upload/complete" \
  -H "X-API-Key: <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "5aa9d43d-4fab-4eb8-a85f-c852541685f3",
    "file_ids": [
      "14fdd88e-cea2-4145-a042-e3aaee25f19f"
    ]
  }'

A successful response reports the number of files accepted for processing:

{
  "confirmed": 1
}

Upload a smaller file in one request

For smaller files, you can send multipart form data directly to VectorAmp:

curl -X POST \
  "https://api.vectoramp.com/v1/datasets/<dataset_id>/upload" \
  -H "X-API-Key: <api_key>" \
  -F "files=@vectors.parquet;type=application/vnd.apache.parquet"

The request returns 202 Accepted with the ingestion job ID:

{
  "job_id": "5aa9d43d-4fab-4eb8-a85f-c852541685f3",
  "files_received": 1,
  "status_url": "/v1/jobs/5aa9d43d-4fab-4eb8-a85f-c852541685f3"
}

Use the temporary-URL flow for large files so the upload does not depend on one long connection to the VectorAmp API.

Monitor ingestion

Uploading the bytes does not mean indexing is complete. Poll the job returned by the upload flow and wait for it to reach a terminal state. See the Jobs API for status values, file-level progress, retries, and errors.

Do not retry the completion request with different file IDs. If a file upload fails, initialize a new upload for that file. Retain your original Parquet chunks until the job completes so you can retry without regenerating the dataset.

Troubleshooting

Symptom	Check
401 Unauthorized	Send a valid VectorAmp API key in X-API-Key. Do not send it to the temporary upload URL.
404 dataset not found	Use the dataset UUID, and verify that the API key belongs to the dataset's organization.
Temporary upload returns 403	The URL may have expired, or the PUT Content-Type differs from initialization. Initialize again and use the exact same content type.
Vector column cannot be detected	Name it values, embedding, vector, embeddings, or vectors.
Dimension mismatch	Ensure every vector has exactly the dataset's configured dim.
IDs are overwritten or collide	Use a globally unique integer ID column across all files.
Job remains incomplete	Inspect the job's file details and error message with the Jobs API.

Related references

• Datasets API
• Jobs API
• Metadata schemas