Sources

A source describes an external data location — a web site, an S3 or GCS bucket, Google Drive, Jira, Confluence, or a managed file upload — that VectorAmp can ingest into a dataset. You register a source once, then start jobs or schedules against it. All source management endpoints live under the unprefixed /ingestion/sources namespace.

info

All source and dataset IDs are UUIDs.

Unprefixed paths

Use the unprefixed /ingestion/sources/... paths. The legacy /v1/ingestion/sources/... prefix is deprecated, and /api/v1/... is not routed (404).

End-to-end flow

Register a source with POST /ingestion/sources (optionally validate the config first).
Start a job with POST /ingestion/jobs, passing source_id and dataset_id.
Track progress with the Jobs endpoints, or automate recurring syncs with Schedules.

For file_upload sources, use the presigned upload flow instead of pulling from an external system.

Authenticated sources

Ingesting from Google Drive, GCS, Jira, or Confluence? See Authenticating sources for the two supported ways to grant access — a fully headless service account, or a reusable, browser-granted connection.

Finished with a source, or want to find ones nothing is using? See Delete Source, List Unused Sources, and Clean Up Unused Sources.

Source Object

Field	Type	Description
`id`	UUID	Unique source identifier
`org_id`	UUID	Owning organization
`type`	string	Source type (`web`, `s3`, `gcs`, `gdrive`, `jira`, `confluence`, `file_upload`)
`name`	string	Human-readable name
`description`	string \| null	Optional description
`config`	object	Type-specific configuration (secrets are redacted in responses)
`metadata`	object	Arbitrary client metadata
`created_at`	string	ISO 8601 timestamp
`updated_at`	string	ISO 8601 timestamp

Redaction

Secret-bearing config fields (access keys, tokens, passwords, credentials) are redacted to ******** in every source response. Public source responses also omit created_by.

List Sources

GET /ingestion/sources

Returns registered sources for the authenticated organization.

Query Parameters

Name	Type	Required	Description
`limit`	integer		Max results to return. Defaults to `50`, max `200`
`offset`	integer		Pagination offset. Defaults to `0`

curl "https://api.vectoramp.com/ingestion/sources?limit=50&offset=0" \
  -H "X-API-Key: vsk_<64hex>"

The response shape is { "sources": [...], "total": number, "limit": number, "offset": number }.

Get Source

GET /ingestion/sources/:sourceId

Parameter	Type	In	Description
`sourceId`	UUID	path	Source identifier

Returns a single source object.

curl https://api.vectoramp.com/ingestion/sources/c3d4e5f6-a7b8-9012-cdef-123456789012 \
  -H "X-API-Key: vsk_<64hex>"

Validate Source Config

POST /ingestion/sources/validate

Validate a source configuration without persisting it. Use this to check credentials and surface sample documents before registering a source.

Request Body

Name	Type	Required	Description
`source_type`	string	✅	Source type to validate
`config`	object	✅	Type-specific config (see Source Types)

cURL
Response

curl -X POST https://api.vectoramp.com/ingestion/sources/validate \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "web",
    "config": { "start_urls": ["https://docs.example.com"] }
  }'

{
  "success": true,
  "message": "Configuration is valid",
  "normalized_config": {
    "start_urls": ["https://docs.example.com"],
    "max_depth": 3,
    "max_pages": 1000
  },
  "samples": [
    { "title": "Getting Started", "url": "https://docs.example.com/" }
  ],
  "warnings": []
}

Create Source

POST /ingestion/sources

Request Body

Name	Type	Required	Description
`source_type`	string	✅	One of `web`, `s3`, `gcs`, `gdrive`, `jira`, `confluence`, `file_upload`
`name`	string	✅	Human-readable source name
`config`	object	✅	Type-specific configuration
`description`	string		Optional description
`metadata`	object		Arbitrary client metadata

Web
S3
Confluence
Google Drive
Response 201

curl -X POST https://api.vectoramp.com/ingestion/sources \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "web",
    "name": "Docs crawler",
    "config": {
      "start_urls": ["https://docs.example.com"],
      "max_depth": 3,
      "max_pages": 1000
    }
  }'

curl -X POST https://api.vectoramp.com/ingestion/sources \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "s3",
    "name": "Reports bucket",
    "config": {
      "bucket": "my-reports",
      "prefix": "2026/",
      "region": "us-east-1",
      "access_key_id": "AKIA...",
      "secret_access_key": "..."
    }
  }'

curl -X POST https://api.vectoramp.com/ingestion/sources \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "confluence",
    "name": "Engineering wiki",
    "config": {
      "auth_mode": "basic",
      "base_url": "https://company.atlassian.net",
      "username": "bot@company.com",
      "api_token": "...",
      "spaces": ["ENG", "ARCH"],
      "include_attachments": false
    }
  }'

# Authenticated source via a reusable connection (see Authenticating sources).
curl -X POST https://api.vectoramp.com/ingestion/sources \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "gdrive",
    "name": "Team Drive",
    "config": {
      "folder_ids": ["1AbC2dEf3GhI"],
      "connection_id": "d1e2f3a4-b5c6-7890-abcd-ef0123456789"
    }
  }'

{
  "id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
  "org_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "type": "web",
  "name": "Docs crawler",
  "description": null,
  "config": {
    "start_urls": ["https://docs.example.com"],
    "max_depth": 3,
    "max_pages": 1000
  },
  "metadata": {},
  "created_at": "2026-04-25T12:00:00Z",
  "updated_at": "2026-04-25T12:00:00Z"
}

Authenticating sources

Connectors that read from a private system — gdrive, gcs, jira, and confluence — need credentials. VectorAmp supports two ways to supply them. Pick based on whether a human can complete a one-time browser consent.

Approach	Best for	Browser consent	How
Service account	Servers, CI, fully headless automation	Never	Put a service-account JSON in the source `config`
Connection reference	Interactive setup that is then reused (incl. by schedules)	Once, up front	Grant a connection once, then reference it by `connection_id`

Option 1 — Service account (headless)

For gdrive and gcs, pass a Google service-account JSON directly in config.service_account_json. No browser, no consent screen — ideal for servers and CI. Share the target Drive folder or GCS bucket with the service account's email first.

curl -X POST https://api.vectoramp.com/ingestion/sources \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "gdrive",
    "name": "Drive (service account)",
    "config": {
      "folder_ids": ["1AbC2dEf3GhI"],
      "auth_mode": "service_account",
      "service_account_json": { "type": "service_account", "client_email": "ingest@my-project.iam.gserviceaccount.com", "private_key": "-----BEGIN PRIVATE KEY-----\n…\n-----END PRIVATE KEY-----\n", "project_id": "my-project" }
    }
  }'

Secrets are redacted

service_account_json (and every other secret-bearing field) is redacted to ******** in all source responses. Store the original JSON in your own secret manager.

Option 2 — Connection reference (interactive once, reusable)

Create a connection once — the user grants access through the provider's OAuth consent screen — then reference that connection from any number of sources with connection_id. Works for gdrive, gcs, confluence, and jira.

# 1. Create the connection and open authorization_url in a browser (see Connections).
curl -X POST https://api.vectoramp.com/connections \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{ "provider": "google", "source_type": "gdrive" }'

# 2. After status == "connected", create the source referencing the connection.
curl -X POST https://api.vectoramp.com/ingestion/sources \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "gdrive",
    "name": "Drive (connection)",
    "config": {
      "folder_ids": ["1AbC2dEf3GhI"],
      "connection_id": "d1e2f3a4-b5c6-7890-abcd-ef0123456789"
    }
  }'

Credentials are resolved and refreshed server-side at ingest time, so a source backed by a connection_id keeps working for schedules and long-running syncs without re-consent. See the full Connections reference for the grant flow, polling, and the end-to-end authenticated Google Drive walkthrough.

Delete Source

DELETE /ingestion/sources/:sourceId

Permanently deletes a source.

Query Parameters

Name	Type	Required	Description
`force`	boolean		Force deletion even when the source is in use. Defaults to `false`

If the source is attached to an active schedule or referenced by an in-flight job, the request returns 409 Conflict unless you pass force=true. With force=true, VectorAmp also disables the source's schedules before deleting it. Returns 204 No Content on success.

cURL
Response 409

# Fails with 409 if the source is attached to an active schedule or in-flight job.
curl -X DELETE https://api.vectoramp.com/ingestion/sources/c3d4e5f6-a7b8-9012-cdef-123456789012 \
  -H "X-API-Key: vsk_<64hex>"

# Force: disables the source's schedules, then deletes.
curl -X DELETE "https://api.vectoramp.com/ingestion/sources/c3d4e5f6-a7b8-9012-cdef-123456789012?force=true" \
  -H "X-API-Key: vsk_<64hex>"

{
  "error": "source_in_use",
  "message": "Source is attached to an active schedule or in-flight job. Retry with force=true to delete and disable its schedules.",
  "references": {
    "schedules": [{ "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "name": "Nightly docs sync" }],
    "schedule_count": 1,
    "active_job_count": 0,
    "in_use": true
  }
}

Check before deleting

Call Get Source References first to see exactly what would block a delete, or use List Unused Sources / Clean Up Unused Sources to remove only sources nothing is using.

Get Source References

GET /ingestion/sources/:sourceId/references

Returns everything currently using a source: active schedules and in-flight jobs. Use it to decide whether a source is safe to delete.

Parameter	Type	In	Description
`sourceId`	UUID	path	Source identifier

cURL
Response

curl https://api.vectoramp.com/ingestion/sources/c3d4e5f6-a7b8-9012-cdef-123456789012/references \
  -H "X-API-Key: vsk_<64hex>"

{
  "schedules": [
    { "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "name": "Nightly docs sync" }
  ],
  "schedule_count": 1,
  "active_job_count": 0,
  "in_use": true
}

in_use is true when schedule_count or active_job_count is non-zero — i.e. when a plain delete would return 409.

List Unused Sources

GET /ingestion/sources/unused

Returns sources that are not attached to any active schedule and have no in-flight job — the sources that are safe to delete. The response shape matches List Sources.

curl https://api.vectoramp.com/ingestion/sources/unused \
  -H "X-API-Key: vsk_<64hex>"

Clean Up Unused Sources

POST /ingestion/sources/cleanup

Deletes all unused sources (every source that List Unused Sources would return) in one call. Sources attached to an active schedule or in-flight job are left untouched.

cURL
Response

curl -X POST https://api.vectoramp.com/ingestion/sources/cleanup \
  -H "X-API-Key: vsk_<64hex>"

{
  "deleted": [
    { "id": "c3d4e5f6-a7b8-9012-cdef-123456789012", "name": "Old docs crawler", "type": "web" },
    { "id": "a7b8c9d0-e1f2-3456-7890-abcdef012345", "name": "Stale upload", "type": "file_upload" }
  ],
  "count": 2
}

Source Types

Each source_type has its own config schema. The tables below list the required fields and the most common optional fields with their defaults. You can also discover the supported types and their schemas programmatically.

Discover Source Types

GET /ingestion/source-types

Lists the supported source types and their high-level capabilities. Now exposed publicly through the gateway, so SDKs and agents can stay forward-compatible as new connectors ship.

cURL
Response

curl https://api.vectoramp.com/ingestion/source-types \
  -H "X-API-Key: vsk_<64hex>"

{
  "source_types": [
    { "type": "web", "name": "Web crawler", "supports_auth": false },
    { "type": "s3", "name": "Amazon S3", "supports_auth": true },
    { "type": "gcs", "name": "Google Cloud Storage", "supports_auth": true },
    { "type": "gdrive", "name": "Google Drive", "supports_auth": true },
    { "type": "jira", "name": "Jira", "supports_auth": true },
    { "type": "confluence", "name": "Confluence", "supports_auth": true },
    { "type": "file_upload", "name": "File upload", "supports_auth": false }
  ]
}

GET /ingestion/source-types/:type

Returns the JSON config schema for a single source type, including which fields are required and which are secrets.

curl https://api.vectoramp.com/ingestion/source-types/gdrive \
  -H "X-API-Key: vsk_<64hex>"

`web`

Field	Required	Default	Description
`start_urls`	✅	—	Array of seed URLs to crawl
`max_depth`		`3`	Maximum crawl depth
`max_pages`		`1000`	Maximum pages to fetch
`allowed_domains`		`[]`	Restrict crawling to these domains
`include_assets`		`true`	Crawl linked assets (PDFs, Office docs, images)
`max_assets_per_page`		`5`	Max assets fetched per page
`selectors`		—	`{ content, title, exclude }` CSS selectors
`headers`		`{}`	Extra HTTP request headers

`s3`

Field	Required	Default	Description
`bucket`	✅	—	S3 bucket name
`access_key_id`	✅	—	AWS access key id
`secret_access_key`	✅	—	AWS secret access key
`prefix`		—	Key prefix to scope ingestion
`region`		`us-east-1`	AWS region
`file_patterns`		`[".pdf",".docx",".txt",".csv"]`	Glob patterns to include
`max_file_size_mb`		`100`	Skip files larger than this
`sync_mode`		`incremental`	`full` or `incremental`

`gcs`

Field	Required	Default	Description
`bucket`	✅	—	GCS bucket name
`prefix`		—	Object prefix to scope ingestion
`project_id`		—	GCP project id
`auth_mode`		`oauth`	`service_account`, `oauth`, `connection`, or `adc`
`service_account_json`		—	Service-account JSON (when `service_account`)
`connection_id`		—	Id of a reusable connection (when authenticating with a granted connection)
`oauth_credentials`		—	OAuth credentials object
`file_patterns`		`[".pdf",".docx",".txt",".csv"]`	Glob patterns to include
`sync_mode`		`incremental`	`full` or `incremental`

See Authenticating sources for when to use service_account_json vs connection_id.

`gdrive`

Field	Required	Default	Description
`folder_ids` or `file_ids`	✅	—	Folders or files to ingest
`auth_mode`		`service_account`	`service_account`, `oauth`, or `connection`
`service_account_json`		—	Service-account JSON (headless; see Authenticating sources)
`connection_id`		—	Id of a reusable connection (interactive grant, reused server-side)
`oauth_credentials`		—	OAuth credentials object
`drive_id`		—	Shared drive id
`mime_types`		—	Restrict to these MIME types
`sync_mode`		`incremental`	`full` or `incremental`

For Google Drive, supply either service_account_json (headless) or connection_id (browser-granted, reusable). See Authenticating sources.

`jira`

Field	Required	Default	Description
`cloud_id`	✅	—	Atlassian cloud/site id
`access_token`		—	OAuth access token
`connection_id`		—	Id of a reusable Atlassian connection (resolved & refreshed server-side)
`project_keys`		—	Restrict to these project keys
`jql`		—	JQL filter for issues
`include_comments`		`true`	Include issue comments
`sync_mode`		`incremental`	`full` or `incremental`

`confluence`

Field	Required	Default	Description
`cloud_id` or `base_url`	✅	—	Atlassian cloud id or base URL (e.g. `https://company.atlassian.net`)
`auth_mode`		`basic`	`basic` (username + API token), `oauth`, or `connection`
`username`		—	Username for basic auth
`api_token`		—	API token for basic auth
`connection_id`		—	Id of a reusable Atlassian connection (resolved & refreshed server-side)
`oauth_credentials`		—	OAuth credentials object
`spaces`		`[]`	Space keys (empty = all accessible)
`include_attachments`		`false`	Ingest page attachments
`sync_mode`		`incremental`	`full` or `incremental`

`file_upload`

Field	Required	Default	Description
`storage_provider`		`s3`	Backing storage provider
`sync_mode`		`full`	Sync mode

file_upload sources are normally created and driven by the presigned upload flow below.

Presigned File Upload

To ingest local files, create a file_upload source and use the presigned upload flow. The SDK ingestFiles(...) helpers hide all four steps; the raw HTTP flow is:

1. Initialize the upload

POST /ingestion/sources/:sourceId/upload/init

Name	Type	Required	Description
`files`	array	✅	List of files to upload
`files[].name`	string	✅	File name
`files[].size_bytes`	integer		File size in bytes
`files[].content_type`	string		MIME type

cURL
Response

curl -X POST https://api.vectoramp.com/ingestion/sources/c3d4e5f6-a7b8-9012-cdef-123456789012/upload/init \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "files": [
      { "name": "report.pdf", "size_bytes": 1048576, "content_type": "application/pdf" }
    ]
  }'

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "uploads": [
    {
      "file_id": "f1a2b3c4-d5e6-7890-abcd-ef1234567890",
      "upload_url": "https://s3.amazonaws.com/...&X-Amz-Signature=..."
    }
  ]
}

2. Upload bytes

PUT each file's bytes directly to its presigned upload_url:

curl -X PUT "https://s3.amazonaws.com/...&X-Amz-Signature=..." \
  --upload-file report.pdf

3. Complete the upload

POST /ingestion/sources/:sourceId/upload/complete

Name	Type	Required	Description
`job_id`	UUID	✅	`job_id` returned by `upload/init`
`file_ids`	string[]	✅	`file_id` values for the files you uploaded

curl -X POST https://api.vectoramp.com/ingestion/sources/c3d4e5f6-a7b8-9012-cdef-123456789012/upload/complete \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "file_ids": ["f1a2b3c4-d5e6-7890-abcd-ef1234567890"]
  }'

Once completed, the ingestion job processes the uploaded files into the dataset. Track it with the Jobs endpoints.

Start an Ingestion Job

After registering a source, start ingestion with POST /ingestion/jobs:

curl -X POST https://api.vectoramp.com/ingestion/jobs \
  -H "X-API-Key: vsk_<64hex>" \
  -H "Content-Type: application/json" \
  -d '{
    "source_id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
    "dataset_id": "550e8400-e29b-41d4-a716-446655440000"
  }'

See Jobs for the full job lifecycle, retry, file audit, and statistics.

File Preview Helpers (Dashboard)

The following routes power the dashboard's source preview UI. They use SuperTokens session authentication, not API keys, and are not part of the programmatic SDK surface.

Get Source Info

GET /sources/info

Retrieve preview/download access for a single source path.

Name	Type	Required	Description
`source_path`	string	✅	Source URI (`gdrive://file_id`, `s3://bucket/key`, `https://...`)

curl "https://api.vectoramp.com/sources/info?source_path=gdrive://1abc2def3ghi" \
  -H "Authorization: Bearer <session_token>"

Stream Google Drive File

GET /sources/stream/gdrive/:fileId

Streams the content of a Google Drive file for in-dashboard preview.

Batch Source Info

POST /sources/batch-info

Retrieve preview/download access for multiple source_paths in a single request.

End-to-end flow​

Source Object​

List Sources​

Query Parameters​

Get Source​

Validate Source Config​

Request Body​

Create Source​

Request Body​

Authenticating sources​

Option 1 — Service account (headless)​

Option 2 — Connection reference (interactive once, reusable)​

Delete Source​

Query Parameters​

Get Source References​

List Unused Sources​

Clean Up Unused Sources​

Source Types​

Discover Source Types​

web​

s3​

gcs​

gdrive​

jira​

confluence​

file_upload​

Presigned File Upload​

1. Initialize the upload​

2. Upload bytes​

3. Complete the upload​

Start an Ingestion Job​

File Preview Helpers (Dashboard)​

Get Source Info​

Stream Google Drive File​

Batch Source Info​

End-to-end flow

Source Object

List Sources

Query Parameters

Get Source

Validate Source Config

Request Body

Create Source

Request Body

Authenticating sources

Option 1 — Service account (headless)

Option 2 — Connection reference (interactive once, reusable)

Delete Source

Query Parameters

Get Source References

List Unused Sources

Clean Up Unused Sources

Source Types

Discover Source Types

`web`

`s3`

`gcs`

`gdrive`

`jira`

`confluence`

`file_upload`

Presigned File Upload

1. Initialize the upload

2. Upload bytes

3. Complete the upload

Start an Ingestion Job

File Preview Helpers (Dashboard)

Get Source Info

Stream Google Drive File

Batch Source Info