Skip to main content

Sources

A source is a connection to an external system — a website, an S3 or GCS bucket, Google Drive, Jira, Confluence, or direct file uploads — that VectorAmp ingests into a dataset. You register a source once and then run pipelines, one-off jobs, or recurring schedules against it.

Why sources matter

Search quality depends on source quality. Sources give operators a place to see what was connected, which schedules and jobs depend on it, and whether anything still uses it before they change or remove it.

Source lifecycle​

  1. Create a source once (for example, a Google Drive folder, an S3 bucket, or a Confluence space).
  2. Configure access — upload files, a bucket path, a folder ID, a URL, or connector credentials.
  3. Start an ingestion job or attach a schedule to sync on a cadence.
  4. Monitor processing as VectorAmp parses, chunks, embeds, and indexes content.
  5. Review failures and retry after fixing permissions, file formats, or source paths.

Source types​

Source typeUse for
file_uploadFast dashboard ingestion for PDFs, docs, markdown, CSVs, and other local files.
s3 / gcsLarge document sets, exports, logs, generated artifacts, or customer file buckets.
gdriveTeam knowledge, policies, shared documents, and department folders.
webPublic or internal documentation pages that should be searchable.
jira / confluenceAtlassian issues and knowledge-base spaces.

Connector availability depends on your account and deployment. Use the Sources API for connector-level details.

Manage Sources page​

From the dashboard, open Sources to see every source in your organization. The table lists each source with:

ColumnMeaning
SourceThe source's name and id
Typeweb, s3, gcs, gdrive, jira, confluence, or file_upload
UsageWhether the source is Unused or in use — attached to schedules or with active jobs
ActionsDelete the source (with a force option if it is still attached)

The Usage column flags sources that are Unused, so you can spot what is safe to remove before cleaning up.

Credentials stay hidden

Once a source is saved, the dashboard never shows its stored credentials (access keys, tokens, passwords, or service-account JSON) again.

Deleting and cleaning up sources​

Over time you accumulate sources that nothing uses — a one-off crawl, a finished upload, an experiment. The Manage Sources page makes it easy to tidy these up:

  • Delete a source — removes a single source. If the source is still attached to an active schedule or has an in-flight job, the dashboard warns you and offers to force the delete, which detaches it from those schedules before removing it.
  • Clean up unused — the Clean up unused button removes every source that nothing is using. Sources attached to an active schedule or with an in-flight job are left untouched. The dashboard asks you to confirm before the cleanup runs.

Creating sources for scheduled jobs​

Most production ingestion runs on a schedule rather than a one-off job. The pattern is:

  1. Create the source once (for example, a Google Drive folder or a Confluence space).
  2. Attach a schedule that re-runs that source into a dataset on a cron cadence.
  3. The schedule keeps the source "in use", so a routine Clean up unused never touches it.

Because schedules reuse the same source, configure the source's credentials once and every future run reuses them.

Authenticated sources​

Sources that read from a private system (Google Drive, GCS, Jira, Confluence) need credentials. VectorAmp supports two ways to provide them:

  • Service account — fully headless. Paste a Google service-account JSON when creating the source. Best for unattended, scheduled ingestion where no one logs in. Remember to share the target Drive folder or bucket with the service account's email.
  • Connect (OAuth) — interactive once. Choose Connect when creating the source to open the provider's consent screen in your browser and approve access. VectorAmp stores the grant as a reusable connection and refreshes it server-side, so scheduled jobs keep working without re-consent. One connection can back many sources.

Source metadata​

Attach metadata that your application will need later. Useful fields include:

{
  "source": "google_drive",
  "source_id": "folder_abc",
  "owner": "finance",
  "doc_type": "policy",
  "visibility": "internal",
  "tenant_id": "org_123"
}

Good source metadata makes filters reliable and citations understandable.

Monitoring source health​

For each source, operators should be able to answer:

  • When did this source last sync successfully?
  • How many files or records were processed?
  • Which files failed and why?
  • Did newly added documents make it into the dataset?
  • Are stale or deleted documents still searchable?
  • Is the source still authorized?

Production notes​

  • Keep production and staging sources separate.
  • Prefer stable source IDs over human-readable folder names when possible.
  • Store connector credentials in approved secret systems.
  • Include source and document identifiers in metadata so citations can point back to the right object.
  • Test deletion/update behavior before relying on a source as authoritative.
  • Review failed files regularly; silent ingestion gaps create bad answers.

See also​