Skip to main content

Custom Connectors

Beak AI ships connectors for a long list of enterprise platforms (see System Datasources). This page covers the other half: connecting an application that isn't on that list, without waiting for a product release.

A connector is now described by a manifest — a single definition saying how to authenticate, what connection fields to show, and which collections of records can be ingested. Register a manifest and the connector appears in Studio's connector picker immediately.

Before you start​

You need:

  • API base URL — the application's endpoint, e.g. https://acme.example.com.
  • Credential — an entry in the Credential Vault that the API accepts.
  • Network reachability — the Beak AI environment must reach that URL. Self-hosted systems on a private network need the tenant's egress_allow_private setting enabled; ask your administrator.
  • Connector permission — connectors:manage. Only administrators hold it, because a connector decides which URLs the ingestion service will call.

Step 1 — Create the datasource​

  1. Go to Datasources → Create Datasource.
  2. Choose type System, then pick your connector.
  3. Fill in the connection fields and choose the credential.
  4. Save.

The fields shown come from the connector's manifest, so a newly registered connector renders its own form with no Studio update.

Step 2 — Detect what can be synced​

Open the saved datasource and find the Resource section, then click Detect automatically.

Beak AI probes the API and reports what it finds:

  1. It looks for a published API description — an OpenAPI/Swagger document at a conventional path, or an OData service document.
  2. If there isn't one, enter one or more endpoint paths to sample (e.g. /api/v2/tickets, /api/v2/users) and click Detect data.

For each collection it finds, Beak AI samples live records and reports:

ReportedWhat it means
Records atWhere in the response the records sit (e.g. $.data, $.value)
FieldsEvery field found, with its type and how often it is empty
PaginationHow the API pages through results
IncrementalThe field that marks when a record last changed. Recorded for future use — syncs do not filter on it yet (see Step 3)

Collections that can't be synced are still listed, together with the reason — so you always see everything the API exposes, not a filtered view.

Each collection is marked Syncable or Not syncable.

Note: Detection reads a small sample from the API. It does not ingest anything, does not modify the source system, and does not change your datasource — it only reports what the API exposes.

Detection is a read-only report, not a setting. It tells you what a connector's API offers, so you know what is available before deciding what this datasource should ingest. It deliberately does not fill in the Resource field, because the two values are different things: a detected collection is named after its URL path (/api/v2/tickets is reported as api_v2_tickets), while Resource has to match the resource type configured for that connector (ticket). Copying the detected name into the field would save a value the sync pipeline cannot resolve.

So use detection to decide what to sync, then set Resource to the resource type the connector declares and click Save resource.

Step 3 — Sync​

Use Sync now from the datasource list, or set a recurring schedule from the Knowledge page in Agent Runtime (administrators only). Frequency options run from every 6 hours to monthly, or any custom cron expression with a live preview of the next run.

Every sync is a complete refresh​

A sync re-reads the whole resource, then replaces the datasource's previous records with what it just read. So each run leaves exactly one copy of every record as it exists in the source system right now:

  • Edited records are corrected — an updated ticket is not left beside its older version.
  • Deleted records disappear — a record removed at the source stops being retrievable.
  • Repeated syncs don't accumulate — running twice yields the same count, not double.

Note: Fetching everything and replacing everything are one decision, not two. Replacing after a partial read would delete the records the read skipped, so a run that fetches less than the whole resource never replaces.

If a safety limit cuts a run short, the records read are kept, the run reports incomplete, and the previous records are left in place rather than replaced.

Incremental syncs — reading only what changed — are not offered yet. Where a connector exposes a last-changed field, Beak AI detects and records it, so no history is lost when the mode arrives; it is simply not used to filter a fetch today. Two things have to be true per vendor first: the API must have a last-changed filter at all, and that filter must cover edits rather than only creation. A filter that covers only creation would silently never re-read an edited record — one correct mode beats two where one is quietly lossy.

What happens to the data​

Each record is turned into a short readable document, embedded, and stored in the tenant's vector store. Agents then retrieve it semantically — an agent asked "has anyone reported the VPN dropping this week?" matches on meaning, not keywords.

By default a record renders as Field: value lines, which works for any application without setup. For better retrieval, a connector can define a document template:

Ticket #{id}: {subject}

Status: {status}
Requester: {requester_name}

{description}

Fields chosen as metadata additionally become filters, so an agent can be scoped to (say) only open tickets.

Note: Any kind of record works — tickets, assets, users, groups, contracts, purchase orders. There is no fixed list of supported entity types.

Troubleshooting​

SymptomCause and fix
"No resource set" on the datasourceA resource hasn't been saved yet. Sync will fail until one is. Pick one from the Resource list, or type the connector's resource type if the connector doesn't declare its resources yet.
Sync fails even though a resource is savedThe saved value doesn't match a resource type configured for that connector. It must be the connector's resource type (e.g. ticket), not a detected collection name (e.g. api_v2_tickets).
Detection finds nothingThe API publishes no spec at a conventional path. Enter the endpoint paths directly.
A collection shows "Not reachable"Usually the credential lacks permission for that endpoint, or the path returns something other than a list of records. The reported reason narrows it down.
Detection is blockedThe URL resolves to a private address and the tenant hasn't opted in to private endpoints. Ask an administrator about egress_allow_private.
A collection needs "engine support"The API pages results in a way not yet supported (for example a continuation URL inside the response body). That collection can still be set up to sync — raise it with support.
Sync reports incompleteA safety limit stopped the run (page, record, or size cap). The records read are kept and the run says which limit was hit; a tenant's limits are configurable.

Best Practices​

  • Detect rather than type — let Beak AI report the record path and incremental field instead of entering them by hand; a wrong record path yields a sync that succeeds with zero records.
  • Size the schedule to the resource — every sync re-reads the whole resource, so a large collection on a 6-hourly schedule is real load on the source system's API limits. Start wider and tighten it once you know how long a run takes.
  • Scope the credential narrowly — read-only, and only over the collections you intend to ingest.
  • Start with one collection — confirm the records look right in an agent's answers before adding more.
  • Review what a record contains — anything ingested becomes retrievable by any agent the datasource is attached to. Credential-looking fields are withheld automatically, but business- sensitive content is not.
  • Sync on a schedule, not manually — a recurring schedule keeps agents current; ad-hoc syncs drift.