Creating a pipeline

Pipeline creation starts with a small form and continues on the pipeline detail page. There is no canvas or node editor.

​

Create the pipeline

1. Open Data Pipelines.
2. Click New Pipeline.
3. Enter a name and optional description.
4. Choose the source connection.
5. Create and open the pipeline.

​

Configure Source

1. Open Source.
2. Test the connection and click Discover catalog.
3. Enable the required streams.
4. Choose Full table or Incremental for each supported stream.
5. Enter a cursor for Incremental streams.
6. Inspect fields, preview rows, and click Save stream settings.

Start with one stream for the first run.

​

Add a destination

1. Open Destinations.
2. Click Add destination.
3. Choose a destination connection and set its schema/database or Airtable base/table.
4. Click Save destination.

​

Create a transformation

1. Open Transformations and select the destination.
2. Click New transformation.
3. Choose input streams, enter the output model name, and write SQL against the staged source relations.
4. Save draft, Validate draft, Preview output, and Publish revision.

Published revisions remain isolated to their owning destination.

​

Configure output targets

1. Return to Destinations and open the destination.
2. Under Published output targets, enter each final table and Upsert key.
3. For Airtable, map output columns to writable fields and select merge fields.
4. Use Create or verify destination table for database targets when needed.
5. Save the destination.

​

Validate and run

Open Overview, click Validate pipeline, and complete all readiness checks. Then click Run all in the header and inspect the result in Runs.

​

Add automation later

Open Settings after a successful manual run. Configure a fixed 1–24 hour interval or a five-field UTC cron schedule, save it, and keep the pipeline active.