Pipelines Overview

​

What a Pipeline Does

1. Connects a source (e.g. PostgreSQL, Shopify) to a destination (e.g. BigQuery, PostgreSQL).
2. Selects which streams (tables or data feeds) to sync from the source.
3. Configures the sync mode (full, incremental, or CDC) and write mode (append, replace, or upsert).
4. Runs on demand or on a schedule. Each run reads from the source and writes to the destination.

​

Pipeline Components

• Source connection — The credentials and config for reading data.
• Destination connection — The credentials and config for writing data.
• Stream selection — Which tables or feeds to sync.
• Sync mode — How to read (full, incremental, CDC).
• Write mode — How to write (append, replace, upsert).
• Column mapping — Optional renaming or column selection.
• Transformation rules — Optional transforms (rename, cast, filter, etc.).
• Schedule — Optional automatic run interval.

​

Next Steps