Skip to main content
A MantrixFlow pipeline defines how data moves from one PostgreSQL source connection to one PostgreSQL destination connection.

​
Lifecycle

  1. Create a pipeline shell from Data Pipelines.
  2. Open the builder canvas.
  3. Configure the Source node: test connection, discover tables, include schema.table streams, and preview rows.
  4. Configure the Destination node: choose destination connection, set sync mode, add optional rename/cast rules, write SQL models, validate, preview, and save.
  5. Run the destination manually.
  6. Review run status and run history.
  7. Switch to Incremental sync and add a schedule only after manual validation.

​
Builder nodes

The current public builder workflow uses two node types:
  • Source: connection test, schema discovery, stream inclusion, row preview
  • Destination: final table targets, Normalisation, SQL Layer, preview, schedule, manual run

​
Delivery model

Each selected source stream is identified as schema.table, for example public.orders. During a run:
  1. MantrixFlow stages source rows in temporary storage.
  2. Normalisation applies destination-owned rename/cast rules.
  3. SQL models read from raw.schema__table.
  4. Delivery writes model output into pre-created destination tables.
  5. Checkpoint state is extracted and temporary staging files are deleted.

​
Destination contract

Final delivery tables must already exist. The runner validates table existence and column compatibility before writing. Use final destination table names as schema.table, for example analytics.orders_live.

​
Write behavior

The live write mode is Upsert. Tables with primary keys use merge-style delivery. Tables without primary keys receive appended rows and a no-primary-key warning in run metadata. ta.