> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mantrixflow.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipelines Overview

> Understand the live MantrixFlow pipeline lifecycle: shell creation, source selection, destination-scoped Normalisation and SQL, manual runs, and Incremental schedules.

A MantrixFlow pipeline defines how data moves from one PostgreSQL source connection to one PostgreSQL destination connection.

## Lifecycle

1. Create a pipeline shell from **Data Pipelines**.
2. Open the builder canvas.
3. Configure the **Source** node: test connection, discover tables, include `schema.table` streams, and preview rows.
4. Configure the **Destination** node: choose destination connection, set sync mode, add optional rename/cast rules, write SQL models, validate, preview, and save.
5. Run the destination manually.
6. Review run status and run history.
7. Switch to Incremental sync and add a schedule only after manual validation.

## Builder nodes

The current public builder workflow uses two node types:

* **Source**: connection test, schema discovery, stream inclusion, row preview
* **Destination**: final table targets, Normalisation, SQL Layer, preview, schedule, manual run

## Delivery model

Each selected source stream is identified as `schema.table`, for example `public.orders`.

During a run:

1. MantrixFlow stages source rows in temporary storage.
2. Normalisation applies destination-owned rename/cast rules.
3. SQL models read from `raw.schema__table`.
4. Delivery writes model output into pre-created destination tables.
5. Checkpoint state is extracted and temporary staging files are deleted.

## Destination contract

Final delivery tables must already exist. The runner validates table existence and column compatibility before writing.

Use final destination table names as `schema.table`, for example `analytics.orders_live`.

## Write behavior

The live write mode is **Upsert**.

Tables with primary keys use merge-style delivery. Tables without primary keys receive appended rows and a no-primary-key warning in run metadata.
ta.