Skip to main content
INCREMENTAL uses a stream cursor and saved checkpoint to read newer or changed source rows after each successful execution.

​
Requirements

  • a connector/stream that advertises Incremental support;
  • a stable, populated cursor such as updated_at, created_at, or an increasing ID;
  • published destination-owned SQL;
  • an existing destination table; and
  • a stable destination Upsert key.

​
Configure Incremental

  1. Open Source.
  2. Discover the catalog and enable the stream.
  3. Choose Incremental in the stream’s Sync mode column.
  4. Enter the cursor in the Cursor column.
  5. Preview and click Save stream settings.
  6. Configure and publish the destination’s transformation.
  7. Assign its output table and Upsert keys in the destination editor.
  8. Validate in Overview, click Run all, and inspect Runs.
  9. Add an interval or UTC cron schedule in Settings after the manual run succeeds.

​
Checkpoint state

After a successful run, MantrixFlow saves checkpoint state before deleting the temporary staging file. Failed runs do not advance the checkpoint. Use cursor columns that move forward predictably and are populated on every row. Index timestamp cursors for large database tables.