Quick Start

​

Before you begin

CREATE SCHEMA IF NOT EXISTS analytics;

CREATE TABLE IF NOT EXISTS analytics.orders_live (
  id bigint PRIMARY KEY,
  customer_id bigint,
  order_status text,
  total_amount numeric,
  created_at timestamptz,
  updated_at timestamptz
);

​

1. Create connections and pipeline

1. Open Connections and create/test a PostgreSQL Source connection.
2. Create/test a PostgreSQL Destination connection.
3. Open Data Pipelines, click New Pipeline, enter a name, select the source connection, and create the pipeline.
4. Open the new pipeline. It opens in the tabbed pipeline workspace.

​

2. Configure Source

1. Open Source.
2. Click Test connection.
3. Click Discover catalog.
4. Enable public.orders.
5. Keep Full table for the first run.
6. Click Preview and inspect the rows.
7. Click Save stream settings.

​

3. Add a destination

1. Open Destinations.
2. Click Add destination.
3. Choose the destination connection, enter a display name, and set the target schema to analytics.
4. Click Save destination.

​

4. Create and publish a transformation

1. Open Transformations and select the new destination.
2. Click New transformation.
3. Enter a name, stable key, output model, and select public.orders as input.
4. Use this SQL:

SELECT
  id,
  customer_id,
  LOWER(order_status) AS order_status,
  total_amount,
  created_at,
  updated_at
FROM {{ source('raw', 'public__orders') }}
WHERE order_status IS NOT NULL

5. Click Save draft.
6. Click Validate draft.
7. Click Preview output and inspect the result.
8. Click Publish revision and confirm.

​

5. Assign the output target

1. Return to Destinations and open the destination.
2. Under Published output targets, set the table to analytics.orders_live.
3. Set the Upsert key to id.
4. Click Create or verify destination table when appropriate, then Save destination.

​

6. Validate and run

1. Open Overview.
2. Click Validate pipeline and resolve any incomplete readiness check.
3. Click Run all in the header.
4. Open Runs, select the execution, and confirm the destination, published revision, rows written, duration, and success status.

​

Add automation later