Ingestion Pipelines

Move data from sources to your data warehouse with full control over sync modes, schema selection, and execution.

Overview

Ingestion pipelines extract data from external sources (databases, SaaS platforms, APIs) and load it into your data warehouse. Configure how data is synced, loaded, and how schema changes are handled.

Prerequisites: At least one active source and one active destination.

To access: Navigate to Ingestion in the sidebar, then click Create Ingestion Pipeline.

---

Creating an Ingestion Pipeline

To start:

1. Navigate to Ingestion in the sidebar
2. Click Create Ingestion Pipeline
3. If you have multiple projects, select which project to create the pipeline in (this determines the destination warehouse)
4. The selected project opens with its pipelines in an Explorer panel on the left

The pipeline wizard then opens with 4 steps:

Step 1: Choose Source

Select the data source for your pipeline. Only active sources are available.

The Objects column shows the last available schema refresh count. A dash means the count is not yet available; 0 means no supported objects were available when counted. This count does not replace the field, key, and cursor checks in Step 3.

If the selected source has no supported objects, review its discovery settings and read permissions, then use Refresh Schema from its menu on the Sources page. Monitor the refresh under Activities. Fix and re-test a failed connection on the Sources page before using it.

To proceed: Click Continue.

---

Step 2: Configure Pipeline

The destination is pre-selected based on the project's warehouse (all pipelines belong to a project). Configure your pipeline settings below.

Pipeline Settings

Availability Note

Not every pipeline setting or option is available for every connector. Supaflow reads source and destination capabilities and only enables supported options. For example, some destinations only allow APPEND load mode or LATENCY optimization.

UI Note

To keep the wizard focused, Supaflow shows only 4 core settings by default in this step. All other supported settings are under Advanced Settings.

Destination Section (Schema/Prefix)

At the top of Step 2, Supaflow shows the selected destination with key connection details (destination name, database, schema/prefix, role).

The schema/prefix column in this section is where destination naming is surfaced:

• When Namespace Rule = Destination Defined: the card shows the destination Schema from destination config (read-only in this step)
• Otherwise: the card shows the pipeline namespace value used for naming. The UI label is:
  • Prefix for sources with schema/catalog hierarchy (PostgreSQL, MySQL, etc.)
  • Schema for sources without hierarchy (Salesforce, HubSpot, etc.)

System generates a unique prefix for each pipeline by default (for example, salesforce, salesforce_2) so pipelines do not collide.

How the prefix is used:

• Sources without schemas (Salesforce, HubSpot, etc.): Prefix becomes the schema name where tables are created
  • Example: Prefix salesforce → Schema salesforce → Tables salesforce.Account, salesforce.Contact
• Sources with schemas (PostgreSQL, MySQL, etc.): Prefix is prepended to the source schema name
  • Example: Prefix postgres + Source schema orders → Destination schema postgres_orders

When editable, click the schema/prefix chip (or pencil icon) in the destination section to update it.

If the selected schema/prefix is already used by another pipeline, Supaflow shows a warning. Collision outcomes are then governed by Destination Table Collision Handling.

Sync Settings

Setting	Description	Options
Ingestion Mode	How data should be synced from source	• Historical + Incremental (default) - Full sync first, then ongoing changes • Historical Only - Full data sync every run • Incremental Only - Only new and changed data
Error Handling Mode	How to handle errors for each object	• Continue with Warnings (default) - Log supported row-level errors and continue processing the object • Abort on Any Error - Stop processing the affected object when an error occurs; other objects continue. The overall run completes with warnings unless every object fails
Save error files for debugging	Whether to save rejected-row files to Supaflow for debugging. These files may contain customer data. Available under Advanced Settings.	• Off (default) - Keep error files on the agent • On - Save error files to Supaflow
Full Sync Object Refresh Frequency	How often objects that do not support incremental ingestion are synced. Disabled when Ingestion Mode is Historical Only.	• Every Run - Resync on every pipeline execution • Daily - Resync every day • Weekly (default) - Resync every 7 days • Monthly - Resync every 30 days • Never - Only run once (initial sync), then skip
Full Resync Frequency	How often all objects are forced into a complete resync, regardless of incremental support. Disabled when Ingestion Mode is Historical Only.	• Never (default) - No scheduled full resync; only incremental syncs after initial load • Daily - Force full resync every day • Weekly - Force full resync every 7 days • Monthly - Force full resync every 30 days

Important: For objects that do not support incremental ingestion, Supaflow runs historical sync only when the configured Full Sync Object Refresh Frequency is due (or when a forced full resync applies). On runs where historical sync is not due, those objects can be skipped.

Save error files for debugging

Under Advanced Settings → Sync Settings, choose whether rejected-row files can be saved to Supaflow for debugging. The default is Off, including for existing pipelines that have no saved value for this setting. Local error files continue to be generated when the setting is off.

Turn it On only when you want these files saved to Supaflow. They may contain customer data. This choice is available for ingestion and activation pipelines and is independent of Error Handling Mode and connector capabilities.

For self-hosted deployments, use an agent version that supports this setting. Older agents do not enforce it.

For an existing pipeline, open its Settings tab, expand Advanced Settings, edit Save error files for debugging, and click Save Changes. Turning it off blocks subsequent file uploads; a transfer already in progress may finish, and files already saved to Supaflow are not deleted.

For CLI and MCP configuration, use the JSON boolean property upload_error_files_to_control_plane:

{
  "upload_error_files_to_control_plane": false
}

Set it to true to opt in. Missing, null, or non-boolean values do not permit uploading error files.

Load Settings

Setting	Description	Options
Load Mode	How data should be loaded into the destination	• Merge (default) - Insert new and update existing records when a stable key is available • Append - Add new records only • Truncate and Load - Delete all data then load • Overwrite - Drop and recreate tables
Objects without a primary or unique key	How to identify rows without a stable key. Available under Advanced Settings.	• Generate UUID (default) • Row hash (where the destination supports it)
Destination Namespace Rules	How destination schemas and tables should be mapped from the source	• Mirror Source (default) • Destination Defined - Use destination's default schema • Fivetran Naming - Use Fivetran-like naming
Destination Table Collision Handling	How to handle table name collisions during initial sync	• Merge with Existing (default) • Fail if Table Exists (recommended) • Drop and Recreate
Perform Hard Deletes	Permanently delete records from destination when deleted from source	• No (default) - Keep all records • Yes - Delete records
Load Optimization Mode	Strategy for optimizing pipeline execution	• Optimize for Cost (default) - Batch operations • Optimize for Latency - Process immediately • Adaptive - System decides
Data Validation Level	Level of data integrity validation during sync	• None (default) - No validation • Row Count - Validate row counts • Schema and Count • Column Statistics • Full Integrity

Objects without a primary or unique key

Under Advanced Settings → Load Settings, choose Objects without a primary or unique key:

Option	When the object requests Merge
Generate UUID (default)	Gives each row a new ID and uses Append. Every full read adds another copy of all returned rows. On incremental reads, a changed row is added and its old version stays; overlapping reads can repeat rows too.
Row hash	Identifies rows by their selected data-column values so Merge can skip exact repeats. Changed rows are added as new rows, and identical source rows are merged into one. Available only when the destination supports row-hash merging.

Use a primary or unique key when you need updates to the same record. Row hash does not provide that behavior. The setting also applies to child objects whose IDs depend on a parent without a key; having a key on the child alone does not make those IDs stable.

The choice does not change the read schedule or an explicit Append, Truncate and Load, or Overwrite load mode. Explicit Append still adds rows with either option. Objects read in full on every sync can therefore add a full copy on every sync, not only during a scheduled full resync. With Generate UUID, keyless row IDs also change across full replacement loads. Do not use those IDs for downstream joins that require a stable record identifier.

Before creating or saving: Click the warnings indicator beside Create Pipeline or Save Changes to see applicable warnings. For a saved pipeline, the dialog lists affected objects, with search, schema groups and a CSV download. During creation, the warning is conditional until the saved pipeline's object scope is available. Job Details shows Requested MERGE → APPEND when Merge becomes Append and explains the effect on full or incremental reads.

Changing an existing pipeline: Open Settings → Advanced Settings → Load Settings, change the option, review the warnings, and click Save Changes. Supaflow requests a reset and full reload of affected destination tables when those objects next sync. Saving does not start a sync; the next scheduled or manually started run applies the reset. No manual reset is needed.

Review the tables before saving

Existing rows in the affected tables will be deleted and reloaded from the source. Rows no longer available from the source will not come back, and the next sync may take longer.

A child reset may require reading its parent in full. Tables loaded with Append from that same read may also be reset, even if they have their own keys, to avoid appending another complete copy. Other independently keyed tables are not reset by this setting change. Review the affected-table list for the scope.

If an affected object is skipped or fails, its reset remains pending. Append-only tables read with it may be reset again when it next runs, losing history accumulated between those attempts.

A manual Full Resync also applies pending resets from a settings change, even without choosing Recreate Destination Tables. The resync dialog shows the pending-reset count and explains that append-only tables read with them will also be recreated when they run.

For CLI and MCP configuration, use keyless_object_handling:

{
  "keyless_object_handling": "UUID"
}

Use ROW_HASH to opt into row hashes. Missing or null values default to UUID; other values are rejected. Row hash is rejected at create or save when the destination does not support it. An unrelated CLI edit that omits this field preserves its saved value.

Schema Evolution Settings

Setting	Description	Options
Schema Evolution Mode	How to propagate source schema changes to the destination	• Allow All Changes (default) - Automatically propagate all changes • Block All Changes - Prevent any modifications • Column Level Changes Only - Allow column changes only
Auto Re-Sync on New Table	Automatically do a full re-sync for new tables	• Yes (default) • No
Auto Re-Sync on New Column	Automatically re-sync entire table when new column detected	• No (default) • Yes
Auto Re-Sync on Schema Change	Automatically re-sync when any schema change detected	• No (default) • Yes

To proceed: Click Continue.

---

Step 3: Choose Objects to Sync

Select which tables and fields to replicate from your source.

To select:

• Use the top-level checkbox to select all objects
• Or expand catalogs/schemas and select individual tables and fields
• Use the search bar to filter objects by name

Use Selected, Unselected, Unavailable, and Errors to narrow the list. Unavailable objects remain visible for diagnosis but cannot be selected. Refresh the source schema after correcting discovery settings or permissions, then review the objects again.

Selections are saved automatically as you work.

Changing a field selection does not itself remove a destination column or reload historical rows. Before reselecting a previously excluded field, review Schema Evolution Mode, destination permissions, and whether older rows need a resync. Saving the pipeline does not start a sync.

To proceed: Click Continue.

---

Step 4: Review & Save

Review all pipeline settings and selected objects. If a warnings indicator appears beside Create Pipeline, open it to review the applicable warnings before creating.

Pipeline Name: Auto-generated from source/destination names (e.g., "Salesforce to Snowflake"). You can edit this before creating.

To create: Click Create Pipeline.

You'll be redirected to the pipeline detail page where you can view settings, schema, and run the pipeline.

---

Pipeline Detail Page

After creating a pipeline, you'll see Settings, Schema, and Jobs tabs:

Settings Tab

• View all pipeline configuration and edit supported settings, including Objects without a primary or unique key and Save error files for debugging
• See source, destination, and all sync/load/schema evolution settings
• Click Sync Now button to run the pipeline

Schema Tab

• View all selected objects and their fields
• Filter by: All, Enabled, or Disabled objects
• Search for specific tables
• See total object count (e.g., "594/594 objects selected")

To access: Navigate to Pipelines in the sidebar, then click on any pipeline name.

Notification Preferences

The Settings tab also includes Notification Preferences, which controls email notifications for this specific pipeline. You can enable or disable notifications for failures, warnings, and successes independently.

These settings override your workspace-wide notification defaults. To revert to workspace defaults, click Use workspace defaults.

When you create a pipeline, Supaflow automatically subscribes you to failure and warning notifications.

---

Running a Pipeline

To run manually:

1. Navigate to Pipelines in the sidebar
2. Find your pipeline
3. Click the ... menu → Sync Now

Or from the pipeline detail page, click the Sync Now button.

What happens next:

• A success message appears: "Sync request was successfully submitted"
• Click the View activity link to monitor the pipeline sync
• The activity begins processing immediately

---

Monitoring Pipeline Runs

After clicking Sync Now, navigate to Activities in the sidebar to monitor pipeline execution.

Each activity shows status (Queued → Running → Completed or Failed), object counts, and row metrics. Click a Pipeline Run to view detailed per-object breakdowns with ingestion mode (Historical/Incremental) and stage-level metrics.

For comprehensive monitoring details: See the Activities documentation.

---

Managing Pipelines

Access pipeline management options from the three-dot menu (•••) on the pipeline detail page.

Resync Data

Force a full data resync from the source, ignoring incremental state.

When to use:

• Source data changed outside normal sync process
• Need to refresh all historical data
• Recovering from data quality issues

To resync: Click ••• → Resync Data

Pending resets from a change to Objects without a primary or unique key still apply during Full Resync. Check the dialog for affected-table counts before starting. See changing the keyless-row setting.

Disable Pipeline

Pause the pipeline to prevent it from running.

When to use:

• Temporarily stop syncing without deleting the pipeline
• Source credentials expired and need updating
• Maintenance or troubleshooting

To disable: Click ••• → Disable Pipeline

Note: Disabled pipelines won't run on schedule. Re-enable by clicking ••• → Enable Pipeline.

Edit Pipeline

Modify pipeline settings or object selection.

What you can edit:

• Pipeline name and description
• Sync settings (ingestion mode, error handling, saving error files for debugging)
• Supported load settings, including Objects without a primary or unique key. Load Mode itself is read-only in the saved pipeline Settings view.
• Schema evolution settings
• Object and field selection

To edit: Click ••• → Edit Pipeline

You'll be redirected to the pipeline wizard to make changes.

Delete Pipeline

Permanently remove the pipeline and its configuration.

To delete: Click ••• → Delete Pipeline

⚠️ Warning:

• Deletion is permanent and cannot be undone
• Activity history is preserved for audit purposes
• Data already loaded in the destination is not deleted

---

Troubleshooting

Pipeline Won't Create

Problem: Error about missing sources or destinations.

Solution: Ensure at least one source and one destination have status Active.

Pipeline Run Fails Immediately

Problem: Activity fails within seconds.

Solution: Click the activity to view error details. Common causes include expired credentials, insufficient permissions, or schema mismatches.

No Data in Destination

Problem: Activity shows Completed but destination has no data.

Solution: Verify tables are selected in object selection, check the destination schema name matches your destination prefix, and review row counts in the activity detail page.

---

Related Pages

• Activities - Monitor pipeline execution
• Schedules - Automate pipeline runs
• Deployments - Promote pipelines across workspaces

---

Support

Need help? Contact us at support@supa-flow.io