Skip to main content

Orchestrations

Chain ingestion pipelines, SQL tasks, and activation pipelines into end-to-end workflows.

Overview​

Orchestrations link multiple pipelines and tasks together into a single automated workflow. Define execution order, run steps in parallel, and build complete data flows from ingestion to transformation to activation.

Each orchestration is a series of steps where each step runs a pipeline or task. Control when steps run by setting dependencies - steps can run immediately, wait for specific steps, or wait for all previous steps to complete.

Common use cases:

  • Ingest from multiple sources in parallel, then transform
  • Sequential transformation: raw → staging → mart → activation
  • Fan-out: one ingestion feeds multiple activations

Prerequisites: At least one active pipeline (ingestion or activation) or task.

To access: Navigate to Orchestrations in the sidebar, then click + Create Orchestration.

Creating an Orchestration​

To start:

  1. Navigate to Orchestrations in the sidebar
  2. Click + Create Orchestration
  3. The orchestration builder opens

Add Steps​

Click + Add Step to add pipelines and tasks to your workflow.

  1. Click Add Step button
  2. Choose category:
    • Replication Pipelines (ingestion pipelines)
    • Activation Pipelines
    • Tasks (SQL tasks)
  3. Select pipeline or task from the list
  4. Repeat to add more steps

The orchestration builder shows a visual graph at the top and a table below with all steps.

Configure Dependencies​

Control when each step runs by setting dependencies.

To edit a step's dependency:

  1. Click the pencil icon in the "Runs after Step #" column
  2. Select which steps this step should wait for (checkboxes)
  3. Click Apply to save

Dependency options:

  • Select specific steps: Step runs after selected steps complete
  • Select all previous steps: Step waits for all prior steps
  • Click Clear: Step runs immediately (no dependency / parallel execution)

Default behavior: New steps run after the previous step (sequential)

Parallel Execution Example​

By default, steps run one after another. To run steps in parallel:

Before (Sequential):

  • Step 1: SFMC to Snowflake (no dependency)
  • Step 2: Salesforce to Snowflake (runs after Step 1)
  • Step 3: Oracle to Snowflake (runs after Step 2)
  • Step 4: Enrich Leads (runs after Step 3)

After (Parallel ingestion):

  1. Click pencil on Step 2 → Click Clear → Click Apply
  2. Click pencil on Step 3 → Click Clear → Click Apply
  3. Click pencil on Step 4 → Select all checkboxes (Steps 1, 2, 3) → Click Apply

Result: Steps 1-3 run in parallel, Step 4 waits for all to complete.

Name & Save​

Before saving, provide a name and optional description:

Orchestration Name (required): Descriptive name (e.g., "End to End Ingestion Transformation and Activation")

API Name (auto-generated): Lowercase identifier

Description (optional): Add notes about workflow purpose

To create: Click Create Orchestration.

Running an Orchestration​

To run manually:

  1. Navigate to Orchestrations in the sidebar
  2. Find your orchestration
  3. Click the ••• menu → Run Now

The orchestration executes immediately, creating activities for each step. Steps run according to their dependencies.

To monitor execution: Navigate to Activities in the sidebar to view activity status for each step.

Managing Orchestrations​

Access management options from the three-dot menu (•••) on each orchestration row.

Edit​

Modify steps, dependencies, or orchestration name.

To edit: Click ••• → Edit

You can add/remove steps, change dependencies, or update the orchestration name and description.

Run Now​

Manually execute the orchestration.

To run: Click ••• → Run Now

All steps execute according to their dependency configuration.

Delete​

Permanently remove the orchestration.

To delete: Click ••• → Delete

⚠️ Warning:

  • Deletion is permanent and cannot be undone
  • Ensure the orchestration is not scheduled before deleting
  • Historical activities remain visible in Activities page

Troubleshooting​

Steps Running in Wrong Order​

Problem: Steps execute sequentially when they should run in parallel.

Solution: Click the pencil icon on each step and click Clear to remove dependencies. Steps with no dependencies run in parallel.

Orchestration Won't Start​

Problem: Click "Run Now" but no activities are created.

Solution: Check that all steps reference active pipelines/tasks. Verify pipelines and tasks are not disabled.

Activity Failed​

Problem: Orchestration activity shows failed status.

Solution: Navigate to Activities, click the failed activity, and check which step failed. Fix the underlying pipeline or task, then re-run the orchestration.

Cannot Delete Orchestration​

Problem: Want to delete but orchestration is scheduled.

Solution: Navigate to Schedules, find schedules using this orchestration, delete or edit them, then return to delete the orchestration.

Support​

Need help? Contact us at support@supa-flow.io

Last updated on by pungaa
AI Tools
Ask ChatGPTClaudeAsk Claude
On this page