Skip to main content

Snowflake Destination

Connect Snowflake as a destination for loading data from your sources into your warehouse.

For a commercial overview of the connector, see the Snowflake connector page. If you are building a source-to-Snowflake pipeline, start with the Redshift to Snowflake, Google Drive to Snowflake, SQL Server to Snowflake, or Oracle TM to Snowflake guides.

Prerequisites

Before you begin, ensure you have:

  • An active Snowflake account with admin privileges (Don't have one? Sign up for a free trial)
  • A warehouse configured for data loading (or ability to create one)
  • Target database and schema created
  • Appropriate role with CREATE TABLE permissions
  • Network policies configured (if applicable)
  • Whitelist Supaflow IP (If Required)

If your Snowflake account has IP restrictions enabled, add this IP to your Network Access settings:

18.214.240.61

For reference: https://docs.snowflake.com/en/user-guide/network-policies

Run Script in Snowflake Warehouse

Follow these steps to set up your Snowflake environment:

  1. Log in to your Snowflake data warehouse
  2. Click "Copy Script" button below to copy the snowflake setup script
  3. In Snowflake, click ProjectsWorksheets+ (New Worksheet)
  4. Paste the entire script into the new worksheet
  5. IMPORTANT: Modify these values as needed:
    • Change the user_password (required)
    • Optionally modify role_name, user_name, warehouse_name, database_name, or schema_name
  6. Click the dropdown arrow next to the Run button ▼ and select "Run All"
note

Don't use the regular "Run" button — it only executes the current statement. Use "Run All" to execute the entire script.

Snowflake Setup Script (Key Pair Auth)
-- create variables for user / password / role / warehouse / database (needs to be uppercase for objects)
set role_name = 'SUPA_ROLE';
set user_name = 'SUPA_USER';
set warehouse_name = 'SUPA_WH';
set database_name = 'SUPA_DB';
set schema_name = 'SUPA_SCHEMA';
set fqn_schema_name = concat($database_name,'.',$schema_name);

-- change role to securityadmin for user/role steps
use role securityadmin;

-- create role for Supaflow
create role if not exists identifier($role_name);
grant role identifier($role_name) to role SYSADMIN;

-- create a user for SupaFlow
create user if not exists identifier($user_name)
type = SERVICE
default_role = $role_name
default_warehouse = $warehouse_name;

grant role identifier($role_name) to user identifier($user_name);

-- set binary_input_format to BASE64
ALTER USER identifier($user_name) SET BINARY_INPUT_FORMAT = 'BASE64';

-- change role to sysadmin for warehouse/database steps
use role sysadmin;

-- create a warehouse for Supaflow
create warehouse if not exists identifier($warehouse_name)
warehouse_size = xsmall
warehouse_type = standard
auto_suspend = 60
auto_resume = true
initially_suspended = true;

-- create database for Supaflow
create database if not exists identifier($database_name);

create schema if not exists identifier($fqn_schema_name);



-- grant supaflow role access to warehouse
grant USAGE on warehouse identifier($warehouse_name) to role identifier($role_name);

-- grant supaflow access to the database
grant ALL on database identifier($database_name) to role identifier($role_name);
grant ALL on ALL schemas in database identifier($database_name) to role identifier($role_name);
grant ALL on schema identifier($fqn_schema_name) to role identifier($role_name);

-- change role to ACCOUNTADMIN for STORAGE INTEGRATION support (only needed for Snowflake on GCP)
use role ACCOUNTADMIN;

-- transfer ownership of database and schema to SUPA_ROLE
GRANT OWNERSHIP ON DATABASE identifier($database_name)
TO ROLE identifier($role_name) COPY CURRENT GRANTS;
GRANT OWNERSHIP ON SCHEMA identifier($fqn_schema_name)
TO ROLE identifier($role_name) COPY CURRENT GRANTS;

grant CREATE INTEGRATION on account to role identifier($role_name);
grant CREATE EXTERNAL VOLUME on account to role identifier($role_name);
grant CREATE DATABASE on account to role identifier($role_name);
GRANT EXECUTE TASK ON ACCOUNT TO ROLE identifier($role_name);
GRANT EXECUTE MANAGED TASK ON ACCOUNT TO ROLE identifier($role_name);
GRANT APPLICATION ROLE SNOWFLAKE.EVENTS_VIEWER TO ROLE identifier($role_name);

grant all on future schemas in database identifier($database_name) to role identifier($role_name);
grant all on future tables in database identifier($database_name) to role identifier($role_name);
grant all on future views in database identifier($database_name) to role identifier($role_name);

-- grants for querying Iceberg tables via external volume + catalog integration
grant CREATE TABLE on all schemas in database identifier($database_name) to role identifier($role_name);
grant CREATE TABLE on future schemas in database identifier($database_name) to role identifier($role_name);

-- grant usage on future file formats and stages
GRANT USAGE ON FUTURE FILE FORMATS IN DATABASE identifier($database_name) TO ROLE identifier($role_name);
GRANT USAGE ON FUTURE STAGES IN DATABASE identifier($database_name) TO ROLE identifier($role_name);

Python Task Permissions

The current Snowflake setup script includes the permissions used by Python tasks:

  • Permission to create procedures and tasks in the configured database and schema
  • EXECUTE TASK and EXECUTE MANAGED TASK for Snowflake-managed serverless execution
  • The SNOWFLAKE.EVENTS_VIEWER application role for reading Python log messages from the default event table

If you configured the Snowflake role with an older copy of the script, run the current script again or ask a Snowflake administrator to apply the missing grants. Your Python code also runs with the datasource role's privileges, so grant that role access to every table, view, schema, and operation the task needs.

The Logs tab in Supaflow reads SNOWFLAKE.TELEMETRY.EVENTS_VIEW. If the account uses a custom active event table, the Python task still runs, but its log entries are not displayed in Supaflow. See Snowflake's event table overview and CREATE TASK access requirements.

Optional: Direct S3 External-Stage Loads

Most Snowflake destinations do not need an external stage. Leave External Storage Integration blank unless the source connector writes Snowflake-readable load files to S3 and the Snowflake destination should load directly from that S3 path.

In this flow, the source writes each job's files to an S3 bucket and prefix. The Snowflake destination creates a temporary external stage over that same S3 location and runs COPY INTO from the stage. This avoids downloading the source output and re-uploading it into a Snowflake internal stage.

The first supported use case is Amazon Redshift to Snowflake: the Redshift source writes files with UNLOAD, and Snowflake reads those files through a storage integration. For the full source and destination walkthrough, see the Redshift to Snowflake tutorial. The same Snowflake destination setting can be used by other Supaflow source connectors when their docs say they support direct S3 external-stage loads.

Use this option when all of the following are true:

  • The source connector supports direct S3 external-stage loads.
  • The destination is Snowflake.
  • You want Snowflake to read the source files directly from S3.
  • Your Snowflake administrator can create a storage integration and your AWS administrator can update the IAM role trust policy.

Do not use this field for pipelines whose source connector does not explicitly support direct S3 external-stage loads. Those pipelines use the standard Snowflake loading path.

Step 1: Confirm the Source S3 Location

In the source connector configuration, note the S3 bucket and prefix that will hold the direct-load files. For Redshift sources, these fields are:

  • S3 Staging Bucket
  • S3 Staging Prefix
  • Bucket Region

The Snowflake storage integration must allow the same S3 bucket and prefix. For example, if the source writes to s3://company-pipeline-stage/redshift/, the Snowflake integration's STORAGE_ALLOWED_LOCATIONS must include s3://company-pipeline-stage/redshift/ or a parent prefix that covers it.

Step 2: Create the Snowflake Storage Integration

Run the script below once in Snowflake. Update the variables before running it:

  • integration_name: the Snowflake integration name you will enter in Supaflow, for example SUPAFLOW_EXTERNAL_S3_INTEGRATION
  • storage_role_arn: the AWS IAM role Snowflake will assume to read the S3 staging location
  • storage_allowed_location: the S3 bucket and prefix used by the source connector's direct-load files
Create Snowflake Storage Integration for Direct S3 Loads
-- create_supa_storage_integration.sql
--
-- OPTIONAL. Enables Redshift -> Snowflake direct S3 external-stage loads.
-- Run this ONCE per Snowflake account, as SUPA_ROLE (which already holds
-- CREATE INTEGRATION; see create_supa_user.sql). The Supaflow connector does
-- NOT create this integration at runtime -- it only references it by name via
-- the Snowflake datasource advanced property `externalStorageIntegration`.
--
-- Two-step setup:
-- 1. Run this script. Fill in the three variables below first.
-- 2. Run DESC INTEGRATION (at the bottom) and copy STORAGE_AWS_IAM_USER_ARN
-- and STORAGE_AWS_EXTERNAL_ID into the AWS IAM role's trust policy so that
-- Snowflake's principal can assume the role (sts:AssumeRole with the
-- matching sts:ExternalId condition). The integration only works AFTER the
-- AWS trust policy is updated.
--
-- IMPORTANT: storage_allowed_location MUST cover whatever S3 bucket/prefix
-- Redshift UNLOADs to (the per-job stage URL must fall within it).

set integration_name = 'SUPAFLOW_EXTERNAL_S3_INTEGRATION';
set storage_role_arn = 'arn:aws:iam::<account-id>:role/<s3-read-role>'; -- s3:GetObject/ListBucket on the UNLOAD bucket; may be the SAME role your Redshift source already uses
set storage_allowed_location = 's3://<bucket>/<prefix>/'; -- MUST cover the Redshift UNLOAD bucket/prefix (the per-job stage URL must fall within it)

use role SUPA_ROLE;

CREATE STORAGE INTEGRATION IF NOT EXISTS identifier($integration_name)
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = 'S3'
ENABLED = TRUE
STORAGE_AWS_ROLE_ARN = $storage_role_arn
STORAGE_ALLOWED_LOCATIONS = ($storage_allowed_location);

-- Redundant when SUPA_ROLE created (and therefore owns) the integration above, but
-- required if an operator runs this as ACCOUNTADMIN instead: without USAGE, SUPA_ROLE
-- cannot create the external stage at load time. Safe to keep here -- the CREATE above
-- guarantees the integration exists, and GRANT has no IF EXISTS form.
GRANT USAGE ON INTEGRATION identifier($integration_name) TO ROLE SUPA_ROLE;

-- Read these two values and add them to the IAM role's trust policy:
-- Principal AWS = STORAGE_AWS_IAM_USER_ARN
-- sts:ExternalId = STORAGE_AWS_EXTERNAL_ID
DESC INTEGRATION identifier($integration_name);

The IAM role must be able to list and read the source files in S3. If you later enable Purge External Stage After Load, the same role also needs permission to delete objects from the staging prefix.

note

The standard Snowflake setup script grants CREATE INTEGRATION to SUPA_ROLE. If your Snowflake administrator creates the storage integration with a different role, grant the Supaflow role access to it:

GRANT USAGE ON INTEGRATION SUPAFLOW_EXTERNAL_S3_INTEGRATION TO ROLE SUPA_ROLE;
GRANT CREATE STAGE ON SCHEMA SUPA_DB.SUPA_SCHEMA TO ROLE SUPA_ROLE;

Step 3: Update the AWS Trust Policy

After you run the script, Snowflake returns STORAGE_AWS_IAM_USER_ARN and STORAGE_AWS_EXTERNAL_ID from DESC INTEGRATION.

Add those values to the AWS IAM role's trust policy so Snowflake can assume the role:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "<STORAGE_AWS_IAM_USER_ARN>"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<STORAGE_AWS_EXTERNAL_ID>"
}
}
}
]
}

If you use the same IAM role for Redshift, Supaflow, and Snowflake, keep the existing trust statements for Redshift and Supaflow and add Snowflake as an additional trusted principal. Do not replace the existing trust policy unless Snowflake is the only service that needs the role.

For Snowflake's full storage integration flow, see Configuring a Snowflake storage integration for Amazon S3.

Step 4: Configure the Snowflake Destination in Supaflow

In your Snowflake destination, open Advanced Settings and set:

  • External Storage Integration: enter the Snowflake integration name, for example SUPAFLOW_EXTERNAL_S3_INTEGRATION
  • Purge External Stage After Load: leave disabled unless the Snowflake IAM role has s3:DeleteObject permission and you want Snowflake to delete the source files after a successful load

The field expects only the Snowflake integration name. Do not enter an S3 bucket, role ARN, or credentials in this field.

Configuration

Step 1: Connection

Account Identifier*

Your full Snowflake account identifier.
Example: orgname-account_name.snowflakecomputing.com
or
Example: account_locator.cloud_region_id.cloud.snowflakecomputing.com

Warehouse*

Compute warehouse for queries
Example: SUPA_WH (created by script)

Database*

Target database name
Example: SUPA_DB (created by script)

Schema*

Target schema name
Example: SUPA_SCHEMA (created by script)


Step 2: Authentication

Username*

Your Snowflake username
Example: SUPA_USER (created by script)

Authentication Type*

Choose authentication method
Options: keypair, basic (password), or spcs_oauth
Default: keypair

Auth TypeUse Case
keypairRecommended for all standard connections. Uses RSA key pair authentication.
basicPassword-based authentication. Requires a Snowflake user with password login enabled.
spcs_oauthMost secure option. Used by Native App destinations where the agent runs inside Snowflake (SPCS). No credentials are stored -- authentication uses Snowflake-injected OAuth tokens.

When keypair is selected, you have two options for providing the private key:

Private Key*
  • Create Key -- Click to have Supaflow automatically generate a key pair and passphrase. This is the easiest option.
  • Upload Key -- Click to upload an existing private key file.
Important: Register the Public Key in Snowflake

After clicking Create Key, Supaflow generates a key pair and displays a SQL statement:

ALTER USER SUPA_USER SET RSA_PUBLIC_KEY='MIIBIjAN...';
  1. Click Copy to copy the SQL statement
  2. Open a Snowflake worksheet as SECURITYADMIN (or ACCOUNTADMIN)
  3. Paste and run the SQL statement

You must complete this step before the connection test will succeed.

Private Key Passphrase

Passphrase for the encrypted private key
Auto-generated when using Create Key. Only required if your uploaded key is encrypted.

Password Authentication

When basic is selected:

Password*

Your Snowflake password
Use the password set in the setup script

SPCS OAuth Authentication

When a Supaflow agent runs inside Snowflake as a Native App (via Snowpark Container Services), it uses spcs_oauth authentication. This is the most secure option:

  • Zero stored credentials -- No passwords, keys, or secrets are configured in the datasource. Authentication uses OAuth tokens that Snowflake injects directly into the running container at runtime.
  • No credential exposure risk -- Since no credentials exist in the datasource configuration, there is nothing that can be leaked, stolen, or rotated.
  • Private network path -- Data flows directly from the agent to Snowflake within the Snowflake infrastructure, without traversing the public internet.
info

After you approve a Native App agent in Settings > Agents, Supaflow shows a Create Snowflake Destination prompt. Click Create Destination to create the spcs_oauth destination. You can also create it later from the agent menu. You do not need to select this auth type manually in the destination form. See the Snowflake Native App Deployment Guide for the full setup walkthrough.


Step 3: Advanced Settings (Optional)

Role

The Snowflake role to use
Example: SUPA_ROLE (created by script)
Other options: ACCOUNTADMIN, SYSADMIN

Noop Query

Query to test connectivity
Default: SELECT 1

Query Timeout

Maximum seconds a Snowflake query may run. Use 0 for no timeout.
Default: 0 (Range: 0 to 86400)

Query Tag

Tag to apply to all queries for tracking
Default: Supaflow

Log Query Stats

Enable query statistics logging
Default: Disabled

External Storage Integration

Snowflake storage integration name for direct S3 external-stage loads. Leave blank for the standard Snowflake loading path.
Create the integration in Snowflake before entering the name here. For Snowflake's setup flow, see Configuring a Snowflake storage integration for Amazon S3.
Example: SUPAFLOW_EXTERNAL_S3_INTEGRATION

Purge External Stage After Load

Delete files from the external S3 stage after a successful direct load. Only enable this when External Storage Integration is set and the Snowflake IAM role has s3:DeleteObject permission.
Default: Disabled

Schema Refresh Interval

Interval in minutes for schema metadata refresh
0 = refresh before every pipeline execution
-1 = disable schema refresh
Default: 30 (Range: -1 to 10080)

JDBC Read Controls

The Snowflake connector inherits these JDBC read controls. They can generally be left at their defaults for Snowflake destination loads. Change them only when Supaflow support asks you to tune a Snowflake read path.

Chunked Read Enabled

Enable keyset-based chunked reads for full-table reads with a deterministic primary key.
Default: Disabled

Chunked Read Window Limit Enabled

Use SQL window limits for chunked reads.
Default: Disabled

Chunked Read Chunk Size

Maximum rows per chunk when chunked read window limits are enabled.
Default: 1,000,000 (Range: 1,000 to 5,000,000)

Chunked Read Memory Aware Limit Enabled

Reduce chunk size for very wide rows based on the configured memory target.
Default: Disabled

Chunked Read Target Memory MB

Target memory budget used when memory-aware chunk limits are enabled.
Default: 50 MB (Range: 1 to 4096)


Step 4: Test & Save

After configuring all required properties, click Test & Save to verify your connection and save the destination.

Loading Behavior

Supaflow creates destination tables when needed and applies the pipeline's configured load behavior. Keyed merge paths update existing rows and add new rows. Overwrite and reset paths can recreate a target table when the selected operation requires it.

Schema Evolution

Supaflow adds compatible columns and applies supported type changes when schema evolution permits them. Removed source columns remain in Snowflake instead of being dropped automatically.

When an existing table must be recreated, Supaflow preserves a compatible Snowflake clustering key whose referenced columns remain available. It also restores a suspended automatic reclustering state when Snowflake allows it. Review the activity if a table recreation reports that its previous physical design could not be reapplied.

Troubleshooting

Common issues and their solutions:

Invalid account identifier

Problem:

  • Connection fails with "Invalid account identifier" error
  • Cannot connect to Snowflake account

Solutions:

  1. Use the full account identifier including region and cloud provider
    • ✅ Correct: xy12345.us-east-1.snowflakecomputing.com
    • ❌ Wrong: xy12345
  2. Find your account identifier:
    • Log into Snowflake Console
    • Look at the bottom left corner
    • Copy the full identifier shown
  3. Verify format:
    • New format: orgname-account_name.snowflakecomputing.com
    • Legacy format: account_locator.region.cloud.snowflakecomputing.com

Warehouse suspended or not running

Problem:

  • Queries fail with "Warehouse not running" error
  • Connection test times out
  • Data loading fails intermittently

Solutions:

  1. Check warehouse status in Snowflake:
    • Navigate to: Admin → Warehouses
    • Verify warehouse state is "Started" or "Auto-Resume"
  2. Enable auto-resume:
    ALTER WAREHOUSE SUPA_WH SET AUTO_RESUME = TRUE;
  3. Manually start warehouse:
    ALTER WAREHOUSE SUPA_WH RESUME;
  4. Verify warehouse name:
    • Ensure the warehouse name in your config matches exactly (case-sensitive)

Network policy blocking connection

Problem:

  • Connection refused due to IP restrictions
  • "IP address not allowed" error
  • Connection timeout

Solutions:

  1. Add Supaflow IP to your network policy:
    -- Create network policy (if doesn't exist)
    CREATE NETWORK POLICY supaflow_policy
    ALLOWED_IP_LIST = ('18.214.240.61');

    -- Or modify existing policy
    ALTER NETWORK POLICY your_existing_policy
    SET ALLOWED_IP_LIST = ('your.existing.ip', '18.214.240.61');
  2. Verify network policies:
    • Go to: Admin → Security → Network Policies
    • Ensure 18.214.240.61 is in the allowed list
  3. Contact your Snowflake administrator if you don't have permission to modify network policies

Insufficient permissions

Problem:

  • Cannot create tables in target schema
  • "Access denied" or "Insufficient privileges" error
  • Data loading fails with permission error

Solutions:

  1. Re-run the setup script provided in Prerequisites section
  2. Manually verify permissions:
    -- Check current grants
    SHOW GRANTS TO ROLE SUPA_ROLE;

    -- Grant necessary permissions
    GRANT CREATE TABLE, INSERT, UPDATE, DELETE ON SCHEMA SUPA_DB.SUPA_SCHEMA TO ROLE SUPA_ROLE;
    GRANT USAGE ON DATABASE SUPA_DB TO ROLE SUPA_ROLE;
    GRANT USAGE ON WAREHOUSE SUPA_WH TO ROLE SUPA_ROLE;
  3. Verify role assignment:
    -- Check user's default role
    SHOW USERS LIKE 'SUPA_USER';

    -- Assign role if missing
    GRANT ROLE SUPA_ROLE TO USER SUPA_USER;

Authentication failed

Problem:

  • "Invalid username or password" error
  • Key-pair authentication fails
  • Connection refused

Solutions:

  1. For password authentication:
    • Verify password matches what you set in setup script
    • Check for special characters that might need escaping
    • Reset password if needed:
      ALTER USER SUPA_USER SET PASSWORD = 'your_new_password';
  2. For key-pair authentication:
    • Ensure private key is in correct PEM format
    • Verify public key is assigned to user in Snowflake
    • Check if private key is encrypted (requires passphrase)
    • Test key pair:
      -- Assign public key to user
      ALTER USER SUPA_USER SET RSA_PUBLIC_KEY='your_public_key';

Database or schema does not exist

Problem:

  • "Object does not exist" error
  • Cannot find database or schema
  • Setup test fails

Solutions:

  1. Verify database exists:
    SHOW DATABASES LIKE 'SUPA_DB';
  2. Create database if missing:
    CREATE DATABASE IF NOT EXISTS SUPA_DB;
  3. Verify schema exists:
    SHOW SCHEMAS IN DATABASE SUPA_DB;
  4. Create schema if missing:
    CREATE SCHEMA IF NOT EXISTS SUPA_DB.SUPA_SCHEMA;
  5. Check naming:
    • Database and schema names are case-sensitive
    • Ensure exact match with configuration

External storage integration cannot access S3

Problem:

  • A pipeline fails during the direct external-stage load
  • Error mentions a storage integration, allowed locations, CREATE STAGE, COPY INTO, or S3 access denied

Solutions:

  1. Confirm the Snowflake destination has the integration name.
    • In Advanced Settings, External Storage Integration should contain only the Snowflake integration name, such as SUPAFLOW_EXTERNAL_S3_INTEGRATION
    • Leave the field blank for pipelines whose source does not support direct S3 external-stage loads
  2. Confirm the allowed S3 location covers the source staging path.
    • The integration's STORAGE_ALLOWED_LOCATIONS must include the S3 bucket and prefix used by the source connector's direct-load files
    • For Redshift sources, this means the Redshift source S3 Staging Bucket and S3 Staging Prefix
    • If the source writes to s3://company-pipeline-stage/redshift/jobs/..., an allowed location of s3://company-pipeline-stage/other-prefix/ will fail
  3. Update the AWS IAM role trust policy after DESC INTEGRATION.
    • The trust policy must allow the Snowflake STORAGE_AWS_IAM_USER_ARN
    • The trust policy must use the matching STORAGE_AWS_EXTERNAL_ID
  4. Check Snowflake privileges.
    GRANT USAGE ON INTEGRATION SUPAFLOW_EXTERNAL_S3_INTEGRATION TO ROLE SUPA_ROLE;
    GRANT CREATE STAGE ON SCHEMA SUPA_DB.SUPA_SCHEMA TO ROLE SUPA_ROLE;
  5. Keep purge disabled unless delete access is configured.
    • If Purge External Stage After Load is enabled, the IAM role must have s3:DeleteObject on the staging prefix
    • If delete access is not approved, leave purge disabled and use an S3 lifecycle policy or external cleanup process

Connection timeout

Problem:

  • Connection times out after waiting
  • No response from Snowflake
  • Intermittent connectivity issues

Solutions:

  1. Check Snowflake service status:
  2. Verify network connectivity:
    • Ensure your network allows outbound HTTPS (443)
    • Check if corporate firewall is blocking Snowflake
  3. Test from different network:
    • Try from a different internet connection
    • Verify if issue is network-specific
  4. Contact support if timeout persists

Support

Need help? Contact us at support@supa-flow.io