Amazon S3 Data Lake Connector

Read Parquet and Iceberg tables from Amazon S3, or load pipeline data into your own S3 data lake for open analytics.

SourceDestinationBronze

Start for Free Source Documentation Destination Documentation

Evaluating against Fivetran? See how Supaflow handles Amazon S3 Data Lake pricing, connector quality, and Snowflake deployment side-by-side.

Supaflow vs Fivetran

Why Supaflow

All connectors included

Every connector is available on every plan. Pricing does not increase with connector count.

Pay for compute, not rows

Credit-based pricing. Usage scales with your pipelines, not with row counts.

One platform

Ingestion, dbt Core transformation, reverse ETL, and orchestration in a single workspace.

Capabilities

Parquet and Apache Iceberg Table Formats

Read or write Parquet files for broadly compatible data lake storage, or work with Apache Iceberg tables through AWS Glue or Snowflake Open Catalog.

Catalog and Prefix-Based Discovery

Discover source tables from AWS Glue, Snowflake Open Catalog, or S3 prefix layouts. Prefix-based Parquet sources support directory, Hive, and table-mapping layouts.

Cross-Account IAM Role Assumption

Supaflow assumes an IAM role in your AWS account for S3 and catalog access. Your credentials stay in your account, and the external ID mechanism prevents confused deputy attacks.

Open Destination Writes

Load pipeline output as Parquet files or Iceberg tables in your own bucket, with optional Glue registration for Parquet and catalog-managed Iceberg writes.

Supported Objects

Source Table Formats

Parquet

Read Parquet files from AWS Glue table locations or prefix-based S3 layouts.

Apache Iceberg

Read Iceberg tables through AWS Glue or Snowflake Open Catalog.

Catalog Options

AWS Glue Data Catalog

Register tables as Glue databases and tables. Query with Athena, Redshift Spectrum, Spark, or EMR. Works with both Parquet and Iceberg.

Snowflake Open Catalog

Iceberg REST catalog backed by Snowflake. Query with Snowflake, Spark, Trino, or any Iceberg-compatible engine. Requires OAuth2 credentials.

Destination Table Formats

Parquet

Write compressed Parquet files to S3 with optional Glue catalog registration.

Apache Iceberg

Write catalog-managed Iceberg tables with snapshot history and multi-engine access.

How It Works

Create IAM policies in your AWS account

Create an S3 permissions policy for the source or destination prefix. Add Glue permissions when using AWS Glue for source discovery, Parquet registration, or Iceberg catalog access.

Create an IAM role with trust policy

Create a role that trusts the Supaflow AWS account (805595753828) with an external ID you choose. Attach the S3 and Glue policies to the role.

Configure the source or destination in Supaflow

Enter your S3 bucket name, region, IAM role ARN, and external ID. Choose Parquet or Iceberg. For sources, configure Glue or prefix discovery; for destinations, choose the write format and catalog options.

Test and save

Click Test & Save to verify IAM role assumption, S3 write access, and catalog connectivity. Supaflow validates all permissions before saving.

Use Cases

Build an open data lake on your own S3

Replicate data from Salesforce, HubSpot, PostgreSQL, and other sources into S3 as Parquet or Iceberg. Your data stays in your bucket under your control -- no vendor lock-in on storage.

Ingest existing lake tables

Read curated Parquet or Iceberg datasets from S3 into a warehouse destination while preserving your existing bucket and catalog ownership model.

Query with any engine

Once data lands in S3 with Glue or Open Catalog registration, query it from Athena, Snowflake, Spark, DuckDB, Databricks, Dremio, Trino, Redshift Spectrum, or BigQuery. One write, many readers.

Iceberg time travel and schema evolution

Use Iceberg table format for ACID transactions, snapshot-based time travel queries, and automatic schema evolution as your source systems change.

Frequently Asked Questions

What query engines can I use to read data from my S3 data lake?

Amazon Athena, Apache Spark, Snowflake, DuckDB, Databricks, Dremio, Trino, Starburst Galaxy, Redshift Spectrum, Azure Synapse Analytics, BigQuery, and Bauplan. Any engine that reads Parquet files or supports the Iceberg table format will work.

When should I use Iceberg vs Parquet?

Use Parquet for simple data lake files that you query with Athena or Spark. Use Iceberg when you need catalog-managed tables, ACID transactions, time travel, concurrent reads during writes, or multi-engine access. Iceberg requires a catalog such as Glue or Snowflake Open Catalog.

Does my data leave my AWS account?

No. Supaflow assumes an IAM role in your AWS account to write data directly to your S3 bucket. The data never passes through Supaflow storage. You control access, encryption, and lifecycle policies.

What is the difference between S3 Data Lake and the legacy S3 connector?

S3 Data Lake adds source support, Apache Iceberg table format support, Snowflake Open Catalog integration, and improved Glue catalog management. New pipelines should use S3 Data Lake. Existing legacy S3 pipelines continue to work.

Can Supaflow read existing Parquet files from S3?

Yes. Use S3 Data Lake as a source with Table Format set to Parquet. Supaflow can discover tables from AWS Glue or from a configured S3 prefix layout.