Amazon S3 Data Lake Source

Connect Amazon S3 Data Lake as a source to read Parquet files and Apache Iceberg tables from S3 into your destination. The source can discover tables from AWS Glue, an Iceberg catalog, or a configured S3 prefix.

To load pipeline data into S3 instead, see Amazon S3 Data Lake Destination.

Table Format

The S3 Data Lake source supports two table formats:

Format	How Tables Are Found	Incremental Sync
Parquet	AWS Glue table locations or S3 prefix layout	Automatic, based on each file's last-modified time
Iceberg	AWS Glue or Snowflake Open Catalog	Selected date/timestamp cursor field

Use Parquet when your source data is a set of Parquet files in S3. Use Iceberg when the source is an Apache Iceberg table managed by AWS Glue or Snowflake Open Catalog.

Prerequisites

Before you begin, ensure you have:

• An S3 bucket containing Parquet files or Iceberg table data
• An IAM role Supaflow can assume with read access to the bucket and prefix
• An External ID for secure role assumption
• For AWS Glue discovery: Glue permissions to list databases, list tables, and read table metadata
• For Snowflake Open Catalog: catalog endpoint, catalog name, client ID, and client secret

How Data is Organized

How Supaflow finds source objects depends on the table format and discovery method you choose. Catalog-based sources use catalog metadata. Prefix-based Parquet sources use the folder layout beneath S3 Prefix Path.

Parquet with AWS Glue Catalog

When Table Format is PARQUET and Use AWS Glue Catalog is enabled, Supaflow discovers Glue databases and tables that match the configured Glue Database Prefix. Each Glue table becomes a source object, and Supaflow reads the Parquet files from the S3 location stored on that Glue table.

Use this mode when Glue is already your table catalog for Athena, Redshift Spectrum, Spark, or another query engine.

Parquet with S3 Prefix Discovery

When Table Format is PARQUET and Use AWS Glue Catalog is disabled, Supaflow discovers tables by scanning objects under S3 Prefix Path.

The Data Lake Layout setting controls how S3 object keys become source objects:

Layout	Source Object Boundary	Use When
Auto-detect	Detects a directory-partitioned or Hive-partitioned {schema}/{table}/... layout	The prefix contains a conventional data lake export
Directory partitions	The first folder is the source schema and the second folder is the source object	Partition folders use positional values such as 2026/08/07
Hive partitions	The first folder is the source schema and the second folder is the source object	Partition folders use key=value names such as year=2026/month=08
Table mapping	A folder at the configured depth or an individual file becomes a source object	The prefix is a Parquet drop zone without a {schema}/{table} hierarchy

All examples below are relative to S3 Prefix Path.

Auto-detect

Auto-detect chooses between the directory-partitioned and Hive-partitioned layouts shown below. It does not select Table Mapping automatically. Files you want Supaflow to discover must follow a {schema}/{table}/... hierarchy beneath the prefix.

If the prefix contains both positional and key=value partition folders, choose the intended layout explicitly or narrow S3 Prefix Path to one consistent layout.

Directory-partitioned Layout

Use Directory partitions when partition folders are positional values. The first folder beneath the prefix becomes the source schema, the second becomes the source object, and any remaining folders are treated as partitions within that object.

S3 Prefix Path = lake
Data Lake Layout = Directory partitions

s3://my-bucket/lake/
+-- sales/                                  --> Source schema: "sales"
|   +-- orders/                             --> Source object: "orders"
|   |   +-- 2026/08/07/part-000.parquet        (rows in "orders")
|   |   `-- 2026/08/08/part-000.parquet        (rows in "orders")
|   `-- customers/                          --> Source object: "customers"
|       `-- part-000.parquet                    (rows in "customers")
`-- finance/                                --> Source schema: "finance"
    `-- invoices/                           --> Source object: "invoices"
        `-- batch-42/part-000.parquet           (rows in "invoices")

Parquet files directly beneath lake/ or lake/sales/ do not have both a schema and object folder, so they are not discovered as source objects in this layout.

Hive-partitioned Layout

Use Hive partitions when partition folders use key=value names. The schema and source-object folders remain the first two levels beneath the prefix; the partition keys do not become part of the object name.

S3 Prefix Path = lake
Data Lake Layout = Hive partitions

s3://my-bucket/lake/
`-- sales/                                  --> Source schema: "sales"
    +-- orders/                             --> Source object: "orders"
    |   +-- year=2026/month=08/day=07/
    |   |   `-- part-000.parquet                (rows in "orders")
    |   `-- year=2026/month=08/day=08/
    |       `-- part-000.parquet                (rows in "orders")
    `-- customers/                          --> Source object: "customers"
        `-- region=us/part-000.parquet          (rows in "customers")

Table Mapping: One Table per Folder

Use Table mapping with Table Grouping: One table per folder when folders at a consistent depth define your source objects. Table Folder Depth controls how many folder levels beneath the prefix form the object name. Files below each object folder are included recursively.

With Table Folder Depth = 1, each immediate child folder becomes one source object. Files directly beneath the prefix are above the configured depth and are not included.

S3 Prefix Path = drop-zone
Data Lake Layout = Table mapping
Table Grouping = One table per folder
Table Folder Depth = 1

s3://my-bucket/drop-zone/
+-- summary.parquet                         --> not included: above folder depth 1
+-- orders/                                 --> Source object: "orders"
|   +-- orders_july.parquet                     (rows in "orders")
|   `-- archive/
|       `-- orders_june.parquet                 (rows in "orders")
`-- customers/                              --> Source object: "customers"
    `-- customers.parquet                       (rows in "customers")

With Table Folder Depth = 2, the first two folder names are joined to form the source-object name:

S3 Prefix Path = drop-zone
Table Folder Depth = 2

s3://my-bucket/drop-zone/
`-- us/
    +-- orders/                             --> Source object: "us_orders"
    |   `-- orders.parquet
    `-- refunds/                            --> Source object: "us_refunds"
        `-- refunds.parquet

Table Mapping: One Table per File

Use Table mapping with Table Grouping: One table per file when every Parquet file should be a separate source object. The object name is derived from the file's path relative to S3 Prefix Path, without the .parquet extension, and normalized to lowercase with underscores.

S3 Prefix Path = drop-zone
Data Lake Layout = Table mapping
Table Grouping = One table per file
Include Subfolders = On

s3://my-bucket/drop-zone/
+-- summary.parquet                         --> Source object: "summary"
`-- orders/
    +-- orders_july.parquet                 --> Source object: "orders_orders_july"
    `-- orders_june.parquet                 --> Source object: "orders_orders_june"

When Include Subfolders is disabled, only Parquet files directly beneath S3 Prefix Path become source objects. Nested files such as orders/orders_july.parquet are not included.

Folder- and file-derived object names are normalized to lowercase with underscores. Spaces and punctuation become underscores, and repeated underscores are collapsed.

Normalized Object Name Collisions

With One table per folder, different folder paths that normalize to the same name are combined into one source object. For example, orders-prod/ and orders_prod/ both become orders_prod, so files from both folders are read together. Rename one of the folders if the datasets should remain separate.

With One table per file, each file remains a separate source object. Names that collide after normalization receive numeric suffixes such as _2 and _3.

File Filtering and Supported Formats

In Table Mapping mode, File Pattern can limit which object keys are considered. The pattern filters files only; it does not change how object names are derived.

Only Parquet data files are supported in Parquet source mode. Do not use a plain-Parquet source to read Delta Lake, Apache Hudi, or catalog-managed Iceberg table folders. Use Table Format: ICEBERG for Iceberg tables.

Iceberg Catalog Tables

When Table Format is ICEBERG, Supaflow reads tables through the configured Iceberg catalog:

• AWS Glue: discovers Iceberg tables from Glue databases that match Glue Database Prefix
• Snowflake Open Catalog: discovers Iceberg tables from the configured catalog

Iceberg tables are read through the catalog, not by scanning raw metadata files under an S3 prefix.

Incremental Sync

Parquet Sources

Parquet sources sync automatically based on each S3 object's last-modified time. There is no cursor column to select.

• The initial sync reads matching files in the selected table
• Later syncs read files modified since the previous successful cursor
• Each file is read as a whole

Treat Parquet source files as immutable file drops. If an old object key is overwritten after it has already been synced, Supaflow may read that file again as a new version of the data. If you intentionally replace historical files, run a full resync or use a fresh prefix.

Iceberg Sources

Iceberg incremental sync uses a date or timestamp cursor column selected in the pipeline. During schema discovery, Supaflow recommends eligible cursor fields when they are present.

If an Iceberg table does not have a reliable date or timestamp cursor, run it in historical mode. Non-initial Iceberg incremental runs require the selected cursor state to be present.

AWS Setup

Supaflow assumes an IAM role in your AWS account. For source reads, the role needs:

• s3:ListBucket on the bucket and source prefix
• s3:GetObject on the Parquet files or Iceberg metadata/data files
• Glue read permissions when using AWS Glue discovery or AWS Glue as the Iceberg catalog

If you also use the same connection as an S3 Data Lake destination, follow the broader IAM setup in Amazon S3 Data Lake Destination.

Configuration

In Supaflow, create a new S3 Data Lake source with these settings:

Connection

S3 Bucket Name*

Name of the S3 bucket.
Example: my-data-lake

Bucket Region*

AWS region where the bucket is located.
Example: us-east-1

S3 Prefix Path

Root path inside the bucket. Prefix-based Parquet sources scan and read objects under this path. Glue-catalog sources read table locations from Glue.
Example: analytics/exports

IAM Role ARN*

IAM role Supaflow assumes for S3 access and catalog access when enabled.
Example: arn:aws:iam::123456789012:role/SupaflowS3Access

External ID*

External ID for role assumption. Must match the value in your trust policy.
Stored encrypted

S3 Endpoint

Custom S3-compatible endpoint URL for MinIO, DigitalOcean Spaces, or another S3-compatible service. Leave empty for standard AWS S3.

Storage Format

Table Format*

Format of the data at this S3 location.
Options:

• PARQUET - Read Parquet files from S3.
• ICEBERG - Read Apache Iceberg tables through a catalog.

Default: PARQUET

Parquet Settings

Use AWS Glue Catalog

Use AWS Glue as the Parquet table catalog. Sources discover Parquet tables from Glue metadata and read the S3 table locations stored there.
Default: Disabled

Glue Database Prefix

Prefix for Glue databases. Sources show databases named exactly like the prefix or beginning with prefix_. Leave empty to show all Glue databases.
Default: supaflow

Iceberg Settings

These settings appear when Table Format is ICEBERG.

Iceberg Catalog*

Catalog Supaflow uses to find existing Iceberg tables.
Options:

• GLUE - Use AWS Glue as the Iceberg catalog.
• SNOWFLAKE_OPEN_CATALOG - Use Snowflake Open Catalog.

Default: GLUE

Catalog Endpoint URL*

Snowflake Open Catalog REST endpoint. Required only when Iceberg Catalog is SNOWFLAKE_OPEN_CATALOG.

Catalog Name*

Snowflake Open Catalog catalog that contains the Iceberg tables.

Client ID*

OAuth client ID for Snowflake Open Catalog.
Stored securely

Client Secret*

OAuth client secret for Snowflake Open Catalog.
Stored encrypted

Principal Role

Principal role to use in Snowflake Open Catalog. Leave as ALL unless your catalog administrator provided a narrower role.
Default: ALL

Source Read Options

These settings apply to prefix-based Parquet sources when Use AWS Glue Catalog is disabled.

Data Lake Layout

How your files are arranged under the bucket prefix. Start with AUTO_DETECT for conventional data lake exports.
Options:

• AUTO_DETECT - Choose between a directory-partitioned and Hive-partitioned {schema}/{table}/... layout.
• DIRECTORY_PARTITIONED - Use positional partition folders beneath each {schema}/{table} path.
• HIVE_PARTITIONED - Use key=value partition folders beneath each {schema}/{table} path.
• TABLE_MAPPING - Use folder- or file-based grouping for a Parquet drop zone.

Default: AUTO_DETECT

Table Grouping

How files become tables in table-mapping mode.
Options:

• TABLE_PER_FOLDER - Folders at Table Folder Depth define source objects. Files beneath each object folder are included recursively.
• TABLE_PER_FILE - Each file becomes its own table.

Default: TABLE_PER_FOLDER

Table Folder Depth

How many folder levels beneath S3 Prefix Path form the source-object name in TABLE_PER_FOLDER mode. For example, depth 1 maps orders/ to orders; depth 2 maps us/orders/ to us_orders.
Default: 1

Include Subfolders

Include Parquet files nested beneath S3 Prefix Path in TABLE_PER_FILE mode. When disabled, only files directly beneath the prefix become source objects.
Default: true

File Pattern

Optional regular expression matched against the full S3 object key to choose which Parquet files to sync. Applies to TABLE_MAPPING mode.

Advanced Settings

Schema Refresh Interval

Interval in minutes for schema metadata refresh.
0 = refresh before every pipeline execution
-1 = disable automatic schema refresh
Default: 60

Test & Save

After configuring all required properties, click Test & Save to verify role assumption, S3 access, and catalog access when enabled.

Troubleshooting

No objects found during schema discovery

Problem:

• Schema discovery returns no tables
• Expected Parquet or Iceberg tables do not appear

Solutions:

1. Confirm S3 Bucket Name, Bucket Region, and S3 Prefix Path are correct.
2. Confirm the IAM role can list the bucket and read objects under the prefix.
3. For Glue sources, confirm Glue Database Prefix matches the database names you expect Supaflow to scan.
4. For prefix-based Parquet sources, confirm Data Lake Layout matches the object key structure.
5. For Iceberg sources, confirm Table Format is ICEBERG and the catalog settings point to the catalog that owns the tables.

Unsupported files inside a discovered table

Problem:

• Schema discovery fails because non-Parquet files are present inside a discovered Parquet table
• The prefix contains Delta Lake, Hudi, or raw Iceberg metadata

Solutions:

1. Point the source at a prefix that contains only the Parquet files for the selected tables.
2. Move logs, manifests, CSV files, and other non-Parquet files outside the discovered table folder.
3. For Iceberg tables, set Table Format to ICEBERG and read through the catalog instead of scanning raw S3 files.

Layout is ambiguous or tables are grouped incorrectly

Problem:

• Table names do not match your expected S3 folder structure
• Auto-detect cannot classify the layout you intended

Solutions:

1. Set Data Lake Layout explicitly to DIRECTORY_PARTITIONED, HIVE_PARTITIONED, or TABLE_MAPPING.
2. For table mapping, adjust Table Grouping and Table Folder Depth.
3. Use File Pattern to narrow the files considered for a table-mapping source.

Iceberg incremental read requires a cursor field

Problem:

• An Iceberg incremental sync fails because no cursor field is selected or the cursor state is missing

Solutions:

1. Refresh schema and choose a reliable date or timestamp cursor field for the Iceberg table.
2. Run the object in historical mode if no reliable cursor exists.
3. If pipeline state was manually changed or lost, run a full resync.

Incremental Parquet sync reads more files than expected

Problem:

• A later Parquet sync re-reads old data

Solutions:

1. Treat source files as immutable after they are synced.
2. Avoid overwriting old object keys in place.
3. If you intentionally replaced historical files, run a full resync or use a fresh prefix.

---

Related Pages

• Amazon S3 Data Lake connector overview
• Google Drive source
• SFTP source
• Create an ingestion pipeline
• Snowflake destination
• Amazon S3 Data Lake destination

---

Support

Need help? Contact us at support@supa-flow.io