Google BigQuery Source

Connect Google BigQuery as a source to replicate warehouse data into any supported Supaflow destination.

For capabilities and use cases, see the Google BigQuery connector overview. To load pipeline data into BigQuery, see Google BigQuery Destination.

Prerequisites

Before you begin, ensure you have:

• A billing-enabled Google Cloud project with the BigQuery API enabled
• A BigQuery data project and location for the datasets you want to read
• An optional billing project if query jobs should be billed separately from the data project
• A supported Google Cloud identity with permission to create jobs and read the selected datasets

GCS Staging Bucket is required for both source and destination connections. Provide an existing, customer-owned Cloud Storage bucket, even for a source-only connection. The bucket location must match the connection’s Location. Supaflow checks bucket metadata during initialization; grant the same bucket roles for source and destination use.

Authentication

Supaflow-managed service account (recommended)

1. Create the Google BigQuery connection in Supaflow and leave Authentication Method set to managed service account.
2. Click the Grant this email access to BigQuery field to copy the generated email.
3. Grant that identity the project and bucket roles in Source IAM.
4. Return to Supaflow and click Test & Save.

Supaflow manages short-lived credentials for this identity. Supaflow-hosted customers do not run gcloud auth or upload a key for the managed-service-account path.

Customer service-account key

Choose service account only when your organization deliberately uses a user-managed service-account key. Upload the complete JSON credential through Service Account Key. Supaflow stores the credential encrypted.

Google recommends avoiding user-managed keys where a managed identity is available. Review Google's service-account key guidance before choosing this method.

Application Default Credentials

Choose ADC only for a private or self-hosted agent whose runtime already provides Application Default Credentials. This is not part of the Supaflow-hosted setup path.

gcloud auth login authenticates the Google Cloud CLI; it does not configure ADC for an agent runtime. Use the identity mechanism approved for the host where the agent runs, preferably an attached service account rather than a local key.

Source IAM

Use the same standard grants for source and destination connections so the identity is ready for both reading and writing. Grant these roles to the email shown in the connection form:

Scope	Google Cloud role
BigQuery data project (Project ID)	BigQuery Data Editor (roles/bigquery.dataEditor)
Billing/job project (Billing Project ID, or Project ID when blank)	BigQuery Job User (roles/bigquery.jobUser) and BigQuery Read Session User (roles/bigquery.readSessionUser)
Bucket entered in GCS Staging Bucket	Storage Object User (roles/storage.objectUser) and Storage Bucket Viewer (roles/storage.bucketViewer)

Grant the BigQuery roles at project level and the storage roles on the bucket. Storage Bucket Viewer allows the metadata check performed during initialization. Storage Object User prepares the bucket for destination staging. Follow the required-permissions guide for the complete setup.

Cross-project sources

For every additional data project:

1. Add the project ID to Additional Source Projects.
2. Grant the connector identity BigQuery Data Viewer on each dataset to expose.
3. Grant job permissions in the configured billing/job project. If a different project should own query jobs, set Billing Project ID explicitly.

Projects that are not listed in Project ID or Additional Source Projects are not included in discovery.

Set Up Google BigQuery as a Source

1. In Supaflow, navigate to Sources and click Create Source.
2. Select Google BigQuery, enter a source name, and leave Authentication Method set to managed service account for the recommended setup.
3. Click Grant this email access to BigQuery to copy the generated email.
4. In Google Cloud, grant that identity the project and bucket roles in Source IAM. Use the same grants for source and destination connections.
5. Return to Supaflow and enter Project ID and Location. Set Billing Project ID only when a different project should own query jobs.
6. Under Dataset Discovery, leave Enable Source Schema Discovery enabled. It is enabled by default for new source connections; existing connections keep their saved setting.
7. Add any Additional Source Projects and optional dataset inclusion or exclusion patterns.
8. Under Destination Staging, enter an existing bucket name in GCS Staging Bucket, without gs://. The bucket must match the configured Location and have both storage roles listed above, including for source-only connections.
9. Click Test & Save.

What Gets Synced

Supaflow discovers accessible BigQuery objects in a project -> dataset -> table hierarchy. Supported objects include:

• Tables, including accessible snapshots and clones
• External tables supported through BigQuery
• Views, when Include Views is enabled
• Materialized views, when Include Views is enabled

Object visibility follows Google Cloud IAM. You choose the projects, datasets, tables, and columns to include in each pipeline.

Sync Modes

Full refresh reads all selected rows on each sync. Use it for small tables or objects without a reliable cursor.

Incremental sync reads rows in the selected cursor window. Use a date or timestamp column that advances whenever a row is inserted or updated.

The Google BigQuery source does not read change logs for CDC and does not automatically detect hard deletes in source tables. Use full refresh when the destination must reflect source-side removals and no suitable incremental strategy exists.

Dataset Discovery

important

Enable Source Schema Discovery is enabled by default when you add BigQuery as a source. Destinations, and connections created without a role, start with it off. Existing connections keep their setting. If discovery is off, a refresh does not discover source projects, datasets, tables, or views.

Dataset Inclusion Patterns and Dataset Exclusion Patterns accept one Java-compatible regular expression per line:

• Each expression must match the complete dataset ID. Use .* when you need a partial match.
• Exclusions take precedence over inclusions.
• Leaving inclusion patterns blank includes every accessible dataset in the configured projects, except excluded datasets.
• Default Dataset provides connection and destination context; it does not restrict source discovery.

After changing project access or patterns, refresh the source schema in Supaflow.

Creating from the CLI or MCP

Pass --role SOURCE to supaflow datasources init or supaflow datasources create, or use role: "SOURCE" in hosted MCP, to apply source defaults. Values already in the connection file take precedence: if it contains enableSourceSchemaDiscovery=false, change it to true or remove the line.

Configuration

Authentication

Authentication Method*

Authentication used by the connection
Options: managed service account, service account, ADC; default: managed service account

Service Account Key*

Complete JSON service-account credential, required only when Authentication Method is service account
Uploaded as a file and stored encrypted

Grant this email access to BigQuery

Google Cloud identity generated for managed-service-account authentication
This field is read-only in Supaflow

---

Connection

Project ID*

Default BigQuery data project
Example: my-gcp-project

Billing Project ID

Project that owns query and load jobs
Defaults to Project ID

Default Dataset

Optional source connection context for unqualified queries; does not restrict discovery
Required when the same connection is used as a destination

Location*

BigQuery dataset and job location
Examples: US, EU, us-central1

Additional Source Projects

Comma-separated project IDs allowed during source discovery

---

Dataset Discovery

Enable Source Schema Discovery

Discover source projects, datasets, tables, and views during schema refreshes. Keep this on for every connection used as a source.
Default: enabled for new source connections; disabled for destinations and connections created without a role. Existing connections keep their setting.

Dataset Inclusion Patterns

Optional dataset-ID regular expressions, one per line; each pattern must match the complete ID

Dataset Exclusion Patterns

Optional dataset-ID regular expressions, one per line; exclusions take precedence

---

Advanced Settings

Include Views

Include accessible views and materialized views during source schema discovery. Enable Enable Source Schema Discovery as well, then refresh the source schema.
Default: Off

Enable High Throughput Reads

Allow the JDBC driver to use the BigQuery Storage Read API for large query results
Default: enabled

Maximum Bytes Billed

Optional per-query billed-byte safeguard
Default: 0, meaning no connector-supplied limit

Job Timeout

Maximum BigQuery read or write job runtime in seconds
Default: 0, meaning no connector timeout; maximum: 86,400 seconds

JDBC Connection Properties

Optional allowlisted, non-secret BigQuery JDBC properties as semicolon-separated key=value pairs

Test & Save

Before clicking Test & Save, confirm Enable Source Schema Discovery is turned on. Test & Save verifies authentication, project access, the configured location, and source metadata access. After saving, refresh the source schema and confirm the expected datasets and objects appear.

Costs and Quotas

BigQuery query usage is billed to the project that owns the job: Billing Project ID, or Project ID when no billing project is set. Review BigQuery pricing, quota controls, and cost-control guidance before large initial syncs.

Set Maximum Bytes Billed when you want a per-query safeguard. BigQuery rejects a query before execution when its estimated billed bytes exceed the limit. Use incremental sync, narrower object selection, and off-peak scheduling to control large workloads.

Data Type Support

Supaflow converts BigQuery values into portable pipeline values. This preserves values across destinations but does not imply identical physical types in every target.

BigQuery source type	Supaflow source representation
BOOL	Boolean
INT64	Integer
FLOAT64	Floating-point number
NUMERIC	Decimal
BIGNUMERIC	Decimal when it fits the portable decimal envelope; otherwise a lossless string
STRING	String
BYTES	Binary value
DATE	Date
TIME	String preserving microsecond precision
DATETIME	Local date and time
TIMESTAMP	UTC instant
JSON	JSON value
STRUCT	JSON object preserving field names and nulls
ARRAY / repeated RECORD	JSON array; nested records preserve field names
GEOGRAPHY	String representation
INTERVAL	String representation
RANGE	String representation

When these values are loaded into another destination, that destination's type mapper chooses the physical type. See Google BigQuery Destination for BigQuery destination mappings.

Private and Self-Hosted Agents

Supaflow-hosted agents and the official Docker agent include the JVM access required by the Arrow-based BigQuery read path. If you launch the agent JAR through a custom command, include:

--add-opens=java.base/java.nio=ALL-UNNAMED

This is needed only for custom agent launchers. It is not a hosted Supaflow setup step.

Troubleshooting

Missing bigquery.readsessions.create

Problem:

• Source reads fail with a permission error that mentions bigquery.readsessions.create

Solutions:

1. Grant BigQuery Read Session User to the connector identity on the configured billing/job project.
2. Confirm the role was granted on Billing Project ID, or on Project ID when the billing project is blank.
3. If you intentionally cannot grant the role, disable Enable High Throughput Reads and retest with the standard read path.

High-throughput reads cannot access the Storage Read API

Problem:

• High-throughput reads fail because the BigQuery Storage Read API is unavailable

Solutions:

1. Confirm the BigQuery API is enabled; Google automatically enables the BigQuery Storage Read API with it.
2. Confirm the connector identity has BigQuery Read Session User on the billing/job project.
3. Retry after Google Cloud propagates the API and IAM changes.

An expected dataset is missing

Problem:

• The connection succeeds, but an expected dataset does not appear during source discovery

Solutions:

1. Confirm the dataset's project is in Project ID or Additional Source Projects.
2. Confirm the dataset matches the configured Location. Use a separate BigQuery source connection for datasets in another location.
3. Grant BigQuery Data Viewer on the dataset.
4. Check Dataset Inclusion Patterns and Dataset Exclusion Patterns. Patterns match the complete dataset ID and exclusions take precedence.
5. Refresh the source schema.

The billing project cannot create jobs

Problem:

• Test & Save or a source sync fails because the connector identity cannot create BigQuery jobs

Solutions:

1. Grant BigQuery Job User on Billing Project ID, or on Project ID when the billing project is blank.
2. Confirm billing is enabled and the BigQuery API is available in that project.
3. Retry after Google Cloud propagates the IAM change.

Authentication fails

Problem:

• Test & Save or a source sync fails with a credential, token, or ADC error

Solutions:

1. Managed service account: confirm the email shown under Grant this email access to BigQuery received the required grants.
2. Service account: upload a valid, active JSON key for the intended service account and confirm the key has not been disabled.
3. ADC: confirm the private agent runtime, not only your interactive CLI session, receives ADC.

A custom agent reports an Arrow or Java module-access error

Problem:

• A custom agent launcher reports an Arrow or Java module-access error during a high-throughput read

Solutions:

1. Add --add-opens=java.base/java.nio=ALL-UNNAMED to the custom Java launcher.
2. Use the official Supaflow Docker agent when you do not need a custom launcher; it already includes the required option.

A query exceeds Maximum Bytes Billed or Job Timeout

Problem:

• Job Details reports that a query exceeded Maximum Bytes Billed or Job Timeout

Solutions:

1. Review the failed object's expected query cost or runtime.
2. Reduce the selected data or use a suitable incremental cursor.
3. Increase the configured safeguard only after confirming the expected workload.

Related Pages

• Google BigQuery connector overview
• Create an ingestion pipeline
• Google BigQuery destination guide
• Amazon Redshift source
• PostgreSQL source

Support

Need help? Contact us at support@supa-flow.io