Prepare Google Cloud for a BigQuery Destination

Use this guide when you need to create the Google Cloud resources required by the Supaflow Google BigQuery destination. It covers the Google Cloud console workflow, an equivalent CLI handoff, and the final Test & Save validation in Supaflow.

What You Will Create

Resource	Example used in this guide	Requirement
Google Cloud project	acme-bigquery-production	Billing enabled
BigQuery dataset	supaflow_data	Same location as the staging bucket
Cloud Storage bucket	acme-bigquery-production-staging	Globally unique name; customer owned
Staging prefix	supaflow/	Reserved for Supaflow temporary objects
Supaflow identity	Generated in the destination form	Managed service account recommended

This guide uses the US multi-region. Choose the location required by your organization, then use that exact location for the dataset, bucket, and Supaflow Location field. US and us-central1 are different locations.

Before You Begin

You need permission to:

• Create a project in the intended organization or folder
• Link the project to an active Cloud Billing account
• Enable Google Cloud APIs
• Create BigQuery datasets and Cloud Storage buckets
• Update project, dataset, and bucket IAM policies

If your organization separates these duties, give the CLI handoff to the Google Cloud administrator and provide the Supaflow-managed service-account email after Supaflow generates it.

Console Setup

1. Create and bill the project

1. In the Google Cloud console, open the project selector and click New Project.
2. Enter a descriptive Project name.
3. Click Edit beside Project ID and enter a permanent, globally unique project ID.
4. Select the required Billing account, Organization, and Parent resource.
5. Click Create and wait for the project to become available.
6. Select the new project in the project selector before continuing.

Project IDs cannot be changed after project creation. Bucket names are also globally unique, so a common pattern is <project-id>-staging.

2. Enable the BigQuery API

1. Go to APIs & Services > Library.
2. Search for BigQuery API.
3. Open BigQuery API and click Enable.
4. Confirm the page shows API enabled.

3. Enable the Cloud Storage API

1. Return to APIs & Services > Library.
2. Search for Cloud Storage API.
3. Open Cloud Storage API and click Enable.
4. Confirm the page shows API enabled.

4. Create the destination dataset

1. Go to BigQuery > Studio.
2. In Explorer, select the project and click Create dataset.
3. Enter the Dataset ID, such as supaflow_data.
4. For Location type, choose the location required by your organization. This guide uses US (multiple regions in United States).
5. Keep Google-managed encryption unless your organization requires a customer-managed key.
6. Click Create dataset.

5. Create the staging bucket

1. Go to Cloud Storage > Buckets and click Create.
2. Enter a globally unique bucket name. Do not include gs://.
3. Select the same location as the BigQuery dataset.
4. Choose Standard as the default storage class.
5. Select Uniform access control and keep Public access prevention enabled.
6. Review your organization's protection requirements. The validated setup shown here disables soft delete and object versioning because the bucket contains short-lived staging data. If your policy requires either control, account for its retention and cost implications.
7. Keep Google-managed encryption unless your organization requires CMEK, then click Create.

6. Add a staging cleanup rule

Successful loads remove their staged objects. A lifecycle rule is a backstop for interrupted or failed jobs.

1. Open the bucket and select the Lifecycle tab.
2. Click Add a rule.
3. Choose Delete object and click Continue.
4. Select Object name matches prefix and enter supaflow/.
5. Select Age and enter 3 days.
6. Click Create.

The trailing slash matters: it scopes deletion to objects below the Supaflow staging prefix instead of similarly named objects elsewhere in the bucket.

7. Start the destination and copy its identity

1. In Supaflow, go to Destinations and click Create Destination.
2. Select Google BigQuery.
3. Enter a destination name and leave Authentication Method set to Managed service account.
4. Copy the generated Service account email. It is an identity, not a private key.

Do not create or download a Google Cloud service-account key for this path. Supaflow manages short-lived credentials for the generated identity.

8. Grant project-level job access

1. In Google Cloud, go to IAM & Admin > IAM in the billing or job project.
2. Click Grant access.
3. Paste the generated Supaflow service-account email into New principals.
4. Select BigQuery Job User (roles/bigquery.jobUser).
5. Click Save.

If Billing Project ID will be different from Project ID in Supaflow, grant this role on the billing project instead.

9. Grant dataset-level write access

1. Go to BigQuery > Studio and open the destination dataset.
2. Click Share > Manage permissions.
3. Click Add principal and paste the generated service-account email.
4. Select BigQuery Data Editor (roles/bigquery.dataEditor).
5. Click Save, then close the permissions panel.

Grant this role on the dataset, not the entire project, for the normal least-privilege setup.

10. Grant staging-bucket access

1. Go to Cloud Storage > Buckets and open the staging bucket.
2. Select Permissions and click Grant access.
3. Paste the generated service-account email.
4. Add both roles:
   • Storage Object User (roles/storage.objectUser)
   • Storage Bucket Viewer (roles/storage.bucketViewer)
5. Click Save.

Storage Object User lets Supaflow manage staged objects. Storage Bucket Viewer supplies storage.buckets.get, which Supaflow uses to validate the bucket's existence, location, and lifecycle metadata.

11. Complete and validate the Supaflow destination

Return to the destination form and enter:

Supaflow field	Value
Project ID	The Google Cloud project containing the destination dataset
Billing Project ID	Leave blank when it is the same as Project ID
Default Dataset	The dataset ID, such as supaflow_data
Location	The exact dataset and bucket location, such as US
Enable Source Schema Discovery	Disabled for a destination-only connection
GCS Staging Bucket	Bucket name only, without gs://
GCS Staging Prefix	supaflow

Click Test & Save. A successful connection test verifies authentication, job creation, destination-dataset metadata access, and the staging bucket's existence and location. It does not upload a staging object or write a destination table. Supaflow then returns to the destination list; the new connection can briefly show Syncing Schema while its initial post-save work completes.

CLI Handoff

The following commands provide the same setup for macOS, Linux, or Google Cloud Shell. Run them as an administrator with permission to create projects, link billing, create resources, and update IAM.

1. Set customer-specific values

export PROJECT_ID="acme-bigquery-production"
export PROJECT_NAME="Acme BigQuery Production"
export BILLING_ACCOUNT_ID="000000-000000-000000"
export ORGANIZATION_ID="123456789012"
export LOCATION="US"
export DATASET_ID="supaflow_data"
export BUCKET_NAME="${PROJECT_ID}-staging"
export STAGING_PREFIX="supaflow"

Replace every example value. Billing account and organization IDs are identifiers, not secrets, but they should still come from the customer's approved Google Cloud administrator.

2. Create the project and resources

gcloud projects create "$PROJECT_ID" \
  --name="$PROJECT_NAME" \
  --organization="$ORGANIZATION_ID"

gcloud billing projects link "$PROJECT_ID" \
  --billing-account="$BILLING_ACCOUNT_ID"

gcloud services enable \
  bigquery.googleapis.com \
  storage-api.googleapis.com \
  --project="$PROJECT_ID"

bq --project_id="$PROJECT_ID" mk \
  --dataset \
  --location="$LOCATION" \
  "$PROJECT_ID:$DATASET_ID"

gcloud storage buckets create "gs://$BUCKET_NAME" \
  --project="$PROJECT_ID" \
  --location="$LOCATION" \
  --default-storage-class=STANDARD \
  --uniform-bucket-level-access \
  --public-access-prevention \
  --soft-delete-duration=0s

If the project belongs under a folder, replace --organization="$ORGANIZATION_ID" with --folder="FOLDER_ID". If the account creates projects without an organization or folder, omit the parent flag.

3. Apply the lifecycle rule

Create supaflow-lifecycle.json:

{
  "rule": [
    {
      "action": {
        "type": "Delete"
      },
      "condition": {
        "age": 3,
        "matchesPrefix": [
          "supaflow/"
        ]
      }
    }
  ]
}

Apply it to the bucket:

gcloud storage buckets update "gs://$BUCKET_NAME" \
  --lifecycle-file=supaflow-lifecycle.json

If you change STAGING_PREFIX, update matchesPrefix to the same value with a trailing slash.

4. Generate the Supaflow identity

In Supaflow, start a Google BigQuery destination with Managed service account authentication and copy its generated service-account email. Then set it in the same shell:

export SUPAFLOW_SERVICE_ACCOUNT="service-account-from-supaflow@example.iam.gserviceaccount.com"

5. Apply least-privilege IAM

gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --member="serviceAccount:$SUPAFLOW_SERVICE_ACCOUNT" \
  --role="roles/bigquery.jobUser"

bq query \
  --project_id="$PROJECT_ID" \
  --location="$LOCATION" \
  --use_legacy_sql=false \
  "GRANT \`roles/bigquery.dataEditor\` ON SCHEMA \`$PROJECT_ID.$DATASET_ID\` TO \"serviceAccount:$SUPAFLOW_SERVICE_ACCOUNT\""

gcloud storage buckets add-iam-policy-binding "gs://$BUCKET_NAME" \
  --member="serviceAccount:$SUPAFLOW_SERVICE_ACCOUNT" \
  --role="roles/storage.objectUser"

gcloud storage buckets add-iam-policy-binding "gs://$BUCKET_NAME" \
  --member="serviceAccount:$SUPAFLOW_SERVICE_ACCOUNT" \
  --role="roles/storage.bucketViewer"

6. Verify the handoff

gcloud services list \
  --enabled \
  --project="$PROJECT_ID" \
  --filter="name:(bigquery.googleapis.com OR storage-api.googleapis.com)"

bq show \
  --project_id="$PROJECT_ID" \
  --format=prettyjson \
  "$PROJECT_ID:$DATASET_ID"

gcloud storage buckets describe "gs://$BUCKET_NAME"

gcloud projects get-iam-policy "$PROJECT_ID" \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:$SUPAFLOW_SERVICE_ACCOUNT"

gcloud storage buckets get-iam-policy "gs://$BUCKET_NAME"

Complete the Supaflow fields from step 11, then click Test & Save. This is the final connection validation because it runs as the generated connector identity instead of the administrator who provisioned the resources. To validate the complete write path, run a small pipeline and confirm that Supaflow creates and loads a destination table.

Cleanup and Ownership

The Google Cloud project, dataset, bucket, billing relationship, IAM policies, and lifecycle rules remain customer owned. Removing the Supaflow destination does not delete those resources. Follow the customer's retention and change-management procedures before deleting the project or removing the service-account grants.

For connector behavior, configuration fields, and troubleshooting, return to Google BigQuery Destination.