8 posts tagged with "Data Engineering"

A user can sign in successfully and still see or change another tenant's data. Authentication proves who the user is; it does not tell Postgres which organization rows that user may access.

The risky shortcut is to trust an organization ID sent by the browser. A caller can change that value. Another common mistake is to use auth.uid(), which represents a Supabase Auth user UUID rather than Clerk's string user ID. Membership lookups inside RLS policies can also become recursive and slow.

This tutorial shows how to make the verified Clerk session token the root of the authorization decision:

• Clerk authenticates the user and supplies the active organization context.
• Supabase verifies the Clerk token and makes its claims available to Postgres.
• Postgres derives the user and tenant from those claims.
• Row-Level Security applies indexed, non-recursive policies to every query.

By the end, you will have a reusable schema, JWT helper functions, a controlled tenant-bootstrap function, explicit read/write policies, and tests for personal accounts, organizations, role boundaries, and cross-tenant attacks. The integration uses Clerk and Supabase's native third-party authentication—without a Clerk JWT template, a shared Supabase JWT secret, or auth.uid().

The complete runnable implementation is in the supaflow-labs/clerk-supabase-demo repository. The snippets below are intentionally small enough to study; use the repository migration and tests when building the complete example.

Moving Google Drive files into Snowflake can mean several different things: loading a folder of recurring CSV exports, syncing every worksheet in a set of Google Sheets, replacing one named range, or running a one-time import. Those workloads should not share one blanket recommendation.

This guide compares Supaflow, Fivetran, Hevo, Airbyte, manual Snowflake loads, and a custom Google API pipeline. It then walks through a Supaflow setup using CSV files; the same pipeline workflow applies to TSV, Excel (.xlsx), and native Google Sheets sources.

Need MySQL data in Snowflake for analytics, reporting, or a migration? Supaflow handles the initial load and keeps new and updated rows in sync without a custom export job. This guide walks through the setup, the first sync, and the checks that confirm your Snowflake tables match the source.

Before you start, decide how each table records changes and how you want to handle deleted rows. Supaflow uses a date, datetime, or timestamp column to find inserts and updates. It does not read the MySQL binary log, so hard deletes require a separate approach.

For the shorter product and capability overview, see the MySQL to Snowflake connector page.

Need to move Redshift tables into Snowflake? Supaflow can replicate Redshift tables into Snowflake without a custom export script, copy job, scheduler, or table-by-table loader.

This guide walks through the full setup end to end: create the Snowflake destination, create the Redshift source, build the pipeline, run the first sync, and verify the data in Snowflake.

For the shorter product overview before the walkthrough, see the Redshift to Snowflake migration page.

Need to replicate your SQL Server tables into Snowflake? Supaflow's SQL Server connector supports two query modes: Standard for simple cursor-based sync, and Change Tracking for capturing every insert, update, and delete -- including hard deletes that standard mode can never detect.

This guide walks through setting up a SQL Server to Snowflake pipeline with Change Tracking enabled, so your Snowflake tables stay in sync with every change made in SQL Server.

Most people who want to build a Supaflow connector already have deep domain expertise. They know the Asana API inside out, or they understand exactly how Stripe's pagination works, or they have spent years working with Dynamics 365. What they do not know -- and should not have to learn from scratch -- is the Supaflow connector SDK: how schema discovery works, how incremental sync state is managed, what lifecycle methods to implement, and what contracts the pipeline engine expects.

That is the gap this project fills. We codified everything we know about the Supaflow connector SDK into an AI skill framework that any coding agent can follow -- so you can focus on your domain expertise while the skill handles the SDK plumbing.

It is open source and available now on GitHub: supaflow-labs/supaflow-connector-dev-skills.

Introducing Supaflow​

A single platform to replace multiple stitched-together pipeline tools.

Modern data teams are drowning in vendor sprawl. The average company uses four to five different tools just to move data from Salesforce to Snowflake—and back again. Each vendor adds another contract, another integration to maintain, and another potential point of failure.

That's why we built Supaflow: a unified data movement platform that collapses ingestion, transformation, activation, and orchestration into one secure, flexible solution—without your data ever leaving your environment.

Export Salesforce to S3 in 10 Minutes​

Export Salesforce to S3 and query it in Athena — no ETL tools, no long‑running servers.

This open‑source, serverless pipeline uses AWS Lambda, Step Functions, and the Salesforce Bulk API 2.0 to export every Salesforce object to S3. After each run, it automatically updates the AWS Glue Data Catalog, so your data is immediately queryable in Athena.

After a single deployment, you can query your Salesforce data directly in Athena:

SELECT * FROM salesforce_export.account_raw LIMIT 10;