9 posts tagged with "Data Engineering"

Oracle OTM analytics should answer four questions quickly: What are we spending? Are shipments on time? Are we using capacity well? Where is performance getting worse?

The difficult part is rarely drawing the dashboard. It is agreeing on the calculation behind each number and mapping that definition to the right Oracle Transportation Management data.

This guide defines 12 practical OTM KPIs, the formulas behind them, and the shipment, cost, stop, and location data needed to calculate them in a warehouse or BI tool. To move that source data into your analytics platform, start with the Supaflow Oracle Transportation Management connector.

Migrating SQL Server to Amazon Redshift is not a backup-and-restore job. SQL Server is an operational database; Redshift is an analytical warehouse. The safest approach is to move table data continuously, validate it while SQL Server stays live, and switch downstream analytics only after the Redshift copy passes your checks.

This guide shows how to build that migration with Supaflow's SQL Server connector and Amazon Redshift connector. You will configure Redshift as the destination, connect SQL Server as the source, run an initial load, test ongoing changes, and use a controlled cutover checklist instead of betting everything on one migration window.

Moving Google Drive files into Snowflake can mean several different things: loading a folder of recurring CSV exports, syncing every worksheet in a set of Google Sheets, replacing one named range, or running a one-time import. Those workloads should not share one blanket recommendation.

This guide compares Supaflow, Fivetran, Hevo, Airbyte, manual Snowflake loads, and a custom Google API pipeline. It then walks through a Supaflow setup using CSV files; the same pipeline workflow applies to TSV, Excel (.xlsx), and native Google Sheets sources.

Need MySQL data in Snowflake for analytics, reporting, or a migration? Supaflow handles the initial load and keeps new and updated rows in sync without a custom export job. This guide walks through the setup, the first sync, and the checks that confirm your Snowflake tables match the source.

Before you start, decide how each table records changes and how you want to handle deleted rows. Supaflow uses a date, datetime, or timestamp column to find inserts and updates. It does not read the MySQL binary log, so hard deletes require a separate approach.

For the shorter product and capability overview, see the MySQL to Snowflake connector page.

Need to move Redshift tables into Snowflake? Supaflow can replicate Redshift tables into Snowflake without a custom export script, copy job, scheduler, or table-by-table loader.

This guide walks through the full setup end to end: create the Snowflake destination, create the Redshift source, build the pipeline, run the first sync, and verify the data in Snowflake.

For the shorter product overview before the walkthrough, see the Redshift to Snowflake migration page.

Need to replicate your SQL Server tables into Snowflake? Supaflow's SQL Server connector supports two query modes: Standard for simple cursor-based sync, and Change Tracking for capturing every insert, update, and delete -- including hard deletes that standard mode can never detect.

This guide walks through setting up a SQL Server to Snowflake pipeline with Change Tracking enabled, so your Snowflake tables stay in sync with every change made in SQL Server.

Most people who want to build a Supaflow connector already have deep domain expertise. They know the Asana API inside out, or they understand exactly how Stripe's pagination works, or they have spent years working with Dynamics 365. What they do not know -- and should not have to learn from scratch -- is the Supaflow connector SDK: how schema discovery works, how incremental sync state is managed, what lifecycle methods to implement, and what contracts the pipeline engine expects.

That is the gap this project fills. We codified everything we know about the Supaflow connector SDK into an AI skill framework that any coding agent can follow -- so you can focus on your domain expertise while the skill handles the SDK plumbing.

It is open source and available now on GitHub: supaflow-labs/supaflow-connector-dev-skills.

Introducing Supaflow​

A single platform to replace multiple stitched-together pipeline tools.

Modern data teams are drowning in vendor sprawl. The average company uses four to five different tools just to move data from Salesforce to Snowflake—and back again. Each vendor adds another contract, another integration to maintain, and another potential point of failure.

That's why we built Supaflow: a unified data movement platform that collapses ingestion, transformation, activation, and orchestration into one secure, flexible solution—without your data ever leaving your environment.

Export Salesforce to S3 in 10 Minutes​

Export Salesforce to S3 and query it in Athena — no ETL tools, no long‑running servers.

This open‑source, serverless pipeline uses AWS Lambda, Step Functions, and the Salesforce Bulk API 2.0 to export every Salesforce object to S3. After each run, it automatically updates the AWS Glue Data Catalog, so your data is immediately queryable in Athena.

After a single deployment, you can query your Salesforce data directly in Athena:

SELECT * FROM salesforce_export.account_raw LIMIT 10;