Data pipelines that don't break every time a schema changes

A brittle data pipeline that breaks whenever an upstream system changes a field isn't a rare edge case — it's the default outcome unless you specifically design against it.

Why this happens so often

Data pipelines commonly get built with implicit assumptions about upstream schema — a script that expects a specific column to always exist, in a specific format, in a specific position. This works until an upstream team (often one you don't control directly, or don't coordinate closely with — a SaaS vendor updating their API, a different internal team modifying their database) makes a change, and the pipeline breaks, often silently producing wrong data rather than an obvious error, which is worse.

The specific patterns that make pipelines resilient

Schema validation at ingestion — checking incoming data against an expected schema before processing it, and failing loudly (alerting someone) rather than silently processing malformed data through the rest of the pipeline. This turns a silent data corruption issue into an immediate, visible alert.

Explicit typing and defaults — rather than assuming a field will always be present and correctly typed, defining explicit handling for missing or unexpected values, so a new nullable field upstream doesn't crash the whole pipeline.

Decoupling via a staging layer — raw data lands in a staging area exactly as received, and transformation logic operates on that staged copy rather than directly on the live upstream source. This means a pipeline failure during transformation doesn't lose or corrupt the raw data — you can always reprocess from the staged raw copy once the transformation logic is fixed.

Testing that catches schema drift before it reaches production

We build automated tests (typically using dbt's built-in testing framework) that run on every pipeline execution, checking not just "did the pipeline run without erroring" but "does the output data match expected patterns" — row counts within expected ranges, key fields not unexpectedly null, referential integrity between related tables. This catches a broader category of problems than simple error handling alone, including cases where an upstream change doesn't cause an outright failure but silently changes the meaning of the data.

A concrete example

A retail client's pipeline ingesting product data from their e-commerce platform broke roughly once every 6-8 weeks, typically when the platform vendor pushed an API update, and each break required manual investigation to identify what had changed. We rebuilt the ingestion layer with explicit schema validation (failing loudly and specifically identifying which field changed, rather than a generic pipeline failure), a staging layer preserving raw data independent of transformation logic, and automated data quality tests running on every load. In the following year, the pipeline experienced 4 upstream schema changes from the vendor — each one was caught immediately by validation, with a specific, actionable error message, and none required more than about 20 minutes to diagnose and fix, compared to the multi-hour investigations the previous brittle pipeline had required.

How Ndakum approaches it

Building resilient pipelines is standard practice in our Data Engineering & AI work — we design for upstream change from the start, because it's not a matter of if a schema changes, but when.