Getting your data ready for machine learning

Most ML projects that stall don't stall on the modeling — they stall because the underlying data wasn't actually ready, and that gap usually isn't visible until well into the project.

What "ready for ML" actually requires, specifically

Volume and history — most useful ML approaches need a meaningful history of labeled examples to learn from; a business that's only recently started capturing the relevant data may simply not have enough history yet, regardless of modeling technique. Consistency — if the same event or entity has been recorded inconsistently over time (a product category that's been renamed twice, a customer status field with values that mean different things in different eras of the data), the model will learn from that noise. Labeled outcomes — for supervised learning specifically, you need historical examples where the outcome you're trying to predict was actually recorded reliably, not inferred after the fact from incomplete records.

The data audit that should happen before any modeling work

We run a structured readiness assessment before committing to an ML project: how much history exists for the relevant data, what's the actual completeness rate (what percentage of records have all the fields the model would need), and how consistent is labeling/categorization over the historical period. This regularly surfaces that a project client expected to be primarily a modeling exercise is actually, first, a data cleanup and pipeline-building exercise — which changes the timeline and budget conversation upfront rather than mid-project.

A concrete example

A retail client wanted to build a demand forecasting model to optimize inventory ordering. Our readiness audit found: three years of sales history existed (sufficient volume), but product categorization had changed twice during that period with no mapping between old and new category schemes, meaning "the same" product category in different years wasn't actually comparable without reconciliation work. Additionally, roughly 22% of historical orders were missing a promotional-flag field that would matter for accurate forecasting (an order made during a promotion behaves very differently than a normal-price order, and a model that can't distinguish them will learn a distorted pattern).

We spent the first phase of the engagement — before any modeling — reconciling the category taxonomy across the historical period and backfilling the promotional flag where it could be reliably inferred from linked pricing data. This phase took roughly 5 weeks, longer than the client had initially budgeted, but the resulting forecasting model, once built, achieved meaningfully better accuracy than an earlier internal attempt that had skipped this step and trained directly on the inconsistent raw history.

Why skipping this step doesn't actually save time

Teams under time pressure sometimes want to skip straight to modeling on data "as-is." This doesn't actually save time — it produces a model that performs poorly for reasons that are hard to diagnose (is it the model architecture, or is it the data), and the data cleanup work usually ends up happening anyway, later, under more pressure, once poor model performance forces the question.

How Ndakum approaches it

Data readiness assessment is the first phase of every ML project in our Data Engineering & AI work — we'd rather set an honest timeline upfront than discover the real scope mid-project.