Data readiness: the cleanup that has to happen before any AI project

Key Takeaways

• An AI project does not fail loudly on bad data. It produces confident, wrong answers that look fine until someone checks. That is more dangerous than an outright crash.
• The four places data quietly rots in a UAE SME: duplicate customer records, unstructured knowledge trapped in spreadsheets and WhatsApp threads, no agreed source of truth across systems, and inconsistent product or pricing data.
• RAND Corporation's interviews with 65 data scientists put data quality among the top five root causes of AI project failure, ahead of model choice or infrastructure.
• Sequence the cleanup: customer and product master data first, then the business rules, then the automation. Cleaning unstructured documents before the master data is stable wastes the effort twice.

An AI project built on messy data does not usually fail in a way anyone notices right away. It ships, it works in the demo, and then it quietly generates duplicate customer statements, misquotes a price, or drops a supplier's outstanding balance because the name was spelled three different ways across two systems. Nobody gets an error message. Someone just gets an angry phone call weeks later and has to reconstruct what went wrong.

That is the real argument for doing the cleanup first. Not because it is good practice in the abstract, but because the failure mode of AI on dirty data is silence, not crash, and silent failures are the expensive kind.

Why "garbage in, garbage out" is not just a cliché here

Picture a Dubai trading company that rolls out an AI agent to auto-match incoming payments against open customer invoices. In testing, against a curated sample of twenty clean records, it works perfectly. In production, it meets the real customer list: "Al Futtaim Trading LLC," "Al-Futtaim Trading," and "ALFUTTAIM TRADING L.L.C." sitting as three separate accounts, each with its own partial payment history, because three different salespeople created the customer record over three different years and nobody deduplicated afterward.

The agent does not know these are the same company. It matches a payment to whichever record scores highest on its fuzzy logic, which is sometimes the wrong one. The customer's real balance is now split across three ledgers, and the agent (confident, consistent, and wrong) keeps making the same misallocation every time a payment comes in. A human doing this manually would have recognized the company name from memory and fixed it on the spot. The agent has no memory of that kind and no reason to doubt itself.

This is the pattern RAND Corporation found when it interviewed 65 experienced data scientists across government and industry about why AI projects fail: production databases carry encoding issues, unexpected nulls, and duplicate records accumulated over years, and a system trained and tested against clean data breaks on contact with the real thing (RAND Corporation, "Why AI Projects Fail and How They Can Succeed," 2024, retrieved 2026-09-03). Gartner's research points the same direction from a different angle: it expects organizations to abandon a majority of AI projects that are not built on data the vendor would call "AI-ready" (Gartner, "Lack of AI-Ready Data Puts AI Projects at Risk," February 2025, retrieved 2026-09-03). Neither source is describing a modelling problem. Both are describing exactly the mess above.

The four places data rots in a UAE SME

Duplicate and inconsistent customer records. Every sales handover, every new CRM, every rushed quotation adds a variant. Trade names get typed differently, phone numbers get updated in one system and not another, and nobody owns the job of merging them. An AI tool built to summarize customer history, draft renewal emails, or flag at-risk accounts will treat each variant as a different customer, which means its output is wrong in proportion to how bad the duplication is.

Unstructured knowledge trapped in spreadsheets, PDFs, and WhatsApp. A meaningful share of what a UAE SME "knows": the special pricing agreed verbally with a long-standing client, the delivery exception noted in a WhatsApp thread, the discount schedule buried in a PDF someone emailed two years ago: exists nowhere a system can query it. An AI project that assumes this knowledge is retrievable will keep missing it, and because the miss is invisible, nobody flags it as an error.

No agreed source of truth across systems. Pricing lives in the accounting software, stock levels live in a warehouse spreadsheet, and customer contact details live in three inboxes. When two systems disagree (and they always eventually do) nobody has decided in advance which one wins. An automation has to make that call every time it runs, and without a rule, it makes it inconsistently.

Missing or inconsistent product and pricing data. SKUs entered differently across the POS system, the online store, and the accountant's spreadsheet; prices quoted in AED in one place and left ambiguous in another; units of measure that mean "per box" in one record and "per piece" in the next. A quote-generation or reorder tool sitting on top of this will produce numbers that are wrong in ways a person would catch immediately and a system will not.

A practical audit: how bad is it, actually

Before scoping any AI project, spend a week finding out how much of this applies to you. This does not require a consultant or a data platform. It requires someone willing to open the actual files.

1. Export the full customer list from whichever system is primary and sort it alphabetically. Count how many near-duplicate names you can spot in five minutes. If it is more than a handful in a list of a few hundred, the duplication problem is systemic, not incidental.
2. Pick ten recent transactions and trace each one back to its supporting record, pricing, customer details, product code. Note every place the number in the transaction disagrees with the number in the master record.
3. Ask three people in different roles where they'd look up "the current price for this customer" or "this customer's correct company name." If you get three different answers, you do not have a single source of truth, whatever the org chart says.
4. Estimate what fraction of pricing or policy exceptions live only in someone's head or inbox. This is necessarily rough, but even a rough estimate tells you whether unstructured knowledge is a small tail case or the bulk of how the business actually runs.

The output of this audit is not a score. It is a short list of exactly which of the four problems above is worst in your business, because that determines where cleanup starts.

Sequencing: what to clean first

The instinct is to start with whichever mess is most visible, usually the unstructured documents, because there's a folder full of them and it feels tractable. That is normally the wrong place to start.

Master data comes first: deduplicate customers, agree which system is the source of truth for pricing, and standardise product codes and units of measure. Everything downstream depends on this being stable, because any matching logic (whether it's an AI agent or a plain fuzzy-match script) needs a clean list to match against. Cleaning unstructured PDFs or WhatsApp exports before the master list is fixed means re-doing that extraction work once the underlying records change, which they will.

Business rules come second: decide, in writing, what happens when two systems disagree, which price wins in an edge case, how an exception gets logged. This is the step most businesses skip, and it's the one that turns automation from a shortcut into a liability, because whatever ambiguity exists in the rule gets applied by the automation consistently, at volume, instead of inconsistently by a person who at least might notice something looks wrong.

Only after both are settled does it make sense to build the automation itself, and only then does connecting an ERP or extraction platform: the kind of infrastructure covered in our guide on operational business scaling: actually pay off rather than automate the mess faster. If you haven't yet worked out which process to automate first or what it should cost against a hire, our AI readiness guide walks through that decision in detail. It assumes the data groundwork covered here is either done or underway.

What "clean enough" looks like

Perfection is not the target, and chasing it delays a project indefinitely. "Clean enough" means: one system is the agreed source of truth for each data type, the obvious duplicates are merged, and the remaining exceptions are logged somewhere queryable rather than living only in someone's memory. You can run a rough return-on-effort estimate for the cleanup itself the same way you'd size any operational investment: the ROI calculator is built for exactly that kind of before-and-after comparison.

The cleanup checklist

• Export customer and supplier lists; flag and merge near-duplicate names
• Agree, in writing, which system is the source of truth for pricing, stock, and contact details
• Standardise product codes and units of measure across every system that touches them
• Trace a sample of recent transactions back to their supporting records and log every mismatch found
• Inventory where pricing exceptions and special agreements currently live (spreadsheet, PDF, WhatsApp, memory)
• Write down the rule for what happens when two systems disagree. Don't leave it implicit
• Assign an owner for ongoing data hygiene, not just the one-time cleanup

Frequently asked questions

How long does data cleanup take before an AI project can realistically start?

For a business with a few hundred customers and one or two core systems, a focused cleanup of customer and product master data typically takes two to six weeks of dedicated effort. The unstructured-document problem takes longer and can run in parallel once the master data is stable. It doesn't need to finish first.

Can we start the AI project in parallel with the data cleanup?

You can scope and build the automation in parallel, but do not connect it to live data or start relying on its output until the master data it depends on is stable. Building against a moving target means re-testing every time the underlying records change, which costs more than waiting.

What's the single most common data problem in UAE SMEs specifically?

Duplicate customer and supplier records from years of manual entry across sales, accounting, and CRM tools, often compounded by trade names that get transliterated or abbreviated differently each time someone types them. It is rarely dramatic on its own, but it undermines almost anything built on top of it.