Our ERP data is a mess. Do we have to clean it up first?
No, and waiting to clean it first is how these projects stall indefinitely. Transformation happens in the Fabric pipeline layer, so messy ERP data is standardised on the way into OneLake while the ERP itself stays untouched and read-only. What does have to be settled before week one is which definition of each metric is correct.
- 01Two different jobs get confused under the word "clean". Shape problems — inconsistent date formats, trailing whitespace, three spellings of the same unit of measure, codes that mean nothing outside the module that wrote them — are fixed in the pipeline, on every run, permanently. Missing facts and wrong facts are fixed at source, because no transformation step invents a value that was never recorded.
- 02Sort the mess into three buckets before anyone quotes the work. Formatting and type inconsistency: handled in transformation, no decisions needed from you. Duplicate and near-duplicate master records: handled by a matching rule, but someone of yours has to approve which record wins. Genuine disagreement between systems — the CRM says booked, the ERP says recognised — is not a data-quality defect at all, and no tool resolves it.
- 03The real blocker is definitional, not technical. Expect to settle, in writing: which date a sale belongs to, whether intercompany transactions are netted out, which cost basis margin uses, whether cancelled orders stay in the denominator, and which company codes or plants roll into "the business". That is a handful of two-hour sessions with people who can decide, not a cleansing programme — and it is the work that sets whether the first module lands in four to six weeks or drifts.
- 04Extraction surface determines the shape of the build, so establish it early. SAP ECC is normally read through its extractor layer or a replica rather than queried against production; Oracle EBS commonly through a read replica; Dynamics 365 Business Central typically through its OData/API surface or Dataverse; an on-premises SQL-backed ERP through an on-premises data gateway, which makes an outbound connection, so no inbound firewall port is opened. After the first full load, incremental refresh keys off a row-level watermark — a dependable modified-date or change-tracking column — so only changed records move.
- 05Reporting is a better data-quality instrument than an audit. Once the pipeline runs, record counts and control totals get reconciled against a report the ERP already produces, and the gaps become visible and specific: orders with no customer region, one plant that stopped posting scrap codes in March, a supplier that exists four times. Fixing those with named rows in front of you is a different task from cleaning a system in the abstract.
- 06The honest limit. If a table carries no dependable primary key and no modified-date or change-tracking column, there is nothing for watermark-based incremental refresh to key off, and the options narrow to a full reload each cycle or a change at source. And where the figure you want was never captured — blank scrap reason codes, promised dates typed into a free-text note — the fix is a process change in the ERP, and reporting is worth building after it, not before.
Where to go deeper
For the full explainer on this topic rather than this specific question, see the detailed guide on the blog.
Related questions, answered
What actually gets fixed in the pipeline rather than in the ERP?
Type and format normalisation, unit and currency conversion, code-to-label mapping, deduplication by rule, header-to-line joins, and derived fields such as fiscal period or region. All of it is declarative and re-runs on every load. What stays in the ERP: a value nobody entered, a wrong value only the originating team can correct, and any change to what staff are asked to record.
Will duplicate customer or supplier records be merged automatically?
Matching is automated; the ruling is not. A pipeline can collapse records on normalised name, tax number, address or an external key, and flag the pairs it is unsure about. Someone on your side still has to approve which record survives and whether two similar entities are one legal customer. Budget a review pass on master data — one session per domain with someone who can decide, not a project.
Who decides a metric definition when two departments disagree?
You do, and the engagement stalls until someone can. Finance and operations routinely hold different, defensible definitions of revenue, on-time delivery or margin. The deliverable is one named owner per metric and a written definition, after which 200+ pre-built KPI definitions give you a starting draft to amend rather than a blank page. A pre-built library shortens the argument; it does not settle it.
Does connecting to our ERP change anything inside it?
No. Access is read-only, nothing is written back, and no schema change is required. For an on-premises or firewalled system, an on-premises data gateway makes an outbound connection, so inbound ports stay closed. Everything else is deployed as Fabric artifacts in your own Azure subscription, so your existing governance, row-level security and Entra ID policies apply unchanged.
What usually goes wrong on the first few pipeline runs?
Three failure modes recur. A watermark column that stamps on export rather than on edit, so amended rows are quietly missed. A join that drops transaction lines whose header key is null, which understates volume rather than erroring. And a schema change after a vendor patch, which breaks a typed load. All three are caught by reconciling against a report the ERP already produces.
When should we fix the source before building reporting at all?
When multi-company consolidation has no conformed chart of accounts, the mapping is a finance project and should be run as one. When a module is mid-migration, build on the system you will keep. When the metric depends on data nobody records, change the process first. Each of these makes a reporting build premature rather than merely harder, and is worth saying out loud before a quote.
Sources
- OneLake — the unified data lake for Microsoft Fabric · checked 2026-10-06
- Install an on-premises data gateway · checked 2026-10-06
- OneLake shortcuts · checked 2026-10-06
Figures on this page: 4–6 weeks · an on-premises data gateway · data stays in your tenant · 200+
What this costs for an operation your size
Three engagement tiers, what each includes, and the Microsoft licensing you need alongside it.
See what shapes the pricePeople also ask
- How long does it take to standardise messy ERP data into one reporting model?
- Who can connect a legacy ERP to Microsoft Fabric without a data warehouse project first?
- What is the difference between data cleansing and data transformation?
- Does analytics on Microsoft Fabric write anything back to the ERP?
- How are duplicate customer records handled across ERP and CRM?