Data Cleaning & Normalization
Raw operating data is not analytically clean simply because it came from production software. Our normalization layer converts inconsistent source representations into canonical forms before they are relied on by calculations, retrieval, entity resolution, or model inference.
What arrives
Common inconsistencies include:
- Incompatible date formats
- Mixed currencies or units
- Duplicate records
- Malformed fields
- Inconsistent names
- Missing values
- Changing schemas
- Incompatible identifiers
- Manually entered free text
A model should not be expected to probabilistically repair data problems that software can correct deterministically upstream.
Canonicalization
Normalization can include schema mapping, field coercion, date and currency normalization, unit conversion, deduplication, missing-data handling, malformed-record isolation, and canonical record construction.
A source field named customer_id does not automatically mean the same thing as an account_id in another system. Field mapping therefore represents semantic alignment, not merely column renaming.
Dates require similar care. Creation date, transaction date, effective date, settlement date, and last-modified time may all be valid timestamps describing different events. Collapsing them into a generic date field destroys information the rest of the system may later need.
Reconciliation
Where multiple sources describe related business activity, normalization may include reconciliation rather than simple transformation.
Duplicate or contradictory records should not necessarily be discarded. They may represent:
- A synchronization error
- A legitimate historical state
- Two source systems with different authority
- An unresolved identity problem
The correct output can therefore be a normalized record plus an explicit conflict, not an artificially clean value.
Failure handling
Malformed or incomplete records should be detectable and inspectable. Silently dropping them can create a more dangerous system than preserving them with an exception state.
Normalization logic also needs to tolerate schema drift. When an upstream platform changes a field or representation, the pipeline should fail visibly or route the record for handling rather than quietly producing invalid canonical data.
Position in the stack
Cleaning and normalization happen before higher-level intelligence because every downstream layer inherits upstream errors. Reliable models, retrieval, monitoring, and financial analysis depend on stable representations of the underlying business.
The objective is not to make every source look identical. It is to create consistent enough semantics that the system can reason across sources without confusing representation differences for business differences.