Data Cleaning and Validation Service: Duplicates, Errors and Documentation

What this service covers

• Duplicate detection, including near-duplicates with spelling variations
• Range, logic and consistency checks, such as dates out of order or impossible ages
• Standardising categories, spellings, units and date formats
• Checking skip logic and routing in survey data
• Flagging suspicious response patterns such as straight-lining
• Validation against reference lists or source documents
• A cleaning log and data dictionary documenting every change

What goes wrong in raw data

Raw datasets contain surprises: the same person entered twice, “Lahore”, “lahore” and “LHR” as three cities, ages of 250, or end dates before start dates. Each problem distorts results in a different way. We systematically check each variable, not only the ones that look suspicious.

Our approach: check, flag, fix, record

We run automated checks, review flagged records and decide on each case with rules agreed with you. Where a value is clearly an entry error and the source is available we correct it; where it is uncertain we flag instead of guessing. The original data are never overwritten, and a cleaning log lists what changed, why and how many records were affected.

Preparation for analysis

Clean data feed straight into analysis. We often continue with missing data and outlier handling, coding and recoding or merging and reshaping. Download your own quick checks with our free data tools and follow with survey data analysis or statistical analysis.

Reproducible and auditable

We provide the script or steps used so the cleaning can be repeated when new data arrive, which is valuable for panel surveys and ongoing monitoring.

How the work runs

1. You send the raw dataset and any codebook or questionnaire.
2. We profile the data and report the problems found.
3. We clean using agreed rules and keep a log.
4. You receive the clean dataset, log and data dictionary.

What you receive

• A cleaned dataset in your preferred format
• A cleaning log listing every change
• A data dictionary with variable labels and codes
• A summary of problems found and how they were resolved
• Scripts for repeating the cleaning