Data Quality
Data quality measures and improves the accuracy, completeness, consistency, timeliness, and validity of data assets. It applies profiling, validation rules, monitoring, and remediation processes so that downstream consumers can trust the data they build on.
itData engineering and analytics | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — Data Quality
Data quality is the managed claim that a dataset is fit for a stated purpose. That sounds dignified, which is useful because the alternative is a spreadsheet full of numbers making confident faces while nobody knows whether they are safe to use. The question is not whether data is universally good. It is whether it can support this decision at an acceptable level of risk.
The key is purpose. A live operations feed may need to arrive within minutes, while monthly reporting may wait for reconciliation. Neither is morally superior. They have different risks. Once the purpose is known, you can write a rule: which data is in scope, what condition it must meet, how it is measured, when it is measured, and who responds when it fails.
A metric is the calculation. A measurement is its result for one particular run. This distinction prevents an alarming but common magic trick: treating the formula as if it were evidence. A measurement needs its population, time, method, and limitations attached. Otherwise a cheerful percentage can wander in wearing a hat and claim to be the truth.
The six familiar dimensions help keep the claim honest. Completeness asks whether required values exist. Uniqueness asks whether unintended duplicates exist. Consistency asks whether related values agree. Timeliness asks whether the data arrives soon enough. Validity asks whether it follows expected rules. Accuracy asks whether it matches reality. Validity is not accuracy, and completeness is not accuracy either. A date can be perfectly formatted and still be the wrong date. Computers are very good at admiring a well-formatted mistake.
The practical loop is less mysterious than it first appears. Identify critical data. Define scoped rules and realistic targets. Measure a baseline. Record failures. Trace causes toward collection or creation. Communicate limitations. Measure again. Automation can run checks and retain evidence, but it cannot decide what the data means or which trade-off is acceptable. That part remains inconveniently human.
Read the Introduction for the full operating model and roles. Use the Slides for the relationships among dimensions, rules, and evidence. Keep the Cheatsheet nearby when writing a rule or issue record. Then use the Practice tab to turn one small, disposable dataset into measurements you can inspect. The goal is not a spotless score. It is a visible, owned claim that can survive a real decision.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework
Supports
- Data quality is fitness for purpose and depends on user needs
- Data quality is broader than data cleaning
- Quality should be assessed and communicated throughout the data lifecycle
- The six core dimensions are completeness, uniqueness, consistency, timeliness, validity, and accuracy
- Completeness and accuracy are distinct, and validity does not establish accuracy
- Quality dimensions can trade off according to intended use
- Continuous improvement includes regular assessment, communication, and early correction
- https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance
Supports
- Action plans identify critical data, rules, baselines, goals, and ongoing monitoring
- Root-cause analysis should trace the data journey and move fixes toward the source
- Quality results and caveats should be communicated to users
- Metadata helps users interpret data and its limitations
- https://www.gov.uk/government/publications/implement-a-data-quality-action-plan/data-quality-action-plan-implementation-guide
Supports
- A data quality action plan follows seven steps from critical-data identification through repeated measurement
- Rules align with user needs, business goals, dimensions, methods, and percentage targets
- Targets should describe realistic fitness for purpose instead of absolute perfection
- Data owners or process owners oversee plans with stewards, custodians, analysts, and subject experts
- Stewards and subject experts define business rules while technical experts implement checks
- Repeated assessment, user feedback, issue tracking, and change management sustain quality
- https://www.w3.org/TR/vocab-dqv/
Supports
- Dataset quality information helps users judge fitness for purpose
- A dimension is a criterion for assessing quality
- A metric defines how a quality dimension is measured
- A quality measurement evaluates specific data against a metric and records a value
- Quality metadata can group policies, measurements, certificates, and annotations
- Provenance can connect derived metrics, measurements, and annotations
- https://www.iso.org/standard/81745.html
Supports
- ISO 8000-1 provides the current overview of the ISO 8000 series
- The overview covers data-quality principles, a path to data quality, the series structure, and relationships to other standards
- https://docs.greatexpectations.io/docs/core/run_validations/
Supports
- Great Expectations associates a batch definition with an expectation suite and runs validation definitions against data
- Great Expectations supports retrieving unexpected rows from a validation
- https://docs.soda.io/soda/product-overview.html
Supports
- Soda runs prepared data quality checks as scans against datasets and surfaces failed results for investigation
- Soda checks can address freshness, completeness, duplicates, transformation failures, validity, and anomalies
- https://info.montecarlodata.com/solutions/data-observability-platform
Supports
- Monte Carlo combines automated data quality and pipeline checks with profiling, alerts, lineage, triage, and root-cause investigation
- https://productresources.collibra.com/docs/collibra/latest/Content/UnifiedDataQuality/co_about-data-quality.htm
Supports
- Collibra Data Quality and Observability supports automatic and custom monitoring, profiling, alerting, custom SQL, and scheduled analysis
- https://docs.informatica.com/content/dam/source/GUID-2/GUID-246B42EF-FCF6-4084-B958-240052C7C2EF/40-1-2/en/DQ_1056_DataQualityGettingStartedGuide_en.pdf
Supports
- Informatica Data Quality supports profiling, mappings, and reference tables for data-quality work
- https://andrew-jones.com/blog/we-had-an-incident-and-it-was-great/
Supports
- A data-platform incident can lose data in transit while requiring a structured postmortem and follow-up actions