Data Wrangling
Data wrangling is the process of cleaning, reshaping, and enriching raw data into a form suitable for analysis or modeling. It handles missing values, inconsistent formats, duplicates, and structural mismatches that prevent data from being used directly.
itData engineering and analytics | OpenSkills.info
Intro
Data Wrangling
Data rarely arrives in the shape that analysis or a downstream system needs. Names vary. Dates use several formats. Missing values carry different meanings. Two files may describe the same customer with different identifiers.
Data wrangling is the work of turning that source data into a defined, usable dataset. You inspect the source, reshape its structure, standardize representations, resolve quality problems, combine sources, and validate the result. You also preserve enough evidence for another person to understand what changed.
The goal is not to make data look clean. The goal is to make it fit for a stated purpose without hiding uncertainty.
Why the work starts with purpose
Quality depends on use. A dataset can be complete enough for a weekly trend report but unsafe for individual billing decisions. The UK Government Data Quality Framework defines quality as fitness for purpose and separates six dimensions: completeness, uniqueness, consistency, timeliness, validity, and accuracy.
Those dimensions answer different questions:
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework
Supports
- Data quality as fitness for purpose
- Completeness, uniqueness, consistency, timeliness, validity, and accuracy as distinct dimensions
- Quality trade-offs based on user needs
- Assessment throughout the data lifecycle
- Treating recurring quality problems at their source
- https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance
Supports
- Quality action plans tied to user needs
- Baseline assessment, metrics, and automated checks
- Root-cause analysis and metadata practices
- Communication of cleaning, missingness, deduplication, and known limitations
- https://tidyr.tidyverse.org/articles/tidy-data.html
Supports
- Tidy data structure with variables in columns and observations in rows
- Reshaping data between wide and long representations
- Separating and combining variables during structural normalization
- https://pandas.pydata.org/docs/user_guide/missing_data.html
Supports
- Multiple missing-value representations across data types
- Detection, dropping, and filling of missing data
- The behavior and consequences of missing values in calculations
- https://pandas.pydata.org/docs/user_guide/indexing.html#duplicate-data
Supports
- Identification of duplicate rows with selected fields
- Duplicate removal and explicit first, last, or all survivor behavior
- https://pandas.pydata.org/docs/reference/api/pandas.merge.html
Supports
- Merge indicators for left-only, right-only, and matched rows
- Validation of one-to-one, one-to-many, and many-to-one key relationships
- Join behavior based on declared keys
- https://pandas.pydata.org/docs/user_guide/merging.html
Supports
- Merge, join, concatenate, and comparison operations
- Relational join semantics and key selection
- Cardinality and source-indicator checks for joins
- https://openrefine.org/docs/manual/facets
Supports
- Facets for summarizing values and selecting subsets
- Interactive inspection of textual, numeric, and timeline distributions
- https://openrefine.org/docs/technical-reference/clustering-in-depth
Supports
- Clustering as detection of possible alternate string representations
- Syntactic limits of clustering for semantic reconciliation
- Key-collision and nearest-neighbor clustering approaches