Chapter 4: Data Cleaning and Exploratory Analysis¶
Part I — Scientific Python for Researchers
Learning Objectives¶
By the end of this chapter you will be able to:
- Explore distributions, relationships, and outliers visually
- Use matplotlib and seaborn for quick validation plots
- Establish visual validation as a prerequisite for spatial analysis
# Standard imports — add chapter-specific imports below
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
Univariate Distributions¶
Chapter 3 treated world_data.csv as already trustworthy: it had one row per country, sensible types, and no missing values, because someone — invisibly, before you ever saw it — had already checked. Real datasets rarely arrive that way. This chapter uses a deliberately messier sibling of the same file, world_data_messy.csv, to walk through what that invisible checking actually involves, before getting to the visual exploration the chapter's title promises.
John Tukey, who coined the term exploratory data analysis in 1977, argued that looking at your data — really looking, with plots, before any modelling — is not optional polish but the first analytical step. In a spatial context this matters even more: a bad histogram is a wasted afternoon, but a bad spatial model built on unnoticed missing values or a badly-typed column is a wrong conclusion that looks confident on a map.
messy = pd.read_csv("world_data_messy.csv")
messy.info()
<class 'pandas.DataFrame'> RangeIndex: 181 entries, 0 to 180 Data columns (total 4 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 country 181 non-null str 1 continent 181 non-null str 2 population 176 non-null str 3 gdp 177 non-null float64 dtypes: float64(1), str(3) memory usage: 5.8 KB
Three problems are already visible in that output. First, population is not typed as a number — pandas shows it as object (or, on pandas 3.0+, str) rather than float64 or int64 — a strong hint that some values are stored as text (a thousands-separated string like "58,005,463" will do this to an entire column, not just the affected rows). Second, population and gdp both have fewer non-null entries than the row count, meaning missing values. Third, worth checking separately, is whether any row is a straight duplicate of another:
print("Missing values per column:")
print(messy.isna().sum())
print()
print("Fully duplicated rows:", messy.duplicated().sum())
Missing values per column: country 0 continent 0 population 5 gdp 4 dtype: int64 Fully duplicated rows: 4
Each of these needs a decision, not just a fix. A duplicated row is usually safe to drop outright — messy.drop_duplicates() — since it adds no information and would silently double-count in any aggregation. Missing values are less automatic: dropping every row with a NaN (messy.dropna()) is the simplest option, but it throws away the continent and country you do have for that row along with the value you don't. Whether that's the right call depends on how many rows are affected and what you intend to do with the column — a question this chapter returns to explicitly in the EDA Checklist.
The typing problem is fixed with pd.to_numeric(), which converts a column to a numeric dtype and, with errors="coerce", turns anything it can't parse into NaN rather than raising an exception:
messy['population'] = (
messy['population']
.astype(str)
.str.replace(',', '', regex=False)
)
messy['population'] = pd.to_numeric(messy['population'], errors='coerce')
clean = messy.drop_duplicates().dropna(subset=['population', 'gdp']).copy()
print(f"{len(messy)} rows -> {len(clean)} rows after cleaning")
clean.info()
181 rows -> 169 rows after cleaning <class 'pandas.DataFrame'> Index: 169 entries, 0 to 176 Data columns (total 4 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 country 169 non-null str 1 continent 169 non-null str 2 population 169 non-null float64 3 gdp 169 non-null float64 dtypes: float64(2), str(2) memory usage: 6.6 KB
With clean now numeric, complete, and duplicate-free, a histogram is the natural first look at any single column — the univariate view the section title promises:
clean['population'].plot(kind='hist', bins=30)
plt.xlabel('Population')
plt.title('Distribution of country population')
plt.show()
The shape is heavily right-skewed: most countries cluster near zero on this axis, and a handful — India, China — stretch the x-axis so far that the rest become indistinguishable. This is extremely common with population, GDP, and most other quantities that scale with the size of a place, and it is precisely the kind of pattern a histogram reveals in one line that a table of summary statistics would hide. A log-scaled x-axis usually makes the shape far more legible:
clean['population'].plot(kind='hist', bins=30, logx=True)
plt.xlabel('Population (log scale)')
plt.title('Distribution of country population, log scale')
plt.show()
Bivariate Relationships¶
A histogram describes one column at a time. As soon as the question becomes "does GDP scale with population, and how tightly?", you need a bivariate view — one that plots two columns against each other. A scatter plot is the default choice:
fig, ax = plt.subplots(figsize=(6, 5))
ax.scatter(clean['population'], clean['gdp'], alpha=0.6)
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel('Population (log scale)')
ax.set_ylabel('GDP, million USD (log scale)')
ax.set_title('Population vs. GDP, by country')
plt.show()
On log-log axes, the relationship is visibly close to a straight line — larger populations tend to go with larger GDP, roughly proportionally, though with plenty of scatter around the trend (small, wealthy economies sit well above the line; large, lower-income ones sit below it). seaborn, built on top of matplotlib, adds a fitted trend line and a continent color grouping with very little extra code, which is often enough to spot whether a relationship holds uniformly or is really being driven by one region:
sns.scatterplot(
data=clean, x='population', y='gdp',
hue='continent', alpha=0.7
)
plt.xscale('log')
plt.yscale('log')
plt.title('Population vs. GDP, colored by continent')
plt.show()
Outlier Detection¶
Every plot above has points sitting visibly apart from the main cloud. The question EDA asks is not just "which points are extreme?" but "is this an error, or is it real?" — and the two demand opposite responses: an error gets fixed or dropped, a real outlier gets investigated and, usually, kept.
A common quantitative rule of thumb is the interquartile range (IQR) method: flag any value more than 1.5×IQR beyond the 25th or 75th percentile. seaborn's boxplot draws exactly this rule:
fig, ax = plt.subplots(figsize=(8, 3))
sns.boxplot(x=clean['gdp'], ax=ax)
ax.set_xlabel('GDP, million USD')
ax.set_title('GDP outliers by the 1.5x IQR rule')
plt.show()
q1, q3 = clean['gdp'].quantile([0.25, 0.75])
iqr = q3 - q1
upper_fence = q3 + 1.5 * iqr
outliers = clean.loc[clean['gdp'] > upper_fence, ['country', 'continent', 'gdp']]
outliers.sort_values('gdp', ascending=False)
| country | continent | gdp | |
|---|---|---|---|
| 4 | United States of America | North America | 21433226.0 |
| 155 | Japan | Asia | 5081769.0 |
| 121 | Germany | Europe | 3861123.0 |
| 98 | India | Asia | 2868929.0 |
| 143 | United Kingdom | Europe | 2829108.0 |
| 43 | France | Europe | 2715518.0 |
| 141 | Italy | Europe | 2003576.0 |
| 29 | Brazil | South America | 1839758.0 |
| 3 | Canada | North America | 1736425.0 |
| 18 | Russia | Europe | 1699876.0 |
| 96 | South Korea | Asia | 1646739.0 |
| 137 | Australia | Oceania | 1396567.0 |
| 132 | Spain | Europe | 1393490.0 |
| 27 | Mexico | North America | 1268870.0 |
| 140 | Taiwan | Asia | 1127000.0 |
| 8 | Indonesia | Asia | 1119190.0 |
| 130 | Netherlands | Europe | 907050.0 |
| 158 | Saudi Arabia | Asia | 792966.0 |
| 124 | Turkey | Asia | 761425.0 |
| 127 | Switzerland | Europe | 703082.0 |
By this rule, essentially every large economy — the United States, China, and similar — gets flagged as an "outlier." Mechanically, the rule is correct: these values genuinely sit far from the bulk of the distribution. But there is nothing wrong with the data here; the world's GDP really is dominated by a small number of very large economies, and dropping these rows would not clean the dataset, it would delete the most important countries in it. Contrast that with a value like a population of -1 or a GDP ten orders of magnitude larger than every other country: that would be a data-entry error worth fixing or removing.
The IQR rule is a starting point for where to look, not an automatic instruction for what to do. This is exactly why outlier detection belongs in an EDA chapter about looking rather than a cleaning chapter about deleting: the decision needs a human, and usually some domain knowledge, in the loop.
EDA Checklist¶
Before treating any dataset as ready for modelling — spatial or otherwise — it is worth running through the same short sequence every time, in this order:
- Shape and types.
.info()— right number of rows and columns? Are numeric columns actually numeric? - Missing values.
.isna().sum()— how many, and in which columns? Missing at random, or concentrated in a pattern (e.g. every small island territory)? - Duplicates.
.duplicated().sum()— exact duplicates are usually safe to drop; near-duplicates (same country, slightly different spelling) need a judgment call. - Univariate shape. A histogram per numeric column — skewed? Does a log scale help? Any impossible values (negative population, GDP of zero)?
- Bivariate relationships. A scatter plot of the pairs you plan to model together — does the relationship look linear, log-linear, or is there no visible relationship at all?
- Outliers. Flag the extremes, then decide, case by case, whether each is an error to fix or a real, important data point to keep.
None of these steps require geography specifically — they apply to any tabular dataset. Part II adds a second layer on top of this checklist once geometry enters the picture: invalid polygons, mismatched coordinate reference systems, and geometries that don't actually overlap the data you're trying to join them to. The habit of looking before modelling, though, is exactly the same one you are building here.
Exercises¶
- Re-run the pipeline. Starting from
world_data_messy.csv, reproduce the cleaning steps in Univariate Distributions on your own, then check thatclean.duplicated().sum()andclean.isna().sum()are both zero. - A different missingness strategy. Instead of
dropna(), try filling missingpopulationvalues with the median population of the samecontinent(hint:.groupby('continent')['population'].transform('median')). How does the histogram from Univariate Distributions change? - Your own bivariate plot. Using
clean, plot GDP per capita (gdp / population) against population. Does the relationship look different from raw GDP vs. population? - Defend an outlier decision. Pick one country flagged in Outlier Detection and write two sentences: one arguing it should be treated as an error, one arguing it should be kept as-is. Which argument do you find more convincing, and why?
# Your code here
Summary¶
Key concepts introduced¶
- Cleaning as a prerequisite to exploration: fixing dtypes (
pd.to_numeric(..., errors="coerce")), handling missing values, and dropping duplicates - Univariate exploration with histograms, including when a log scale clarifies a skewed distribution
- Bivariate exploration with scatter plots, and
seabornas a higher-level layer on top of matplotlib - The IQR rule for flagging outliers mechanically — and why flagging is not the same decision as removing
- A repeatable EDA checklist to run before any modelling step, spatial or otherwise
This closes out the non-spatial half of Part I. Chapter 5 turns to software engineering practices — testing, packaging, version control — that keep an analysis like this one reproducible; Part II then reopens every step of this chapter's checklist with geopandas, where a "row" is a geometry as much as it is a set of numbers.
Further Reading¶
- John W. Tukey, Exploratory Data Analysis (Addison-Wesley, 1977) — the foundational text that gives this chapter its name and its philosophy: look at the data before you model it
- Sergio J. Rey, Dani Arribas-Bel, and Levi J. Wolf, Geographic Data Science with Python (CRC Press) — freely available at https://geographicdata.science/book/. Their "Datasets" appendix keeps every cleaning step in the open, in its own notebook per dataset, which is the same spirit behind the separate
world_data_messy.csvused here; see in particular their treatment of a countries dataset at https://geographicdata.science/book/data/countries/countries_cleaning.html, which picks the story back up once geometry — not just attributes — needs cleaning too - seaborn documentation, An introduction to seaborn — https://seaborn.pydata.org/tutorial/introduction.html