Chapter 5: Software Engineering for Scientific Python¶

Part I — Scientific Python for Researchers

Learning Objectives¶

By the end of this chapter you will be able to:

Track a small project's history with Git and push it to GitHub
Separate reusable code from one-off exploration, and test the reusable part with pytest
Build a reproducible environment with venv and requirements.txt
Explain why a script-and-tests layout scales better than a single notebook, and when a notebook is still the right tool

In [1]:

Copied!





# Standard imports — add chapter-specific imports below
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# Standard imports — add chapter-specific imports below
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

Every chapter so far has run inside a notebook that already existed, in this book's own repository, with the data already sitting next to it. This chapter asks you to leave that sandbox: create an empty folder on your own machine, and build a small, real project in it — the same shape of project every package in the DisSModel ecosystem this book builds toward is made of, right down to the venv + pip + pytest combination its own contributing guide uses.

You will need three things installed and working in a terminal: Python 3.10+, git, and a GitHub account. On Linux and macOS, your regular terminal is exactly what you need. On Windows, the commands below run identically inside WSL; if you'd rather stay on native Windows, Git Bash (installed alongside Git for Windows) understands the same source-style commands too — with one small difference for venv, flagged when it comes up below.

Version Control with Git¶

Git tracks the history of a folder as a sequence of commits — named snapshots you can always return to. That alone is worth adopting for any analysis: "what did this script look like before I changed it" stops being a question you answer by digging through email attachments or files named analysis_final_v3.py.

Start by creating the project folder and turning it into a Git repository:

mkdir geo-book-project
cd geo-book-project
git init

git init creates a hidden .git/ folder — that's the entire repository, nothing more. Nothing is tracked yet; git status at this point reports an empty repo with no commits. Let's give it something to track:

echo "# Geo Book Project" > README.md
git status

git status now reports README.md as an untracked file. Git noticed it exists, but isn't watching it yet. Two commands move it into history:

git add README.md
git commit -m "Initial commit"

git add moves a file into the staging area — a holding pen for "changes I'm about to commit." git commit then takes everything staged and seals it into a permanent snapshot, labeled with the message you gave it. This two-step add-then-commit process is deliberate: it lets you build a commit out of exactly the changes that belong together, even if you've been editing five files at once.

Did you know?

A good commit message describes why, not just what — "Initial commit" is a fine exception since a first commit is self-explanatory, but "Fix off-by-one in top-N filter" tells a future reader (often you, in six months) something the diff alone won't.

Before adding any real code, tell Git what to permanently ignore. A Python project accumulates files that should never be committed: the virtual environment you'll create later in this chapter, compiled bytecode caches, and editor-specific clutter. Create a .gitignore file with:

venv/
__pycache__/
*.pyc
.pytest_cache/
.ipynb_checkpoints/

Commit it the same way as before — git add .gitignore && git commit -m "Add .gitignore" — and every file matching these patterns will be invisible to git status and git add . from now on, no matter how many times you regenerate them.

GitHub Workflows for Research¶

A Git repository so far lives only on your machine. GitHub hosts a copy remotely, which gets you two things a local-only repo doesn't have: a backup that survives a dead laptop, and a URL you can hand to a collaborator — or cite in a paper.

Create a new, empty repository on GitHub (no README, no .gitignore — you already have both locally, and GitHub initializing its own would conflict with yours on the first push). GitHub will show you a remote URL; connect your local repo to it and push:

git remote add origin https://github.com/<your-username>/geo-book-project.git
git branch -M main
git push -u origin main

git remote add registers the GitHub URL under the short name origin. -u on the first push sets origin main as the default target, so every push after this one is just git push.

Watch out

Never git add a data file containing personal information, credentials, or an API key. Once committed, it stays in the repository's history even after you delete it in a later commit — removing it properly requires rewriting history, which is far more disruptive than not committing it in the first place.

A commit message convention worth adopting early: Conventional Commits prefixes each message with the kind of change it makes — feat: for a new capability, fix: for a bug fix, docs: for documentation only, test: for tests. git log --oneline on a repository that follows this convention reads almost like a changelog for free.

Writing Tests with pytest¶

So far, README.md and .gitignore are the only tracked files — nothing to actually run yet. Copy world_data.csv from this book's repository into a new data/ folder inside your project, then create analysis.py:

# analysis.py
import pandas as pd


def compute_gdp_per_capita(path="data/world_data.csv"):
    df = pd.read_csv(path)
    df["gdp_per_capita"] = df["gdp"] * 1000 / df["population"]
    return df


def top_n_by_gdp_per_capita(df, n=5):
    return df.sort_values("gdp_per_capita", ascending=False).head(n)


if __name__ == "__main__":
    df = compute_gdp_per_capita()
    print(top_n_by_gdp_per_capita(df))

This is exactly the kind of pandas code Chapter 3 taught — nothing new there. What is new is the shape: two small functions instead of one long script, each doing one thing, callable independently of whether the file is run directly (if __name__ == "__main__": guards the part that only makes sense as a script).

Before writing a test, it's worth confirming what the function actually returns — this book's own copy of world_data.csv is a stand-in for the one you'd copy into data/:

In [2]:

Copied!





df = pd.read_csv("world_data.csv")
df["gdp_per_capita"] = df["gdp"] * 1000 / df["population"]
print(len(df), "rows")
df.sort_values("gdp_per_capita", ascending=False).head(5)[["country", "population", "gdp_per_capita"]]
df = pd.read_csv("world_data.csv")
df["gdp_per_capita"] = df["gdp"] * 1000 / df["population"]
print(len(df), "rows")
df.sort_values("gdp_per_capita", ascending=False).head(5)[["country", "population", "gdp_per_capita"]]

177 rows

Out[2]:

	country	population	gdp_per_capita
159	Antarctica	4490.0	200.000000
128	Luxembourg	619896.0	114.703111
23	Fr. S. Antarctic Lands	140.0	114.285714
20	Falkland Is.	3398.0	82.989994
127	Switzerland	8574832.0	81.993676

Notice the top five are dominated by tiny territories rather than the economies you'd probably guess — the same pattern Chapter 4's Outlier Detection section warned about: a population of a few thousand makes almost any GDP look enormous per capita. That's not a bug in analysis.py; it's a real, expected property of the data, and precisely the kind of thing a test should pin down rather than silently assume.

A test doesn't need to be complicated to be useful. test_compute_gdp_per_capita_adds_column below checks a structural property (the new column exists), and test_top_n_returns_requested_rows checks a behavioral one (asking for 3 rows gives you 3 rows) — together they'd catch a surprising number of real mistakes, like a typo in a column name or an off-by-one in .head():

# tests/test_analysis.py
from analysis import compute_gdp_per_capita, top_n_by_gdp_per_capita


def test_compute_gdp_per_capita_adds_column():
    df = compute_gdp_per_capita()
    assert "gdp_per_capita" in df.columns
    assert len(df) == 177


def test_top_n_returns_requested_rows():
    df = compute_gdp_per_capita()
    top = top_n_by_gdp_per_capita(df, n=3)
    assert len(top) == 3

Run both with:

pytest tests/ -v

pytest discovers any file matching test_*.py, runs every function inside it prefixed with test_, and reports pass/fail per function — no test runner boilerplate, no class hierarchy to inherit from. An assert that fails is enough to fail the test; nothing more ceremonial is required.

Reproducible Environments¶

Two collaborators — or you, six months from now, on a different laptop — need the exact same package versions to reproduce a result reliably. venv, part of the Python standard library, creates an isolated environment for exactly this purpose:

python -m venv venv

Activating it puts that isolated Python first on your PATH, so pip install and python both refer to the isolated copy instead of your system-wide one:

Did you know?

On Linux, macOS, and inside WSL, activation is identical: source venv/bin/activate. On native Windows through Git Bash, the same source command works, but the folder venv creates is named Scripts/ instead of bin/, since Git Bash runs a Python built for Windows underneath its Unix-like shell: source venv/Scripts/activate. (PowerShell and cmd use a different activation file entirely, Scripts\Activate.ps1 or Scripts\activate.bat — not covered here, since this book assumes WSL or Git Bash on Windows.)

With the environment active — your terminal prompt now shows (venv) — install exactly what the project needs, then freeze those versions to a file:

pip install pandas pytest
pip freeze > requirements.txt

requirements.txt now pins the specific version of every installed package. Commit it, and anyone — including future you — reproduces the same environment with three commands:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Commit analysis.py, tests/test_analysis.py, data/world_data.csv, and requirements.txt the same way as README.md earlier — git add, then git commit -m "feat: add GDP per capita analysis with tests" — and push. Your project now lives on GitHub, in full, reproducible by a stranger who has never seen your machine.

Software Design Principles for Scientists¶

It would have been possible to do everything in this chapter inside a single Jupyter notebook — load the data, compute the ratio, print the top five, all in one cell. Three things would have been lost by doing that, and all three are reasons this book itself, notebook-based as it is, still keeps a companion repository (dissmodel) built entirely as ordinary .py files:

Reuse. A function in analysis.py can be imported by another script, or by next year's version of this same analysis, without copy-pasting notebook cells. A notebook has no import-able functions unless you extract them into a module anyway.
Testability. pytest can call compute_gdp_per_capita() directly and check its output. It cannot meaningfully "call" a notebook cell in isolation — testing notebook code almost always means testing it indirectly, after first pulling it out into a function, which is exactly what analysis.py already is.
Diff-ability. A .py file's Git history is a readable, line-by-line diff. A .ipynb file is JSON underneath, and a one-line code change can produce a diff full of unrelated metadata and cell-output noise — a real cost, and one this very book accepts deliberately, in exchange for the didactic benefit of prose and code living side by side for a reader.

None of this means notebooks are the wrong tool — Chapters 1 through 4 of this book, and most exploratory work, are exactly where a notebook's fast, visual, iterative loop earns its keep. The judgment call this chapter is really teaching is when to graduate a piece of exploratory code into a tested, reusable script: usually the moment you find yourself about to copy-paste a cell into a second notebook.

Exercises¶

Build it for real. Work through every command in this chapter in an actual terminal, on an actual empty folder, ending with a real push to GitHub. Confirm pytest tests/ -v shows two passing tests.
A third test. Add a test that checks every value in the gdp_per_capita column is positive (hint: (df["gdp_per_capita"] > 0).all()). Does it pass on the real dataset? Should it?
Extend .gitignore. Add a rule that ignores any file ending in .log, then create an empty debug.log and confirm git status no longer reports it.
A second function. Add bottom_n_by_gdp_per_capita() to analysis.py, alongside a test for it, and commit the change with a Conventional Commits message starting with feat:.

Summary¶

Key concepts introduced¶

Git's add-then-commit workflow, and .gitignore for what should never be tracked
Pushing a local repository to GitHub with git remote add and git push, and Conventional Commits as a message convention
pytest, and the difference between a structural test and a behavioral one
venv + requirements.txt as the minimum needed for another machine to reproduce your environment exactly
Script-plus-tests as a layout that trades notebook convenience for reuse, testability, and clean diffs — a trade-off worth making once exploratory code needs to be trusted again later

Every one of these practices carries forward unchanged once Part II swaps pandas for geopandas and the data grows geometry — nothing about testing a spatial join or pinning a requirements.txt for geopandas is any different from what you just did with pandas. And if you're curious what this same discipline looks like on a real research codebase, dissmodel's own contributing guide follows this identical venv + pytest pattern — you'll meet the framework itself properly in Chapter 21.

Optional: a taste of Streamlit

If tests/ is green and the repo is pushed, here is a small reward. Add streamlit to your environment (pip install streamlit), save this as app.py, and run streamlit run app.py:

# app.py
import streamlit as st
from analysis import compute_gdp_per_capita, top_n_by_gdp_per_capita

st.title("GDP per Capita Explorer")
n = st.slider("How many countries?", 3, 20, 5)
df = compute_gdp_per_capita()
st.dataframe(top_n_by_gdp_per_capita(df, n=n))

Eight lines turn analysis.py into an interactive table with a live slider, in a browser tab, no HTML or JavaScript involved. This is exactly the pattern the dissmodel-ca package uses to turn every one of its cellular automaton models into an explorable app (examples/streamlit/ca_all.py) — Chapter 21 puts it to real use.