Chapter 28: Reproducibility and Experiment Provenance¶
Part VI — Data & Infrastructure
Learning Objectives¶
By the end of this chapter you will be able to:
- Explain what "provenance" means for a simulation result, beyond just saving the output
- Read every field of an
ExperimentRecordand know what each one is for - Register a
ModelExecutorand discover it later by name - Validate an executor automatically with
ExecutorTestHarness, before ever running a real experiment
# Standard imports — add chapter-specific imports below
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
Chapter 22 built one complete ModelExecutor and ran it once. This chapter asks what happens after that one run — six months later, when a reviewer asks "are you sure this is the same input that produced Figure 3," or a collaborator asks "can I rerun exactly what you ran." Chapter 22's checksums were the first piece of the answer; this chapter is the rest of it.
Why Provenance Matters¶
A result without provenance is a claim, not evidence. "I ran a fire model and got this map" is a claim; "I ran forest_fire version a3f9c1e, on input forest.gpkg with SHA-256 eba1aa64..., with initial_fire_density=0.1, and got output with SHA-256 3335784f..." is evidence — every piece of it independently checkable by someone who wasn't in the room when it ran.
Three failure modes provenance specifically guards against, none of them requiring anyone to have acted in bad faith:
- Silent input drift. A shared input file gets edited — a bug fix, a data correction — after an experiment ran against it. Without a checksum, nothing distinguishes "the exact file that produced this result" from "a file that looks similar now."
- Silent code drift. The model's own logic changes between when a result was produced and when someone tries to reproduce it. Without recording which version of the code ran, "rerun the experiment" silently means "run a different experiment that happens to share a name."
- Parameter amnesia. Six months on, even the person who ran the experiment often can't recall the exact parameter values used — which
seed, whichinitial_fire_density— from memory alone.
ExperimentRecord in Depth¶
Chapter 22 touched three or four of ExperimentRecord's fields in passing. Here's the complete picture — every field exists to close one of the three gaps above:
from dissmodel.executor import ExperimentRecord, DataSource
print("ExperimentRecord fields:")
for name in ExperimentRecord.model_fields:
print(" -", name)
ExperimentRecord fields: - experiment_id - created_at - model_name - model_commit - code_version - resolved_spec - source - input_format - column_map - band_map - parameters - period - output_path - artifacts - metrics - status - logs
Grouped by what they're for:
| Field(s) | Closes which gap |
|---|---|
experiment_id, created_at |
identity — which run is this, and when did it happen |
model_name, model_commit, code_version |
code drift — which model, which exact version of its code |
resolved_spec |
the fully-resolved configuration actually used, defaults and all — not just what the caller explicitly passed |
source (a DataSource: type, uri, collection, version, checksum) |
input drift — exactly which file, and proof via checksum |
input_format, column_map, band_map |
how the raw input was interpreted — which column meant what |
parameters, period |
parameter amnesia — every value the model actually ran with |
output_path, output_sha256, artifacts, metrics |
the result itself, and proof it's the exact bytes save() produced |
status, logs |
did it actually complete, and what happened along the way |
model_commit deserves a second look: pinning a commit hash, not just a version string like "1.2.0", means "run against exactly this code" survives even a model package that hasn't cut a formal release yet — exactly the situation every extension package since Chapter 22 has been in, installed straight from a GitHub branch rather than a PyPI release.
Checksums as Proof, Not Just Metadata¶
A checksum only does its job if it's computed at the moment of use, not trusted from a filename or a timestamp. Chapter 22's load() already did this correctly — load_dataset() hashes the file's actual bytes as it reads them, not the string "forest.gpkg":
from dissmodel.io import load_dataset, save_dataset
from dissmodel.geo import vector_grid
gdf = vector_grid(dimension=(10, 10), resolution=1, attrs={"state": 0})
save_dataset(gdf, "checksum_demo.gpkg")
_, checksum_before = load_dataset("checksum_demo.gpkg")
print("Checksum, unchanged file:", checksum_before[:16], "...")
# Now change the file — flip a single cell's state and re-save
gdf.loc[0, "state"] = 1
save_dataset(gdf, "checksum_demo.gpkg")
_, checksum_after = load_dataset("checksum_demo.gpkg")
print("Checksum, one cell changed:", checksum_after[:16], "...")
print("Same file?", checksum_before == checksum_after)
Checksum, unchanged file: 20cacd2ee0588e0a ... Checksum, one cell changed: 85134edc977001b3 ... Same file? False
/home/sergio/develop/github/lambdageo/ebooks/geospatial-modeling-python/.venv/lib/python3.12/site-packages/pyogrio/geopandas.py:948: UserWarning: 'crs' was not provided. The output dataset will not have projection information defined and may not be usable in other systems. write( /home/sergio/develop/github/lambdageo/ebooks/geospatial-modeling-python/.venv/lib/python3.12/site-packages/pyogrio/raw.py:200: RuntimeWarning: File /vsimem/pyogrio_dff57c104bf5491ebdcd69bcaee4220c has GPKG application_id, but non conformant file extension return ogr_read( /home/sergio/develop/github/lambdageo/ebooks/geospatial-modeling-python/.venv/lib/python3.12/site-packages/pyogrio/geopandas.py:948: UserWarning: 'crs' was not provided. The output dataset will not have projection information defined and may not be usable in other systems. write( /home/sergio/develop/github/lambdageo/ebooks/geospatial-modeling-python/.venv/lib/python3.12/site-packages/pyogrio/raw.py:200: RuntimeWarning: File /vsimem/pyogrio_0239e32cc1514b41be9114b85419f4df has GPKG application_id, but non conformant file extension return ogr_read(
One value in one cell, and the checksum changes completely — that's the point. record.source.checksum, recorded once at load() time and never recomputed silently later, is what lets someone six months from now confirm "yes, this really is the file the experiment ran against," rather than a file that merely shares its name.
# A sample input for the executor examples below
sample_gdf = vector_grid(dimension=(15, 15), resolution=1, attrs={"state": 0})
save_dataset(sample_gdf, "forest.gpkg")
/home/sergio/develop/github/lambdageo/ebooks/geospatial-modeling-python/.venv/lib/python3.12/site-packages/pyogrio/geopandas.py:948: UserWarning: 'crs' was not provided. The output dataset will not have projection information defined and may not be usable in other systems. write(
'b59288f36b4f2a6a5141014a9a65e49d1f1b5375fc49105e2a4bed4a8a341bf3'
Registering and Discovering Executors¶
A real project accumulates more than one ModelExecutor — the fire model from Chapter 22, whatever Chapter 26's LUCC models eventually add, an ensemble driver from Chapter 31. ExecutorRegistry gives every one of them a name, so calling code can look one up without importing its class directly:
from dissmodel.executor import ModelExecutor, ExecutorRegistry
class ForestFireExecutor(ModelExecutor):
name = "forest_fire"
def load(self, record):
gdf, checksum = load_dataset(record.source.uri)
record.source.checksum = checksum
return gdf
def run(self, data, record):
from dissmodel.core import Environment
from dissmodel_ca.models import FireModel
env = Environment(start_time=0, end_time=record.parameters.get("end_time", 20))
FireModel(
gdf=data,
initial_fire_density=record.parameters.get("initial_fire_density", 0.05),
seed=record.parameters.get("seed", 42),
)
env.run()
return data
def save(self, result, record):
uri = record.output_path or "output.gpkg"
record.output_path = uri
record.output_sha256 = save_dataset(result, uri)
record.status = "completed"
return record
ExecutorRegistry.register(ForestFireExecutor)
print("Registered:", ExecutorRegistry.list())
retrieved_cls = ExecutorRegistry.get("forest_fire")
print("Looked up by name:", retrieved_cls)
Registered: ['forest_fire'] Looked up by name: <class '__main__.ForestFireExecutor'>
This is the same registration pattern run_cli() (Chapter 22) relies on internally to route a --model forest_fire command-line flag to the right class — a lookup by string name, rather than every caller needing to know and import every executor class directly.
Testing Executors¶
An executor is code, and code that's never been tested is code you don't actually know works. ExecutorTestHarness runs two distinct checks, deliberately kept separate:
from dissmodel.executor.testing import ExecutorTestHarness
harness = ExecutorTestHarness(ForestFireExecutor)
passed = harness.run_contract_tests()
print("\nContract tests passed:", passed)
ExecutorTestHarness — ForestFireExecutor ──────────────────────────────────────────────────── ✅ name attribute exists ✅ name is a non-empty string ✅ name has no whitespace ✅ load() is implemented ✅ run() is implemented ✅ save() is implemented ✅ run() signature is correct ✅ save() signature is correct ──────────────────────────────────────────────────── All 8 checks passed ✅ Contract tests passed: True
Contract tests check the executor's shape, not its behavior: does it declare a name, does it implement load/run/save, do the method signatures match what ModelExecutor expects. None of this runs a single tick of simulation — it's the equivalent of Chapter 5's pytest checking that a function exists and is callable, before ever checking what it computes.
A sample-data cycle goes further: it actually runs load → run → save end to end, against a real ExperimentRecord, and confirms the result looks like a completed experiment should — status == "completed", an output_sha256 actually set:
from dissmodel.executor import DataSource
record = ExperimentRecord(
experiment_id="harness_demo",
model_name="forest_fire",
source=DataSource(type="file", uri="forest.gpkg"),
parameters={"initial_fire_density": 0.1, "seed": 7, "end_time": 5},
)
cycle_passed = harness.run_with_sample_data(record)
print("\nSample-data cycle passed:", cycle_passed)
▶ Running forest_fire... validate()... load()... run()... Running from 0 to 5 (duration: 5)
/home/sergio/develop/github/lambdageo/ebooks/geospatial-modeling-python/.venv/lib/python3.12/site-packages/pyogrio/raw.py:200: RuntimeWarning: File /vsimem/pyogrio_affeef2a7dd44ebeae47aa7a93143e31 has GPKG application_id, but non conformant file extension return ogr_read(
save()... ✅ Cycle OK — status=completed sha256=daec616077ab... Sample-data cycle passed: True
/home/sergio/develop/github/lambdageo/ebooks/geospatial-modeling-python/.venv/lib/python3.12/site-packages/pyogrio/geopandas.py:948: UserWarning: 'crs' was not provided. The output dataset will not have projection information defined and may not be usable in other systems. write(
Both checks belong in a project's own test suite — the same pytest-based suite Chapter 5 first introduced — run automatically before any executor is trusted with a real experiment, not eyeballed once by hand and forgotten.
Exercises¶
- Break it on purpose. Remove the
save()method from a copy ofForestFireExecutorand rerunrun_contract_tests(). Which specific check fails, and does the failure message tell you what's missing without you having to guess? - Trace a field to its gap. For
model_commitspecifically, write one sentence describing a real scenario where having it recorded — versus not — would have mattered to someone trying to reproduce a result. - A second executor. Register a second, trivial
ModelExecutor(it can even reuseForestFireExecutor's logic under a differentname) and confirmExecutorRegistry.list()now returns both names. - Checksum a real change. Using the Checksums as Proof pattern, load any
GeoDataFramefrom earlier chapters, checksum it, add a single new column, re-save, and checksum again. Confirm the two checksums differ — and explain, in a sentence, why a checksum can prove a file changed but can't by itself prove how.
# Your code here
Summary¶
Key concepts introduced¶
- Provenance as evidence, not a claim: three specific failure modes (input drift, code drift, parameter amnesia) it guards against
- Every
ExperimentRecordfield, grouped by which failure mode it closes — includingmodel_commitpinning an exact commit rather than a version string - Checksums computed from actual file bytes at the moment of use, proven to change the instant a single value does
ExecutorRegistry, giving everyModelExecutora string name thatrun_cli()and other calling code can look up without a direct importExecutorTestHarness's two-layer check: contract tests (does the executor have the right shape) and a sample-data cycle (does a real run actually complete and produce a checksummed output)
Chapter 29 puts a registered, tested executor to work at scale — submitting it as a job to the DisSModel Platform instead of running it by hand.
Further Reading¶
- Wilkinson, M. D. et al. (2016). "The FAIR Guiding Principles for scientific data management and stewardship." Scientific Data, 3, 160018 — the FAIR principles this chapter's provenance discipline is built to satisfy
- Sandve, G. K. et al. (2013). "Ten Simple Rules for Reproducible Computational Research." PLOS Computational Biology, 9(10)
- DisSModel executor API reference: https://dissmodel.github.io/dissmodel/