Chapter 30: Spatial Data Cubes¶
Part VI — Data & Infrastructure
Implemented by the disscube package.
Watch out
disscube is Alpha software, and this chapter deliberately uses a small synthetic raster instead of a real Earth-observation source like the Brazil Data Cube — no BDC-backed pipeline has been run for this book yet. Every code cell below was tested against that synthetic data; the underlying mechanics (grid registration, derivation, the bridge into a simulation) are identical either way.
Learning Objectives¶
By the end of this chapter you will be able to:
- Explain why a data cube is a semantic guarantee, not a file format
- Understand why the ecosystem needs two cube implementations — one at rest, one in motion
- Register a grid, register a source, and derive a variable
- Bridge a derived variable into a running DisSModel simulation
# Standard imports — add chapter-specific imports below
import numpy as np
import matplotlib.pyplot as plt
Chapter 29's platform moves a ModelExecutor from your machine to a shared service. This chapter is about what feeds that executor in the first place — the raw rasters, vectors, and satellite scenes a real model needs prepared, aligned, and ready before a single tick runs. disscube is the piece of the DisSModel ecosystem responsible for that preparation.
What Is a Data Cube¶
In the Earth-observation literature, a data cube is a regular multidimensional array — typically (variable, time, y, x) — with three properties an arbitrary folder of files doesn't have:
- Spatial alignment. Every variable, every date, shares the same grid — same CRS, resolution, and origin. Cell
(i, j)is the same physical location no matter which variable or which date you're looking at. - An explicit, indexable time axis. Time is a dimension you can query, not a filename suffix.
- Analysis-ready. The cost of reprojecting and aligning data is paid once, at ingestion — not recomputed on every single query.
A common misconception is that a cube has to be one giant file. It doesn't — a cube can be virtual (nothing computed yet, the array a promise resolved only when queried) just as legitimately as materialized (pixels already computed and written to disk). What makes something a data cube is the guarantee — alignment plus indexability — not any particular storage format.
Two Regimes: The Cube at Rest, the Cube in Motion¶
The DisSModel ecosystem doesn't build one cube implementation — it builds two, on purpose, because a cube gets used for genuinely different things depending on where it sits in a pipeline:
disscube (rest) |
RasterBackend (motion) |
|
|---|---|---|
| Role | cube infrastructure | one cube instance |
| Persistence | yes — Zarr + a catalog | no — lives in process memory |
| Scope | a family of cubes, multi-grid | one cube, one grid, one run |
| Typical consumer | analysis, cataloging | a running Environment (Chapter 21) |
Both satisfy the three-property definition above — RasterBackend enforces alignment by construction and exposes an indexable time axis, exactly like disscube's own cubes do. The difference that actually matters is lazy versus materialized. A lazy cube (xarray backed by Dask) is ideal for analysis: chain several operations, and only move real bytes once, at the very end, optimized as a whole. A cellular automaton can't work that way — it needs the concrete value at step t to decide the transition to t+1; a lazy chain run through a CA would rebuild its entire pending-operation graph every single tick, recomputing history it already paid for once. RasterBackend is deliberately materialized for exactly this reason. Neither is the "better" substrate in general — each fits the regime it was built for, and the boundary between them is a single .compute() call, the exact moment a cube crosses from rest into motion.
disscube: The Cube at Rest¶
disscube turns raw sources into named, grid-aligned, cataloged variables through one consistent pipeline:
SpatialSource → Derivation → Variable → DerivedVariable (Zarr)
GridSpecis the alignment contract itself — an ID, a CRS, a resolution, a bounding box. Every variable derived on the sameGridSpecis comparable cell-by-cell by construction; a variable derived on a different grid is a different cube, full stop, even if it covers the same physical area.SpatialSourceis a raw input registered in the catalog — local file, HTTP, or S3 — with its own CRS, bounding box, and atimevalue.Derivationis a declarative recipe: which operator, applied to which source, produces which named target variable. ADerivationis grid-independent — the identical recipe can run against a 1km grid or a 5km grid, producing two distinct, separately-cataloged cubes.DerivedVariableis what a recipe actually produces on a specific grid — a Zarr file on disk, plus aspec_hash: a deterministic hash of the recipe and the grid together, uniquely identifying that variable. Ask for the same derivation on the same grid twice, and the catalog returns the cached result instead of recomputing it — reproducibility built into the architecture, not left to a modeler's notes.
Basic Flow: Registering a Grid and Deriving a Variable¶
Three steps, working entirely from a small synthetic raster — a stand-in for real Earth-observation data, chosen specifically so this section doesn't depend on the Brazil Data Cube or any other external source:
import rasterio
from rasterio.transform import from_origin
# A synthetic land-cover-style raster: 50x50 cells, 5 classes (0-4), near São Luís
np.random.seed(0)
synthetic_data = np.random.randint(0, 5, size=(50, 50)).astype(np.uint8)
transform = from_origin(-44.5, -2.0, 0.01, 0.01)
with rasterio.open(
"synthetic_landcover.tif", "w",
driver="GTiff", height=50, width=50, count=1,
dtype=synthetic_data.dtype, crs="EPSG:4326", transform=transform,
) as dst:
dst.write(synthetic_data, 1)
plt.imshow(synthetic_data, cmap="tab10")
plt.title("Synthetic land cover (5 classes)")
plt.colorbar(label="class")
plt.show()
from disscube.client import CubeClient
from disscube.utils.grids import register_local_grid
from disscube.models import SpatialSource
from disscube.derivation import Derivation
cube = CubeClient(catalog="catalog.db", store="./data/")
grid = register_local_grid(
cube, name="DEMO",
bbox_geo=(-44.5, -2.5, -44.0, -2.0),
resolution=1000.0,
)
print("Grid registered:", grid.id)
cube.register_spatial_source(SpatialSource(
id="synthetic_lc_2024",
name="Synthetic land cover",
format="raster",
asset_url="synthetic_landcover.tif",
crs="EPSG:4326",
time=2024,
))
derivation = Derivation(
target="class3_pct", source_id="synthetic_lc_2024",
operator="percentage", class_code=3, role="driver",
valid_from="2024", valid_until="2024",
)
derived = cube.derive_declarative(derivation, grid_id=grid.id)
print("Derived:", derived[0].name, "| spec_hash:", derived[0].spec_hash[:12], "...")
The percentage operator answers "what fraction of each output cell is covered by class 3," resampling from the fine synthetic raster onto the coarser DEMO/1km grid — the exact aggregation problem Chapter 16's raster-vector integration touched from the other direction. Every other operator (mean, majority, min_distance, and others) follows the identical Derivation shape, just with a different operator name and its own relevant parameters.
loaded = cube.load("class3_pct", grid_id=grid.id)
print(type(loaded), loaded.dims, loaded.shape)
loaded.isel(time=0).plot(cmap="Greens")
plt.title("class3_pct — derived and reloaded from the catalog")
plt.show()
cube.load() returns a plain xr.DataArray — reloaded from the catalog, not recomputed, since the spec_hash from the previous cell already matched an existing entry the moment derive_declarative() ran a second time (try rerunning the derivation cell above and compare the spec_hash — identical, every time).
Bridging to a Running Simulation¶
A DataArray is the cube at rest. Chapter 21's RasterModel needs the cube in motion — a RasterBackend. CubeClient.to_lucc_data() is the dedicated bridge between the two, loading one or more named variables at once and assembling them directly into that substrate:
backend = cube.to_lucc_data(["class3_pct"], grid_id=grid.id)
print(type(backend))
print("class3_pct shape:", backend.get("class3_pct").shape)
That backend is the exact RasterBackend type Chapter 21's RasterModel and Chapter 24's FireModelRaster both construct against — nothing downstream needs to know its data originated from disscube's catalog rather than a hand-built raster_grid() call. RasterBackend.from_xarray() is the second, independent door into the same motion regime: any xr.Dataset with the right dimensions and named variables crosses into a simulation through it too, regardless of whether disscube produced it or another tool — like odc-stac pulling directly from a STAC catalog such as the Brazil Data Cube — did. Both doors lead to the identical substrate; a model never has to know, or care, which one was used.
Multiple Cubes and Coupling¶
Three practical questions come up as soon as more than one grid enters a project:
- Does a different grid mean a different cube? Yes — the grid is a cube's identity. The same
Derivationrun on two different grids produces two separately-cataloged results, each with its ownspec_hash. - Does one model use only one cube? One model instance does, since its transition rule assumes every driver variable is aligned to the same grid its own state lives on. An
Environment(Chapter 21) can still coordinate several models at once, each on its own grid — Chapter 25's two-kind agent models are one example of several models sharing one substrate; nothing stops two entirely different models from running on two different grids in the sameEnvironment. - Do two grids in play always need runtime coupling? Not if the crossing happens before the simulation starts — resampling one variable onto a different grid ahead of time is just another
Derivation, with the model itself never seeing more than one grid at once. Real runtime coupling — two models on different grids exchanging state every tick — is a harder, still-open problem in the ecosystem today, one Chapter 33's architecture chapter returns to as an area of active work rather than a solved one.
Exercises¶
- A second derivation. Using the synthetic raster from Basic Flow, derive a second variable with the
majorityoperator instead ofpercentage(noclass_codeneeded formajority). Compare itsspec_hashtoclass3_pct's — confirm they differ, since they're different recipes on the same grid. - Cache, proven. Rerun the exact same
Derivationfrom Basic Flow a second time and time both calls withtime.perf_counter(). Is the second call meaningfully faster? What does that tell you about whatspec_hashactually buys you? - A finer grid. Register a second grid at
resolution=500.0instead of1000.0, covering the samebbox_geo, and rederiveclass3_pcton it. Do the two derived variables'spec_hashvalues collide, or differ? Why must they differ, structurally? - Rest or motion? For each of the following, decide whether it belongs to the "rest" regime or the "motion" regime, and justify it in one sentence: (a) computing a five-year average of a derived variable, (b) a fire model reading a wind-speed raster every tick, (c) resampling a vector road layer onto a new grid before a run starts, (d) two coastal models exchanging flooded-area state every tick on different grids.
# Your code here
Summary¶
Key concepts introduced¶
- A data cube as a semantic guarantee — spatial alignment, an indexable time axis, analysis-ready — not a specific file format or storage layout
- Two deliberate implementations for two regimes:
disscubeat rest (cataloged, persistent, multi-grid) andRasterBackendin motion (materialized, mutable, one grid per running model) - The
SpatialSource → Derivation → Variable → DerivedVariablepipeline, andspec_hashmaking reproducibility architectural rather than a matter of documentation discipline to_lucc_data()andRasterBackend.from_xarray()as two independent doors into the same motion-regime substrate — one tied todisscube's own catalog, one open to any correctly-shapedxarray.Dataset, including one sourced from the Brazil Data Cube- The grid as a cube's identity: a different grid is always a different cube, whether crossed statically (a
Derivation, before a run starts) or dynamically (real runtime coupling, still an open problem in the ecosystem)
Chapter 31 leaves infrastructure behind and returns to modeling directly — running the same model many times over, systematically, for ensemble scenarios and sensitivity analysis.
Further Reading¶
- disscube on GitHub: https://github.com/DisSModel/disscube
- Open Data Cube project — the reference implementation this chapter's "alignment plus indexability" definition draws on: https://www.opendatacube.org/
- Brazil Data Cube: https://brazildatacube.org/
- Zarr documentation — the chunked, compressed array format
disscubepersists derived variables in: https://zarr.readthedocs.io/