The Analysis Graph

Tidemark computes its analysis as steps, each a Python function that returns one result. A step's arguments name the earlier results it needs, and Apache Hamilton reads those names to work out which step has to run before which. Nothing in the project writes that order down, so a new step joins the analysis by declaring the function and naming what it reads.

Declaring a Step

A step is an ordinary function in the subpackage of whatever it produces, so the step that separates the gauges sits beside the catalog it reads. tidemark run computes every step the graph declares, reading a result back wherever Each Step's Key finds it kept. Naming one on the command line computes that step and whatever it reads:

tidemark run targets

Hamilton's own glossary, like the package's docstrings, calls a step a node, the steps and the order among them a graph, and the object that runs them a driver, which tidemark run builds. The drawing below is read from those functions, so it holds exactly the package's steps.

$tidemark run

an entry

an object

a table

came from

ends up in

Anchors and Targets

The station catalog holds two kinds of gauge, the anchors and the targets, and its anchor column says which is which. The graph's first steps read the catalog the package ships, and two of them narrow it to each kind, anchors to the anchors and targets to the targets. The comparison holds an anchor's vertical reference rather than estimating it, and judges a target that has a Global Navigation Satellite System (GNSS) survey against that survey.

The split matters because the comparison reads a satellite pass across every gauge that pass crosses, and an anchor constrains what that pass contributes. The NOAA stations⁠¹⁠ all sit on the open coast, whereas the USGS gauges⁠²⁠ sit up two rivers, where a pass covers only the one at Camp Ellis⁠³⁠, as Across the Network sets out.

Catalog.shipped() reads the catalog, and its low_cost, anchors, and unsurveyed properties hold the targets, the anchors, and the gauges carrying no surveyed elevation.

Each Step's Key

tidemark run reads a step's result back from Hamilton's cache under .cache/hamilton/ wherever an earlier run computed it under the same key, and computes the step again otherwise. Hamilton versions a step's code by the step's own source alone and, as its guide on caching says, does not version the functions that source calls, so the package's StepCache widens the key to cover:

• The code, the step's own source beside every Python file of the package, the name and version of each installed library, and the interpreter's version
• The data, the fingerprint of each result the step reads, a table's covering each column's name and type beside its rows
• The settings, every field of PipelineConfig, for a step that reads config

A step reading a directory under data/ reads it through a snapshot, which holds the text of the directory's manifest.json and the SHA-256 of every other file its record names, the directory's revisions.json among them, and reads exactly the files the snapshot lists other than those two records. Four steps each take one:

• parts, of the passes the fetch kept, which tracks reads
• samples, of the samples table once it brings the table up to date, which coasts, heights, and selection read
• geoid, of the GEOID18 grid, which heights reads
• gnss, of the Nevada Geodetic Laboratory's table of GNSS station velocities, which heights reads

Those four, catalog, which reads the table the package ships, and each saver writing a table into a run's directory are computed on every run, since no key covers the files they read or write. Since tracks reads the snapshot of the parts in place of the settings, its result is read back whatever setting moves that the fetch's request leaves out.

tidemark cache names the file holding each kept result beside whether the most recent run computed it, and under --check computes every kept step again with nothing read back, naming each whose result differs from the one kept under the same key. mise run data:clean reaches data/ alone, so the cache stays until .cache/hamilton/ is deleted by hand.

Then change

NothingA settingNew passesA Python file

One block per computing step, illustrative, not measured. The five steps no key covers compute on every run and are left out, and the steps reading a setting are named by hand.

Every Setting Declared Once

Every analytic choice the steps depend on is a field of PipelineConfig, each with a default, a note on where that default comes from, and the bound or the closed set of values it accepts, so a value outside what a setting accepts is reported against that setting. The Command Line lists every setting under tidemark run beside its default. The quality band defaults to the good samples alone, which the mission's Science Data Products User Handbook names as the ones a reader would normally use.

The settings reach the steps as the single input config, and changing one takes two forms:

• For one run, pass it to tidemark run as the flag named for its field
• For every run, set it in a [tool.tidemark] table in pyproject.toml

Each Run's Record

Every tidemark run records what it computed with under data/runs/, through Hamilton's experiment tracker, so a number in a finding traces back to the run behind it, and two runs compare without computing either again. Each run's record holds:

• The settings, every field of PipelineConfig as the run read it
• The commit, the git commit the clone had checked out
• Whether the tree was dirty, meaning it differed from that commit, a file git does not track included unless .gitignore covers it, as it covers data/ and .cache/
• The lockfile, a SHA-256 digest of uv.lock, which pins the version of every library the run imported
• The derived tables, each table a step derives from another table, written as Parquet into the run's own directory with every column as text unless it holds floats or integers no wider than 64 bits, a list as its items joined by commas and an instant in ISO 8601, and named in the record wherever the table holds a row, so Hamilton's viewer shows each table the record names

Hamilton records its own hash of every step's source beside these, which its viewer numbers as the graph's version, and the commit and the lockfile name what that hash leaves out, meaning the code the steps call and the libraries they run on. The request behind each fetched file is recorded beside that file rather than by the run, as The Ranges on Disk and The Parts on Disk set out. The record stays on the machine that ran it, where mise run data:browse opens Hamilton's viewer over every recorded run at http://127.0.0.1:8123, with no account. mise run data:clean leaves data/runs/ in place, so clearing the history means deleting that directory by hand.

Checking Each Table

Every step that returns a table is checked against a pandera schema naming the columns the table carries and the values each column accepts. A table that breaks its schema stops the run at the step that computed it.

catalognot checked

catalog returns an object rather than a table, so the run checks nothing at this step.

anchorsStationSchema

• No two rows share a station
• coops_id, where filled, holds digits alone
• datum, where filled, is mhhw, mhw, mllw, mlw, msl, mtl, navd, or stnd
• latitude lies between 43 and 47.5
• longitude lies between −71.1 and −66.9
• navd88_offset_route, where filled, is agency, mark, or vdatum
• sensor, where filled, is acoustic, microwave, radar, or ultrasonic
• source is coops or neracoos
• vendor, where filled, is brown, greenstream, hohonu, or usgs
• Every row the CO-OPS metadata API supplied reads as an anchor and names the station id NOAA operates it under
• anchor, latitude, longitude, name, operator, source, and station are never empty
• It carries no column beyond its 29, each of its declared type or one that casts to it

A table that breaks any of these stops the run at anchors.

partsnot checked

parts returns an object rather than a table, so the run checks nothing at this step, and tracks is checked instead.

targetsStationSchema

• No two rows share a station
• coops_id, where filled, holds digits alone
• datum, where filled, is mhhw, mhw, mllw, mlw, msl, mtl, navd, or stnd
• latitude lies between 43 and 47.5
• longitude lies between −71.1 and −66.9
• navd88_offset_route, where filled, is agency, mark, or vdatum
• sensor, where filled, is acoustic, microwave, radar, or ultrasonic
• source is coops or neracoos
• vendor, where filled, is brown, greenstream, hohonu, or usgs
• Every row the CO-OPS metadata API supplied reads as an anchor and names the station id NOAA operates it under
• anchor, latitude, longitude, name, operator, source, and station are never empty
• It carries no column beyond its 29, each of its declared type or one that casts to it

A table that breaks any of these stops the run at targets.

stationsStationSchema

• No two rows share a station
• coops_id, where filled, holds digits alone
• datum, where filled, is mhhw, mhw, mllw, mlw, msl, mtl, navd, or stnd
• latitude lies between 43 and 47.5
• longitude lies between −71.1 and −66.9
• navd88_offset_route, where filled, is agency, mark, or vdatum
• sensor, where filled, is acoustic, microwave, radar, or ultrasonic
• source is coops or neracoos
• vendor, where filled, is brown, greenstream, hohonu, or usgs
• Every row the CO-OPS metadata API supplied reads as an anchor and names the station id NOAA operates it under
• anchor, latitude, longitude, name, operator, source, and station are never empty
• It carries no column beyond its 29, each of its declared type or one that casts to it

A table that breaks any of these stops the run at stations, before coasts, counts, crossings, heights, and selection read it.

tracksTrackSchema

• No two rows share the same cycle_number, pass_number, and time
• longitude_nadir, where filled, lies between −180 and 180
• Only latitude_nadir, longitude_nadir, and velocity_heading may be empty
• It carries exactly its 7 columns, each of its declared type

A table that breaks any of these stops the run at tracks, before crossings, cycles, and ground_tracks read it.

geoidnot checked

geoid returns an object rather than a table, so the run checks nothing at this step, and heights is checked instead.

gnssnot checked

gnss returns an object rather than a table, so the run checks nothing at this step, and heights is checked instead.

samplesnot checked

samples returns an object rather than a table, so the run checks nothing at this step, and coasts, heights, and selection are each checked instead.

crossingsCrossingSchema

• No two rows share the same station, cycle_number, and pass_number
• cross_track_distance lies between −60,000 and 60,000
• half_swath, where filled, is left or right
• Only half_swath may be empty
• It carries exactly its 6 columns, each of its declared type

A table that breaks any of these stops the run at crossings, before counts, distances, heights, and looks read it.

cyclesCycleSchema

• No two rows share a cycle_number
• No column is ever empty
• It carries exactly its 2 columns, each of its declared type

A table that breaks any of these stops the run at cycles, before looks reads it.

heightsPassHeightSchema

• No two rows share the same station, cycle_number, and pass_number
• cross_track_distance, where filled, lies between −60,000 and 60,000
• half_swath, where filled, is left or right
• mad is 0 or more
• mean_distance is 0 or more
• product is l2 or l3
• sample_uncertainty, where filled, is 0 or more
• samples is 1 or more
• spread_excess, where filled, is 0 or more
• uncertainty, where filled, is 0 or more
• uncertainty_analytic, where filled, is 0 or more
• uncertainty_bootstrap, where filled, is 0 or more
• uncertainty_swot, where filled, is 0 or more
• cycle_number, pass_number, height_ellipsoid, mad, mean_distance, product, samples, station, time, time_end, and time_start are never empty
• It carries exactly its 26 columns, each of its declared type

A table that breaks any of these stops the run at heights, before selection reads it.

coastsCoastSchema

• No two rows share a station
• samples is 1 or more
• sides_differ lies between 0 and 1
• No column is ever empty
• It carries exactly its 5 columns, each of its declared type

A table that breaks any of these stops the run at coasts.

countsCountSchema

• No two rows share a station
• passes is 0 or more
• passes_per_cycle, where filled, is 0 or more
• Only passes_per_cycle may be empty
• It carries exactly its 5 columns, each of its declared type

A table that breaks any of these stops the run at counts.

distancesDistanceSchema

• No two rows share the same station and pass_number
• cross_track_distance lies between −60,000 and 60,000
• half_swath, where filled, is left or right
• Only half_swath may be empty
• It carries exactly its 4 columns, each of its declared type

A table that breaks any of these stops the run at distances, before ground_tracks reads it.

looksLookSchema

• No two rows share the same station, cycle_number, and pass_number
• No column is ever empty
• It carries exactly its 5 columns, each of its declared type

A table that breaks any of these stops the run at looks.

selectionSelectionSchema

• No two rows share the same station and product
• chosen_passes is 0 or more
• coast_removed, where filled, lies between 0 and 1
• passes is 1 or more
• product is l2 or l3
• quality_removed, where filled, lies between 0 and 1
• samples is 1 or more
• surface_removed, where filled, lies between 0 and 1
• usable is 0 or more
• usable_passes is 0 or more
• Only coast_removed, quality_removed, and surface_removed may be empty
• It carries exactly its 10 columns, each of its declared type

A table that breaks any of these stops the run at selection.

ground_tracksGroundTrackSchema

• longitude lies between −180 and 180
• No column is ever empty
• It carries exactly its 4 columns, each of its declared type

A table that breaks any of these stops the run at ground_tracks.

---

1. 1NOAA CO-OPS 2026. "CO-OPS Data Retrieval API and Metadata API." https://tidesandcurrents.noaa.gov/web_services_info.html.↩︎

2. 2United States Geological Survey 1994. "USGS Water Data for the Nation." https://doi.org/10.5066/F7P55KJN.↩︎

3. 3SWOT 2025. "SWOT Level 2 KaRIn Low Rate Sea Surface Height Data Product." Version D. NASA Physical Oceanography Distributed Active Archive Center. https://doi.org/10.5067/SWOT-SSH-D.↩︎