The Gauge Series ​

Each gauge records the water beneath it, one reading after another, and those readings are what the comparison sets beside each satellite pass. This page follows the series of every gauge in the station catalog from the service publishing it, through the fetch and the files a clone keeps, to the one table it becomes, whatever gauge it came from.

Readings and Flags ​

Every reading pairs an instant with a water surface elevation, and the gauges read at steps of their own, the NOAA stations⁠1⁠ and the USGS gauges⁠2⁠ every six minutes, the Greenstream gauges every six or fifteen, the Brown University gauges about once a minute, and a Hohonu gauge anywhere from every half minute to every twelve minutes⁠3⁠. One gauge's step can change partway through its series, so each reading carries its own instant rather than sitting on a fixed step.

Beside each elevation sit the flags the QARTOD quality tests set, one column for each test and an aggregate flag summarizing them. The gauge at Scarborough and the USGS gauges publish none, the NOAA stations publish their own in place of QARTOD's, and the Brown University gauge on Lindsey Brook publishes every flag but the flat line test's.

Graded by a test

Not graded

No reading

1 Good

In the manual: Pass

Data have passed critical real-time quality control tests and are deemed adequate for use as preliminary data.

Each QARTOD column holds one of these codes on every reading of a gauge whose dataset publishes it.

The manual's name and definition for each code are quoted from Table 2 of the Manual for Real-Time Oceanographic Data Quality Control Flags, version 1.2, which the U.S. Integrated Ocean Observing System published in 2020.

Every value as a table
CodeMeaningManual's NameManual's Definition
1GoodPass"Data have passed critical real-time quality control tests and are deemed adequate for use as preliminary data."
2UnknownNot evaluated"Data have not been QC-tested, or the information on quality is not available."
3SuspectSuspect or Of High Interest"Data are considered to be either suspect or of high interest to data providers and users. They are flagged suspect to draw further attention to them by operators."
4FailFail"Data are considered to have failed one or more critical real-time QC checks. If they are disseminated at all, it should be readily apparent that they are not of acceptable quality."
9MissingMissing data"Data are missing; used as a placeholder."

A NOAA Station's Reading ​

A NOAA station's reading carries columns of its own beside its instant and its elevation, meaning the standard deviation of the one-second samples behind the elevation, the flags NOAA sets on it, and whether NOAA has verified it, each read as the Data API's response help page defines it and listed under The Series Table.

NOAA serves each elevation above the station datum, and the series reads it on the datum the station catalog names for the station, so Bar Harbor's heights are read above mean lower low water. Each height read is the elevation above the station datum less the catalog's height of that datum above it, worked out as the series is read, so a revision of the catalog's heights reaches every reading the station holds at once. Bar Harbor's heights reach the NAVD 88 column as well, each its height above mean lower low water plus the NAVD 88 offset the catalog measures from the station's bench mark K 22, worked out the same way.

Fetching the Series ​

mise run data:fetch:gauges fetches every gauge's whole series, from its first reading to now. Each NOAA station's comes from the CO-OPS Data API, as A NOAA Station's Months and The First Fetch set out, each USGS gauge's from the USGS Water Data API, as A USGS Gauge's Windows sets out, and every other gauge's from the NERACOOS ERDDAP, as the Parquet response the ERDDAP answers with through erddapy. tidemark fetch gauges, the subcommand behind the task, lists its options and what it prints.

The Ranges on Disk ​

Each gauge fetched from the ERDDAP keeps a directory under data/gauges/ (data/gauges/Hohonu_tide_Bath_ME/), holding one Parquet file for each range of its series, named for the instants the range starts and ends, beside a manifest.json naming the columns every range carries and, once a rerun replaces a run of days whose tally changed, a revisions.json recording the days of each before the newest reading's day, as The Weekly Release sets out.

Yes
No
Not taken
On diskThe fetch decidesAt the ERDDAPAsk which columnsthe dataset publishesAsk which columnsthe dataset publishesAsk which columnsthe dataset publishesReadings saved froman earlier fetch?The route never reaches “Readings saved from an earlier fetch?”.YESThe route never reaches “Readings saved from an earlier fetch?”.NOReport this gaugeas failedJoin the small filesinto oneLeave the savedreadings as they areSame columns aslast time?The route never reaches “Same columns as last time?”.YESThe route never reaches “Same columns as last time?”.NOSame columns aslast time?The route never reaches “Same columns as last time?”.YESThe route never reaches “Same columns as last time?”.NOAsk for every readingsince 1970Look at the filessaved for this gaugeAnything saved forthis gauge already?Yes is the answer this route takes to “Anything saved for this gauge already?”.YESAnswer no to “Anything saved for this gauge already?” instead.NORun with --refresh?Answer yes to “Run with --refresh?” instead.YESNo is the answer this route takes to “Run with --refresh?”.NOSmall files under30 days and 30 files?Yes is the answer this route takes to “Small files under 30 days and 30 files?”.YESAnswer no to “Small files under 30 days and 30 files?” instead.NOAsk for each month’scount and averageDid the ERDDAPaccept the request?Yes is the answer this route takes to “Did the ERDDAP accept the request?”.YESAnswer no to “Did the ERDDAP accept the request?” instead.NODoes any month differfrom what is saved?Yes is the answer this route takes to “Does any month differ from what is saved?”.YESAnswer no to “Does any month differ from what is saved?” instead.NOAsk for each day’scount and averageAsk for the readingsof the days that differDid any readingscome back?Yes is the answer this route takes to “Did any readings come back?”.YESAnswer no to “Did any readings come back?” instead.NOSave what came backinto the gauge’s files
Every value as a table
FetchAsks the ERDDAP For
The firstThe whole series, from 1970-01-01, the instant ERDDAP counts time from
Every otherA tally of each month, then of each day from the first month whose tally differs from the gauge's, and the readings of the days that differ
Each under --refreshThe tallies every other fetch asks for, then the whole series

The short ranges reruns leave are joined on disk into one range once they span 30 days together or number 30. A gauge therefore holds one file for its series up to its first fetch, about one for each month after, and at most 30 for the month since, however often it is fetched.

Catching Up ​

A rerun asks the ERDDAP how many readings each month or day holds and what each column averages there, through the orderByCount and orderByMean filters its tabledap service offers, and compares both with the same tally of the readings the gauge holds. The ERDDAP answers a tally as CSV, since its Parquet response to an orderBy filter holds no bytes, and each tally of a whole series runs to a few kilobytes.

  • A reading that reaches the ERDDAP after a rerun has already asked past its time, and one the ERDDAP revises, each change the tally of their day, so the next rerun finds them wherever they fall, with no --refresh needed.
  • The newest reading's spike flag reads 2 until the next reading arrives, since the QARTOD spike test judges a reading against the ones on either side of it, so its day's tally changes, and the next rerun reads that day again.
  • The first run of days reaching past the newest reading held is asked for up to now and saved as a range of its own, so a day the gauge was silent stays inside a range for a reading the ERDDAP receives late. An earlier day that differs is written into the file already holding it, so each instant is still read from the file fetched last.
  • The fetch asks for no reading at a gauge whose tallies all agree.

Revised Readings ​

The USGS publishes its gauges' readings as provisional, and a Water Mission Area policy memorandum requires their approval within 150 days, within 240 days, or by June 1 of the following year, by the category of the site, so a USGS reading can change long after it was first fetched, and the next rerun reads it again among every reading USGS modified since, as A USGS Gauge's Windows sets out.

Columns and Failures ​

  • A whole fetch and each join read the columns from the information table the ERDDAP keeps for the dataset, and every other fetch reads them from manifest.json, turning to that table where the server refuses a tally or a query naming them.
  • A gauge that publishes other columns than its manifest.json names, such as one that starts publishing a flag, is fetched whole again, so every range carries the same columns.
  • A fetch or a join that stops partway, or an answer that is not Parquet, leaves no file under a range's name, so the next fetch asks again.
  • A gauge whose dataset holds no reading keeps its series as an empty table, whereas an answer holding no reading leaves the readings a gauge already holds in place, and where no other answer adds one, the gauge's row in the ledger the fetch prints reads Found.

A NOAA Station's Months ​

Each NOAA station's series comes from the CO-OPS Data API as its six-minute water_level product, in meters and GMT, a calendar month at a time, since the API serves at most a month of six-minute data a request. Every month is asked for on the station datum, which the API documents as the original reference all of a station's data is collected to, so a revision of another datum, such as a new tidal epoch or NAPGD2022 in place of NAVD 88, changes that datum's height in the station catalog rather than any reading held. The station keeps a directory under data/gauges/ named for its catalog key (data/gauges/NWLON_8418150/), holding:

  • One Parquet file for each month, named for the instants the month spans, whose footer records the instant it was retrieved, the query, the datum heights the catalog kept then, and the commit of the code that fetched it
  • means.parquet, the verified monthly mean of every month as the Data API gave it at the last fetch
  • manifest.json, naming the station, the datum, and the columns each month holds
  • revisions.json, each month a fetch replaced once NOAA verified it or revised its verified monthly mean, where a fetch has replaced any
FetchAsks the Data API For
The first, each under --refresh, and any whose manifest.json records another requestEvery month a stretch of six-minute water levels reaches into, as the CO-OPS data inventory lists the stretches, gaps left out
Every otherEach month the inventory lists that the station lacks, that holds a reading past the newest one held, that NOAA has verified since, whose verified monthly mean the Data API gives otherwise than means.parquet records, and, at a station NOAA still operates, that holds a reading past the newest verified one the inventory lists
  • The inventory, rather than the date the metadata API gives for a station's establishment, sets where a series starts, since Portland, established in 1910, holds its first six-minute reading in 1996.
  • CO-OPS verifies each month in the month after it, replacing its preliminary readings, as the Data API's documentation states, so a month past the newest verified reading is read again, and once more as verified once the inventory's newest verified reading passes it.
  • A month holding readings NOAA never verified sits before the newest verified reading, as Bar Harbor's March 2000, Cutler's November 2010, and Seavey Island's June 2020 do, so a rerun leaves it as held.
  • NOAA reprocesses verified months after the fact, changing the verified monthly mean the Data API's monthly_mean product gives for up to 200 years in one request, so a rerun asks for that product once per station and compares each month's mean with the one means.parquet recorded, since a month's own six-minute mean differs from NOAA's, which averages hourly heights, by up to a few centimeters, most of all in a month with gaps.
  • Wells, which NOAA has removed, keeps its readings from 2021 on preliminary, and its inventory lists nothing past its last reading, so a rerun asks for no month there.
  • At a station with nothing new, the fetch asks the Data API for the station's monthly means alone.
  • A month the Data API holds no reading for is kept as an empty file wherever the station holds no reading in that month, so a rerun counts it as held, whereas an empty answer for a month holding readings is set aside, since it would otherwise read as their removal.

The First Fetch ​

Fetching every NOAA station's whole record takes about 12 minutes, including the waits after each refusal of the CO-OPS throttle, so each weekly release carries every station's months as they stood when the release was cut, and a fetch seeds from the newest one carrying them first, whereas a rerun asks NOAA only for the months a station lacks or NOAA has changed.

  • Except under --refresh, the fetch reads through gh the newest release carrying the stations' tar, and where a station holds no month, records another request, or is still operated with its newest reading from before the release was published, it downloads the release's tar of the NOAA stations, checks it against the SHA-256 digest GitHub lists for it, and moves into place each month the station lacks, wherever the release records the request the station is fetched with.
  • tidemark fetch gauges names how many months it seeded from which release, or why it seeded none, such as a gh holding no login that reads the private repository, then asks NOAA for what is left.
  • Where a station still holds no month, records another request, or is still operated with its newest reading more than a week old, or under --refresh, the fetch first prints about how long the months it asks NOAA for take.

The CO-OPS Throttle ​

CO-OPS limits the requests it accepts from one customer while its services are under heavy load, and the Data API's documentation names no rate. The fetch therefore sends every month of every station at once and lets each refusal set the pace of the host that refused it, which every request to that host shares, whether to the metadata API, the inventory, or the Data API.

  • A refusal, meaning the throttle's 403, a 429 other than the USGS's, a 500, 502, 503, or 504, an answer slower than the timeout, a body cut short, or a dropped connection, halves how many requests the host takes at once.
  • A refusal also closes the host for a wait drawn at random between none and a window that starts at 1 second and doubles with each refusal since the host last answered, up to 2 minutes, lengthened by any Retry-After the refusal carries.
  • Each round of answers lets the host take one more request at once, which beside the halving is the congestion avoidance RFC 5681 sets out for TCP, and the wait is the full jitter tenacity documents for uncoordinated clients contending for a shared resource.
  • A refused request is sent again through tenacity once its host reopens, up to 10 attempts in all, or until the host has answered nothing for 10 minutes, when its last refusal raises. Every refusal beside the throttle's 403 is a transient error Google Cloud's retry strategy names, whereas the deadline counts from the host's last answer rather than from the request's first attempt.

Each month in flight holds a thread, a connection, and the file it writes, so the fetch raises its own limit on open files to fit every month, as far as the system's hard limit allows, and keeps each station's months oldest first, whatever order their answers land in. An inventory answer listing no stretch is asked again 5 seconds later, twice at most, and a month the Data API refuses, or an inventory answer that is not XML or still lists no stretch, fails its station, so the next fetch asks again for every month the station still lacks.

fetch16 at oncehostHold to refuse
Longest wait
1 s2 s4 s8 s16 s32 s64 s120 s

The rules are the fetch’s own, the sizes illustrative, not measured, since CO-OPS names no rate. Long waits run fast.

ERDDAP Request Limits ​

A fetch runs every gauge fetched from the ERDDAP at once, beside every NOAA station, and leaves the ERDDAP's own limits to set the pace, since the server processes two requests at once from one address by default and queues the rest. A 429 the server answers past its limit on the requests it accepts carries no Retry-After header and narrows the ERDDAP's pace as any refusal narrows its host's, and a gauge the server still refuses fails alone while every other gauge is fetched.

A USGS Gauge's Windows ​

Each USGS gauge's series comes from the USGS Water Data API as the gage height its monitoring location publishes under parameter 00065, in feet, from the continuous collection, which answers a query in pages of GeoJSON, each naming the address of the next. The gauge keeps a directory under data/gauges/ named for its catalog key (data/gauges/USGS_tide_Hallowell_ME/), holding one Parquet file for each range of its series beside a manifest.json naming the monitoring location, the altitude of its gage datum above NAVD 88, and the columns each range holds, so a gage datum USGS changes fetches the series whole again. Each elevation is the gage height plus that altitude, converted at 0.3048 meters to the international foot, the one foot the Federal Register notice at 85 FR 62698 left in use once the U.S. survey foot was deprecated on 2022-12-31. At Camp Ellis, whose gage datum no survey set, the series reads each elevation instead as the gage height in meters plus the NAVD 88 offset the station catalog takes from that altitude, worked out as the series is read, so a catalog rebuilt on a changed altitude reaches every reading the gauge holds at once.

FetchAsks the Water Data API For
The first, each under --refresh, and any whose manifest.json records another requestThe whole record from the first reading the series' metadata names, in windows of at most 1,100 days, the longest time the collection accepts, every window sent at once and each kept as a range of its own once every window lands
Every otherEvery reading USGS modified after the newest modification the gauge holds, across the whole record, kept as one range
  • A range keeps each reading's last_modified, and where two ranges hold one instant the series reads the reading USGS modified last, so a revision reaches the series without rewriting the range that held it.
  • A whole fetch that stops partway keeps no window and leaves every range the gauge held in place, so a gauge holding none asks for its whole record again on the next fetch.
  • Except under --refresh, a gauge holding no range or recording another request first takes each range and the revisions.json the newest release holds for it under the same request, as The First Fetch sets out for the NOAA stations.
  • Each range's footer records the instant it was retrieved, the query, and the commit of the code that fetched it, as a NOAA month's does.
  • A reading USGS removes returns no modification, so it stays as held until a fetch under --refresh reads the whole record again.
  • Once a fetch of a gauge holding readings saves a reading USGS modified after the newest modification the gauge held, a revisions.json beside the manifest.json records each run of days holding one before the newest reading's day, as The Weekly Release sets out.

USGS Request Limits ​

The API accepts a limited number of requests an hour from each address sending no API key, the number its X-RateLimit-Limit header names, and a key raises that limit. Past it, the API refuses with a 429 whose Retry-After header names the seconds until the limit resets, which can run to an hour, so the fetch sends no such request again and fails the gauge, naming when to rerun, as a 403 rejecting a key fails it at once, naming where the key came from. Every request counts against it, and a whole fetch of every USGS gauge asks for about a hundred pages. The API can also leave a request unanswered past the session's timeout, cut its answer off partway, or drop the connection. Such a page is a refusal like any other, so it narrows the API's pace and is sent again from its own address once the host reopens, as The CO-OPS Throttle sets out for every host. A fetch runs every USGS gauge at once beside every other gauge, and every window of a whole fetch at once, leaving the API's own refusals to set the pace.

The Stakeholder's Series ​

The stakeholder may provide a gauge's series as a file, above all for the months before the gauge joined the ERDDAP. Place it at data/provided/gauges/<station>.parquet, named for the gauge's key in the station catalog, carrying time in UTC, navd88_meters, and any flag column under the names the ERDDAP uses, and no other column. Across each stretch of its readings, meaning readings no more than an hour apart, the file's readings replace the ones the gauge's source answered with, and the source's readings outside every stretch stay. The stretches are read from the whole file, so one reaching past either end of a span replaces the source's readings up to that end. A fetch of the gauge still has to run, since the loader reads the provided file beside what the source answered.

The Series Table ​

One row per

reading

station

Shares station with 4 other tables.

18columns

Commit47945eaDatastations.jsonUnreleased1.2.0

load_series reads one gauge's series over a span from every range it holds into one table, carrying the columns below in the same order for every gauge.

  • Each instant reads from the range fetched last among those covering it, except at a USGS gauge, which reads the reading USGS modified last, so a reading the server revised or removed between two fetches reads as the later fetch answered, though a reading USGS removes stays until a --refresh.
  • A column a gauge's source never publishes stays empty rather than absent, so the QARTOD columns stay empty at a NOAA station, the NOAA columns at every gauge fetched from the ERDDAP, and the USGS columns at every gauge but a USGS one.
  • The ERDDAP sometimes repeats the first reading of a month, every column alike, and the table keeps one.
  • An elevation the server wrote as its fill value, which marks a missing reading, reads as empty.
  • Every reading and every flag of one range stay, including two readings at one instant that differ.

mise run docs:render writes the table's columns below from SeriesSchema, and those of the monthly means a NOAA station's fetch records from MeansSchema.

ColumnHolds
stationA shared key the tables join on
timeThe instant the reading was taken, in UTC
navd88_metersThe water surface elevation in meters above NAVD 88, as the network publishes it, at a NOAA station as worked out from its elevation above the station datum, or at a USGS gauge as its gage height plus the altitude of its gage datum, and at a gauge without a surveyed NAVD 88 tie as its elevation above its datum plus the NAVD 88 offset the station catalog measures
mllw_metersThe water surface elevation of a NOAA station without a NAVD 88 tie, in meters above mean lower low water, worked out from its elevation above the station datum
stnd_metersThe water surface elevation a NOAA station publishes, in meters above its station datum
coops_flat_toleranceThe NOAA flag set to 1 where the reading exceeded the flat tolerance limit
coops_height_limitThe NOAA flag set to 1 where the reading exceeded the maximum or the minimum water level height expected at the station
coops_outliers_or_inferredHow many of the one-second samples behind a preliminary NOAA reading fall outside three standard deviations of their mean, or on a verified one, 1 where NOAA inferred the value
coops_rate_of_changeThe NOAA flag set to 1 where the reading exceeded the rate of change tolerance limit
gage_feetThe gage height a USGS gauge publishes, in feet above the gage datum, the altitude its monitoring location records
navd88_meters_qartod_flat_line_testThe QARTOD flag the flat line test set on the reading
navd88_meters_qartod_gross_range_testThe QARTOD flag the gross range test set on the reading
navd88_meters_qartod_rate_of_change_testThe QARTOD flag the rate of change test set on the reading
navd88_meters_qartod_spike_testThe QARTOD flag the spike test set on the reading
qartod_qc_rollupThe aggregate QARTOD flag, which summarizes the flags the tests set on the reading
sigma_metersThe standard deviation of the one-second samples behind a NOAA reading, in meters
usgs_qualifiersThe codes USGS qualifies a reading with, such as FORCEINTERPOLATION, which stays empty where USGS gives none
verifiedWhether the agency has verified the reading, which stays empty where the gauge's source publishes no such status

means.parquet ​

One row per

month

Joins no other table on a shared key.

2columns

Commit47945eaDatastations.jsonUnreleased1.2.0

The verified monthly mean water levels NOAA's CO-OPS Data API gives for one NOAA station, one row per month, which means.parquet in the station's directory records.

ColumnHolds
mslThe month's mean sea level, in meters above the station datum
timeThe month's first instant, in UTC

  1. 1NOAA CO-OPS 2026 CO-OPS Data Retrieval API and Metadata API https://tidesandcurrents.noaa.gov/web_services_info.html↩︎

  2. 2United States Geological Survey 1994 USGS Water Data for the Nation https://doi.org/10.5066/F7P55KJN↩︎

  3. 3Gulf of Maine Research Institute & Brown University 2026 NERACOOS ERDDAP Northeastern Regional Association of Coastal Ocean Observing Systems. https://data.neracoos.org/erddap/↩︎