sensor-points-checks

Daily data-quality checks on the sensor_points table in the WindMC read replica. A cron job (1pm America/Los_Angeles, on sk-femur under the jake user) pulls the last CHECK_LENGTH (48) hours of data and posts a summary to Zulip (#eng-code-sensors > database quality check) as the SHODAN bot — tagging Jake Spisak if anything failed, plain "passed" message otherwise.

How it works

check_sensor_points.py COPYs the window out of Postgres as CSV into numpy arrays (one query, ~2.5 min for 48h / ~9M rows), then runs every check vectorized in Python. Each dataset is checked independently, grouped by sensor_points.mission_id UUID; dropsonde rows (source LIKE '%dropsonde%') are exempt for now. Labels in the report are the mission W-number when one exists, else the dataset UUID.

Checks

• Bounds (*_out_of_bounds): value inside its physical range (and not NULL) wherever its validity flag is set — altitude 50–26000 m, |speed_x/y| ≤ 150 m/s, |longitude| ≤ 180, |latitude| ≤ 90, pressure 2000–104000 Pa, temperature -88.5–48.5 C, humidity 0–100%.
• Jumps (*_jump): no big change between adjacent points (≤10s apart, same dataset) — altitude 200 m, speed_x/y 10 m/s, lat/lon 1° (exempt near poles/equator; longitude is date-line aware), pressure 2000 Pa, temperature 10 C.
• Stale (*_stale): no run of ≥100 identical consecutive valid values spanning ≥24h (humidity only counts between 10 and 90).
• baro_altitude: ISA barometric altitude from pressure (met_funcs.pressure_to_geometric_height) within 1 km of GPS altitude.
• gfs_temperature: measured temperature within 15 C of GFS where a forecast value exists.

Running it

python3 check_sensor_points.py --local   # print the table instead of posting
python3 check_sensor_points.py           # post to Zulip

Needs WINDMC_RO_URL, WINDMC_RO_KEY, and SHODAN_ZULIP_KEY in the environment (the cron sources them from ~/.bashrc). met_funcs.py is a trimmed copy of the sensor-processing version — just the pressure-to-height chain. Run outputs land in outputs/ (gitignored).