Weather without
lab history.
This page documents a one-off backtest from 2026-10-15. It asks how well the last six days of weather alone can separate a beach's dirty lab days from its clean days. It does not change the forecast users see today.
Question and target.
Question. With only the last six days of weather, how well can a model tell a beach's dirty days from its clean days?
Target. Each lab result is divided by its own limit (104 for culture tests, 1413 copies for ddPCR), on a log scale. We subtract that beach's average from the training years only. Both lab methods share one scale.
Inputs. 36 numbers per beach and day: for each of the 6 days before 5 AM, rain, shortwave, uv, cloud_cover, onshore_wind, max_wind. No lab history, no beach record, and no beach identity.
Models. We tested linear (ridge), boosted trees (XGBoost), a small neural network (two layers of 32 units), and random forest. We also tested rain-only versions and the county rain rule (72 h rain ≥ 0.1 in). Each model is refit monthly on the four years before that month.
Within-beach AUROC.
Within-beach AUROC measures how well a score ranks dirty days above clean days at the same beach; 0.50 is a coin flip. Scores are for forecasts one to three days ahead.
| Model | All beaches | Outside San Diego | Wet season (Nov–Apr) | Dry season (May–Oct) | San Diego |
|---|---|---|---|---|---|
| County rain rule (72 h rain ≥ 0.1 in) | 0.563 | 0.585 | 0.583 | 0.510 | 0.507 |
| Rain-only linear (ridge) | 0.585 | 0.630 | 0.622 | 0.538 | 0.472 |
| Weather neural network (2×32) | 0.615 | 0.647 | 0.642 | 0.526 | 0.535 |
| Weather linear (ridge) | 0.603 | 0.641 | 0.632 | 0.522 | 0.507 |
| Served v2 (reference) | 0.647 | 0.643 | 0.610 | 0.561 | — |
- Neural network vs rain rule (95% beach-cluster interval): +0.043 to +0.059
- All weather vs rain-only linear: +0.009 to +0.028
- Neural network vs served v2 outside San Diego: −0.017 to +0.022
Rain dominates the linear model.
Linear weights on the full weather vector: rain in the last 24 h (0.136), rain the day before (0.062). Cloud cover and UV today are small; wind terms are near zero.
Weather plus served v2.
We also fit the weather score as an extra input to logistic v2, with all weights fit together and cross-fitted. It did not pass our promotion rule on forecasts one to three days ahead.
Leakage controls.
- Shuffling weather within each beach drops within-beach AUROC to 0.500.
- A deliberately leaky version scores at most 0.01 higher.
- Inputs stop at 5 AM on the forecast day.
Serving is unchanged.
Weather alone carries real day-to-day signal, mostly from rain in the last two days, and it does not improve the served model enough to change it. The live forecast still uses logistic v2 on the per-beach lookup with lab history and rain.
Read methodology →