Track record · Tornado
How accurate is the tornado forecast?
How the published tornado model scored on every thunderstorm NOAA tracked over the US in 2025: 1,469,977 storm observations, 1,579 of them followed by a tornado. Every modelling choice was fixed before 2025 was scored, and it was scored once. Every number below is read from the programme’s results files.
The model
- Model version
tornado_v3-05a06c843c87- What it forecasts
- The chance that this storm produces a tornado within 60 minutes (its inputs include the live NWS tornado-warning state)
- What counts as a tornado
- A tornado report starts within 10 km of this storm's tracked polygon within 60 min
- Inputs (30)
- 28 ProbSevere storm attributes, 2 NWS tornado-warning state
- Model form
- Gradient-boosted trees (LightGBM, 339 trees), scored in NumPy from a JSON file in the repository
- Calibration
- Platt scaling fitted on leave-one-year-out scores; a Venn–Abers band per storm
- Training
- 2020-10-15–2024-12-31, fixed rounds chosen by early stopping on validation 2023
How the test was run: every choice — inputs, model type, settings, calibration and the event definition — was made on 2023; 2024 was a development check; the model was then refitted on 2020–2024 and scored once on 2025, against SPC storm reports matched to each storm’s own tracked radar outline. Protocol: docs/TORNADO_MODEL_PROGRAM.md.
Final test: every storm of 2025
| Forecast | AUC | 95% interval | PR-AUC | Brier skill | Storm observations / tornadic |
|---|---|---|---|---|---|
| Published model (with the NWS warning state) | 0.970 | [0.964, 0.976] | 0.175 | +0.107 | 1,469,977 / 1,579 |
| Fallback, when the NWS warnings feed is down | 0.969 | [0.963, 0.975] | 0.166 | +0.097 | 1,469,977 / 1,579 |
| Tornado within 30 min | 0.974 | [0.968, 0.979] | 0.139 | +0.084 | 1,469,977 / 978 |
| Tornado within 90 min | 0.968 | [0.961, 0.974] | 0.189 | +0.111 | 1,469,977 / 1,993 |
| EF2+ tornado within 60 min | 0.994 | [0.989, 0.997] | 0.151 | +0.061 | 1,469,977 / 284 |
| NOAA ProbTor, as issued | 0.879 | [0.850, 0.902] | — | — | same storms |
| NOAA ProbTor, ties broken | 0.939 | [0.924, 0.951] | — | — | same storms |
AUC: how often a storm that produced a tornado is ranked above one that did not (0.5 is chance, 1 is perfect). PR-AUC: the same idea, focused on the rare tornadic storms. Brier skill: improvement over always forecasting 2025’s average rate. Intervals: 95%, bootstrapped over days. NOAA ProbTor is issued in whole percents and is 0 for most storms, so its ties are broken with ProbSevere’s own severe probability for a fair ranking. Development year 2024: AUC 0.969 [0.963, 0.974], Brier skill +0.114.
Against NOAA and the NWS, on the same storms
- NOAA ProbTor. NOAA ProbTor scores 0.879 as issued and 0.939 with its integer ties broken; the model is +0.031 [+0.022, +0.042] AUC above the tie-broken version, paired by day, over 365 days; Brier −0.000036 [−0.000055, −0.000021].
- NWS tornado warnings. At the NWS warnings’ own false-alarm rate it catches 30.5% of tornadic storm observations vs the warnings’ 22.7% (+7.7 [+3.8, +10.1] points, using the warning state as an input); without that input it catches 27.5% (+4.7 [−0.2, +8.5] points). The false-alarm rate is the warnings’ own (0.090% of storm observations without a tornado). A model that reads the warning state measures what it adds on top of the warnings; the figure without that input is the contest on its own. HazardPulse does not issue warnings; NWS warnings remain the authority for protective action.
- Our previous model (v2). +0.007 [+0.003, +0.010] AUC on the same storms and the same event.
By region, season, time of day, storm size and lead time
The lowest AUC in any group is 0.929; in 17 of 19 groups the model’s whole interval lies above calibrated ProbTor’s.
| Group | Storm observations / tornadic | Model AUC | ProbTor (calibrated) AUC |
|---|---|---|---|
| Plains (105°W to 94°W, north of 30°N) | 273,395 / 546 | 0.969 [0.961, 0.976] | 0.859 [0.824, 0.891] |
| Midwest (94°W to 80°W, north of 37°N) | 244,581 / 401 | 0.966 [0.952, 0.977] | 0.893 [0.847, 0.923] |
| Southeast (94°W to 75°W, south of 37°N) | 551,398 / 540 | 0.974 [0.961, 0.982] | 0.907 [0.859, 0.940] |
| Outside these three regions | 400,603 / 92 | 0.929 [0.890, 0.953] | 0.731 [0.666, 0.800] |
| Winter (Dec–Feb) | 71,657 / 63 | 0.979 [0.967, 0.987] | 0.965 [0.931, 0.988] |
| Spring (Mar–May) | 305,354 / 948 | 0.970 [0.963, 0.977] | 0.919 [0.895, 0.941] |
| Summer (Jun–Aug) | 783,458 / 440 | 0.957 [0.945, 0.968] | 0.764 [0.717, 0.802] |
| Autumn (Sep–Nov) | 309,508 / 128 | 0.952 [0.924, 0.968] | 0.872 [0.818, 0.913] |
| Night (20:00–06:00 local solar time) | 492,724 / 382 | 0.984 [0.975, 0.989] | 0.942 [0.909, 0.963] |
| Day (06:00–20:00 local solar time) | 977,253 / 1,197 | 0.964 [0.956, 0.970] | 0.859 [0.828, 0.883] |
| Smallest third of storms | 498,712 / 148 | 0.933 [0.902, 0.954] | 0.728 [0.642, 0.794] |
| Middle third of storms | 481,316 / 144 | 0.943 [0.926, 0.959] | 0.760 [0.707, 0.809] |
| Largest third of storms | 489,949 / 1,287 | 0.969 [0.963, 0.975] | 0.901 [0.873, 0.923] |
| Fine-grid (9 km) environment analysis available | 1,466,639 / 1,575 | 0.970 [0.964, 0.976] | 0.880 [0.851, 0.904] |
| Tornado 0–15 minutes ahead | 1,468,932 / 534 | 0.977 [0.970, 0.982] | 0.904 [0.873, 0.926] |
| Tornado 15–30 minutes ahead | 1,468,842 / 444 | 0.971 [0.962, 0.978] | 0.882 [0.846, 0.911] |
| Tornado 30–60 minutes ahead | 1,468,999 / 601 | 0.964 [0.956, 0.971] | 0.854 [0.823, 0.883] |
| Strong tornadoes (EF2 and above) | 1,468,682 / 284 | 0.995 [0.989, 0.998] | 0.977 [0.949, 0.990] |
| Weak tornadoes (EF0–EF1) | 1,469,438 / 1,040 | 0.968 [0.963, 0.974] | 0.875 [0.848, 0.897] |
Does 10% mean 10%?
In a well-calibrated row the average forecast and the rate at which a tornado followed agree. Intervals are 95% Jeffreys intervals.
| Forecast range | Storm observations | Average forecast | Tornado followed [95% interval] |
|---|---|---|---|
| 0% – 0.10% | 1,340,592 | 0.010% | 0.012% [0.010%, 0.014%] |
| 0.10% – 0.20% | 44,834 | 0.14% | 0.13% [0.099%, 0.17%] |
| 0.20% – 0.50% | 39,904 | 0.32% | 0.29% [0.24%, 0.35%] |
| 0.50% – 1.0% | 18,840 | 0.70% | 0.58% [0.48%, 0.70%] |
| 1.0% – 2.0% | 11,724 | 1.4% | 1.2% [0.99%, 1.4%] |
| 2.0% – 5.0% | 8,184 | 3.1% | 2.3% [2.0%, 2.6%] |
| 5.0% – 10% | 3,231 | 7.0% | 7.0% [6.1%, 7.9%] |
| 10% – 20% | 1,595 | 14% | 17% [15%, 19%] |
| 20% – 30% | 430 | 24% | 22% [18%, 26%] |
| 30% – 50% | 641 | 38% | 35% [31%, 38%] |
| 50% – 100% | 2 | 51% | 50% [6.1%, 94%] |
Tested and not used
Each was measured on 2023 under the same protocol. None improved the forecast, so none is in the published model.
- The HRRR environment at the storm, timed as it is available live: +0.0001 validation AUC (difference of two validation runs).
- The coherence field (80 km) on top of those HRRR fields: −0.0011 [−0.0032, +0.0007] validation AUC (paired by day; measured before amendment 8 corrected the HRRR timing, which both arms shared).
- The coherence PDE solution against a Gaussian-smoothing control (9 km): +0.0004 [−0.0008, +0.0015] validation AUC (paired by day; measured before amendment 8 corrected the HRRR timing, which both arms shared).
Live forecast ledger
Every live tornado forecast is appended to a ledger in which each entry carries the hash of the one before it, so any change to a past entry breaks every link after it.
| Issued | Storms | Model | Highest storm chance | Entry hash |
|---|---|---|---|---|
| 2 | tornado_v3-05a06c843c87 | <0.1% | 1f0cfa791a66… | |
| 1 | tornado_v3-05a06c843c87 | <0.1% | 160b48426b70… | |
| 10 | tornado_v3-05a06c843c87 | <0.1% | 12a22995440a… | |
| 21 | tornado_v3-05a06c843c87 | <0.1% | dee8d71cf02e… | |
| 16 | tornado_v3-05a06c843c87 | <0.1% | 8cf6fdab9108… | |
| 17 | tornado_v3-05a06c843c87 | <0.1% | 2b0e9261daae… | |
| 11 | tornado_v3-05a06c843c87 | <0.1% | 61db9759d782… | |
| 11 | tornado_v3-05a06c843c87 | <0.1% | 6f89d9d35140… | |
| 11 | tornado_v3-05a06c843c87 | <0.1% | 7fc6dfd9f3ac… | |
| 10 | tornado_v3-05a06c843c87 | <0.1% | 7b5f3a43ca9c… | |
| 18 | tornado_v3-05a06c843c87 | 0.1% | 6ff7fd272a58… | |
| 19 | tornado_v3-05a06c843c87 | <0.1% | 47b5748182bf… | |
| 15 | tornado_v3-05a06c843c87 | 0.1% | c869327b10ef… | |
| 24 | tornado_v3-05a06c843c87 | <0.1% | c2a24a2de4fa… | |
| 22 | tornado_v3-05a06c843c87 | <0.1% | 9fa3efb1c7d9… | |
| 36 | tornado_v3-05a06c843c87 | 0.1% | 266cbdf4d0a1… | |
| 38 | tornado_v3-05a06c843c87 | <0.1% | 9f307e632c6e… | |
| 35 | tornado_v3-05a06c843c87 | <0.1% | 7012fc930371… | |
| 40 | tornado_v3-05a06c843c87 | <0.1% | f274db9c2dae… | |
| 43 | tornado_v3-05a06c843c87 | <0.1% | 0813d37adb9c… |
Why you can check these numbers
Fixed in advance, scored once
Every modelling choice was fixed on 2023 and checked on 2024 before 2025 was scored, once. The protocol and its dated amendments are public, including the ones that corrected our own earlier numbers.
Scored against NOAA on the same storms
The comparison with NOAA ProbTor and the NWS warnings uses the same storm observations, the same event and intervals paired by day.
Open and reproducible
The model is a JSON file in the repository, scored in pure NumPy; its identity is the hash of its bytes, and every live forecast is appended to a hash-chained ledger.
What did not work is published too
The HRRR environment and the coherence field were tested and did not improve the forecast; the measurements are above.