Methods

How the forecasts are made

Each forecast follows the same path: collect public data, run a fixed and versioned model, run automatic quality checks, and publish the result with a record of exactly what produced it. This page describes each step, the models, the tests that chose them, and their limits.

What each number means

Every number is a chance: how likely an event is within a stated window, given only the data available when the forecast was issued and the model version named beside it. The three forecasts answer three different questions, over three different windows, so their numbers are not comparable with each other.

ForecastThe chance ofWindowScored against
Earthquakeat least one magnitude 6+ earthquake with its epicentre in a 2° grid cell30 daysthe USGS catalog (ComCat)
Hurricanerapid intensification: maximum sustained winds rising 30 knots or more24 hoursthe National Hurricane Center’s best track
Tornadoa tornado report starting within 10 km of the storm’s tracked radar outline60 minutesStorm Prediction Center storm reports

A chance is not a verdict. A 30% chance means the event is expected in about 3 of every 10 such situations and not in the other 7. The tornado model also publishes chances for 30 and 90 minutes and for a strong (EF2+) tornado.

Data

Public data from government agencies and published science. We make no observations of our own.

Earthquake

Live catalog
USGS ANSS Comprehensive Catalog (ComCat): every M2.5+ event of the last 30 days, and every M5+ event since 1973 (a frozen copy, brought up to date from ComCat at each forecast). ComCat lists events of every type, so a few are not earthquakes: see Limits
Long-term rates
GEAR1, a global model of earthquake rates from crustal strain and past seismicity (Bird et al. 2015, CC BY 4.0)
Outcomes
M6+ earthquakes in ComCat after each forecast window closes

Hurricane

Storms
Positions and intensities from the National Hurricane Center’s and Joint Typhoon Warning Center’s advisories
NOAA guidance
The NHC’s SHIPS text for each forecast cycle: DTOPS, SHIPS-RII and the other rapid-intensification probabilities
Our models
The early intensity guidance in NHC’s forecast files; for v10.3 and v10.4, also NOAA’s GMGSI satellite infrared; for J1, in the other basins, v8.2’s own probability and inputs with the same GMGSI infrared; for TC1, the track and intensity guidance in NHC’s real-time forecast files
Outcomes
The NHC best track (IBTrACS for the historical tests)

Tornado

Storms
NOAA ProbSevere v3, which tracks every thunderstorm on radar over and near the contiguous US every two minutes, with its radar, lightning, satellite and environment attributes
Warnings
The National Weather Service’s active tornado warnings (an input of the published model)
Outcomes
Storm Prediction Center storm reports

The models

Every figure in this section is read from the results files bound to the exact model file that publishes the live number. If a result cannot be bound to that file, the section says so instead of quoting it.

Earthquake

The chance of a magnitude 6+ earthquake in each 2° cell of the globe over the next 30 days, from where earthquakes happen in the long run, how they cluster after recent ones, and boosted trees on both, plus GEAR1, a published global model of where the crust is straining, weighted most where recent earthquakes are few. Scored on 2023–2025 (a second look at those years): +2.61 nats per M6+ cell-window (a 2° cell over a 30-day window in which an M6+ earthquake occurred) of information over a uniform map (S1, GEAR1 with one weight: +2.60). Earthquakes cannot be predicted; this ranks where the odds are higher.

Hurricane

The chance of rapid intensification: maximum sustained winds rising 30 knots or more in 24 hours. For storms the National Hurricane Center covers we publish NOAA DTOPS, which beat NOAA SHIPS-RII, NOAA RIOC consensus and our previous model (v8.2) in a test written down in advance and run once on 2025: AUC 0.928, against 0.849 for our v8.2 model on the same cycles. Everywhere else we publish v8.3 (tested itself on 2,967 2025–2026 cycles in the basins where it publishes, none used to fit it, with best-track inputs: AUC 0.852, log loss 0.1659 against v8.2’s 0.1656; it replaced v8.2 by a rule written down before the test). Each storm says which. Our newer model, v10.1, scored better than DTOPS over 2022–2025, each season forecast only from earlier ones (log loss 0.144 against 0.165); it is shown beside DTOPS, labelled experimental, while a test written down in advance scores it on new forecasts.

Tornado

For every thunderstorm NOAA’s ProbSevere tracks over the contiguous US, the chance that it produces a tornado within the next hour, from NOAA’s own storm attributes (radar, lightning, satellite, environment) and the live NWS tornado-warning state. On every storm observation of 2025, scored once after every choice was fixed: AUC 0.970 (NOAA ProbTor: 0.879).

Earthquake

Earthquake eq_operational_S2_gear1_v2-ecb87f020f99

Target
P(an M6.0+ epicentre in the 2° cell within 30 days), every one of 11,700 cells, from events before the issue time only
Architecture
C0 plus GEAR1's long-term rate (geodetic strain and smoothed seismicity, Bird et al. 2015), weighted by how active C0 says the cell is (156 trees, 22 inputs, scored in NumPy)
How it was chosen
Pre-registered comparison of smoothed seismicity, aftershock clustering, two boosted models and the previous model (docs/EARTHQUAKE_FORECAST_PROGRAM.md); GEAR1 (Bird et al. 2015) was added by a later pre-registered test: three weights fitted on 2018–2020, decided on 2021–2022 against C0 recalibrated on the same years, +0.016 [+0.002, +0.030] nats per M6+ cell-window; a further pre-registered test added a fourth weight so that GEAR1’s weight follows how active the cell is (0.73 in a typical cell, 0.11 where M6+ earthquakes happen), decided against GEAR1 with one weight, +0.014 [+0.002, +0.026] nats per M6+ cell-window
Test 2023–2025 (a second read)
+2.61 [2.49, 2.73] nats per M6+ cell-window (a 2° cell over a 30-day window in which an M6+ earthquake occurred) over a uniform map (1,384 such cell-windows); AUC 0.969 [0.964, 0.973] over all cells, 0.769 [0.735, 0.799] among recently active cells; Brier skill +0.022
Against the standard reference
vs long-term smoothed seismicity (the standard reference forecast): +0.081 [+0.027, +0.132] nats per M6+ cell-window, AUC +0.0019 [+0.0009, +0.0029] (paired, by month)
Replaced
S1, GEAR1 with one weight scored +2.60 nats per M6+ cell-window on the same test; 74% of the test’s M6+ cell-windows fell outside recently active cells
Limits
Forecasts 13% more M6+ cell-windows than occurred in the test period (ratio 1.13 [1.03, 1.25]); live catalogs are preliminary in the first days after a large quake

Hurricane

Hurricane, NHC areas: NOAA DTOPS hurricane_ri_stack_v1-35938d6c09b7

What is served
NOAA DTOPS (the NHC's deterministic-to-probabilistic RI guidance) as issued (SHIPS-RII where it is missing: 21 of 601 cycles in the test), read from NHC’s SHIPS text for the same cycle
How it was chosen
Pre-registered comparison of NOAA’s aids and pools of them on 2020–2024 by forward chaining (docs/HURRICANE_RI_PROGRAM.md)
Final test (2025, read once)
AUC 0.928 [0.890, 0.966], Brier skill +0.311 [+0.183, +0.398]; 601 cycles, 44 RI events, 33 storms (intervals by storm)
Beats
NOAA SHIPS-RII, NOAA RIOC consensus, our previous model (v8.2) (paired 95% intervals on Brier and log loss below zero)
Independent review
An adversarial re-check, by a reviewer that did not build the test, confirmed the result (scoped to the 601-cycle case set)

Hurricane, our model (experimental): v10.1 hurricane_ri_v10-8a5ba5896731

What it is
Our own model: one exceedance curve P(a rise of k kt in 24 h), k = 15–45, from the early intensity guidance (NOAA’s statistical models, the hurricane and global models, NHC’s consensus aids), NOAA’s RI probabilities at six thresholds and the official forecast; NOAA’s DTOPS wherever that guidance is missing
2022–2025, each season forecast only from earlier ones
log loss 0.144 vs DTOPS 0.165 (paired by storm [−0.031, −0.010]); over DTOPS’s four 24-h thresholds, Brier 0.155 vs 0.172 [−0.028, −0.006]
Against every RI probability NOAA publishes
2022–2025, as served, each season predicted from earlier ones; 30-kt log loss SHIPS-RII 0.200 (ours 0.144) · RI logistic 0.197 (ours 0.144) · RI Bayesian 0.232 (ours 0.144) · RI consensus 0.183 (ours 0.144) · DTOPS 0.166 (ours 0.148): ours is lower than every one, each paired by storm with its 95% interval below zero
Against intensity forecasts as yes/no RI calls
at each call’s own false-alarm rate (a forecast 24-h rise of 30 kt or more), ours catches more RI events than SHIPS (DSHP) (+19 points [+6, +31]); and is within noise of NHC official forecast · HCCA · IVCN · LGEM · NNIC · GEFS mean · hurricane-model mean · most aggressive hurricane model · global-model mean (gaps −3 to +9 points)
2026 so far
log loss 0.129 vs DTOPS 0.150, AUC 0.925 (a later look at a season that informed its design, so not a proof)
Variant under test (v10.2)
hurricane_ri_v10_2-3e9c1b766454: the same inputs, never lowering the odds when the guidance or NOAA’s probability rises. 2022–2025 against v10.1: log loss 0.1436 vs 0.1443 [−0.0022, +0.0008], four-threshold Brier 0.1544 vs 0.1554; a gain whose interval still includes zero; 2026 (a later look at a season that informed the design, not a proof): 0.1262 vs 0.1291. Tested live under its own rule, written before it scored a forecast (confidence level 99.375%); live so far, 94 cycles both scored, four-threshold Brier difference +0.010 [−0.004, +0.020]; it replaces the shown model only if it meets that rule
Variant under test (v10.3)
hurricane_ri_v10_3-f941a72e2682: v10.2 plus the storm’s cloud-top structure from satellite infrared (NOAA GMGSI, two hours after the cycle and six hours earlier). 2022–2025 against v10.2: log loss 0.1418 vs 0.1436 [−0.0054, +0.0015], four-threshold Brier 0.1527 vs 0.1544; a gain whose interval still includes zero; 2026 (a later look at a season that informed the design, not a proof): 0.1178 vs 0.1262 [−0.0152, −0.0018]. Tested live under its own rule, written before it scored a forecast (confidence level 99.6875%); live so far, 94 cycles both scored, four-threshold Brier difference +0.035 [+0.002, +0.060]; it replaces the shown model only if it meets that rule
Variant under test (v10.4)
hurricane_ri_v10_4-9fd8cc8dcc7b: v10.3 plus how much of that convective heating the vortex can hold: the coherence equation solved as the balanced response inside the local Rossby radius. 2022–2025 against v10.3: log loss 0.1416 vs 0.1418 [−0.0009, +0.0005], four-threshold Brier 0.1526 vs 0.1527; a gain whose interval still includes zero; 2026 (a later look at a season that informed the design, not a proof): 0.1172 vs 0.1178 [−0.0028, +0.0020]. Tested live under its own rule, written before it scored a forecast (confidence level 99.8438%); live so far, 62 cycles both scored, four-threshold Brier difference +0.057 [−0.005, +0.077]; it replaces the shown model only if it meets that rule
Status
in prospective verification: shown beside NOAA’s number, not instead of it, until the pre-registered prospective test on cycles from 2026-10-04 meets its rule at 98.75% (verdicts 2026-12-01 and 2027-12-01); cycles scored so far: 94. Which of our models is shown is named in one file, results/hurricane_prospective/shown_model.json

Hurricane, NHC areas, the entrant that could replace NOAA DTOPS: v9.1 hurricane_ri_v9-ce02551062e7

What it is
Gradient-boosted trees (166 trees, trained on NHC cycles 2020–2025) on the early intensity guidance, NOAA’s RI probabilities and the official forecast, wherever DSHP, IVCN and NNIC have a 24-hour forecast; NOAA DTOPS where they do not
2026, read once
log loss 0.1316 against NOAA DTOPS’s 0.1495, Brier 0.0349 against 0.0423, AUC 0.924 against 0.893 (586 cycles, 35 RI events): better on all three; the claim rule, an interval wholly below zero, was not met, so NOAA DTOPS stays published
Why it still matters
It is the one entrant whose claim would change what is published: if its prospective test meets its rule at a look (97.5% intervals; 2026-12-01 and 2027-12-01), it replaces NOAA DTOPS as the published number in the NHC areas
Live so far (descriptive, no claim before a look)
94 cycles from 4 storms (17 RI events): log loss 0.215 against NOAA DTOPS’s 0.310, difference −0.095 [−0.233, +0.008] at 95%
Status
in shadow: recorded on every live NHC cycle, never shown as the forecast

Hurricane, other basins: HazardPulse v8.3 hurricane_ri_v8_3

Architecture
Histogram-GBT (depth 3 + 4) + L2 logistic + bagged logistic ensemble, Newton-calibrated
Key inputs
Analysis intensity and pressure and their 6–24 h tendencies, position, time of year, climatological potential intensity, motion, storm age (no forecast aids)
Its registered test
AUC 0.852 on 2,967 2025–2026 cycles (158 RI events, 130 storms) in the basins where it publishes (South Indian 1,302, West Pacific 1,078, South Pacific 440, North Indian 147), none used to fit it; log loss 0.16586 against v8.2’s 0.16560 on the same cycles (difference +0.00027 [−0.00070, +0.00137], a 95% interval by storm), Brier 0.04556 against 0.04543. It replaced v8.2 by a rule written down before the test (amendment 15, docs/HURRICANE_RI_V9_PROGRAM.md): the upper end of that interval at most +0.002; it was +0.00137
Used for
West Pacific, Indian Ocean and Southern Hemisphere storms (no public RI guidance), and NHC cycles without SHIPS text
Limits of that test
The test read every input from the best track; until 2026-10-05 live West Pacific storms were scored from one JTWC warning, leaving 9 of 17 inputs unknown (and still are when RAL’s real-time best track cannot be read); scored that way the 2025–2026 West Pacific cycles had log loss 0.2269 against 0.2403 for climatology (0.1771 with every input; 0.1834 with the history but this cycle’s fix not yet published, 4 inputs filled); the live history is the working best track, which the post-season one the test read revises.

Hurricane, other basins, our satellite model (in test): J1 hurricane_ri_j1-6ffdec7356c1

What it is
Our own rapid-intensification model for the West Pacific, North Indian and Southern Hemisphere basins, where NOAA publishes no such guidance: gradient-boosted trees (5 seeds, averaged) on v8.2’s own probability, its 17 inputs, 14 measures of the storm’s cloud tops from NOAA’s GMGSI satellite infrared (2 hours after the cycle and 4 hours before) and the basin; trained on storms of 2022–2025, all six basins (v8.2 is out of sample on every one). It is recorded beside the published v8.3, never instead of it; it is built on v8.2 and tested against it
Registered test, 2024–2025 (a hindcast)
Every cycle of the West Pacific, North Indian, South Indian and South Pacific basins, pooled, each season forecast from earlier ones only, with the satellite crops centred on post-season best-track positions.
J1 against v8.2, 2024–2025 (lower log loss and Brier are better)
CyclesRI eventsLog lossBrierAUC
v8.2 as published (the bar)3,5461580.141210.037680.867
J1 without the satellite inputs (a control)3,5461580.142530.037940.870
J13,5461580.133600.036860.898
J1 minus v8.2: log loss −0.00761 [−0.01472, −0.00034], Brier −0.00083 [−0.00304, +0.00133] (95% intervals by storm; below zero is better). Both point estimates are below v8.2’s, the rule written before the result, so J1 was carried on this pooled hindcast; the claim is not made basin by basin, nor for live forecasts
What the satellite adds
J1 minus the same model without the satellite inputs: log loss −0.00892 [−0.01380, −0.00390]. The control alone scored 0.14253 against v8.2’s 0.14121: re-learning v8.2’s inputs does not help, the gain is the satellite’s. With the 14 satellite columns shuffled within each season (a noise control, 3 seeds), the same comparison gave −0.00026 to +0.00067
How narrow the pooled interval is
From the program (amendment 14), after an independent pass that did not build J1: “It excludes 0 by 0.0003. Each of these keeps about 77-85% of the point gain, but every interval then includes 0:”
  • moving the test crops 15 km: -0.0064 [-0.0133, +0.0005];
  • moving them 30-60 km at random: -0.0059 to -0.0061;
  • dropping one storm (Neoguri 2025): -0.0064 [-0.0127, +0.0001].
(docs/HURRICANE_RI_V9_PROGRAM.md)
Each region, 2024–2025
The registered outcome’s table merged the South Indian and South Pacific into one Southern Hemisphere row; here each region is apart.
J1 minus v8.2 by region, 2024–2025 (95% intervals by storm; below zero is better)
RegionCyclesRI eventsLog lossBrierShown
West Pacific1,16676−0.0096 [−0.0247, +0.0045]−0.0007 [−0.0051, +0.0038]yes
North Indian2990+0.0041 [−0.0055, +0.0159]+0.0023 [+0.0001, +0.0059]no
South Indian1,69081−0.0139 [−0.0237, −0.0047]−0.0031 [−0.0063, −0.0002]yes
South Pacific3911+0.0164 [+0.0051, +0.0362]+0.0062 [+0.0022, +0.0129]no
2026 so far (a further read, no claim)
1,130 cycles, 90 RI events, every JTWC region pooled: log loss 0.1953 against v8.2’s 0.2150 (difference −0.0197 [−0.0355, −0.0034]), Brier 0.0582 against 0.0622, AUC 0.893 against 0.858
Where it is shown
West Pacific and South Indian storms only, by the rule written before the result (amendment 14): the regions where its 2024–2025 log loss against v8.2 is at or below zero. North Indian: J1’s log loss was above v8.2’s on 2024–2025, with an interval that includes zero (0 RI events in 299 cycles), so it is not shown there; on 2026, J2 (a variant of J1 registered in amendment 14, not carried) scored log loss +0.0211 against v8.2 there (0 RI events in 2 cycles), the same direction. South Pacific: J1 tested worse than v8.2 on 2024–2025, beyond its interval (1 RI event in 391 cycles), so it is not shown there; on 2026, J2 (a variant of J1 registered in amendment 14, not carried) scored log loss −0.0482 [−0.0953, +0.0011] against v8.2 there (20 RI events in 222 cycles): the rule excludes it although that season points the other way. J1 is still recorded on every West Pacific, North Indian and Southern Hemisphere storm, so the record stays complete, but it is shown, and its claim is judged, only in its scope (results/calibration/hurricane_ri_j2.json)
Untested: North Indian
The North Indian basin had no RI event in the test seasons (299 cycles), so J1 is untested there on RI events; where nothing happened it forecast higher than v8.2 (Brier +0.0023 [+0.0001, +0.0059])
Status
In test beside v8.3: never the published number. Its test against v8.2 (published until v8.3 replaced it; its comparator is unchanged) was written before its first live cycle (scripts/score_hurricane_j1_prospective.py): at each look (2026-12-01 and 2027-12-01), on live cycles in its scope only, the claim is met if J1 minus v8.2 has a 97.5% storm-bootstrap interval wholly below zero on log loss or on Brier, with both point estimates at or below zero; a met claim makes J1 the published number. The hindcast above is not evidence for that. Live so far, in its scope (descriptive, no claim before a look): 37 cycles from 3 storms (4 RI events), log loss 0.139 against 0.172, difference −0.033 [−0.057, −0.013] at 97.5%

Hurricane track and intensity, NHC areas: HazardPulse TC1 hurricane_tc1-589cd13ad9b6

What it is
Our own track and intensity forecast to five days for every storm the National Hurricane Center follows: the real-time computer guidance NHC publishes, each model weighted by how its recent forecasts verified, re-measured every six hours; a new model earns weight as it verifies
How it was chosen
Pre-registered: settings chosen on 2021–2022, tested on 2023–2025 against NHC’s official forecast (docs/HURRICANE_TRACK_INTENSITY_PROGRAM.md)
Track, 2023–2025
mean error 90.74 n mi over the 5 leads from 24 to 120 h; NHC official 92.08; NHC’s corrected consensus HCCA 97.61
Track against the official forecast (the claim)
−2.76 [−8.18, +1.95] n mi at 98.75%: not met; the interval includes zero, so this is a tie
Track against HCCA (descriptive, not a claim)
−6.24 [−11.14, −1.36] n mi at 95%; ahead at every lead from 72 h
Intensity, 2023–2025
mean error 12.83 kt against NHC official 12.63; against the official forecast +0.25 [−0.63, +1.11] kt at 98.75%: not met; the interval includes zero, so this is a tie
TC1+O, also recorded
The same weighting with NHC’s official forecast as one more member, issued after the advisory: track 90.47 n mi, against the official forecast −3.05 [−8.22, +1.40] at 98.75% (not met); intensity 12.60 kt, −0.00 [−0.69, +0.70] (not met)
2026 so far (no claim)
against NHC’s official forecast, on operational best tracks (31 storms, read 2026-10-08): track −0.90 [−3.95, +2.22] n mi, intensity +1.04 [−0.64, +2.22] kt (95% intervals)
Status
Shown beside NHC’s official forecast on every NHC storm, never instead of it; NHC is the authority. Its live record is scored at each look against NHC’s best tracks

Tornado

Tornado, storm by storm tornado_v3-05a06c843c87

Target
A tornado report starts within 10 km of this storm's tracked polygon within 60 min (also served: tornado within 30 min, tornado within 90 min, ef2+ tornado within 60 min)
Architecture
LightGBM (339 trees), exported to a JSON payload scored in pure NumPy; Platt calibration on leave-one-year-out scores; Venn–Abers band per storm
Inputs (30)
28 ProbSevere storm attributes, 2 NWS tornado-warning state
How it was chosen
Pre-registered programme: every choice on 2023; development test 2024; refit 2020–2024; final test 2025 read once (docs/TORNADO_MODEL_PROGRAM.md)
Final test 2025
AUC 0.970 [0.964, 0.976] on all 1,469,977 storm observations (1,579 tornadic); Brier skill +0.107. 2024: 0.969
Against NOAA
NOAA ProbTor scores 0.879 as issued and 0.939 with its integer ties broken; the model is +0.031 [+0.022, +0.042] AUC above the tie-broken version, paired by day; at the NWS warnings’ own false-alarm rate it catches 30.5% of tornadic storm observations vs the warnings’ 22.7% (+7.7 [+3.8, +10.1] points, using the warning state as an input); without that input it catches 27.5% (+4.7 [−0.2, +8.5] points)
NOAA’s 2025 format change
A test written down in advance scored the published model on every storm observation from 2026-01-01 to 2026-09-30 (1,262,285 of them, 1,500 followed by a tornado), all in the new format: AUC 0.9699, against 0.9702 on its 2025 test, so no loss of ranking is visible (2026 is scored on SPC’s preliminary reports). A model built without the inputs the feed lost did not rank better (−0.0003 [−0.0012, +0.0007] AUC, paired by day), so the published model stays. It may cost calibration: over that period the model’s average forecast was 0.81 times the rate at which tornadoes followed. A recalibration for the new format is an open question, to be judged on forecasts made after it is registered (docs/TORNADO_MODEL_PROGRAM.md, amendment 10; results/tornado_program/t2_2026.json).
Coherence field
Tested on top of the HRRR fields: −0.0011 [−0.0032, +0.0007] validation AUC (paired); no lift, so not served
Status
Published

Research record: carried and not carried

The registered tests whose results are in, since the published models were chosen, including the ones that changed nothing. Each was written down, with the rule that decides it, before its result existed. None of them changes a published number on its own: a carried model runs in shadow and must still pass its own test on new forecasts.

Registered tests, decided by rules written before their results existed
What was testedResultDecisionResults file
The coherence equation solved for the convective heating the vortex can hold inside its local Rossby radius, from the satellite infrared and NHC’s radius of maximum wind (amendment 8)4 features: against v10.3 on 2022–2025 (2,871 cycles, 193 RI events): log loss −0.00016 [−0.00089, +0.00054], four-threshold Brier −0.00013 [−0.00083, +0.00060] (below zero is better)Carried: it is v10.4, in shadow (both point estimates below the model it was tested against)results/calibration/hurricane_ri_h8.json
A memory of each storm’s recent convection, symmetry, heating retention and intensity trend (its “coherence state”) (amendment 10)8 features: against v10.4 on 2022–2025 (2,871 cycles, 193 RI events): log loss +0.00275 [+0.00111, +0.00455], four-threshold Brier +0.00235 [+0.00025, +0.00465] (below zero is better)Not carried: worse on both measures, beyond its interval on both measuresresults/calibration/hurricane_ri_h9.json
Every RI probability, ours and NOAA’s, weighted by how each has verified, with old verifications forgotten (amendment 11)9 sources; the fusion minus the best single source on the choosing season (v10.4) on 2024–2025, 1,220 cycles: log loss +0.0036 [−0.0014, +0.0082]Claim not met: the interval includes zeroresults/calibration/hurricane_ri_fusion.json
The inner core seen hourly at 2 km by GOES: core and ring convection, the eye, the eyewall’s edge, symmetry, persistence and their trends (amendments 12, 12b)12 features: against v10.4 on 2022–2025 (2,871 cycles, 193 RI events): log loss +0.00061 [−0.00027, +0.00155], four-threshold Brier +0.00116 [+0.00008, +0.00233] (below zero is better); the same columns shuffled within each season (a noise control) gave log loss −0.00005 to +0.00052Not carried: worse on both measures, beyond its interval on Brierresults/calibration/hurricane_ri_g1.json
Input check for the 2 km GOES test: is the eye where ADT says it is?the 2 km eye detector against CIMSS’s Advanced Dvorak Technique on 29,010 matched images: skill (HSS) 0.834 on the held-out 2024–2026 images (required 0.80); an eye called in 1.1% of images below 50 kt (allowed 3.0%); centre 9.6 km from ADT’s (median); eye temperature rank correlation 0.959Passed every gate, so the satellite input was sound before any outcome was readresults/goes/adt_check.json
Satellite infrared (the storm’s cloud tops from NOAA GMGSI) for the West Pacific, North Indian and Southern Hemisphere, where NOAA publishes no RI guidance and we publish our own model (amendment 13)36 features: against v8.2 as published on 2024–2025, every JTWC cycle pooled (3,546 cycles, 158 RI events; satellite crops on best-track positions): log loss −0.00761 [−0.01472, −0.00034], Brier −0.00083 [−0.00304, +0.00133] (below zero is better); what the satellite adds, against the same model without it: log loss −0.00892 [−0.01380, −0.00390]; the satellite columns shuffled within each season (a noise control) gave log loss −0.00026 to +0.00067Carried on the pooled hindcast: it is J1, in test beside v8.2, shown only for West Pacific and South Indian storms (both point estimates below the model it was tested against)results/calibration/hurricane_ri_j1.json
Separate South Indian and South Pacific basins in that satellite model, after an independent pass found it worse than the published model in the South Pacific (amendment 14)J2 minus J1 on 2026 (1,130 cycles, 90 RI events): log loss +0.00038 [−0.00112, +0.00172], Brier −0.00002 [−0.00046, +0.00038] (below zero is better)Not carried: its log loss is not below J1’s; J1 stays the live entrant, in its scope (West Pacific and South Indian)results/calibration/hurricane_ri_j2.json
Tornado: a model without the inputs NOAA’s 2025 format change removed (amendment 10)AUC −0.0003 [−0.0012, +0.0007] against the published model, 2026-01-01 to 2026-09-30, paired by dayNot carried: the published model staysresults/tornado_program/t2_2026.json

How models are chosen and tested

Each model was chosen by a test written down before the data that decides it was scored: the candidates, the measure, the years and the rule for a winner are fixed first, then the test is run once. The protocols, and every dated amendment to them, are in the source repository.

  • Held-back years. A model is scored on years it never saw while it was being built. Where a year has informed a design, a later look at it is labelled “a second look”, not a test.
  • Paired comparisons. Two models are compared on the same cases, with intervals that resample whole days, months or storms, so that one busy week cannot decide the result.
  • Live records. Every published forecast is frozen with its inputs, then scored when its window closes. A live score is quoted only once enough events have been observed for it to mean something.
  • Experimental models. A new model can be shown beside the published number, labelled experimental, while a test written down in advance scores it on new forecasts. It replaces the published model only if it meets that test’s rule.

The measures

AUC (ranking accuracy)
How often the forecast ranks a case that had the event above one that did not. 0.5 is a coin flip; 1 is perfect.
Brier score
The average squared gap between the forecast and the outcome (0 or 1). Lower is better.
Brier skill score
The improvement over always forecasting the average rate. Above 0 beats that baseline.
Log loss
Penalises confident wrong forecasts heavily. Lower is better.
Information gain
For earthquakes: how much better than a uniform map the forecast predicted where M6+ earthquakes happened, in nats per M6+ cell-window (a 2° cell over a 30-day window in which an M6+ earthquake occurred). Higher is better.
Calibration
Whether events happen as often as forecast: of many 10% forecasts, about 1 in 10 should come true.

Checks before publishing

Every forecast goes through automatic quality checks when it is published: its numbers are well formed, its inputs are fresh, its record is complete, its values are sane, it can be re-run, and its labels never borrow the terms of official warnings. Each check ends in one of three outcomes:

  • Passed: the forecast is published.
  • Published with warnings: a non-blocking check flagged something; the forecast is published and the warning is shown beside it.
  • Blocked: the forecast is withheld from every page, and the page says why.

What each check tests, and how often each outcome occurred this week

Limits

  • These are research forecasts, not warnings. Follow the National Weather Service, the National Hurricane Center, the USGS, your national services and your local authorities for every safety decision.
  • Earthquakes cannot be predicted. The forecast ranks where the odds are higher; it does not say when or exactly where an earthquake will happen. Where few earthquakes have been recorded, its numbers are less certain.
  • Coverage. Earthquake: the globe between 60°S and 70°N on a 2° grid. Hurricane: every active tropical cyclone, with NOAA’s guidance in the Atlantic, East and Central Pacific and our v8.3 model elsewhere. v8.3 was tested on 2,967 2025–2026 cycles in the basins where it publishes (South Indian 1,302, West Pacific 1,078, South Pacific 440, North Indian 147), none used to fit it, with best-track inputs. Tornado: every storm NOAA’s radar tracks over and near the contiguous US, including northern Mexico and the Gulf; tornado reports, and so the tests, cover the US only.
  • Tornado inputs. NOAA changed the format of its ProbSevere feed on 5 August 2025, and some inputs the model learned from are no longer in it. A test written down in advance scored the published model on every storm observation from 2026-01-01 to 2026-09-30 (1,262,285 of them, 1,500 followed by a tornado), all in the new format: AUC 0.9699, against 0.9702 on its 2025 test, so no loss of ranking is visible (2026 is scored on SPC’s preliminary reports). A model built without the inputs the feed lost did not rank better (−0.0003 [−0.0012, +0.0007] AUC, paired by day), so the published model stays. It may cost calibration: over that period the model’s average forecast was 0.81 times the rate at which tornadoes followed. A recalibration for the new format is an open question, to be judged on forecasts made after it is registered (docs/TORNADO_MODEL_PROGRAM.md, amendment 10; results/tornado_program/t2_2026.json).
  • Preliminary data. Live catalogs and storm reports are revised after the fact; forecasts use what was available at issue time, and scores use the revised records.
  • Model changes. A new model version can shift the numbers for reasons that reflect the model, not a change in the hazard. Every version change is listed in the model registry.
  • Calibration is measured on finite samples. Rare events make rare-event calibration uncertain; intervals are shown wherever they are computed.