AI tournament · closed

Five AIs forecast the market every morning for six weeks. None did better than chance.

Haiku 4.5, Sonnet 5 and Opus 5 (and, from 25 September, one model from DeepSeek and one from Alibaba) received the same data package on 10 S&P 500 names every session from 3 September to 16 October 2026 and returned, before the open, the probability that each one would close up and how much it would move. At the close we scored them with Brier and CRPS and published the result whatever it said.

Tournament closed on Oct 18. It ended with 315 resolved predictions per model, more than the 200 pre-registered before claiming anything. No model forecast direction better than always saying the base rate and, with enough data, none sized the move better than a mechanical forecast based on volatility. Last session: Oct 16. What you see is the final state.

What is measured

Calibration, not hit rate: Brier and its decomposition (resolution) for direction; a three-quantile CRPS and the coverage of the 80% band for the size of the move.

What we expect

The pre-registered expectation is that no AI discriminates one-day direction: in this universe it is measured as null. Where value may exist is in how much a name moves, not which way.

What it is not

Not a recommendation or a signal. No model sees headlines or searches the web; only prices, volatility, catalyst news counts, the main macro calendar (Fed, CPI, jobs) and, in variant B, quarterly-results context.

Backtest results (simulation), not real trading. Not investment advice. Past performance does not guarantee future results.

Reading

Sentences generated by fixed rules from the numbers, using the pre-registered thresholds. Nobody writes them by hand.

Updated 2026-10-18 09:18 UTC

Paper P&L

Fixed, pre-registered rule: each name is held long or short in proportion to how far the probability sits from 0.50, equal capital per name, one-session holding, close to close, 10 bp round-trip cost. The basket is the same list of names bought equal-weight. 🏆 = best Sharpe among models with the same sessions; returns are not compared across models or against the basket, because each book has a different exposure and the basket is 100% invested. Below 250 sessions the leader is noise.

ModelVar.SessionsNet returnbp/daySharpeMax drawdownBasketvs basketp
Haiku 4.5E10.11 %11.1–0.0 %1.78 %-1.67 %–
deepseek-v4.1-flashA160.05 %0.3–-0.1 %-2.86 %2.91 %0.61
deepseek-v4.1-flashR30.05 %1.6–0.0 %-2.96 %3.01 %–
Opus 5A300.05 %0.21.48-0.0 %-6.54 %6.58 %0.35
Sonnet 5R50.05 %0.9–-0.0 %-3.78 %3.82 %0.07
EnsembleA160.04 %0.2–-0.1 %-2.86 %2.90 %0.61
deepseek-v4.1-flashB150.03 %0.2–-0.1 %-0.91 %0.94 %0.85
EnsembleR30.03 %1.1–0.0 %-2.96 %2.99 %–
Sonnet 5B290.03 %0.10.79-0.1 %-5.21 %5.24 %0.43
Opus 5B280.03 %0.11.00-0.1 %-6.78 %6.81 %0.34
qwen3.8-flashR20.02 %1.2–0.0 %-1.63 %1.65 %–
qwen3.8-flashX20.02 %1.0–0.0 %-1.16 %1.18 %–
Haiku 4.5X60.01 %0.2–-0.0 %-2.10 %2.11 %0.27
EnsembleB150.01 %0.0–-0.1 %-0.91 %0.92 %0.85
Sonnet 5A310.00 %0.00.03-0.1 %-4.97 %4.97 %0.48
Sonnet 5E10.00 %0.0–0.0 %1.78 %-1.78 %–
qwen3.8-flashE10.00 %0.0–0.0 %1.78 %-1.78 %–
EnsembleE1-0.00 %-0.2–-0.0 %1.78 %-1.79 %–
Opus 5R5-0.00 %-0.1–-0.0 %-3.78 %3.77 %0.07
Sonnet 5X5-0.01 %-0.1–-0.0 %-2.10 %2.10 %0.28
EnsembleX3-0.01 %-0.3–-0.0 %0.08 %-0.08 %–
Opus 5E1-0.01 %-1.3–-0.0 %1.78 %-1.80 %–
qwen3.8-flashA15-0.02 %-0.1–-0.1 %-1.53 %1.51 %0.77
Opus 5X6-0.02 %-0.3–-0.0 %-2.10 %2.08 %0.27
deepseek-v4.1-flashX3-0.02 %-0.6–-0.0 %0.08 %-0.09 %–
deepseek-v4.1-flashE1-0.04 %-4.0–-0.0 %1.78 %-1.82 %–
qwen3.8-flashB12-0.04 %-0.3–-0.1 %-0.46 %0.42 %0.93
Haiku 4.5R5-0.06 %-1.2–-0.1 %-3.78 %3.72 %0.08
Haiku 4.5A31-0.09 %-0.3-0.66-0.4 %-4.97 %4.88 %0.48
Haiku 4.5B29-0.10 %-0.3-3.25-0.1 %-5.21 %5.11 %0.44

This is a paper book measured close to close: a real execution would enter at the open, and open prices are not available. None of this is a recommendation.

One-session totals

A = without results context, B = with it, X = control with the context shuffled across names. Bets are the names where the model moved away from 0.50.

ModelVar.nHits/betsBrierResolutionvs base rateCRPSBaseline80% coverage
Haiku 4.5A315119/2520.25170.0034-0.00171.58 %1.11 %0.54
Haiku 4.5X609/210.25040.0024-0.00071.81 %0.84 %0.40
Haiku 4.5B29073/1670.25050.0021-0.00052.15 %1.18 %0.46
Haiku 4.5E105/100.24380.0171-0.08381.11 %1.50 %0.60
Haiku 4.5R5014/270.25100.0020-0.00742.01 %1.34 %0.42
Opus 5A30530/590.25000.0000-0.00001.15 %1.10 %0.68
Opus 5X606/130.25020.0053-0.00050.91 %0.84 %0.72
Opus 5B28043/820.24970.00050.00031.25 %1.17 %0.68
Opus 5E100/10.25100.0000-0.09101.22 %1.50 %0.80
Opus 5R503/50.25000.0039-0.00641.43 %1.34 %0.58
Sonnet 5A31574/1400.24950.00060.00041.22 %1.11 %0.74
Sonnet 5X5012/240.24950.0054-0.00110.91 %0.68 %0.74
Sonnet 5B290114/2250.24980.00000.00021.39 %1.18 %0.69
Sonnet 5E100.50 everywhere0.25000.0000-0.09001.36 %1.50 %0.60
Sonnet 5R5012/190.24810.0015-0.00451.43 %1.34 %0.66
deepseek-v4.1-flashA16058/1150.25020.0014-0.00061.16 %1.04 %0.74
deepseek-v4.1-flashX305/120.25040.0171-0.00150.95 %0.65 %0.70
deepseek-v4.1-flashB15051/970.24920.0002-0.00081.38 %1.22 %0.67
deepseek-v4.1-flashE102/100.25610.0000-0.09611.11 %1.50 %0.80
deepseek-v4.1-flashR3018/300.24810.0000-0.00811.89 %1.70 %0.60
EnsembleA16075/1580.25030.0000-0.00071.14 %1.04 %0.73
EnsembleX3016/290.24960.0142-0.00070.90 %0.65 %0.73
EnsembleB15071/1440.24950.0001-0.00111.37 %1.22 %0.65
EnsembleE104/100.25140.0100-0.09141.22 %1.50 %0.80
EnsembleR3018/300.24840.0000-0.00841.79 %1.70 %0.57
qwen3.8-flashA15045/1030.25090.0041-0.00131.10 %0.89 %0.71
qwen3.8-flashX2013/200.24660.0189-0.00661.07 %0.64 %0.60
qwen3.8-flashB12053/1070.24970.0006-0.00031.35 %1.10 %0.62
qwen3.8-flashE100.50 everywhere0.25000.0000-0.09001.21 %1.50 %0.70
qwen3.8-flashR2013/200.24710.0000-0.01960.99 %0.93 %0.65

Resolution > 0.02 = the probabilities discriminate. vs base rate > 0 = beats always saying the average rate. CRPS: lower is better; the baseline is a mechanical forecast from volatility, no AI. Target coverage: 0.80.

A/B of the quarterly-results context

The same package with and without the results block (date, whether the recent move is already the reaction, the name's historical move size). CRPS difference paired by session and name.

A/B re-registered on 9 October 2026. Until then, when the calendar had no entry, the block took the first “earnings” news item as the announcement date. Checked on 8 October against the real record of earnings reports, none of the 35 news-based dates matched a report (calendar dates: 28 of 28). From now on dates come only from the calendar and the count of pairs with an event starts again from zero.
ModelSubsetPairsCRPS ACRPS BChangep
Haiku 4.5all601.47 %1.82 %+24 %0.055
Haiku 4.5with event120.99 %1.23 %+24 %0.284
Opus 5all600.96 %0.90 %−7 %0.360
Opus 5with event120.82 %0.85 %+3 %0.742
Sonnet 5all600.90 %0.96 %+7 %0.491
Sonnet 5with event120.81 %0.94 %+17 %0.262
deepseek-v4.1-flashall600.92 %0.92 %+1 %0.931
deepseek-v4.1-flashwith event120.81 %1.01 %+25 %0.097
Ensembleall600.89 %0.89 %−0 %0.839
Ensemblewith event120.71 %0.94 %+32 %0.045
qwen3.8-flashall600.86 %0.89 %+4 %0.523
qwen3.8-flashwith event120.66 %0.92 %+39 %0.194

Pre-registered criterion: improvement ≥ 10% with p < 0.05 and 200 pairs with an event. p is hidden with fewer than 5 sessions.

Earlier epochs (they do not count towards the criterion)

v2 (24 September) and v1 (original rule) were measured with results dates that sometimes came from news. They are kept so nothing is lost, but they are not evidence for the criterion.

ModelEpochSubsetPairsCRPS ACRPS BChangep
Haiku 4.5v2all1101.66 %2.34 %+41 %0.001
Haiku 4.5v2with event381.26 %2.41 %+91 %0.001
Haiku 4.5v1all1201.55 %2.15 %+39 %0.001
Haiku 4.5v1with event221.56 %2.13 %+36 %0.132
Opus 5v2all1101.24 %1.44 %+16 %0.064
Opus 5v2with event380.90 %1.65 %+84 %0.001
Opus 5v1all1101.16 %1.24 %+7 %0.246
Opus 5v1with event221.21 %1.56 %+29 %0.273
Sonnet 5v2all1101.23 %1.62 %+32 %0.001
Sonnet 5v2with event380.94 %1.83 %+95 %0.001
Sonnet 5v1all1201.38 %1.39 %+1 %0.960
Sonnet 5v1with event221.27 %1.64 %+29 %0.446
deepseek-v4.1-flashv2all901.35 %1.69 %+25 %0.001
deepseek-v4.1-flashv2with event331.09 %2.11 %+94 %0.001
Ensemblev2all901.34 %1.69 %+26 %0.001
Ensemblev2with event331.00 %1.95 %+94 %0.001
qwen3.8-flashv2all601.40 %1.81 %+29 %0.001
qwen3.8-flashv2with event251.19 %2.43 %+104 %0.001

By session

Each cell: hits/bets · Brier · magnitude CRPS. ½ = the model said 0.50 for every name.

SessionClosed up|ret| medianHaiku 4.5 AHaiku 4.5 BSonnet 5 ASonnet 5 BOpus 5 AOpus 5 B
Oct 166/101.47 %4/7 · 0.248 · 1.30 %2/2 · 0.246 · 1.74 %½ · 0.250 · 1.04 %6/9 · 0.245 · 1.00 %½ · 0.250 · 0.94 %2/3 · 0.247 · 0.97 %
Oct 158/102.25 %4/6 · 0.249 · 1.18 %2/10 · 0.256 · 1.02 %0/1 · 0.254 · 1.27 %2/9 · 0.255 · 1.17 %0/2 · 0.253 · 1.29 %0/6 · 0.256 · 1.47 %
Oct 144/101.23 %4/6 · 0.247 · 0.93 %1/1 · 0.248 · 1.59 %½ · 0.250 · 0.66 %7/10 · 0.245 · 0.73 %1/1 · 0.249 · 0.61 %3/4 · 0.248 · 0.65 %
Oct 133/101.18 %4/5 · 0.246 · 1.82 %2/8 · 0.257 · 2.70 %7/10 · 0.246 · 0.94 %6/10 · 0.249 · 0.91 %1/1 · 0.249 · 1.06 %6/10 · 0.249 · 0.93 %
Oct 126/100.53 %4/5 · 0.246 · 2.36 %4/8 · 0.251 · 2.09 %½ · 0.250 · 0.68 %4/10 · 0.252 · 1.15 %0/3 · 0.253 · 0.98 %4/8 · 0.250 · 0.62 %
Oct 94/102.22 %1/6 · 0.259 · 1.26 %2/5 · 0.251 · 1.80 %½ · 0.250 · 0.78 %5/8 · 0.247 · 0.78 %0/1 · 0.251 · 0.91 %1/2 · 0.249 · 0.76 %
Oct 84/102.11 %2/8 · 0.257 · 1.21 %2/5 · 0.252 · 1.37 %½ · 0.250 · 1.01 %4/7 · 0.250 · 1.45 %0/2 · 0.252 · 1.01 %0/2 · 0.253 · 1.13 %
Oct 74/101.37 %5/9 · 0.245 · 1.87 %3/6 · 0.251 · 2.04 %½ · 0.250 · 1.06 %4/7 · 0.250 · 1.47 %1/2 · 0.251 · 0.93 %2/5 · 0.252 · 1.19 %
Oct 64/100.85 %4/9 · 0.252 · 2.02 %1/2 · 0.250 · 2.03 %1/3 · 0.251 · 1.08 %2/5 · 0.251 · 1.69 %2/3 · 0.249 · 0.98 %1/4 · 0.252 · 1.41 %
Oct 56/101.03 %2/8 · 0.260 · 1.79 %4/10 · 0.250 · 3.18 %1/2 · 0.250 · 1.25 %6/10 · 0.246 · 2.32 %1/2 · 0.250 · 1.28 %4/7 · 0.249 · 2.08 %
Oct 26/103.79 %6/9 · 0.244 · 1.37 %3/9 · 0.253 · 1.81 %3/9 · 0.259 · 1.14 %3/8 · 0.252 · 1.43 %½ · 0.250 · 1.17 %1/2 · 0.250 · 1.40 %
Oct 18/101.88 %1/8 · 0.267 · 1.19 %3/7 · 0.248 · 2.27 %½ · 0.250 · 0.99 %4/8 · 0.247 · 2.09 %2/3 · 0.249 · 0.98 %3/3 · 0.244 · 1.72 %
Sep 308/102.27 %5/8 · 0.244 · 1.11 %2/2 · 0.246 · 1.84 %0/3 · 0.253 · 0.87 %3/7 · 0.249 · 1.20 %½ · 0.250 · 0.68 %3/5 · 0.247 · 0.83 %
Sep 295/100.99 %5/8 · 0.248 · 3.47 %5/10 · 0.250 · 5.73 %6/8 · 0.242 · 3.30 %5/9 · 0.248 · 3.34 %2/3 · 0.249 · 3.32 %4/5 · 0.246 · 3.38 %
Sep 282/102.01 %5/10 · 0.251 · 1.26 %6/7 · 0.245 · 2.46 %2/5 · 0.251 · 0.82 %7/8 · 0.244 · 0.94 %2/2 · 0.248 · 0.84 %3/4 · 0.248 · 0.83 %
Sep 255/103.35 %6/9 · 0.246 · 1.33 %4/8 · 0.251 · 1.65 %5/6 · 0.243 · 0.98 %3/9 · 0.253 · 0.99 %1/2 · 0.250 · 0.98 %½ · 0.250 · 1.01 %
Sep 245/101.05 %5/9 · 0.257 · 1.63 %4/7 · 0.249 · 1.34 %2/3 · 0.249 · 1.01 %3/7 · 0.251 · 0.88 %1/2 · 0.250 · 1.44 %½ · 0.250 · 0.87 %
Sep 234/102.00 %4/8 · 0.250 · 1.04 %4/7 · 0.249 · 1.58 %½ · 0.250 · 0.60 %3/6 · 0.250 · 0.62 %½ · 0.250 · 0.59 %½ · 0.250 · 0.61 %
Sep 227/101.28 %7/9 · 0.241 · 1.77 %0/2 · 0.254 · 2.62 %1/4 · 0.253 · 1.13 %1/9 · 0.259 · 2.12 %0/4 · 0.255 · 1.12 %0/2 · 0.254 · 1.76 %
Sep 216/103.07 %4/7 · 0.245 · 1.55 %½ · 0.250 · 2.87 %2/3 · 0.250 · 1.57 %4/7 · 0.249 · 1.59 %––
Sep 185/101.67 %4/9 · 0.241 · 1.23 %5/10 · 0.242 · 2.91 %5/10 · 0.250 · 0.84 %5/8 · 0.247 · 1.24 %2/3 · 0.249 · 0.79 %1/2 · 0.249 · 0.90 %
Sep 179/102.13 %2/9 · 0.264 · 1.28 %2/9 · 0.254 · 1.42 %4/8 · 0.249 · 0.97 %1/7 · 0.254 · 0.94 %½ · 0.250 · 1.00 %1/1 · 0.248 · 0.96 %
Sep 163/101.56 %6/10 · 0.245 · 1.51 %2/4 · 0.251 · 1.32 %4/6 · 0.247 · 1.29 %7/10 · 0.246 · 0.83 %2/3 · 0.250 · 0.97 %½ · 0.250 · 0.80 %
Sep 152/101.58 %3/9 · 0.255 · 1.39 %3/9 · 0.255 · 1.57 %8/10 · 0.244 · 0.70 %2/7 · 0.256 · 0.65 %1/1 · 0.249 · 0.79 %1/1 · 0.249 · 0.67 %
Sep 144/105.30 %1/7 · 0.257 · 1.29 %3/9 · 0.253 · 1.23 %2/6 · 0.252 · 1.96 %5/7 · 0.247 · 1.69 %½ · 0.250 · 1.79 %½ · 0.250 · 2.09 %
Sep 115/102.32 %6/8 · 0.233 · 1.75 %2/4 · 0.250 · 1.87 %3/7 · 0.250 · 1.75 %2/5 · 0.251 · 1.65 %½ · 0.250 · 1.65 %1/3 · 0.251 · 1.68 %
Sep 103/102.42 %2/8 · 0.266 · 1.25 %0/3 · 0.253 · 1.92 %4/8 · 0.248 · 1.86 %4/8 · 0.250 · 1.73 %1/3 · 0.251 · 1.44 %1/1 · 0.249 · 1.27 %
Sep 96/101.69 %6/9 · 0.244 · 2.34 %1/1 · 0.248 · 3.55 %3/4 · 0.248 · 1.96 %2/3 · 0.248 · 1.85 %2/3 · 0.249 · 1.35 %1/2 · 0.250 · 1.42 %
Sep 84/103.67 %1/8 · 0.273 · 2.21 %1/2 · 0.249 · 2.94 %1/4 · 0.252 · 1.88 %4/7 · 0.249 · 1.75 %½ · 0.250 · 1.32 %½ · 0.250 · 1.52 %
Sep 44/101.97 %1/9 · 0.285 · 1.78 %–2/9 · 0.255 · 2.98 %–2/5 · 0.251 · 4.04 %–
Sep 39/152.92 %5/12 · 0.242 · 1.91 %–8/11 · 0.241 · 2.01 %–6/8 · 0.246 · 2.92 %–

Names are picked by recent movement (top and bottom movers over 1, 5 and 21 sessions), not by conviction: it is the sampling frame, not a signal.

Latest resolved session by name (Oct 16)

Cell: probability of closing up / expected move.

NameActual returnHaiku 4.5 AHaiku 4.5 BSonnet 5 ASonnet 5 BOpus 5 AOpus 5 B
BNY0.81 %0.50 / 2.5 %0.50 / 1.7 %0.50 / 1.5 %0.50 / 1.1 %0.50 / 1.3 %0.50 / 1.2 %
CIEN-6.67 %0.46 / 7.8 %0.50 / 9.5 %0.50 / 4.8 %0.49 / 3.5 %0.50 / 4.2 %0.50 / 3.7 %
FITB0.04 %0.50 / 2.0 %0.50 / 1.3 %0.50 / 1.3 %0.49 / 0.9 %0.50 / 1.2 %0.49 / 0.9 %
HPE-5.97 %0.46 / 6.5 %0.50 / 4.7 %0.50 / 4.0 %0.49 / 2.8 %0.50 / 3.4 %0.50 / 3.0 %
KSS-3.66 %0.54 / 7.2 %0.50 / 7.0 %0.50 / 2.8 %0.49 / 2.0 %0.50 / 2.6 %0.50 / 2.2 %
MRNA-7.82 %0.53 / 8.5 %0.50 / 6.5 %0.50 / 5.0 %0.49 / 3.8 %0.50 / 4.5 %0.50 / 4.0 %
PNC1.14 %0.48 / 2.6 %0.50 / 1.4 %0.50 / 1.3 %0.49 / 1.0 %0.50 / 1.2 %0.50 / 1.0 %
RF0.51 %0.50 / 2.0 %0.52 / 3.4 %0.50 / 3.5 %0.52 / 3.4 %0.50 / 2.6 %0.52 / 3.0 %
SIG0.28 %0.52 / 7.0 %0.50 / 10.7 %0.50 / 2.8 %0.49 / 2.0 %0.50 / 2.6 %0.50 / 2.2 %
TRV1.47 %0.52 / 2.2 %0.52 / 2.9 %0.50 / 3.4 %0.52 / 2.9 %0.50 / 2.4 %0.52 / 2.6 %

Method: predictions are recorded before the open and never on sessions already closed; packages are stored exactly as each model saw them; models get no headlines (licensing and memorisation). The directional-skill criterion (resolution > 0 with p < 0.05 over 200 predictions) and the A/B criterion were written down before starting.