We are closing the AI tournament: none beat chance in 315 predictions

Today we are closing the AI tournament. For six weeks, from 3 September to 16 October, Haiku 4.5, Sonnet 5 and Opus 5 (and, from 25 September, one model from DeepSeek and one from Alibaba) received the same data package on 10 S&P 500 stocks every morning before the open. For each stock they gave two numbers: the probability that it would close up, and how much it would move. Before starting we wrote down that we would claim nothing until each model had 200 resolved predictions. We got to 315, over 31 sessions.

Direction: none did better than chance. The honest way to measure it is against a forecaster who always says the average frequency of up days, which in this sample was about half. No model beat it, with or without the quarterly earnings context. When they moved away from 50 %, they were right between 44 % and 53 % of the time: what a coin gives you.

Size of the move: all worse than a rule with no AI. We compared every estimate with a mechanical forecast that only uses each stock's recent volatility. The three Anthropic models missed by more than that rule: Opus by 5 % more (not statistically significant), Sonnet by 10 % and Haiku by 42 %. Giving them the quarterly earnings context as well made them worse: Opus 7 %, Sonnet 17 % and Haiku 82 % worse than the rule. The two outside models and the combination of all of them, with a smaller sample, were also behind, by 10 % to 24 %.

Paper profits: nothing to report. With a fixed rule that turns each probability into a long or short position, every model's book ended between −0.1 % and +0.05 %, with average exposure between 0.4 % and 5 % of capital. That cannot be told apart from zero.

Why we are stopping. The question we asked at the start has an answer now, and it is no. Two tests were still running: whether the earnings context helps, and whether reading the earnings releases companies file with the SEC helps. Both could only tell us whether more context improves a forecast that already loses to a free rule. The tournament cost $13.58 in model calls, and keeping it going cost about $12 a month. We keep every prediction and every data package exactly as each model saw it, and the tournament page stays up with the final result.

What this does not prove. It is a small, narrow experiment: a one-session horizon, 10 stocks picked for their recent moves, no news headlines and no web search. And with 315 predictions per model we would only have spotted a relationship of 0.16 or more between the probability they gave and what happened. It does not prove that an AI cannot forecast the market. It shows that these ones, with this data and one day ahead, did no better than chance.

The second agent we are switching off. On 12 October we put Opus 5.5 to rank the stocks in the Núcleo S&P 500 portfolio every morning, from best to worst, 5 and 21 sessions ahead, next to three rules with no AI. We are retiring it after five sessions, before the year of data we had set, and not because of its result: five sessions cannot tell you whether it is right. The only date that matured, 12 October at five sessions, left its top five 46 basis points behind the basket and the best rule without AI 136 ahead; one date says nothing. We are retiring it because its ranking looked too much like one of those rules, buying what has fallen the most (a correlation between 0.68 and 0.91 depending on the day), and we already measure that rule for free. Even if it were right, it would mostly be the rule being right, and finding out cost about $50 a year.

What stays the same. The strategies keep running as before, in simulation. Along the way we are retiring Tactical Defensive + carry: it moved exactly like Tactical Defensive (correlation 1.00) and returned less (+2.84 % against +3.19 % since June). Its track record stays visible in the lab.

Publishing what does not work is part of the job. This did not work, and we are writing it down with the numbers. Nothing above is investment advice. All figures are simulated, with no real money behind them. The final result of the tournament is at soberquant.com/ai-tournament.