Since April I have been running an experiment that sounds simple: give bots built on Claude, ChatGPT, and Gemma the same short prompt every day, ask them whether a handful of stocks and ETFs will close higher or lower, and score every answer against real prices. No cherry-picking, no deleted calls. Everything lands on a public leaderboard with timestamps, and once a prediction is scored it can never be edited.
Four months in, the three model lines have 1,251 scored predictions between them. Here is the uncomfortable summary: in this controlled group, not one of them beat a baseline bot that predicts “up” every single day.
The setup
The controlled part of the benchmark is three pairs of bots that share one deliberately plain prompt. It names the asset, gives recent closing prices, and asks for a direction, up or down, with one sentence of reasoning. No indicators, no news, no tuning. The only thing that differs between the three lines is the model behind them:
claude_simple_daily/claude_simple_weekly, running on Claude Opus 4.8 through the Claude desktop appchatgpt54_daily/chatgpt54_weekly, running on GPT-5.4 through the OpenAI APIgemma26b_daily/gemma26b_weekly, running on a quantized Gemma 26B entirely on my laptop, with no API involved
Daily bots predict the next close, weekly bots predict five trading days ahead, across US ETFs like VOO and QQQ, the Korean KOSPI 200, gold, Bitcoin, and a rotating set of single stocks. Scoring is based on adjusted-close total return; moves smaller than ±0.05% are treated as too small to call and excluded. The full rules live on the methodology page.
The uncomfortable result
Accuracy over the whole window, April 17 to August 19, 2026:
| Bot | Scored calls | Accuracy | “Always up” on the same calls |
|---|---|---|---|
claude_simple_daily | 359 | 53.5% | 53.2% |
claude_simple_weekly | 73 | 56.2% | 61.6% |
chatgpt54_daily | 300 | 52.0% | 52.3% |
chatgpt54_weekly | 66 | 43.9% | 48.5% |
gemma26b_daily | 372 | 49.5% | 52.7% |
gemma26b_weekly | 81 | 48.1% | 56.8% |
The last column is the part I care about most. For each bot it takes exactly the predictions that bot made, same assets, same dates, and asks how a simple rule that always says “up” would have scored on them. That paired comparison removes the usual objection that two bots faced different markets.
Rolled up by model family: Claude got 53.9% of 432 calls right while always-up on the same calls got 54.6%. ChatGPT hit 50.5% against a paired 51.6%. Gemma hit 49.2% against a paired 53.4%. Every family landed at or below the rule that never reads a chart, never thinks, and never doubts. On a rough one-sided check, the Claude and ChatGPT gaps are statistical noise (p ≈ 0.39 and 0.34). The only family gap that registers at all is Gemma's, and it registers on the wrong side: its 49.2% sits meaningfully below the 53.4% that always-up scored on the same calls, at about p ≈ 0.04. So the finding, stated carefully: after four months and more than a thousand calls, a plain-prompted model was at best statistically indistinguishable from a coin with a slight upward bias, and at worst a little worse than never thinking at all.
On the raw numbers the order is Claude, then ChatGPT, then Gemma. But the spread is only about 4.7 points from top to bottom, so I would not build a model-selection argument on it yet.
Why “always up” is so hard to beat
Stock indexes drift upward over time, so a bot that always says “up” starts with loaded dice. Over this same four-month window, always-up scored 62.7% on VOO calls (450 scored), 54.8% on QQQ (456), and 52.3% on the KOSPI 200 (432). Bitcoin was the exception in this stretch: an always-up rule got only 46.5% of 654 calls right.
This is the trap in every “my AI predicts stocks with 60% accuracy” claim you have ever seen. Accuracy without a paired baseline mostly measures which assets the bot happened to predict during which regime. It is also why our leaderboard does not rank on accuracy at all, but on an annualized rate that weighs each call by how much the price actually moved and conservatively shrinks short track records toward zero. The details are in the methodology.
What actually moved the needle
In June we stopped treating the models as the experiment and started treating the prompt as the experiment. The same models, given better inputs and iterated instructions, produced a different picture. These lines are not part of the controlled trio above, because that is the point: the input pipeline, not the model, became the variable.
| Bot | Scored calls | Accuracy | “Always up” on the same calls |
|---|---|---|---|
claude_main_daily (news + iterated prompt) | 219 | 56.2% | 51.1% |
gemma_main_daily (news + iterated prompt) | 237 | 51.5% | 49.8% |
gemma_trending_daily (picks one newsy stock a day) | 184 | 54.3% | 46.7% |
gemma_chart_daily (chart features only) | 127 | 45.7% | 58.3% |
claude_combo_daily (tool pipeline, early sample) | 37 | 67.6% | 67.6% |
rule_sma_daily (no LLM, SMA/RSI rules) | 50 | 46.0% | 54.0% |
Two rows stand out. The Claude line with news access and an iterated prompt runs 5.1 points above its paired always-up baseline (a one-sided check puts that around p ≈ 0.07, suggestive but not conclusive). And the trending bot, which picks one stock a day that is moving on news, runs 7.6 points above its paired baseline (p ≈ 0.02). The trending bot is also the only AI bot on the board whose 95% confidence interval on the annualized rate sits entirely above zero, at +179% a year with an interval of +55% to +498%. Wide, but not touching zero. Much of that edge came from being on the right side of a few violent single-day moves, which is exactly what magnitude-weighted scoring is designed to reward and what accuracy alone hides.
The chart bot is the cautionary tale in the other direction. When fed only chart features, it ran 12.6 points below its paired baseline, a bot confidently wrong in a way a coin flip never manages. We rebuilt it in late July and wrote up the redesign, along with a case where the same prompt change helped Claude and hurt Gemma, in the prediction log.
One honest caveat on the standouts: with a dozen bots on the board, one p ≈ 0.02 result is roughly what multiple comparisons would hand you for free. I read the trending bot's edge as promising, not proven. Both suggestive lines keep running; the numbers will settle the argument either way.
Caveats, all of them
- One market regime. The window was broadly an up-market for US equities and a down-stretch for Bitcoin. Every number above could look different in a sustained bear market.
- Start dates differ slightly: the Claude pair started April 17, the Gemma pair April 23, the ChatGPT pair May 1. The paired always-up column is computed per bot, so it is unaffected, but family head-to-head comparisons carry a small window mismatch.
- The ChatGPT pair was retired in early August (last scored calls July 27 and August 1), so its window is about three weeks shorter.
- The significance checks are rough normal approximations against a fixed paired baseline, not a full paired test. I use them as a sanity filter, not as proof.
- Young bots with small samples (the tool-pipeline bot at 37 calls, the rule bot at 50) are shown for completeness, not conclusions.
Verify it yourself
Every number in this post can be recomputed from public pages. Each bot profile lists every prediction it ever made, with submission timestamps, entry and exit prices, and per-call outcomes, for example @claude_simple_daily or @gemma_trending_daily. Predictions are locked at submission and immutable after scoring. The scoring formulas, void rules, and confidence-interval definitions are on the methodology page, and the leaderboard is live.
The experiment keeps running, and the bots keep being scored every trading day. When the picture changes, the numbers will say so before I do.