Two sections, never blended
Section 1 is what we tested ourselves. Section 2 is what other groups published. We never turn them into one score or one rank.
Leaderboard · beta
AI model leaderboard, for people who build
Two separate sections: what we tested ourselves, and what public benchmarks say. Every outside number is cited. Where we have not tested, we say "not tested."
Last updated · 22 models · 0 tested by us so far · By BenchSmoke
How to read this page
"Not tested" means not tested
If we have not run a model through our suite, it says so. We do not fill the gap with someone else's number or a guess.
Every outside number has a source
Each public figure links to where it came from and shows the date we retrieved it. Those numbers belong to their publishers, not to us.
1. BenchSmoke hands-on tests
Our own fixed suite of five builder tasks, run the same way on every model. No model has been run yet.
Status: The suite is designed and frozen, but we have not run any model through it yet. Every model below is Not tested. We will add results only after the runs exist, with public evidence.
The suite (version v1)
| Task | What it is | Kind | Checks | Prompt and hash |
|---|---|---|---|---|
| T1 | Small to-do app | coding: build from scratch | 7 | prompt 1825ff5038 |
| T2 | Fix a bug so the tests pass | coding: debug | 5 | prompt 96a5e9655a |
| T3 | Add search and pagination to an existing app | coding: extend | 5 | prompt 03c13c6d7d |
| T4 | Landing page from a brief | design and front end | 6 | prompt 66ef2143b7 |
| T5 | Three-step workflow with tools | agent: tool use | 6 | prompt fe7af2cd9c |
Each task is run 3 times per model. We publish every run, not the best one. Scoring is automatic where we can, with the scripts saved. The exact prompts are published here (see the links in the table) and hashed (SHA-256, first 10 characters shown) so we cannot quietly change them. Full method: How we test.
Our results by model
| Model | Our status | T1Small to-do app | T2Fix a bug so the tests pass | T3Add search and pagination to an existing app | T4Landing page from a brief | T5Three-step workflow with tools | Verdict |
|---|---|---|---|---|---|---|---|
| Qwen3.8 27BAlibaba (Qwen) | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Qwen3.8 Max (0902)Alibaba (Qwen) | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Claude Fable 5.1Anthropic | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Claude Haiku 5.5Anthropic | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Claude Opus 5.5Anthropic | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Claude Sonnet 5.5Anthropic | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| DeepSeek V4 Pro (0813)DeepSeek | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| DeepSeek V4.1 FlashDeepSeek | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Gemini 3.8 FlashGoogle | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Gemini 4 ArgonGoogle | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Gemma 4 31BGoogle | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Muse Spark 1.3Meta | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Kimi K3Moonshot AI | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| GPT-6 AstraOpenAI | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| GPT-6 LunaOpenAI | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| GPT-6.1 SolOpenAI | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Hy4 previewTencent | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| Grok 4.7xAI | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| MiMo-V2.6 FlashXiaomi | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| MiMo-V2.6 ProXiaomi | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| GLM-5.3Z.ai | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
| GLM-5.3 FlashZ.ai | Not tested | not tested | not tested | not tested | not tested | not tested | no verdict yet |
Dash = not tested. Verdicts (Use it, Watch it, Skip it) are only given once a model has run every task. Models are listed alphabetically by vendor, not ranked.
Pro
Full run details and evidence files (settings, timings, cost per run, retries) will be part of BenchSmoke Pro. Locked for now: nothing is for sale yet. Headline results and public evidence links stay free.
2. Public benchmarks (not our tests)
Numbers other groups published. We did not produce them and they are not ours. Each number links to its source, and each column header shows who published it and the day we retrieved it. We show a limited set of figures, not the sources' full tables.
Read with care. Arena scores come from people voting between two answers. They show what people prefer, not whether a model finishes your task. The WebDev score is the closest to building apps; the Text score is general chat. Rank is the source's own rank among its entries. A source's entry is one specific setting of a model (for example "Max" reasoning effort); the exact entry name is shown under each number. We do not show prices yet. We will add vendor list prices, each linked to the vendor's own pricing page.
| Model | LicenseLMArena, retrieved Oct 17, 2026 | WebDev Arena scoreLMArena, retrieved Oct 17, 2026 · rank of 144 | Text Arena scoreLMArena, retrieved Oct 17, 2026 · rank of 416 |
|---|---|---|---|
| Qwen3.8 27BAlibaba (Qwen) | Apache 2.0 | 1,593 rank 29Qwen3.8-27B | 1,441 rank 75qwen3.8-27b |
| Qwen3.8 Max (0902)Alibaba (Qwen) | Proprietary | 1,674 rank 10Qwen3.8-Max (0902) | no matching entry |
| Claude Fable 5.1Anthropic | Proprietary | 1,741 rank 5Claude Fable 5.1 (Max) | 1,512 rank 3Claude Fable 5.1 (Max) |
| Claude Haiku 5.5Anthropic | Proprietary | 1,585 rank 32Claude Haiku 5.5 (High) | no matching entry |
| Claude Opus 5.5Anthropic | Proprietary | 1,807 rank 1Claude Opus 5.5 (Max) | 1,516 rank 2Claude Opus 5.5 (High) |
| Claude Sonnet 5.5Anthropic | Proprietary | 1,768 rank 3Claude Sonnet 5.5 (xHigh) | 1,470 rank 30claude-sonnet-5.5-xhigh |
| DeepSeek V4 Pro (0813)DeepSeek | MIT | 1,582 rank 33DeepSeek-V4-Pro (High, 20260813) | 1,446 rank 61deepseek-v4-pro-high-20260813 |
| DeepSeek V4.1 FlashDeepSeek | MIT | 1,618 rank 24DeepSeek-V4.1-Flash (Max) | 1,463 rank 38deepseek-v4.1-flash-max |
| Gemini 3.8 FlashGoogle | Proprietary | 1,585 rank 31Gemini 3.8 Flash (High) | 1,499 rank 8Gemini 3.8 Flash (High) |
| Gemini 4 ArgonGoogle | Proprietary | 1,678 rank 9Gemini 4 Argon (High) | 1,532 rank 1Gemini 4 Argon (High) |
| Gemma 4 31BGoogle | Apache 2.0 | 1,367 rank 106Gemma 4 31B | 1,443 rank 67gemma-4-31b |
| Muse Spark 1.3Meta | Proprietary | 1,658 rank 12Muse Spark 1.3 (Max) | 1,490 rank 12Muse Spark 1.3 (Max) |
| Kimi K3Moonshot AI | Kimi K3 license | 1,655 rank 14Kimi K3 (Max) | 1,475 rank 25Kimi K3 (Max) |
| GPT-6 AstraOpenAI | Proprietary | 1,777 rank 2GPT-6 Astra (Max) | 1,443 rank 69gpt-6-astra-max |
| GPT-6 LunaOpenAI | Proprietary | 1,578 rank 35GPT-6 Luna (Max) | 1,392 rank 166gpt-6-luna-max |
| GPT-6.1 SolOpenAI | Proprietary | 1,755 rank 4GPT-6.1 Sol (Max) | 1,448 rank 57gpt-6.1-sol-max |
| Hy4 previewTencent | Apache 2.0 | 1,631 rank 18Hy4 preview | no matching entry |
| Grok 4.7xAI | Proprietary | 1,639 rank 16Grok 4.7 (xHigh) | 1,401 rank 153grok-4.7-xhigh |
| MiMo-V2.6 FlashXiaomi | MIT | 1,634 rank 17MiMo-V2.6-Flash | 1,455 rank 45mimo-v2.6-flash |
| MiMo-V2.6 ProXiaomi | MIT | 1,625 rank 20MiMo-V2.6-Pro | 1,491 rank 11mimo-v2.6-pro |
| GLM-5.3Z.ai | MIT | 1,621 rank 23GLM-5.3 (Max) | 1,471 rank 27glm-5.3-max |
| GLM-5.3 FlashZ.ai | MIT | 1,605 rank 27GLM-5.3-Flash | 1,471 rank 29glm-5.3-flash |
On a narrow screen, scroll the table sideways. "No matching entry" means the source has no entry we could match to that model at retrieval time. It is not a score of zero. Machine-readable version with every field: /leaderboard.json.
Sources, licenses and retrieval dates
LMArena WebDev Arena (overall, latest snapshot)
- What it is
- Human preference votes on web-app building tasks, shown as an Elo-style score with a rank.
- Publisher
- LMArena
- Data link
- https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset/viewer/webdev/latest
- License and terms
- CC-BY-4.0. License / terms. Open data: the lmarena-ai/leaderboard-dataset card on Hugging Face declares CC-BY-4.0. We cite name, link and date, show a limited set of figures, and do not republish the full table.
- Retrieved
- Source says it was published
- Dataset revision we read
0ebb411abf00
LMArena Text Arena (overall, latest snapshot)
- What it is
- Human preference votes on general chat answers, shown as an Elo-style score with a rank. Not a coding test.
- Publisher
- LMArena
- Data link
- https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset/viewer/text/latest
- License and terms
- CC-BY-4.0. License / terms. Open data: the lmarena-ai/leaderboard-dataset card on Hugging Face declares CC-BY-4.0. We cite name, link and date, show a limited set of figures, and do not republish the full table.
- Retrieved
- Source says it was published
- Dataset revision we read
0ebb411abf00
Sources we looked at and are not using yet
- OpenRouter model list (list price and context window): Not published for now. OpenRouter terms of service are unresolved, so we show no OpenRouter figures. Vendor list prices will be added by hand, with a link to each vendor pricing page. License: No open license. Public facts only (price, context length), cited and linked..
- SWE-bench leaderboards: not used: the data is NonCommercial and BenchSmoke has sponsors and a planned paid tier. Link only, unless the maintainers give written permission. License: CC BY-NC 4.0 (site data repo).
- Aider polyglot leaderboard: not used yet: the public data file stops at 2025-10-03 and has none of our 22 models. The refresh routine re-checks it. License: Apache-2.0 (repo data file).
- Artificial Analysis free API: not used yet: needs a free account and key that Justin has not created. Responds 401 without a key. License: Free API with required attribution; account and API key needed.
How this is made, and when it was last updated
Last updated: . Dates on this page are calendar days in New York.
- Public figures are fetched by a script from the sources listed above (an open-licensed dataset and a public data feed). The script writes the retrieval date too; nobody types the numbers in. We refresh weekly, and a changed number keeps a record in the change log.
- Our tests are run by us on a frozen suite. Results appear here only after the runs exist and the evidence is public. Until then the model says "Not tested".
- Which models: Scope v1 (2026-10-17): 22 models picked for AI builders and vibe coders: 12 closed (proprietary) frontier or near-frontier models and 10 open-weight or open-license models. Each is pinned to the source entry we cite, and the exact entry name is always shown next to its number.
- What we do not do: we do not copy full tables from anyone, do not scrape pages behind logins, do not invent numbers, and do not present anyone else's data as ours.
- Mistakes: if a number here is wrong or a license says we cannot show it, email hello@benchsmoke.com and we will fix or remove it.
Sponsors and affiliates: No sponsor or affiliate relationship currently affects this leaderboard. If one ever does, it will be named here and labeled Sponsored, and a sponsor never sees or changes results. Same policy as How we test.
BenchSmoke Pro Pro
Locked. Not available yet. This page is free and stays free to read and to crawl. Pro is a possible paid add-on for people who want more depth. Nothing is for sale, and we are not collecting payments or payment details.
If it launches, it would add the items below. Read the Pro page for what stays free and what is planned.
- Full model list and every category More models, more LMArena categories, and score history.
- Side-by-side compare Put any models next to each other, with our test results and cited figures kept in separate columns.
- Full test evidence Exact prompts as sent, settings, timings, cost per run, retries, and the saved run files for every test.
- Change history and alerts What moved since last week, and email alerts when a ranking or a price changes.
- CSV and JSON export The full table for your own spreadsheet or tool.
- New-model alerts A note when a model we track gets a new version, with our test plan for it.
No outcome is promised: these are tools to help you decide, and any model choice is yours to make.
Get the verdicts every Tuesday and Thursday.
Free, every Tuesday and Thursday, with one clear verdict per tool.