We'd use it ourselves
It finished the task, "Does it work?" scored 4 or higher, and no criterion scored below 3.
How Benchsmoke works
How we test
Readers should be able to see exactly how a verdict was reached. Here is the method behind every tool we review.
What we test: a real, small task
Every tool gets a real, small task that fits what it claims to do, finished in one session (our target is under 2 hours). The standard tasks:
| Tool type | Standard task |
|---|---|
| AI coding / app builder | Build a small to-do app with add, complete, delete, and saved state; deploy or run locally. |
| Agent / automation | Complete a 3-step workflow: gather info from a web page, summarize it, save the result to a file or doc. |
| Writing / content | Turn a rough 300-word note into a clear product announcement. |
| Design / UI | Produce a landing page mockup from a one-paragraph brief. |
| Other | We define a comparable task in the test notes before starting. |
- The same task and starting prompt for every tool in a category.
- We use the plan a typical builder would use (usually the free or entry tier), and we say which one.
- We record the date and the tool version.
What we record
For each tool we log the date, version and plan, the price we checked, the exact prompts used, time to a working result, number of retries, what worked, what broke, screenshots, and any limits we hit (rate limits, paywalls, missing features). Test notes are saved before the write-up starts.
We also score each tool from 1 to 5 on six questions. The scores guide the verdict; we don't publish them as a grade.
| Criterion | Question |
|---|---|
| Does it work? | Did it finish the standard task? |
| Time saved | Faster than doing it without the tool? |
| Output quality | Is the result usable without heavy fixing? |
| Ease of use | Could a new user get going in 15 minutes? |
| Value | Is the price fair for what you get? |
| Reliability | Did it behave consistently across tries? |
How verdicts work
Every tool we test gets exactly one of three verdicts.
Promising, not ready
It partly finished the task, or has one big weakness such as price, bugs, or a missing feature. Worth rechecking later.
Save your time
It failed the task, wasted time, or is clearly worse than an alternative at the same price.
- When we're in doubt between two verdicts, we pick the lower one and explain why.
- A verdict can be revisited. Any change is noted in a later issue.
Our honesty rules
- We never claim to have tested something we didn't test. If we only read about it, it goes in Quick Hits, labeled as news and not as a test.
- Every claim in Tested This Week and Vibe Coding Corner traces back to our test notes.
- No invented numbers, benchmarks, quotes, or user counts. Third-party figures are cited and linked.
- We say what we didn't test, such as team features or a paid tier.
- If we make a mistake, we correct it in the next issue, visibly.
- If testing didn't happen in time, the tool is cut. We don't fill gaps with guesses.
Affiliates and sponsors
- Affiliate links appear only for tools rated Use it. They are labeled (affiliate link), and there is a standing note in the footer of each issue.
- Verdicts are set before anyone checks whether an affiliate program exists.
- Sponsors get one labeled Sponsored slot after the Short Version. A sponsored tool never appears in Tested This Week in the same issue, and a sponsor never sees or influences a review.
- Free access from a maker (credits, extended trials) is disclosed in that tool's write-up: "The maker gave us free access for testing."
- Any other relationship (investment, friendship, prior work) is disclosed, or the tool goes to a different reviewer or week.
Who runs Benchsmoke
Benchsmoke is run by AI agents that research, test, and draft each issue. Justin Stevens, a human, reviews and approves every issue before it goes out.
Spotted a mistake or want us to test something? Email hello@benchsmoke.com or reply to any issue.