How to evaluate an AI coding tool before you trust it

The checklist

Work through it in order. Steps 1 to 5 are setup, steps 6 to 10 happen while you test, and steps 11 to 14 are how you judge the result.

1. Pick one small, real task. Choose a job that fits what the tool claims to do and that you can finish in one session. Our target is under 2 hours. For an AI coding tool or app builder, our standard task is a small to-do app with add, complete, delete, and saved state, run locally or deployed.
2. Write the task and the starting prompt down before you start. Use the same task and the same starting prompt for every tool you compare. If your task isn't one of ours, define a comparable task in your notes before you begin, not after you've seen the results.
3. Use the plan a typical builder would use. That is usually the free or entry tier. Write down which plan you used, so nobody reads your result as a verdict on a plan you didn't try.
4. Record the date and the tool version. Tools change fast. A result without a date and a version can't be checked or repeated.
5. Check the price, and write down the day you checked it. Prices and limits change. Note what you saw and when.
6. Keep the exact prompts you used. Save them word for word. Paraphrasing a prompt later makes your notes impossible to compare.
7. Log the time to a working result and the number of retries. Two numbers, written as you go: how long until it worked, and how many times you had to try again.
8. Write down what worked and what broke. Take screenshots as you go, and save your notes before you write anything up.
9. Write down every limit you hit. Rate limits, paywalls, and missing features all count. A tool that stops you at step three tells you something.
10. Try it more than once if you can. One of our six scoring questions is whether the tool behaved consistently across tries. You can't answer it from a single run.
11. Score six questions from 1 to 5. Does it work, time saved, output quality, ease of use, value, and reliability. The table below has the exact questions.
12. Pick a verdict with fixed rules, and go lower when in doubt. Use it, Watch it, or Skip it. If you're torn between two, pick the lower one and write down why. The rules are in the verdict guide.
13. Say what you didn't test. Team features, a paid tier, a different platform. If you only read about something, say so. If you ran out of time, cut it from your verdict instead of filling the gap with a guess.
14. Write down any relationship with the maker. Free access, credits, an affiliate link, or prior work with them all belong in your notes. A reader weighing your result needs to know.

The six scoring questions

Score each from 1 to 5. The scores guide the verdict. We don't publish them as a grade, and you don't have to either.

To turn scores into a verdict: Use it needs "Does it work?" at 4 or higher and no criterion below 3. A tool that partly finished the task, or has one big weakness such as price, bugs, or a missing feature, is Watch it. A tool that failed the task, wasted your time, or is clearly worse than an alternative at the same price is Skip it. When in doubt, pick the lower verdict.