How I test automation tools

Most automation reviews are written from a pricing page. This one is not. Every tool covered here gets a paid account, a real workflow, and a stopwatch before it gets a word written about it.

The five rules I hold myself to

1. I pay for the account

No vendor gives me a free seat in exchange for coverage, and I do not accept sponsored placements. When a tool costs money, it comes out of the same budget that runs this site. That is the only way a paid tier can be judged honestly against a free one.

2. I rebuild a workflow I actually used to run

For fifteen years I ran a wholesale distribution business. The test workflows are the ones that used to eat my week: turning a supplier price list into a quote, keeping stock counts in sync between a spreadsheet and an order form, chasing overdue invoices, and pushing purchase orders to suppliers. Those are the benchmarks, not a toy "send a Slack message when a form is submitted" demo.

3. I time the build, not just the run

Two numbers matter to somebody with a business to run. How long did it take to get the thing working the first time, including reading docs and fixing what broke? And how long does one run take once it works? A tool that executes in 400ms but costs six hours to configure is not the faster tool.

4. Every number on this site has a file behind it

Prices, run times, error counts, screenshots. If I claim it, there is a dated artifact in the evidence folder that produced it. If I could not verify something, the page says so in those words instead of guessing.

5. I write down where the tool I recommend loses

Every comparison table on this site names at least one criterion where my recommended pick is beaten. If a tool won on everything, either the criteria were chosen badly or the review is an advertisement. Both are worth catching.

What a review run looks like

1. Sign up on a paid plan, note the date and the price actually charged.
2. Build the target workflow end to end, logging every point where I got stuck.
3. Run it against real data volumes, not one sample record.
4. Break it on purpose: bad input, expired credential, rate limit. Record what the tool tells you when it fails, because that is what you will spend your Tuesday on.
5. Check what it costs at 10x the volume, since that is where pricing tiers bite.
6. Cancel and record how hard cancelling was.

What I cannot test

I run a one-person operation. I cannot evaluate enterprise SSO, procurement workflows, or how a vendor treats a customer paying five figures a year. Where those matter, the page says the question is out of scope rather than inventing an answer.

Corrections

Pricing moves and features ship. Pages carry the date their pricing was last checked. If something here is wrong, email [email protected]and I will fix it and note the change on the page.