Your test suite
If you don’t have an automated test suite yet, use 25 tests per dev.
To reach 80% coverage, a good rule of thumb is that you'll want ~25 automated tests per developer.
You'll typically need ~25 automated tests per developer for 80% coverage.
This must be ≥1
Test suite runs
This must be ≥1
- ✓per day
- per week
- per month
Tests to run
Run type
Parallel runs cost
Run tests simultaneously to get results back in minutes, not hours.
To test and deploy as often as possible, teams should strive to run 100% of their tests in parallel.
At 3-5 minutes per test, parallelization means that test results come back in minutes, not hours. That's less time waiting and more time shipping.
Unfortunately, most teams have to settle for partial parallelization — running batches of tests at a time — because it’s too expensive to do them all at the same time.
Monthly parallelization rate per test
Major vendors charge about $130/test/mo for up to 25 parallel runs and $100/test/mo for 26 or more.
This can't be $0
Number of tests ran in parallel per run
Example: If you have 200 tests you might run 4 batches of 50.
This must be ≥1
Additional opportunity costs involved with test running which may be less apparent but could be add up.
Babysitting runs
We call it “babysitting runs,” you might call it idle time (and your devs might call it a ping pong break).
Babysitting runs is the hidden cost of lost developer productivity while the tests run. During that time, a developer might do other useful work — or none at all — so this scale lets you adjust the productivity of your developers while the test suite is running.
Idle time for full-time devs
-
This can't be 0
85% efficient
and
Idle time for part-time devs
-
⚠️ This can't be 0
85% efficient
and
Sequential runs cost
Run individual tests one after another.
This is the most common approach, due to the costs and practicality of managing the necessary server infrastructure.
While the fees to run a test suite might be lower, it takes more time to QA (a cost on its own) and creates deployment bottlenecks as a team scales.
Market rate per sequential run
This can't be $0
Average test length
3-5 minutes is a good benchmark.
5 min
Additional opportunity costs involved with test running which may be less apparent but could be add up.
Babysitting runs
We call it “babysitting runs,” you might call it idle time (and your devs might call it a ping pong break).
Babysitting runs is the hidden cost of lost developer productivity while the tests run. During that time, a developer might do other useful work — or none at all — so this scale lets you adjust the productivity of your developers while the test suite is running.
Idle time for full-time devs
-
This can't be 0
85% efficient
and
Idle time for part-time devs
-
This can't be 0
85% efficient
and
3rd party costs
Compare against 3rd party limits & plans
Why would you need to calculate someone else's pricing?
- Overages charged for going over your monthly limit can sneak up on you quickly.
- Within seconds, you'll be able to see how a vendor's pricing stacks up against market rates.
How much are you paying?
This can't be $0
- ✓per year
- per month
This must be ≥1
You've indicated there is no overage fee
This is the number of tests you anticipate running in parallel. This number should not exceed the total number of automated tests you entered above.
This can't be ≤0
3-5 minutes is a good benchmark.
5 minutes
Cost to support infra for parallel runs
Cost to support infra for sequential runs
Cost to support custom or 3rd party infra
Yearly cost
Yearly time spent
Toggle to view how your plan compares to market rates for test running.
Cost comparison vs. market rate
See how your plan stacks up to market rates:
This is the general rate outside of your 3rd party's offering. The market rate for running a single test is $0.16.
This can't be $0
This can't be $0
Compared to market rates, you're paying:
COMPARE WITH QA WOLF