> Some CI runners are single core unless you pay more
I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing.
> some tests are very hard to parallelise (integration, E2E)
No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. You need to go way out of your way to screw up a test suite so badly you have dependencies between tests covering different sets of features.
> If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50.
This argument just proves you're desperately grasping at straws.
The only Windows test runner from GitHub that costs that is their largest, most expensive runner: Windows 64-core runner. Do you need a Threadripper to use the app you're testing?
Even so, that hypothetical cost is only factored in if you run out of minutes from your plan, and you certainly do not order Windows 64-core runners from your free tier plan.
Even in your hypothetical cost-conscious scenario, GitHub actions do support self-hosted runners, which cost zero per minute.
> You could be better with different tests running at PR/Nightly (...)
None of the hypothetical scenarios you fabricated pose a concern. Even assuming you need a Windows machine with 64 processor cores to do an E2E test run of your app, the most basic intro to automated testing tutorial and test pyramids mentions how these tests can and often run on deployments to pre-prod and prod stages. This means that anyone following the most basic intro tutorial on the topic will manage to sidestep any of your hypothetical scenarios by gating auto promotions to prod.
> I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers.
If you’re charging a runtime multiplier per core then there’s a cost per core. Included minutes on most runners are limited to basic versions with limited cores. Try xdist pytest on a default GitHub runner and get any speed up…
> Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing.
Except the cost of the hardware, sysadmin time for setting up and supporting ephemeral runners, monitoring, etc…
> No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario.
In end to end system tests if you have 100 hitting at the same time how do you guarantee the underlying state is the same so the tests are idempotent?
Also another example, I set up testing pipelines for an OS that ran in an FPGA in a HIL CI test. I had three of these due to operating costs. How could I parallelise tests that required flashing firmware AND have the most pipelines running as possible?
I don't know which hypothetical CI runner you have in mind. Back in reality, most CICD services bill you on time spent running tests, not "core". Even core count is reflected as runtime multpliers. Moreover, services like GitHub not only offer a baseline of minutes per month, and on top of that provide support for self-hosted runners that cost you nothing.
> some tests are very hard to parallelise (integration, E2E)
No, that's simply false and fundamentally wrong. You can run any test on any scenario independently of any other test from another scenario. You need to go way out of your way to screw up a test suite so badly you have dependencies between tests covering different sets of features.
> If you need to run a hefty windows runner on GitHub it’s $0.54 per minute, so a 90 minute full suite will cost $50.
This argument just proves you're desperately grasping at straws.
The only Windows test runner from GitHub that costs that is their largest, most expensive runner: Windows 64-core runner. Do you need a Threadripper to use the app you're testing?
Even so, that hypothetical cost is only factored in if you run out of minutes from your plan, and you certainly do not order Windows 64-core runners from your free tier plan.
Even in your hypothetical cost-conscious scenario, GitHub actions do support self-hosted runners, which cost zero per minute.
> You could be better with different tests running at PR/Nightly (...)
None of the hypothetical scenarios you fabricated pose a concern. Even assuming you need a Windows machine with 64 processor cores to do an E2E test run of your app, the most basic intro to automated testing tutorial and test pyramids mentions how these tests can and often run on deployments to pre-prod and prod stages. This means that anyone following the most basic intro tutorial on the topic will manage to sidestep any of your hypothetical scenarios by gating auto promotions to prod.