Path-based selection of tests, in every single CI system I have ever seen it implemented in, is wrong. TFA thankfully gets the "run the downstream dependent tests" which is the biggest miss, but even then, you can get the wrong answer. Say I have the following: paths a/* need to run tests A, and paths b/* need to runs tests B. I configure the CI system as such. Commit A' is pushed, changing paths under a/* only, so the CI runs tests A, only. The tests fail. A separate engineer, quickly afterwards (perhaps, lets say, even before the prior commit has finished; this could just be two quick merges) pushes commit B', changing paths under b/* only. The CI system runs tests B only, and they all pass so the commit is marked green incorrectly. Automated deployment systems, or other engineers, proceed to see passing tests, and use a broken but "green" build.
Since ever rarely do I see the downstream-tests requirement correctly done, and almost always the "successive miss" bug, I'm convinced for these reasons path based is just basically a broken approach. I think a better approach is to a.) compute your inputs to the tests, and cache results, and b.) have the CI system be able to suss out flakes and non-determinism. But I think a.) is actually quite difficult (enough to be one of the hardest problems in CS, remember?) and b.) is not at all well supported by the current CI systems.
(I have seen both of these approaches result in bad (known bad, had the tests run!) builds pushed to production. The common complaint is "but CI is slow" followed by a need to do something, but without care towards the correctness of that something. Responsibility for a slow CI is often diffused across the whole company, managers do not want to do the hard task of getting their engineers to fix the tests they're responsible for, since that just doesn't get a promotion. So CI remains slow, and brittle.)
It's possible to use bazel to do this. You need to be very explicit (The Bazel Way(tm)), but in exchange, you can ask the graph for everything that is an rdep of a given file. This isn't always necessary in bazel (if you avoid weird integration targets, deploy targets, etc) where `bazel test //...` generally does this by default anyway. It's sometimes necessary to manually express due to incomplete graphs, tests that are not executed every time (for non-determinism, execution cost, etc), and a few other reasons but at least it's possible.
Yeah, bazel is sort of the exception that proves the rule, here. I wish it did not have such an absurd learning curve; I've found it next to impossible to get started with it.
Bazel is weird. I work at Google, and it's fundamental to our engineering, and I love it for that, but to me it's an internal tool. Nearly all Bazel code I work with is internal (most open source uses of it I see aren't doing enough customisation to be worth it in my opinion), it integrates into so many parts of our engineering workflow, and uses so many internal services like our build farm.
I'm not sure I see much point in using it outside of Google. Maybe if you're a company within 1/10th the size and have a lot of Xooglers? It seems like the sort of thing where most companies don't need it and therefore shouldn't use it, and those that do need it probably need to build their own that's right for them.
It worked well enough for my last company. It does require a team to teach it and to do the heavy lifting on custom rules and tooling. I'd rather do that than worry about whether my c++ that was exposing me to millions/billions of risk might have skipped some tests in our haste for an intraday release.
Not only is it hard to get started with Bazel, it's hard to continue with Bazel even once you do. Unless you are using Google practices (say, vendoring and compiling all dependencies), you will certainly end up mired in poorly-maintained third-party tools that will cause no end of fun side-quests.
My work became a lot more, ahem, linear, once I moved us off of Bazel.
Some of this is pretty simple, though, at the coarse-grained level. If you have a frontend and a backend, and you change the backend, you run all backend unit tests, backend component tests, and the end to end tests. If you change the frontend, you run the frontend unit tests, the frontend component tests, and the end to end tests.
To stop the problem you mentioned you can either tick the box in Github that says only up to date branches can merge, or in Gitlab you can use merge trains. "Running tests on an older version of the code" is a bigger problem than just this case, but in all cases I can think of enabling those features solves it.
> To stop the problem you mentioned you can either tick the box in Github that says only up to date branches can merge
No, that checkbox doesn't save you. The example above still fails: the second branch, B, after rebase, still only touches files in b/*, and thus still fails to notice the failure in A. The branch CI run would be green, and the merge would be green, both false positives.
The example assumes two branches with a single commit being merged to the repo HEAD, so "testing the branch" and "testing the commit" are equivalent / testing the whole branch does not save you.
(But if you only test the last commit — and some implementations of "did path change?" in CI systems do this — then it is more incorrect.)
Since ever rarely do I see the downstream-tests requirement correctly done, and almost always the "successive miss" bug, I'm convinced for these reasons path based is just basically a broken approach. I think a better approach is to a.) compute your inputs to the tests, and cache results, and b.) have the CI system be able to suss out flakes and non-determinism. But I think a.) is actually quite difficult (enough to be one of the hardest problems in CS, remember?) and b.) is not at all well supported by the current CI systems.
(I have seen both of these approaches result in bad (known bad, had the tests run!) builds pushed to production. The common complaint is "but CI is slow" followed by a need to do something, but without care towards the correctness of that something. Responsibility for a slow CI is often diffused across the whole company, managers do not want to do the hard task of getting their engineers to fix the tests they're responsible for, since that just doesn't get a promotion. So CI remains slow, and brittle.)