As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged Blink into Chromium proper just this week! It is no longer managed by gclient/DEPS, but instead developed alongside Chromium in the same repository. But many other things (pdfium, v8, webrtc, and tons of smaller libraries) are still pulled in via DEPS in exactly the way described.
I'd just like to say this about our development, testing, and release process: it really really works. We keep moving fast, everyone knows what's going on throughout the codebase because everyone is working on master, and there are never any big ugly merge conflicts to resolve when new features land.
There are, of course, problems with the system. We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. The set of configurations on the try-bots doesn't exactly match up with the set of configurations that are run against tip-of-tree. And flaky tests are, of course, a perennial problem for everyone. But we're working on making all of these better, and huge progress has been made in the last couple years.
Ask me anything if you have any questions about our build and test infrastructure!
This is only tangentially related to the topics covered in the article and your comment, but I'm curious about the Chromium project's decision to mostly use static linking.
I've always found it interesting that Chrome is shipped mostly as one big binary (or on Windows, two: chrome.dll and chrome_child.dll), whereas IE and Safari use several (one for the rendering engine, one for the JS engine, one or two for networking, etc.). My guess is that Chrome uses one big binary to minimize startup time, since the OS can just map that binary into memory and go. Am I correct about the rationale? Another way to ask the same thing is this: what would be the downsides of using a component build for releases and not just for development? If you used a component build for releases, you'd have greater parity between dev and production, and that's generally a good thing.
I also notice that there are a few modules that get their own DLLs, at least on Windows. For example, in Chrome 45, we have chrome_elf.dll, libegl.dll, libexif.dll, and libglesv2.dll; I'd think these libraries could be statically linked just like the few hundred others. Are there good reasons for these exceptions? Or are these just little details that tend to get overlooked given the rapid pace of development?
Like Aaron mentioned, the renderer was originally split into a separate DLL because we hit the limits of Microsoft's linker at the time. Later, we ended up relying on that split for disabling win32k inside the sandbox (otherwise we would have had to deal with a bunch of broken USER and GDI initializations).
As for the remaining DLLs, it mostly comes down to licensing concerns. We don't include proprietary code in our main binaries, and in order to simplify IP concerns it's often best to split such code into other DLLs. For example, you asked about libegl.dll and libglesv2.dll, which I believe include a licensed copy of the SwiftShader library.
It's also worth pointing out that the performance impact of having a single, static binary in Windows is much less significant than it used to be. So, were we starting Chrome today I'm not so sure we'd make the same decision.
Since all I've worked on for the last three years is the infrastructure, and not the binary that lands on your desktop/phone, I'm not really qualified to answer that. (Which is just a fancy way of saying I've got no earthly idea :))
But since I've seen @bengoodger in this thread, maybe he can answer your questions.
I can't speak to the current situation, but originally Chrome was one monolithic dll for exactly the reason you describe: we wanted renderer launches to be as fast as possible and having the code mostly already paged in helped a lot. Over time it became impossible to link and so we had to switch to having a separate renderer dll.
I'm not sure why there are now a handful of separate dlls.
Being Google, what exactly keeps you from running every test for every commit? Maybe I am being a bit naive here, but as the owner of multiple data centers across the globe, shouldn't you have enough resources to do that, even for a project as big as Chromium?
Chromium is open source, and will run try jobs for anyone from anywhere around the world (we won't accept your commit without lots of checks, but we'll run your tests). That means that we don't get to rely on a lot of Google-internal technology, including our production data centers.
Also, we are compiling and testing on Mac, Windows, Linux, Android, iOS, ChromeOS, and historical versions of all of the above. That's something that a mostly linux-server-based company doesn't do a lot of. So we maintain our own datacenters with our own hardware, and don't get to take advantage of all of Google's scale.
Many but not all things can be usefully parallelized. The build, and most unit tests and integration tests can, though we struggle with the latter where real hardware is required. Some performance tests are also run on real hardware. This doesn't scale as well.
I'm curious whether you've got any interesting projects underway to work on the flaky test issue? I've been looking at FBs test lifecycle approach with some interest (http://goo.gl/QtUNxY), but would love to hear about any other approaches to solving the same problem.
And FWIW on building MongoDB with Evergreen (https://evergreen.mongodb.com) we've found that a stepback approach seems to give us the finest granularity with minimal cost in identifying the introduction of an error - we batch commits for execution, but when a specific test fails we start running just that test on each previous commit until we hit a passing execution. It obviously doesn't work perfectly in the face of flaky tests (see above question) but it seems to do pretty well.
The first is automatic retries of fast tests. If a test runs quickly and fails, it costs us little to try again just to make sure. Most of our unit tests are configured to run up to 3 times.
Another thing is keeping track of a database of individual test case passes/failures across all time. This will let us automatically mark tests as flaky if they fail often, and ignore their results programatically rather than requiring a human to manually mark the test as ignorable.
A third thing is, obviously, automatically filing bugs against the owners/authors of tests which have been marked flaky. This is controversial -- often a test is just fine until one of its underlying libraries has a race condition introduced, and the real person to fix it should be the author of that change, not the author of the test. But it is still a step in the right direction much of the time.
Many people subscribe to the philosophy that "a flaky test is worse than no test", because you think it is giving you information when in fact it is giving you none. I subscribe to a slightly different philosophy: "A test with a known flaky rate is hugely valuable". If you know how often a test flakes (statistically), then you can measure variances from that rate to detect changes. Of course, a flaky test with an unknown rate of flaky is still useless. Hence the second initiative above: measuring the rate of flake of everything.
Mozilla has a big problem with flaky tests (aka "intermittents") too. The SpiderMonkey VM can run tests in a deterministic mode to make tests less flaky by making things like GC more predictable.
Firefox has an experimental "chaos mode" that takes the opposite approach. It purposely randomizes behavior by adjusting thread priorities, changing hash table iteration order, and randomize timer durations. Unfortunately, many flaky tests fail in chaos mode, so it is not enabled by default.
But we (Mozilla) are also doing many of the same things as the Chromium team here. In particular there is work in progress to automatically "ignore" the results of known-flaky tests until we detect that there has been a change in the rate of flakiness, at which point we will — assuming all goes to plan — trigger new test runs until we can determine the point at which the regression was introduced.
I think one of the lessons we've learnt is that with a browser-type project it's very hard to make test runs fully deterministic, for both technical and human reasons.
The technical reasons are touched on in the original article: these are complex codebases with lots of moving parts and lots of environmental dependencies. Of course there are various tactics to try and combat this; for example there is a wiki page dedicated to innocuous-looking code that leads to intermittent tests [1].
The human reasons centre around the difficulty of getting people to care about spending time fixing a test that fails one time in 1,000 (which is still very noticeable when you are running it hundreds of times a day). Unless the issue is something that fits a known pattern it's hard work, difficult to tell if your fix even worked, and not likely to be considered a top priority due to the diffuse, hard to quantify, nature of the benefits.
I think the fact that both Google and Mozilla still have significant problems with intermittents despite talented engineering staff and it having been a known problem for years implies that some of the standard thinking about making tests fully deterministic simply doesn't apply; for this kind of work you have to embrace — or at least accept — the randomness, and look for ways to get the data you need despite the noise.
That's a good point that making tests fully deterministic is not actually possible. But users aren't deterministic either, so we must accept noisy test environments because that's what users see. Tracking changes in the rate of flakiness is an interesting idea.
Could you run "all" the identify flaky tests by running all the tests 100 times on the same stable build (like the latest ESR)? Is it even possible to write a test that could pass 100 times in a row? :)
Running a test N times will certainly detect some fraction of all the flaky tests. It's something we occasionally do manually to work out if e.g. a certain intermittent is (likely) fixed and it's something that we'd like to do more to quarantine new tests.
Unfortunately there are various confounding factors that mean many intermittent tests would look clean in such a run might nevertheless be problematic. For example if you only run tests that you think are intermittent problems that are triggered by state left from a previous test won't be found. This is one reason that we've been trying to run particularly problematic test types (e.g. firefox browser-chrome tests) in smaller groups restarting the browser with a clean profile between groups to clear the state. A group size of 1 would obviously be ideal here, but when you have thousands of tests and limited resources it's not practical.
The other problem is tests that have unexpected sensitivity to the environment. For example the other day DNS was being slow on the test infrastructure. This isn't a problem for most tests since they use something like /etc/hosts. But some tests were intentionally trying to use a non-resolving domain and those tests sudden started to randomly time out.
I have to dealt with debugging flaky tests before on embedded Linux app with multithreads, GUI, audio, video streaming before.
I debugged those bugs with custom gdb scripts. The test system will run the gdb scripts to start the app. The gdb script setup breakpoints, log stack trace and local variables automatically for a particular bugs.
The debugging environment is scripted and automated. Test failure mark/track the log file with all the gdb info.
I schedule the tests to run continuously overnight and in the morning I analyze the log file with gdb info.
After that I can add more debug command in gdb script or try fixed the bug if I have enough info. The cycle continue until the bugs were all fixed.
I personally think it was very good for try to analyze / fixing the "flaky" tests issue.
Even for the the tests that fails 1 in 1000 times, the bugs can still be debug and fixable.
It was actually more relaxing for me to debug this kind of problem. 99% of time, the computer do all the works.
You guys might want to try it if not done in Google already.
At Mozilla we're actually currently working to run some of our tests continuously under RR (http://rr-project.org/), our record-replay debugger. When a flaky test fails the replay can carefully studied to understand why test only fails say 1 out of 10,000 runs.
You say you have to rely on humans and heuristics to determine which CL to revert. Couldn't you automatically re-run just the failing tests on the batch?
More importantly, you seem to be running all the tests, as often as you can. A commit changing a text label shouldn't require running all the tests, that just doesn't scale. Have you consider splitting Chromium into different projects, where commits for one project would result in the tests of that project, and only that project?
We actually are working on systems which re-run the failing tests on all CLs within a failing batch. Unfortunately, we haven't been able to do this previously due to resource constraints. In particular, if a batch had 10 commits in it, you can either run 10 additional compiles and tests (didn't have the resources), or you can binary-search/bisect to find the failure (takes too long). It is often faster to use a heuristic to say "Test_Foo failed, and Foo_Src was modified in change X, so revert change X". We don't like relying on heuristics, but until recently it was our only choice.
We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.
Do you think a version of this system could work in a team that has mostly "b-players?" I just wonder how much of this stays afloat because the caliber of the engineers is higher than that of your typical company.
I don't think any of our decisions are driven by or dependent upon the caliber of our engineers.
For example:
a) No merges; and the only branches are release branches. This makes sense in almost any context. Whether your team is big or small, high-caliber or college sophomores, this works. The advantages are numerous. No one works on a feature branch in isolation, only to have tons of merge conflicts and surprises when they try to bring it back in. Bug fixes and features are never landed on a release branch and then not merged into master for the next release. Development doesn't slow to a crawl while you try to stabilize master for a release.
b) Keeping dependencies near tip-of-tree. Chromium has two kinds of dependencies: third-party libraries (e.g. yasm, lighttpd) and things we develop on our own but in separate repositories (e.g. pdfium, v8, boringssl). The former tend to have stable releases produced by their own maintainers; the latter is being developed in parallel and you want to be as up-to-date as possible. Most projects are similar: you have third-party dependencies (e.g. python packages) and your own libraries (like a common set of api interfaces). You want to keep the ones you control as close to tip-of-tree as possible so that you can move fast and integrate your own changes quickly.
The caveat to all of this "move quickly" stuff is "test Test TEST". If you are constantly rolling dependencies forward, stuff will break. Run all the tests before landing a dependency change. Revert dependency changes with impunity: since they are effectively merges of many commits made to the other repo, they are more likely to have breaking changes than any single commit to your main repo. So pretest, test, and revert.
None of this requires amazing engineers. It just requires a dedication to quality and good testing hygiene.
I wonder though, there /have/ to be merge conflicts? I suppose you continually rebase on the HEAD of master to keep up to date? So instead of taking care of conflicts at the end of a 'dev cycle' you take care of it in the middle.
The difference is that, in our flow, the merge conflicts are on the scale of a a single developer making a single self-contained commit. In a flow with feature branches, the merge conflicts are on a scale of the entire branch conflicting with any other changes made on master.
I'm sorry if this is obvious, but I would love to understand exactly what you mean by "the only branches are release branches". What I think you mean is that the only branches are release branches and the development branches on each dev's machine.
At my startup, we use feature branches, and we expect devs to continually merge master into their branch while it's under development (ideally daily), and to fix any resulting conflicts or test failures. We have them push these branches to the repo (which triggers our continuous integration server), but they're treated as private works in progress. Then, when they finish and pass code review, we're confident that they will merge seamlessly, since they've been merging master all along. (We also use git merge --squash for the final merge to avoid seeing all the intermediate commits on the master commit log.)
I'm trying to understand if your practices are meaningfully different. Do developers have private branches on their machines for weeks on end? Are they encouraged to regularly merge master into those private branches?
Heh, that's a contentious topic. Personally, I would like to do so. I think I speak correctly when I say that most of the Infrastructure team would like to do so, to increase security/auditability and ease the maintenance burden on ourselves.
However, large portions of the team are opposed to Gerrit because a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. We are working with people to try to fix those, but there's no particular timeline for switching, or even a guarantee that we will.
The bug linked above is about evaluating the ability to make the Polymer-based UI (which is pretty and functional) work on top of Gerrit instead of on top of Rietveld. This is one of the possible paths forward for getting people to use Gerrit, and the one for which external contributors could be the most help.
Other paths forward exist: making Rietveld and the polymer Rietveld UI so good that no one wants to switch to Gerrit anymore; working directly on the Gerrit UI; or convincing devs that the Gerrit UI is good enough. But I think the bug and description above is the path on which external contributors could do the most good.
I know in safety-critical the Astree and SPARK tools have been particularly good at verifying code. Something like Polyspace might be more appropriate for Chrome, though.
We've invested quite a bit of energy in static analysis tools over the years, but so far the return hasn't been there. There are still ongoing efforts, and we'll see if those pan out, but up to this point we've just run into a very low hit rate and a false positive rate that drowns everything else out from a security perspective. Formal methods are a different question, but I can't imagine the size of the undertaking needed to convert even a subset of Chromium over to formal methods.
Instead, we've focused pretty heavily on more conventional anti-exploit mitigations applied to existing C++ code. Our sandboxing is the most obvious one, but we have a variety of other protections on top of typical stuff like ASLR and NX (e.g. heap partitioning, a safe integer handling toolkit, etc). We're also leading efficient control-flow integrity work in clang, and busily working it up to a stable release. And then we have fuzzing as part of our continuous integration, running across thousands of cores of instrumented builds.
Sorry to hear that. Yeah, they're not as strong for C++ yet as it's more complex and has less R&D in that sector. My idea for C++ would be to try to find or even encourage development of those that catch the interface and concurrency errors at the least. So, that's Design-by-Contract [1] [2] along with something like ConSeq [3] if it's available or usable for your build. D-by-C, done right, gets a lot of mileage w/ ConSeq showing promise.
Do keep the exploit mitigation tech, though. The NaCl paper was one of best reads I had that year. Most interesting one right now is Code Pointer Integrity [4]. Curious, is there a configuration that swaps out the sandbox for something with the same interface but no real sandbox? Point is if we can swap out the existing sandbox (esp CFI) for a different technology (eg CPI, Softbound + CETS).
The reason I mentioned SPARK and Astree is because they're excellent at what they do. People Googling them will understand how these approaches work to prove absence of bugs rather than test for their presence. Then, they might experiment with some of the others on the list.
I mentioned PolySpace as an example of something more appropriate for Chrome because it supports C++ and is used in high integrity applications.
Do Chrome for PC and Chrome for iOS come from the same code base, and if so, how do you automate testing for closed systems like iOS? (I ask because from my totally anecdotal perspective, Chrome for PC is a rock-solid dream of an application, but Chrome on my iPad crashes if it sees an animated GIF from across the room.)
It is unfortunate that Apple does not allow us to bring all of Chromium--our full rendering stack (Blink, etc.)--to iOS. If they did, we could build a much better browser for iOS. Instead, we are forced to use their provided WebView, and that means living with its limitations and bugs.
The commit queue can only be bypassed by people who are 'committers' and therefore have direct push access to the repo. Becoming a committer requires landing multiple changes, demonstrating competence and desire to continue to contribute, and being nominated and seconded by your peers. This is true for all contributors, even Google employees.
Even committers are heavily encouraged to use the commit queue, and everyone does. Something like 96% of commits land via the CQ. It's simply a bad idea not to. The main use-case for bypassing the queue is quickly reverting a change which somehow slipped through, and even those usually go through the queue anyway with a special flag that bypasses all of the tests.
Engineers base their local development branches on top of origin/master. They can make as many local commits on that branch as they like. Then they upload the diff of that entire branch vs. its upstream (origin/master, unless they're doing chained changes) to the code review system. As it goes through code review, it is tested on the trybots. Once all the trybots pass and it has received approval from the reviewers, it is landed on master and fully tested again.
Maybe I'm still not understanding fully, but that really sounds just like having branches, except they aren't treated by the system as such. There's still the possibility that while developer A's change is going through code review and testing, developer B's changes (which conflict with developer A's) make it through and get landed.
I'd agree that having long-lived branches is often bad, but that's more a matter of discipline than anything else. I could certainly start working on a feature locally using Chromium's model, take my time, and get far enough behind the tip of the tree that by the time I want to submit for code review and testing, my stuff doesn't apply to the new tip.
Branches vs. no branches doesn't seem to be a factor here.
There are no branches in the golden-source-of-truth repository (except for release branches, but those are never merged back into master and no development is done directly on them). The only branches that exist are on the developer's local machine, and only exist for that developer's personal preference and bookkeeping.
Yes, it is totally possible for two developers to be working in the same place and try to land their changes at the same time, resulting in a conflict. This happens in one of three ways:
a) dev B's change fails tests, because the try bots which try to test dev B's change have already sync'd dev As change (this is the most common)
b) dev B's change passes all the tests, but fails to commit because dev A's change landed seconds earlier. In this case the system sends dev B an email asking them to rebase their change and try again. Simple.
c) dev B's change passes all the tests, then lands and causes tip-of-tree tests to fail because dev A's change landed seconds earlier and they disagree on the name of an API method or something. This happens rarely, and is easily resolved by reverting one of the two and asking the author to try again.
Anyway, the point is that the dev branches are incredibly short lived: they exist only on the developer's machine and end up as a single, coherent, self-contained commit on master. If you get far enough behind that your stuff doesn't apply on master, it is your responsibility to clean it up, and no one else even knows that you had a conflict.
How long does a single dev typically go between applying their changes to master? Minutes, hours, days, weeks? If you have a big change to work on do you put it in master behind a feature flag, do the whole thing and then rebase, or what?
Do developers ever want to collaborate on an in-progress feature before it's ready for master? As far as I can see the only real difference between your workflow and a github-flow-like workflow is that they can't do that because their feature branches are local.
> We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert.
You could automatically bisect with just the failing test to find which commit to revert. At least if it's not a flapping test.
I've contributed to Chromium recently and was amazed by how quickly and painlessly everything was.
I expected to run into a lot of problems with building, missing dependencies etc. but it just worked. This despite the fact that the fully checked out source is 37Gb!
I can highly encourage beginners to take a stab at contributing to Chromium, you can learn a lot.
This was a positive read! About the stress and modularization. It's important that developers feel they are an important part of the team. But if they are working for free, it needs to be fun and not feel like work.
Interacting with others are fun! But not if it gets out of control. So when you modularize, you should try to make a balance for human interaction in mind. Maybe keep the number of module interaction down below five at any given time, example: When a new feature is added, max five modules should need updating.
Sometimes we forget that we are making software for humans, by humans, and not for compilers and computers.
A nit to pick here: Google releases Chrome, not Chromium. We only provide Chromium as a source repository, not binaries. Other independent entities (such as Linux distros) can provide releases of Chromium, but Google does not.
That said, Google doesn't support previous releases for security reasons. We are constantly improving the security profile of Chrome, and hackers (both black hat and white hat) are constantly finding exploits in older versions. (The same is true for all the other major browsers on the market.)
If you want to use Chrome on a previous/older release, please don't. If you want to use Chromium on a previous/older release, also please don't, but you can compile it yourself if you're so inclined.
And for the most part, no, customers don't demand stable old releases of Chrome. They've learned how bad of a security and usability mistake that is from IE6, and they don't want to repeat past mistakes any more than we do.
> And for the most part, no, customers don't demand stable old releases of Chrome. They've learned how bad of a security and usability mistake that is from IE6, and they don't want to repeat past mistakes any more than we do.
I suspect that there are lots of enterprise customers that haven't learned that (encountered it, but not learned it), but they probably aren't Chrome customers, since they are still using IE.
While we acknowledge some people might think they want this feature, it's been something we've been unexcited about helping them with it for the reasons stated. Having the ability to quickly deploy security fixes is fundamental to Chrome's ethos. This is as close to a point of religion for the Chrome team as you can get.
I suspect IT customers actually wanted something in between the two extremes. Your existing IT customers may feel okay about it now, but could change their minds if, for example, a fairly drastic UI change is pushed out.
It depends on what you mean by "customer". The chrome team has traditionally viewed end-users as the customers. Normal people who just want to use the web and have it work well. Those people could care less how the program is updated ... the less they have to worry about, the better. And having Chrome automatically update means that they are always safe from malware, the browser continually gets better, and works with the newest websites, without them ever having to do anything at all.
If you instead view IT departments as the customer, then yes, they care and want to have control over the deployed version of the software on their networks. Chrome does have some controls for this. I'm not that familiar with them, though.
There are certainly end users who are confused when the UI changes. Those users are usually incapable of using the bug tracker or complain otherwise.
Kongregate currently tells you chrome broke Unity with its current version and you should use Firefox instead. If previous versions are not available, switching browsers is an alternative workaround.
Sure. There are downsides to continuous updates too.
Engineering is about tradeoffs. You can't implement everything someone might want. The desire to have the product continuously improve is in direct conflict with the desire to have the product never suddenly break.
In this case the team believed the positives dramatically outweigh the negatives and I agree. Continuous silent updates are one of the best features of Chrome. All consumer software should have it.
I've seen some mild demand for it on occasions where there's a debilitating bug in Chrome. Fixes almost never (never?) get backported, so if you want it, you have to upgrade off stable to the more bleeding edge, and then follow that release as it stabilizes, or wait. Sometimes this up/downgrading is just plain onerous. The two bugs that come to mind are a. a font bug whereby the wrong (fallback) font was selected; this happened quite often on Wikipedia and b. gestures would flat out stop working randomly. AFAIK, these are both long fixed, but the experience on stable during that time was pretty bad. Thankfully in the many years I've used Chrome, such bugs are exceedingly rare.
As a developer, I wouldn't recommend an old browser solely for security reasons, but also so I can stop support ancient browsers sooner…
my experience with projects that do trunk based development, i.e. a single branch and time-based releases, is that not all features can be ready when the release bell "rings", and something is necessarily half-baked. So you end up enclosing some code into "if (FEATURE_READY) {...}" so that it's there, but it isn't executed until a few releases later.
This carries some problems, like the explosion of the quantity of toggles over time, dead code that nobody cleans up, etc.
As the article states, Chromium uses run-time checks (feature toggles) extensively. This lets half-baked features land and start undergoing testing before the feature is ready to be shown to the world.
But we don't say "oh no, this feature isn't ready, let's wrap it in checks". The feature is wrapped in checks from the very first commit, so that everything stays stable on master and on any release branches which are cut from master.
Then, when the feature is ready for prime-time, the developers working on it go through and remove the checks. So they don't accumulate in the codebase.
The feature team is responsible for ensuring their in-progress work is sufficiently tested. This usually means that all their tests are run from the beginning of the features life. So, we don't end up testing every combination of features, but instead the union of all features.
With good software design practices we don't usually find bugs that only occur with some specific non-total combination of features. But of course it does happen.
The article isn't a way to make provably correct software; it's a way to keep the wheels on a massive, fast-moving project -- barely :).
Sure, which is why I'm interested - I work on a massive, fast-moving (proprietary) project, and team and feature branches are a vital part of the workflow, and merges seem less costly than the risks of feature flags.
(Of course, the Knight incident being fresh in our minds, our assessment of those risks might not be entirely objective)
>We run a public buildbot that constantly runs new changes to our code against our test suite on every configuration...
In perforce terminology how is this done? You could use branches, but you could also shelve the change and have the build-bot unshelve it and test.
>We also have a commit queue, which is a way to try a change and have it landed automatically if the try succeeds
What happens if the change becomes incompatible due to other changes over the course of running the test? I assume the commit just fails in this case..
Well, we don't use perforce (we use Git), so I assume you're asking for your own benefit.
The buildbot masters watch the source repository for new commits. When they see a new commit appear, they tell all of their builders (each one representing a different build configuration) to do a build+test sync'ed to the new commit hash. The builders check out the repository at that revision, compile, and upload the binaries for the testers. The testers then download those binaries, test, and report results. Then we have a display (build.chromium.org) which shows the results of the compiles and tests on each configuration for all of the recent commits.
It's actually pretty rare for a patch to become unlandable while the tests are running. But if it does, it just sends a message to the owner of the change that says "Hey, your patch failed to land due to merge conflicts. Please rebase and fix them, then try again."
(i also tried to find all the commits or issues where all the 3 ways to disable sending referrer headers were removed over time, but obviously, there are no issues and the commits are probably mixed up with other things)
I'm not a fan of Google or Chrome either, but this article was amazing, and the Chromium devs in this thread have provided some really neat info. Offtopic flaming isn't useful to HN.
(And as to your bug, I tried --no-referrers in Chrome 45 on Windows 8 and it worked fine.)
As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged Blink into Chromium proper just this week! It is no longer managed by gclient/DEPS, but instead developed alongside Chromium in the same repository. But many other things (pdfium, v8, webrtc, and tons of smaller libraries) are still pulled in via DEPS in exactly the way described.
I'd just like to say this about our development, testing, and release process: it really really works. We keep moving fast, everyone knows what's going on throughout the codebase because everyone is working on master, and there are never any big ugly merge conflicts to resolve when new features land.
There are, of course, problems with the system. We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. The set of configurations on the try-bots doesn't exactly match up with the set of configurations that are run against tip-of-tree. And flaky tests are, of course, a perennial problem for everyone. But we're working on making all of these better, and huge progress has been made in the last couple years.
Ask me anything if you have any questions about our build and test infrastructure!