Heh. I also felt that way, spent the first month after leaving looking for next closest thing.
The most exciting thing I found was Nix <https://nixos.org/nix/>, a hermetic build system + linux distro + config-based service manager, all built around a purely functional language.
The way it's actually used to build a lot of existing projects is coarse-grained — the Nix build actions might be simply "get tarball; untar; configure; make". (I think it may be combined with distcc/ccache for per-object caching, not sure).
But if they run existing tool to do the actual builds, how do they enforce hermetic dependencies? They force a specific absolute path structure where all inputs and outputs dirs include the (transitive) hashes.
[There is also some crazy rewriting of hashes inside binaries, which they claim works fine.]
The drawback is you can't move binaries or libs around; the benefit is multiple versions can coexist in-place.
I was planning to experiment with some pieces I was missing:
- Use bup <https://github.com/bup/bup> to chunk build products for incremental transmission of similar builds.
- Their build farm IIUC assumes long-running slaves; I wanted to build on public cloud only paying for burst activity.
Then I suddenly remembered that I write python and now javascript, which I can use without any kind of build, and gave it up :-)
But if I were to toy with hermetic building now, I'd look at basing it on Docker.
Can you explain why? I pretty much hate all the ones I've worked with; some insight into a system that people actually enjoy to use would be really valuable.
Yes please. After spending way to much time with Ant / Maven / Gradle, I'm convinced they all got it wrong. A build system that doesn't suck is like a unicorn - rumored to exist, but nobody's seen one yet (except Leiningen, people seem to have no complaints about that one).
From what I understand Leiningen is Maven in lisp clothing. Any complaints you have with Maven might apply to Lein as well, unless it is just the XML you don't like. I have only used it a little, but so far it seems nice.
Sorry for the late reply. I got distracted by life :-)
So the qualities that make Google's build infrastructure awesome are:
1. Truly incremental builds. By this I mean the build system can reliably detect when something has actually changed and exactly how much it has to rebuild. It then rebuilds only as much as it has to. There are some systems that claim they do this but almost none of them that I'm aware of other than the aforementioned Nix package manager actually do so.
1a. Googles' system does so by forcing all targets to fully specify their inputs and outputs as well as treating all targets as pure functions. No side effects allowed. The inputs determine the outputs so you know if nothing changed in the inputs then the outputs didn't change either. This means all compiles are as close to fully hermetic as you can get.
2. Distributed builds work really well. Building off the fact aforementioned guarantees for build targets. You can also build a very accurate graph of your build targets and effectively distribute all the of building.
3. Build artifact caching. Since your targets are pure functions you can safely cache them for the whole company. When you have thousands of engineers all building the same codebase frequently and the builds are highly distributed and the cache is shared by everyone the chance that someone else has already compiled a large fraction of the code you are about to compile is high.
All of this combines to make builds really fast, really reliable, and really repeatable. Which is kind of the holy grail of build systems when you think about it.
Google happens to have this because they care enough about their developers productivity that the invested thousands of man hours fixing the time sink that is compiling. That and they also build everything from head so they ran into productivity losses from build times really quickly.
I'd add that having the whole company use a single language to describe build and tests is a trivial but crucial thing.
It allows automatically building and testing everything affected by your change, even if you never heard about those dependencies.
Oh, and "fully specifying inputs" includes the involved compilers, and even the build tool itself! You can reproduce old builds using the same tools that were used then. Upgrading any tools triggers appropriate recompilations.
Heh. In my last job we used Gentoo as the build system for an embedded system. It was a joy to use. In my current job we use naked Makefiles chained together. I waste somewhere near 50% of the day building things I don't need to build, rebuilding things because the last time they were built it was with the wrong options, fixing build bugs where modifications didn't make it into the final binary etc etc etc. Makes me weep.
But seriously, Gentoo was the bomb. Your package doesn't impact anyone else until it's .ebuild is updated. You can precompile most units of the system once a day and download them as binaries. For 4 years of my life I didn't have to worry how things were being built, except for my own very small part that I had mastered. Bliss.
Can you describe more how it worked? I would assume as it's based on bsd ports that one is simply using a common Make file library and then building fairly Independant components.
The problem presumably starts when decomposing a monolithic build and how do you pull it all together again?
I have stalled at pyholodeck.mikadosoftware.com and need some inspiration to pick that over the 100 other things I needs to do :-)
It's a big system, and not something that you can do piecemeal. You basically need management buy-in. So I've been busy recruiting support from other developers and lower-level managers, writing memos that describe the problem to the higher-ups and so on.
