This is soon to be solvable by mounting the cache directory during the build, thankfully: https://github.com/moby/buildkit/pull/442 ("dockerfile: add run --mount support #442 ")

Docker also has a cache, making dependency installs not usually an issue.

Which works on a totally different level and doesn't help at all in terms of sharing dependency snapshots between different projects.

That is not something I'd want to do. Docker's solution is good enough for almost every build that happens. I can eat the time cost not covered by that solution and not have to worry about bugs stemming from improper copying of dep snapshots.