Your reply misses the point. We were already doing so few allocations that the GC only ran because it "had to" at every 2 minute mark. The issue was the large heap of many long lived objects.
When we investigated, there was no way to change that that we could find - barring compiling go from source (something we could have done, but wanted to avoid.)
Yes, you have to rebuild go, but that is literally done in a minute.
It also would be interesting, if you happen to have some conclusive benchmarks, how the latest Go runtime would perform in this sense.
I don't get, why this is downvoted without comments. Compared to a rewrite, this would have been a miniscule change. Furthermore, considering that you wrote that long blog post (which I quite appreciate, as it contains interesting information), it would have been important knowledge, whether the setting of the parameter was the real culprit - and if it was, a good reason to shout out to the Go implementors to look closer at it.
All I'm going to say is that if you think maintaining your own version of a compiler is the reasonable option compared to a rewrite in another language, you are probably deeply invested in the former language. This also applies to kernels and databases.
Well, in this case, "maintaining your own version compiler" concerns a single value change in the code base. At least, as I wrote, it should have been tried to identify the root cause for the observed behavior. If this "fix" significantly improves the behavior, it would have been a good data point to reach out to the Go developers to resolve this issue.
The problem to get down to the core of these issues are test cases. It seems, that neither the Go developers nor many other people have run into this as an issue - I only remember noticing the regular GC some years ago, but it was not an issue for me. As they have a real-life test case exposing this problem, they are possibly the only ones, who could verify a potential fix for the problem.
So, while it is great that they identified the problem and wrote a thorough blog piece about it, the only thing we learn from this is, that in Go 1.9 there was a latency issue every 2 minutes with their style of application/heap usage. Unfortunately, we don't know whether this problem was already addressed in later Go versions, and if not, whether there should be a way to control the automatic gc intervals to address this.
