When I see this kind of GC performance, I wonder why you wouldn't change the implementation to use some sort of pool allocator. I am guessing each Read State object is identical to one another (e.g. some kind of struct) so why not pre-allocate your memory budget of objects and just keep an unused list outside of your HasMap? In a way this is even closer to a ring where upon ejection you could write the object to disk (or Cassandra), re-initialise the memory and then reuse the object for the new entry.
I suppose that won't stop the GC from scanning the memory though ... so maybe they had something akin to that. I assume that a company associated with games and with some former games programmers would have thought to use pool allocators. Honestly, if that strategy didn't work then I would be a bit frustrated with Go.
I have to say, out of all of the non-stop spamming of Rust I see on this site - this is definitely the first time I've thought to myself that this is a very appropriate use of the language. This kind of simple yet high-throughput workhorse of a system is a great match for Rust.
A pool allocator could have reduced the number of existing allocations (1 big one instead of many small ones), making those spikes less significant. (But that depends on how Go handles interior pointers and GC, so I'm not sure.)
Allocations weren't the problem. It was the fact that, every 2 minutes, the GC would trigger because of an arbitrary decision by the Go team and scan their entire heap, find little to nothing to deallocate, then go on its merry way.
Here it wasn't the problem, that the GC was lacking performance when collecting garbage, which a pool allocator would have helped with, but rather, that they didn't produce garbage (good), but the GC ran nevertheless to check whether memory could be returned to the OS. Probably supressing that would have removed the spikes.
In this case, since the lines of code that can touch the manually managed object pool are probably few and easily reviewed and audited, I don't have any problem with your advice.
I realize you're not advocating pervasive use of the technique, but if someone reading this is going to make pervasive use of manually managed object pools in a GC'd language, they should at least consider the possibility of moving to a language with both good language support for manually managed memory and a good ecosystem of tooling around manual memory management.
Manually managed object pools in a language designed around GC don't fully get rid of the costs of GC, and re-expose the program to most of the errors (primarily use-after-free, double-free, and leaks related to poorly reasoned ownership) that motivated so much effort in developing garbage collectors in the first place.
I suppose that won't stop the GC from scanning the memory though ... so maybe they had something akin to that. I assume that a company associated with games and with some former games programmers would have thought to use pool allocators. Honestly, if that strategy didn't work then I would be a bit frustrated with Go.
I have to say, out of all of the non-stop spamming of Rust I see on this site - this is definitely the first time I've thought to myself that this is a very appropriate use of the language. This kind of simple yet high-throughput workhorse of a system is a great match for Rust.