Applications that cache data have to do it on the heap.
Yes, there should be a way for the system to coordinate usage of cache/free memory and coordinate with a VM’s GC. Until we have that, big apps will exhibit the aforementioned memory footprint.
I am not talking of the netflix case from the OP, but rather of the general case where you are likely to have multiple services on a single host. Just imagine if a couple of them were Java-based: you'd have to do figure out how much is the maximum needed for each of those and sum it up and ensure the server has that much memory even if they are unlikely to all need it at the same time: if we are talking about waste, that's wasteful to me.
Even in a more modern setting with containers, you'd have similar issues.
Applications that cache data have to do it on the heap.
Yes, there should be a way for the system to coordinate usage of cache/free memory and coordinate with a VM’s GC. Until we have that, big apps will exhibit the aforementioned memory footprint.