Yeah great question, and I don't fully have an answer yet! For the scenario I need it for, the hostname only does one thing and won't be looked up by the client until it's needed, which helps avoid that problem. Furthermore, thank to K8s nodePort routing, as long as one of the nodes is still alive and answering, the request will be routed even if the first IP is no longer valid. I've also considered as the cluster size grows, only returning the 3 to 5 newest node IPs present since I expect the ones most likely to get killed/recycled will be the older ones (not always true, but often is in my setup).
Has anybody else run into this and solved it? Cloudflare DNS seems to have figured out a decent way to deal with this. I may take a close look at their responses and see what they set for TTL, etc.
I will admit my DNS knowledge is a bit out of date, so I am sure folks much smarter than me have indeed figured this out. I am also curious about how this gets mitigated.
Has anybody else run into this and solved it? Cloudflare DNS seems to have figured out a decent way to deal with this. I may take a close look at their responses and see what they set for TTL, etc.