Interesting. I wouldn't want to shove my base monitoring data over UDP. Without my base monitoring, our product devs are blind and things go to shit. I can't risk that.
And, with a bunch of per-site monitoring frontends with batching, compression and deduplication, thrift/protobuf and long-running connections, the TCP overhead is meh.
If your network is so about to fall apart under load, or so broken due to network issues, or your servers are so overloaded that you start seeing appreciable packet loss with UDP internally at a site, then chances are the last thing you want is loading your system even more and slowing things down by retransmitting samples that are obsolete in a couple of seconds anyway....
Serious exceptions, sure, retry. But serious exceptions are rare. What's not rare is sampled memory usage, cpu usage etc..
Sure, you can't cope without monitoring. But you can absolutely cope if 1 out of 10 of each of those status updates doesn't get to you.
This is the completely standard connection-oriented vs connectionless link tradeoff.
And, with a bunch of per-site monitoring frontends with batching, compression and deduplication, thrift/protobuf and long-running connections, the TCP overhead is meh.