On the other hand, when you have $5M, you may want to protect it because if it goes down to $1M it makes a difference. For those ultra-wealthy though, if their wealth goes down 100x, it does not impact their comfort of living in any way. They just have less power.

I'm sure it still does not feel good psychologically so I'm not completely disagreeing with you.

The problem is that most very rich people have very concentrated positions. The family office might only manage 10% of the wealth with 90% being in one company. The investment then has to be structured so that the 10% survive even if the 90% vanish overnight (unlikely but not impossible). The performance on those 10% isn't that important, ownership in the company drives most of the fluctuation in wealth anyway.