His pseudo-fascist political views, love of eugenics and other fringe stuff might have played a part.
Also you can't really expect a person narcissistic/unhinged enough to name his child "X AE A-XII" to be even remotely likeable (considering he also did it as a joke and possibly forgot about it doesn't make it much better).
Remember that Elon was the overlords' darling up until he decided to spit in their face and buy their favorite mind programming toy from under them.