What's at risk of being taken away is the right of weaker groups to abuse stronger ones, at least to some degree.

Stronger ones always have and always will abuse weaker ones. It's pretty much a tautology. Freedom of speech and classical liberalism in general is the only defense the weak have.