Escalating to physical violence looks a lot riskier if you're smaller than the person you're striking. And you can't know that the escalation will end with your hitting them: they may hit back, and they may not stop. Moreover, assault is a crime: you may feel justified, but if you can't prove the other guy hit first, you can be stuck.

Finally, women in our society are socialized to be nice, to not make waves, to get along. Maybe you don't like that, but you can't just make decades of socialization that millions of people have undergone disappear because you don't like it.