> When the system allowed segregation, whom do you blame more

That is easy - the people who were taking advantage of it.

E.g. I have the right to be a racist dirtbag - but if I act like a racist dirtbag that is on me and not on the system.