I don't think it's likely that all the folks in Vermont are racist or that the system of governance there is supremely racist - yet Vermont has an absolutely tiny percentage of non-white folks (3.1%)
Less than 15% of the U.S. is African American so it's not at all surprising that there are areas where there are very few black folks as a result. That's nobody's fault and shouldn't be described as racist because, well, it isn't.
3.1% is about 1 in 30 people. That's not that small. And this isn't 100 years ago, I'm pretty sure people in Vermont have seen a black face on TV/film.
This is historically ignorant. I don't blame you personally, because the American education system is very bad at covering America's racist history. But you're still basing your estimation of "likely" not on real data, but on what you'd like to be true.
If you'd like to learn more, start with Loewen's "Sundown Towns", which describes how hundreds and possibly thousands of towns across the US were turned and kept white:
It's also not necessary for a lot of white people to be actively racist to preserve white supremacy. As is clear from Loewen's book, all it takes is a small number of people willing to be active (e.g., making threats, burning crosses) to drive off new black arrivals. The majority can be passively accepting of a racist system without lifting a finger.
But that doesn't mean they won't end up biased. White people who don't know many black people are more likely to be biased, and all-white towns produce people that are more likely to have racist views. (For more, see Loewen's Ch 11, "The effects of sundown towns on whites".)
Less than 15% of the U.S. is African American so it's not at all surprising that there are areas where there are very few black folks as a result. That's nobody's fault and shouldn't be described as racist because, well, it isn't.