As a current graduate student, the sunk cost can sometimes be too high just to up and move advisors. In small places, you will simply not have someone else whose interest align with yours, and even at a place like MIT, the advisors have their own specialty, and especially after your second year at Ph.D. moving or changing advisor can be quite difficult and basically "reset" all your progress. So, a lot of time, the choice becomes moving and voiding all your work you've done so far, or gritting your teeth and carrying on for "just a couple more years until I graduate."

Source: was an undergraduate and Masters student at MIT, then left to do a Ph.D somewhere else with a much better/less toxic advisor.