Alcohol even at low doses is toxic that's why your liver has enzymes to break it down. Many other things are not nearly as toxic and most things that don't belong in the brain don't cross the blood brain barrier. Those which do and don't belong usually cause some sort of problems.