Eh, it's all fake internet points anyway. I occasionally get hit by an angry downvoter, but as long as your normal content is quality their effect is a drop in the bucket.
I would actually take the opposite side of this, I think saying "karma is just fake internet points" is missing something important. Karma is a rough metric for social approval, and social approval is one of the most important things for people.
Now, karma isn't actually that great of a measure of social approval (you're unlikely to encounter your downvoters in real life; it's easy to get karma by making short uncontroversial comments; etc.), but it still means something, and it makes perfect sense to me that people would care about how much karma they have.
I would discourage using karma as any measure of worth for the reasons you described. If a user has lots of negative karma that might be something else, but a large positive karma may mean nothing more than the person has been active on the site for a long time and tends to post a lot and doesn't troll frequently.
It doesn't mean they're super smart or some incredible subject matter expert or anything like that. They just have a lot of time to post and aren't a complete shithead.
In fact site operators should put extra scrutiny towards accounts that accumulate karma too quickly. That could be a sign of bot activity or "account optimization" firms making a mockery of your reputation system.
