I was going for a PhD in Math and dropped out for pretty much the same reason. The field was so specialized that nobody really understands anyone else's research, which makes it mostly not-merit-based (politics) who gets hired where. Also, it really wasn't fun. I figured it would be easier to work as programmer and study Math (or other things) in my spare time.
It seems like a waste to spend 5+ years on a PhD and 3-5 years on postdocs just to get a tenured position at the #200 ranked university or to be a permanent-adjunct.
Number theory is an extremely broad field in itself. I'm pretty sure most of it is completely irrelevant to the NSA. If you do a PhD in any theoretical field you'll almost never find a job outside of academia that uses any of your work.
I've seen permanent adjuncts / post-docs. There was a guy at my old school who was there as a post doc. He did his PhD at UC Berkeley. I can't believe that he has been jumping around to different institutions as a post doc or lecturer for over 10 years. It's quite sad. My friend did his Ph.D at University of Oregon. It took him 6 years to finish it. Then he spent 2 years as a post doc at University of Toronto. He finally got a job back home at a small 4 year college. I guess he's doing what he set out to do: be a university math professor. But, I believe him being from Hawaii helped him land that gig. They prefer locals who want to go back home before considering others.
As an outsider I'm a bit frightened. I my (mostly estranged) cousin is a Maths researcher (from wikipedia it looks like he's at UCLA now). Once I was talking to a PhD student friend about him, and actually he had used my cousin's work in his field.
So I don't think there is no hope in research. Also I admit this whole citation, publication and plagiarism hunt is a toxic bullshit.
It seems like a waste to spend 5+ years on a PhD and 3-5 years on postdocs just to get a tenured position at the #200 ranked university or to be a permanent-adjunct.