This is something that has fundamentally bothered me about our current society. I consider all the positions of people who do things that, when all has been said and done, do not provide value to humans. Then I consider all the positions that will advance humanity. The mismatch is jarring and telling. Something is wrong.
I imagine a society where all the funding into ad tech goes into cutting edge research, where we might have to wait a decade or two but before too long we reap the rewards. Even in small cases that that don't matter, even something like creating a video game, how much of the budget goes into the game and how much goes into advertising? Across the gaming industry, how much of the budgets for creating games goes into games that expand the mind, introduce players to new views, and garner interest in new areas of life and how much goes into reskinned slot machines?
This pattern seems to appear in many places. Short term optimization over long term gratitude as the social level leading to and caused by the same as the personal level. While it isn't the worse humanity has done, and it isn't anything worse than seen in other species, I wonder if this is a more pervasive and passive Great Filter that'll cling to us even after we have overcome others like the threat of nuclear war.
From the point of view of wider society, people who don't produce tangible or comprehensible benefit are indistinguishable from parasites.
All the people around them are supporting them, how do you know they're really worth it? If a physicist spent 40 years producing useless stuff, would anyone notice? Does that matter? How many string theorists should we support as a society, anyway? And why say, researchers? Why not artists, interpretive dancers or historians? Critical race theorists? What does all this say about our values?
The plumber is in the physicist's apartment, knee deep in crap, fixing the toilet. Suppose the plumber has interests, things she would do if she didn't have to wade through crap all the time. Suppose she's actually a fairly talented artist. Why does the physicist get a "free ride", intellectual freedom and financial security, while the plumber deals with the crap?
Of course in our society you _do_ get a "free ride" by owning enough stuff. That's a separate, but related conversation.
You don't know anyone is really worth it. That's why it's called "research." You pay smart people to explore being smart, because even though most of the effort and money are wasted every so often something wonderful falls out.
Try building GPS without relativity. Try building modern chip designs, lasers, or many medical applications without quantum theory.
It's no different to funding start ups. Most fail. A few win big.
But... a room sharing startup is still utterly trivial compared to finding life on other planets or inventing a new kind of computing. Or game changers like EM theory, quantum theory, relativity, Shannon's information theory, or Turing/Church.
And a room sharing startup is still vastly more useful than the weaponised parasitism of most of the financial sector.
Academics are not the problem. The problem is poor allocation of resources, especially creativity and intelligence - all of which have been badly warped by the peculiar value system propagated by contemporary economics in ways which have left them far less productive than they have been in the past.
Would you think the same about speculative social science? Some nerds find theoretical physics especially cool for its own sake, as opposed to say pure philosophy, but this is not most people's position. They think even space exploration, something much more concrete, is a waste of money. Your value judgment is right, and theirs wrong? They're the one having to clean the literal toilets. The thing also is that quantum mechanics was developed with a fraction of today's money being spent on physics.
On the one hand I completely agree with everything you wrote.
On the other hand, it sounds like what you are proposing is that there should be socialism for sufficiently talented people (I am not actually averse to this, although it seems fairer if it could somehow be extended to everyone).
Private capital will only fund ideas which a) have quantifiable benefit which b) can be captured. This aligns well with patentable pharmaceuticals or adtech, not so well with theories of quantum gravity. Stuff which benefits everyone conventionally has to be funded by government (although a patronage model does not seem impossible).
In the ideal case, how would we work out who are the sufficiently talented (they get socialism, let's call them the eloi), and who will who make their shoes and clean their toilets (it's capitalism for them, let's call them the morlocks)? What fields would be eligible for this kind of support and who would choose that?
It sounds like your complaint is that the existing systems for doing this (grants, tenure, research impact, prestige) are mis-allocating. I agree that existing systems aren't working well. Is it just that there is not enough funding, or does there need to be a rethink somewhere?
It seems fundamentally to be an information problem. That is, it's hard to figure out who and what are worth funding. The information problem is more difficult if the research has a longer time horizon.
Yet our current system shows we do have tolerance for dead weight, be they geniuses who happened to have bad luck in the theory they chase or be they frauds. One way to rephrase the issue in light of this is to ask if we have optimized the dead weight for the benefit. If we double the scientists and accept that we will increase our percentage of dead weight, won't we still all benefit from the increase in non-dead weight. Have we really reached the point of diminishing returns?
Yet another thing that bothers me is how benefit is viewed. Ability to defraud others, addict them, create fear and then provide the safety for that fear are all seen as benefits where as a scientist who pursued the wrong theory is not. Yet in terms of how better off society is, isn't the scientist winning even if they net contribute nothing?
I don't claim to have answers. The more I ponder the nastier the problem becomes. Some of the answers I've seen other gives, things that could be summarized as variants of socialism or related, fail to account for other flaws in humans.
It somewhat reminds me of trying to figure out how to run a government. Like someone pointing out all the problems with democracy, yet seeing that there isn't a better solution. Well, incremental improvements in the specific variant of democracy aside.
I imagine a society where all the funding into ad tech goes into cutting edge research, where we might have to wait a decade or two but before too long we reap the rewards. Even in small cases that that don't matter, even something like creating a video game, how much of the budget goes into the game and how much goes into advertising? Across the gaming industry, how much of the budgets for creating games goes into games that expand the mind, introduce players to new views, and garner interest in new areas of life and how much goes into reskinned slot machines?
This pattern seems to appear in many places. Short term optimization over long term gratitude as the social level leading to and caused by the same as the personal level. While it isn't the worse humanity has done, and it isn't anything worse than seen in other species, I wonder if this is a more pervasive and passive Great Filter that'll cling to us even after we have overcome others like the threat of nuclear war.