I have a strong feeling that if we went on like that and one day there will be an AI which will optimize you out - "Humans do not add much value to things, they are just cost centres."
Humans add all the value and always will, because that's what value is - what humans want. If an AI has some goal independent of humans, it's no different from a car rolling downhill because the parking brake failed.
Sure, but good luck convincing an AI that after it has power. It's important to build the AI to think that way from the start if it is capable of getting power over us.
It's been a long time since I took a course in philosophy, and all I recalled was John Stuart Mill being associated with utilitarianism.
Looking at his Wikipedia page, it appears that his idea of the utility associated with pleasure was independent of the specific person. If X is the amount of pleasure humanity gets from something, and Y is the number of people that exist, then the pleasure an individual is assumed to get is X/Y. If I understand correctly.
If someone were to abuse the concept of utility by considering it to differ between people, it seems to me a lot more likely that they would say something like "I get a unit of utility from this candy bar, but many other people do/would not, so I don't need to share with them", rather than "I get millions of units of utility from this candy bar and others only get a normal amount".
