Right, we don't know enough about the brain and/or conscious experience to determine where and how morality arises. In a way this is part of the risk. We might stumble on emergent morality with a large enough language model and not know until it starts happening with real consequences.