Ah you have hit the nail on the head. It's not just who defines morality, it is often also circumstances surrounding events that all combine to make life complicated. A man and a woman swear to be faithful to each other, then it is immoral to cheat on each other. The religion you follow also defines what is immoral and what isn't. There are also different ways to intepret religious passages.

The reason I said life is complicated is because it is impossible to define immorality for everyone. There is also the issue of the law, what is legal may not be moral. Then they are personal circumstances. Is it immoral for an abused married woman/man to cheat? Everyone will have their own answer based on their beliefs. This is why life is complicated. It isn't as straightforward as media headlines we see.