There are many legitimate reasons why this is a problem. I'll briefly list a few but there are many more. One is that the government workers who listen to your phone calls can use that information against you, for example if they are a business competitor. They also can create laws which target you specifically that you don't know about or can't avoid breaking. Because they make the laws, there is no recourse against anything that they do with that information.