That's simple - whenever any of your personal or business contacts grants access to their contact list to some app, "everyone in the world" obtains a mapping between your mobile phone number (that they had), your full name and nicknames (that they entered manually) and likely your other identities as well (e.g. emails, various IM providers, etc) that they linked to you in their contact list.

It's the same with facial recognition - to avoid it, it's not enough to never post your pictures, you'd also need to ensure that noone who knows you would ever tag you in some picture they have.