They only do the real number check when you are first creating the account right before number selection. They never ask you again to confirm or do any kind of verification with the real number.

I've had to change my phone number a few times over the decade and my personal gvoice is verified with whatever number I had 10 years ago. I just don't set it as a callback or text receiver.