Managed to make it work. I usually don't put the mask over my nose as, having glasses, makes them all foggy and I can't see anything (protip: I don't care about how effective that is, the mask for me is just the minimum to confirm to current regulations).
Seems that faceID is really anal about the mask being OVER the nose before it's recognized as a mask! This is why it wouldn't work before.
So if you're having problems, be sure to wear a mask as apple expect you to.
It's not like it fails, it literally never even tries to use the watch, goes directly to normal faceid.
Gone are the times when apple software would just work...