Yes that's true you could be linked if using a custom domain and someone can determine that nobody else is sharing that domain with you.

That's why I added the additional username feature to compartmentalise aliases and also the UUID aliases.

The UUID aliases all share the same domain e.g anonaddy.me so it is not possible to link ownership of these to any particular user.