Yes, but actually no: usually setting up those namespaces is done through a privileged daemon or suid-root binaries. Both of those are prone to root exploits, which isn't as bad as a kernel exploit, but only a 'modprobe' away. Group membership in the 'docker' group is famous for being root-equivalent.

It isn't impossible to do things right, but in practice, things are usually done badly.