The Yubikey requires a password by default to use the ssh key stored on it and it will lock itself after 3 failed attempts. So I don’t think your caveat is valid. I rather have my encryption key on hardware design to keep anyone who finds it from brute forcing it than just password protected on a hard drive.