Please read the second part - I use a command restriction in authorized_keys so that the guest can only call certain commands.

Yes, the guest has “unlimited” access to the pasteboard, which does introduce some risks. For example, the guest could set a malicious command line that you paste into the terminal - which is generally mitigated with paste bracketing in zsh, vim, etc. It definitely weakens the isolation to a certain extent, but I don’t think it completely negates the security barrier as you claim.