One thing I like to do with the SSH Agent is also forward my X.509 certificates by adding new opcodes to the SSH Agent protocol, then you can do stuff like PKCS#11 on the remote side with your local smartcard. [0]
This gives you, among other things, passwordless but authenticated "sudo" capability (which is actually required by the DOD, though nobody does it).
Are you aware of any hardware tokens that support PKCS#11 and can be implemented affordably at small scale? The only ones I've found are Yubikey, but Yubikey's other protocols (OTP, U2F) are a lot simpler to implement, so I've never tried.
At one point I bought a few literal smart cards and a USB reader, but not being an enterprise customer of any of the vendors, couldn't find the necessary drivers / passwords to get them working with PKCS#11.
I've used it on macOS to use a PIV cert on a yubikey to ssh to remote linux servers. I have NOT used it to ssh-agent chain through, jump-box style though.
PKCS#11 is extremely versatile, with some configuration you can use Yubikey PIV X.509 keys to authenticate SSH sessions by passing the necessary pkcs11 lib to ssh with the -I flag. PKCS#11 isolates the key signing from the PC by doing any private key computation operations on the secure element, this prevents leakage of the private key by traditional means.
Browser TLS client authentication is also possible with some minimal initial setup of your browser of choice.
You can use openssl with a pkcs11 engine for any X.509 operations you might think of (e.g. PKI, S/MIME email signing, etc.)—technically you could use the PIV applet to store web server TLS keys and have Nginx use those through OpenSSL via the pkcs11 engine, though it would probably slow down handshaking as the throughput and signing speed of a Yubikey is orders of magnitude slower than any modern general purpose CPU.
Many people don't know that Yubikeys also run a OpenPGP card applet for storing up to 3 key pairs. With the ssh option of gpg-agent it can also be used for SSH authentication just like ssh-agent.
I use this in combination with a Yubikey configuration setting requiring all GPG sign operations to be confirmed within 10 seconds by pressing the capacitive touch button on my YK4 nano (LED blinks during this time to prompt). This way I can safely enable agent forwarding globally as all SSH session authentication requires physical interaction to confirm (provided you don't have any file-based keys in the same agent which are still vulnerable to usage without your knowledge via this functionality).
I use the gpg-agent option as well, but hate it (GPG makes me feel icky). That is not, as I understand it, using an X.509 key for authentication though. Which is what I'd like (ssh-agent forward an X.509 based auth mechanism - like remote PKCS#11 over that agent connection, so the key stays on my yubi, and I only need CA certs on the remote hosts.
This is exactly what I do -- I posted some links to both the SSH Agent (well, the ChromeOS version -- I've got C versions as well) and the PKCS#11 module which talks to the SSH Agent. So on a remote system I can do things like sign files using PKCS#11 and passwordless sudo using pam_pkcs11.
There are MANY vendors that supply hardware security modules that make their information available via PKCS#11 modules. From smartcards to TPMs to dedicated large HSM devices.
There are many advantages of something like PKCS#11 over One-Time-Passwords (TOTP/HOTP):
1. Since you have an X.509v3 certificate, you have an identity not just an authenticator -- so when you are using this certificate to login to systems, you don't also have to supply an identification (though most systems let you optionally supply one, e.g., if you can login with multiple usernames and the same credential).
2. PKIX covers a lot more use cases than OTP, things like Kerberos (PKINIT), TLS (TLS Client Certificates), SSH (PKIXSSH et al), digital email signing (S/MIME), code signing, commit signing, and more
3. The device is basically a second computer, so it can run software policies on it to do various things like if you authenticate incorrectly a few times in a row it can zero itself out, with OTP systems that has to be coordinated by the various systems validating the OTP token.
If you're going the Smart card route you can get the PIVKey C910 on Amazon for $15/ea. in quantities of 1. You can also buy them for about $0.10/ea on AliExpress, and then load CoolKey or some other applet.
I forgot to mention that also, of course, CoolKey is open source and the CoolKey PKCS#11 module supports the CoolKey applet (obviously) and makes some attempts to support the US DOD CAC and the US Government PIV -- though I don't know how successful they are. Bugs and feature limitations in CoolKey led me to write CACKey.
The PIVKey C910 complies with NIST SP-800-73 (PIV) -- same as YubiKey PIV mode, so OpenSC supports it. Also, the middleware I wrote (CACKey) is open source and supports it and works on macOS as a tokend driver.
This gives you, among other things, passwordless but authenticated "sudo" capability (which is actually required by the DOD, though nobody does it).
[0] http://cackey.rkeene.org/fossil/artifact/0d0e90bbfdee672c?ln...