That's exactly the same as Wayland, isn't it? Every Wayland compositor also polls HIDs connected to your machine for events. And if your Wayland compositor has permission to open those HIDs when running as your user account, so does any other program you run, regardless of Wayland or X.