That's unusual. You have better specs than I do, and I certainly don't experience any latency on Windows 10 at all -- definitely not any in the seconds range. Anecdotally at my workplace, I haven't heard any complaints either. There might be something that is idiosyncratic to your setup.
I upgraded my Core 2 Duo machine from Windows 7 to Windows 10 and it definitely struggled.
But my Core i7 machine has no trouble with Windows 10.