50-100ms is the time it takes to generate the response on the server. On top of that you have to add the ping time between you and server and whatever other network overhead there is.

That's why it's great to be closer to 50ms so you'll stay under 100ms even with the network overhead.