If that's the case, then why refresh? Might as well just hand out forever tokens, which, you're right, makes less than zero sense.
The point is that you have two tokens, one a refresh token and one a stateless token. You revoke the refresh token on the server, which means the next refresh attempt will fail.
Because it's stored and that would require a round trip to your auth server on every request. Plus, its security requirements are far higher than the access token, so you don't want to be flinging it all over the internet. It only ever goes between your users and your auth server.
The point is that you have two tokens, one a refresh token and one a stateless token. You revoke the refresh token on the server, which means the next refresh attempt will fail.