Usually what happens in cases like this is that the RL network ends up learning a slightly crappier version of A*

Or finds a bug in the environment and learns to teleport, those are fun too