But you've still got to load the chunks from some massive dataset stored somewhere and you can still look/run around in a full sphere at 60fps?

The backdrop pieces are effectively where the rays hit as far as I can tell from your description.

One memory optimisation I can think of is to store a bvh of the empty space and traverse that. Using a signed distance field might also help in the ray marching.