How do you know that they exist, if you're not scanning that memory?

The data structure code can take care of this by registering GC roots with the garbage collector (and de-registering them if an external reference changes). It's no different in principle than any other smart pointer.

> You can treat external references as GC roots.

Which brings you back to doing an expensive scan of the large off-heap data structure you were trying to avoid.