I woke up with the realization that my problem is probably GC pressure. The system is currently written in idiomatic Clojure so doing everything generates garbage. What's more, the garbage is created in one tick, then released in a subsequent tick, so I'm losing the benefit of generational GC.

I think I can make my subgrids algorithm work, but it will have to be using mutable data structures.