I have to admit that I didn't dive 100% into what made this difference exactly. It just worked better without webworkers.
I just dived into the git history, and it turns out I didn't use the buffers as transferable. Perhaps that was it! I'll definitely check that out later, thank you for pointing this out!
No problem. I just happen to be working on a somewhat similar project. It's strictly 360 photos and it's built around our company's foregin language instruction services, but I've had to deal with/am currently dealing with several similar issues. Indeed, the next major tech change I had planned was to also implement my own WebGL code and get rid of Three.js. I'd like to see if I could get the WebGL code running in a WebWorker with OffscreenCanvas, but I'm as-yet unsure if that will work with WebXR.
I just dived into the git history, and it turns out I didn't use the buffers as transferable. Perhaps that was it! I'll definitely check that out later, thank you for pointing this out!