Processing multi-GB files in the browser is... fun. Doing that kind of thing on a server is easier.
*I'm not justifying doing it on the server, especially for an application like this where yes: it can be done in the client.* But I do sympathize because I know from experience why it's easier to do it server-side, without any conspiracies.
I wrote Papa Parse[0] about 10 years ago, and back then at least, it was extremely difficult to stream large files in an efficient, reliable way. Web Workers make things slightly better, but there's so many issues with large-scale local compute in a browser tab.
You get deep enough into the weeds and eventually you realize you can make it work cross-browser if you know which browser you're using (YES, User-Agent does matter for things like this) and call you crazy for trying to find out:
Despite all this, I *100%* agree and local-only processing is also a hard-rule for me as well. (That's why JSON-to-Go[1] does it all client-side. `go fmt` event compiles to WASM and runs in the browser!)
> Processing multi-GB files in the browser is... fun. Doing that kind of thing on a server is easier.
This sounds like a strawman. Not everyone wrangles multi-GB files, let alone JSON documents. Those who do are already readily aware of the implications. I mean,some popular text editors even struggle with multi-GB of plain text files.
You don't need a server to handle JSON. There is no excuse.
*I'm not justifying doing it on the server, especially for an application like this where yes: it can be done in the client.* But I do sympathize because I know from experience why it's easier to do it server-side, without any conspiracies.
I wrote Papa Parse[0] about 10 years ago, and back then at least, it was extremely difficult to stream large files in an efficient, reliable way. Web Workers make things slightly better, but there's so many issues with large-scale local compute in a browser tab.
A few examples:
- https://stackoverflow.com/questions/24708649/why-does-web-wo... (the answer actually came from Google+ which is still linked to, but no longer available; fortunately I summarized it in my post)
- https://stackoverflow.com/questions/27081858/how-can-i-make-...
You get deep enough into the weeds and eventually you realize you can make it work cross-browser if you know which browser you're using (YES, User-Agent does matter for things like this) and call you crazy for trying to find out:
- https://stackoverflow.com/questions/27084036/how-can-i-relia...
Despite all this, I *100%* agree and local-only processing is also a hard-rule for me as well. (That's why JSON-to-Go[1] does it all client-side. `go fmt` event compiles to WASM and runs in the browser!)
[0]: https://www.papaparse.com/
[1]: https://mholt.github.io/json-to-go/