Mozilla has given this a lot of thought, and realized it is extremely valuable. But the value does not dwarf the downsides.
XUL extensions would block most of the Servo work, as an obvious example. You can't rely on the browser internals if those are changing underneath you.
Had they given it enough thought, they'd have pushed for replacement of XUL APIs in WebExtensions in a non-blocking way, for example. But they found killing all of the valuable add-ons of years worth of work of so many developers justified.
> they'd have pushed for replacement of XUL APIs in WebExtensions in a non-blocking way
Which "they" did, and are. Some APIs are more amenable to such replacement than others. For example, APIs that allow you to modify any part of the browser UI in an arbitrary way are not so amenable _and_ have the problem of breaking any time the internal structure of the UI changes. Which has, in the past, prevented improvements to the internal structure of the UI...
It's really easy to see only one side of this (the downside of extensions breaking, or the upside of not having your implementation severely constrained by extensions that keep depending on internal details they shouldn't depend on). But there are in fact people who've thought long and hard about both sides of this.
Except the downsides are mostly for the users (extensions stop working) while the upsides are mostly for the browser developers. (Better ability to refactor and update)
The upsides are for users as well: better performance, less memory usage, better ability to implement web standards. Browser developers aren't just breaking extensions for the fun of it, but because they're in the way of actually producing a better product; the goal is to produce said better product.
The upsides are just somewhat more diffuse than the downsides: the former apply to all users, while the latter only to those who have extensions that stop working.
The thing is, I as an end user shouldn't need to look from both sides (where one side is predominantly of the FF developers or extension developers). I speak for my continued convenience of using Firefox as a browser which had an upper hand to Chrome to Chrome due to its extensions (as far as I'm concerned) which will end soon.
As end users, many (in fact a majority) have already made the decision that the benefits of Firefox's extensions don't outweigh the drawbacks those same extensions impose on Firefox in terms of performance and responsiveness, and have voted with their feet.
I appreciate that your personal tradeoff was different from most, and I know the pain of having an application that was targeting your use cases move away from that. I don't have much to offer other than the goal that Firefox still has to offer extension APIs that are broader than the ones Chrome does. It won't be quite as "yeah, hook into anything you want" as the old extension system, but that was really becoming completely unsustainable and was holding back Firefox not only in things like performance and responsiveness but also web standards: extensions would depend on things working a certain way and changing them to follow the spec would break those extensions...
I find unfortunate to see such trivialization of dev work from outside. They had to deal with years of legacy code piling up, and this was seriously impeding the modernization of Firefox.
Had they given it enough thought, they'd have pushed for replacement of XUL APIs in WebExtensions in a non-blocking way...
They have. It is not finished yet but work is underway.
I also learnes from last(?) discussion about Firefox here on HN that there is even a developer extension that you can install on nightly that will let you create your own extension api.
This is useful 1. if you are a developer that is happy with using nightly and making/copying extensions on top of your own api 2. even more useful as you can sketch out api ideas in js without touching the Firefox C++ codebase.
BTW: Manishearth who is acrive in this discussion often shares knowledge from the inside of Mozilla. If anyone are really interested then reading his comment history might be interesting.
XUL extensions would block most of the Servo work, as an obvious example. You can't rely on the browser internals if those are changing underneath you.