Just a general question: Do you think there is a market for a new operating system? The cost of building something better or equal to mainstream systems seem too high to even consider challenging the status quo.
I think there is, but it'd need to be very different to existing operating systems to justify itself, and in ways that other OS's can't simply adopt for themselves. In practice that means radical architectural change, and even then you'd re-use a lot of code.
For such a project you don't really want to just adopt existing ideas and implement them. I don't really understand why Google is doing Fuschia for this reason: architecturally it's nothing special. Is the GPLd Linux so bad? It took them this far.
If I were to do a new OS I'd explore ideas like all software running on a single language-level VM with a unified compiler (e.g. a JVM), a new take on software distribution ... maybe fully P2P, rethinking filesystems and the shell, making it radically simpler to administer than Linux, etc. You'd need to pick a whole lot of fights and make a lot of controversial decisions to justify such a lot of work: you'd need to be right about things other people are wrong about, not just once but multiple times. Then maybe if people see it working well they'd join you and port existing software, albeit, the porting process would inevitably involve some rewriting or even deeper changes, as if you can just run existing software and get all the benefits you probably didn't change anything important.
Build a new operating system, or build an operating environment that runs on top of an existing operating system, like how e.g. Windows used to run on top of DOS? Or User Mode Linux on top of Linux?
An operating environment means you don't have to worry about stuff like device drivers. It can run inside a Docker container. (At most companies, try suggesting "let's run a brand-new OS that nobody has heard of", and you'll get an emphatic "no"... say "here's this app we want to run, it is packaged as a Docker container", and often nobody will even ask what is inside that Docker container, even if it contains your operating environment with the app running on top.)
You can start out using facilities of the host OS like the filesystem and networking stack. Later, if you want to, you can implement your own filesystem (create a 100GB file on the host filesystem, pretend that is a block device and your custom filesystem exists within it). Or your own networking stack (which can run in user space using facilities like Linux tun/tap drivers.)
An operating environment can always evolve into a standalone operating system at a later date.
Well, there is always a market for a new OS as they all are so complex and have so many baked in assumptions, that they cannot possibly do well on the variety of use cases that people need computing for.
