An operating system which only runs code written in a safe language (without escape hatches) would be useful to enforce capability-based security at a finer level than processes. If you sandbox a program, you could choose which individual functions/classes/modules have access to the filesystem and network by choosing which ones to pass in an object with access to the filesystem. And the program would have no way other than these objects to make raw syscalls (much like you can't call fopen in browser JavaScript).
But from what I heard, Java tried SecurityManager at the language level and it failed. So I'm not sure. Also I haven't researched capability-based security deeply, so I'm not an expert in this field.
When Java was just breaking out, before 1.0, some people proposed changes for capability security, but it didn't get anywhere. Java went with stack inspection instead.
Worth noting that a language-based OS is not necessary for capabilities -- they were invented in the context of the usual hardware memory-protection in the 1960s.
But from what I heard, Java tried SecurityManager at the language level and it failed. So I'm not sure. Also I haven't researched capability-based security deeply, so I'm not an expert in this field.