An OS would have a very hard time determining whether a page is "unused" or not. Normal GCs have to know at least which fields of a data structure contain pointers so it can find unreachable objects. To an OS, all memory is just opaque bytes, and it would have no way to know if any given 8 bytes is a pointer to a page or a 64-bit integer that happens to have the same value. This is pretty much why C/C++ don't have garbage collectors currently.
> To an OS, all memory is just opaque bytes, and it would have no way to know if any given 8 bytes is a pointer to a page or a 64-bit integer that happens to have the same value.
This is like saying to an OS all file descriptors are just integers.
I doubt GC would work on file descriptors either. How could an OS tell when scanning through memory if every 4 bytes is a file descriptor it must keep alive, or an integer that just happens to have the same value?
Not to mention that file descriptors (and pointers!) may not be stored by value. A program might have a set of fds and only store the first one, since it has some way to calculate the others, eg by adding one.
A gargbage collector need not be conservative. Interestingly linux (and most posix compliant unices I guess) implements, as last resort, an actual tracing file descriptor garbage collector to track the lifetime of file descriptors: as they can be shared across processes via unix sockets (potentially recursively), arbitrary cycles can be created and reference counting is not enough.
The OS already does that, though? Your program requests some number of pages of virtual memory, and the OS uses a GC-like mechanism to allocate physical memory to those virtual pages on demand, wiping and reusing it soon after the virtual pages are unmapped.
It's just that programs tend to want to manage objects with sub-page granularity (as well as on separate threads in parallel), and at that level there are infinitely many possible access patterns and reachability criteria that a GC might want to optimize for.
AFAIK, no OS uses a "GC-like mechanism" to handle page allocation.
When a process requests additional pages be added to its address space, they remain in that address space until the process explicitly releases them or the process exits. At that time they go back on the free list to be re-used.
GC implies "finding" unused stuff among something other than a free list.
I was mainly thinking of the zeroing strategy: when a page is freed from one process, it generally has to be zeroed before being handed to another process. It looks like Linux does this as lazily as possible, but some of the BSDs allegedly use idle cycles to zero pages. So I'd consider that a form of GC to reclaim dirty pages, though I'll concede that it isn't as common as I thought.
It is easy to understand how it has grown historically, but the fact that every process still manages its own memory is a little absurd.
If your program __wants__ to manage its own memory, then that is simple: allocate a large (gc'd) blob of memory and run an allocator in it.
The problem is that the current view has it backwards.