You won't hit the worst case, but the common case is that threads will burst stack usage during task execution. If you have 100k threads your effective stack usage will be the high watermark.

It varies by workload, but for mine that is between 128 and 512 kilobytes per thread.

If you churn all your threads no problem. If you are hosting 100k persistent connections I suspect it would detract from available memory.

That's interesting. So if a thread briefly uses 1M stack, and relinquishes most of it, then the kernel still keeps that unused 1M memory in the page table?

I expected the linux kernel to be more intelligent, but maybe there's a reason it behaves like that?

Under pressure, it will eventually page the memory back out, but generally you never want to get your machine into a state where paging happens. Aside from that, there is no mechanism that allows the kernel to know the memory is no longer used unless you explicitly tell it, e.g. via madvise(2).

Though I can't think of a sane way to call madvise(2) involving a thread stack

That makes sense, thanks for clarifying. So if say there are 1000 client connection threads and they monitor some status and say ping their clients every 5 seconds, on wake-up, it could be only a small fraction of the stack would have to be paged-in. And it would depend how deep the stack was when the thread of put to sleep?

Sorry yep, didn't see your reply earlier.

That's not to say threads aren't expensive, they still require several heavyweight struct allocations on the kernel side, e.g. struct task_struct, which I counted to 1k before getting bored (and wasn't even quarter way through the fields)

Edit: Linux git HEAD with Debian unstable .config:

(gdb) print sizeof(struct task_struct)

$1 = 1904