Nah, its a couple KB per connection. the biggest consumer would be the tcp socket control structs and associated data buffers. Ball park 1.5KB for the structs and another 4-16KB for tcp buffers on a typical internet tcp connection.
I think youre talking about timewait states et al. On linux thats dominated by the MSL, which is a compile time constant of 60 seconds. You mentioned sysctls, those are primarily tcp_tw_reuse and tcp_tw_recycle (which is the worlds worst sysctl). Regardless, its a couple KB per connection. How many hundred thousand do you want to support?