The TLB flush is the only part of the context switch that's really slow, and it's the part that's needed for a security boundary.