If your phone gets stolen, then you block your phone number with the carrier, and the thief can't get your SMS - so 2-factor security still works and protects you.
True, I think most snatch and grabs are sold quick for cash and not identity theft anyway.
But in a worst case scenario how quick could someone determined actually compromise you? minutes? less time than you'd wait on hold during work hours, let alone waiting till the next day?