The drawback is if you left your keys on the bench in the garage next to your car. You get in, push the button to start, then drive off. Now you can't get back home. At least with the physical key you are guaranteed to have it with you.
The other thing -- do the key fobs have an on/off switch? If not, you wouldn't be able to have a spare "hidden" under your car somewhere (yes this is insecure anyway, but it works for most people).
People bring this up as an objection all the time, and I think it always comes from people who have never actually had a car with this feature. First, as mentioned earlier, every car I've had with pushbutton start will not let you start the car unless the key is INSIDE the car. Second, if you try to drive off with the key NOT in the car, they've all made an alert sound and shown a message on the display that the key isn't in the car. It's certainly possible to intentionally drive off without the key, but I don't think it's possible to do so accidentally (or if you're so inattentive that it is possible, you probably shouldn't be driving in the first place).
Pop out the battery. You don't need the battery to use the hidden spare if you lose your key: there's a physical key hidden inside the fob to unlock the door, then a slot on the dash to stick the whole fob into to start the car. It's there so you can't be stuck unable to start your car when the fob's battery dies.
I actually got curious and have a friend hold my key outside the car while I try to start the engine, and I couldn't. The car correctly detects that the key is not inside with me.
Also, the dashboard will always tell you if you are inside the car but key is not detected. So even if it miscalculates where the fob is and let you start the engine, that key-not-present light will be on as you drive off.
The other thing -- do the key fobs have an on/off switch? If not, you wouldn't be able to have a spare "hidden" under your car somewhere (yes this is insecure anyway, but it works for most people).