There is a very full worldwide calendar of lower-tier events that take place in the other 3.75 out of four years when the Olympics aren’t happening, some of which would have to be canceled to accommodate. There’s also the issue of clearing all of the massive hotel room blocks needed, and re-working the schedules of all of the Olympic venues.

It could probably be done, but it will be very painful.