Preserving biological/medical specimens with formaldehyde so that they may be re-examined later is a common practice that still occurs today. So it isn't surprising here.
The alternative would be to freeze the specimens, but that is more cumbersome than stacking formaldehyde-fixed plates on a shelf, and specimens may alter upon repeated freezing and thawing.
What's missing from the accounts is why he had those plates there in the first place.
Why was he placing plates streaked with staphylococcus near a window and then peering at them every few days?
Perhaps he was testing some other hypothesis that didn't turn out? The effect of sunlight on staphylococcal growth, maybe? Or perhaps the window is simply incidental?
He only noticed after he got back from vacation, or so the story goes. And it may not have been an open window, could have been a desk near the doorway.
This would also explain why the dish was treated with formaldehyde for preservation, and why the dish still exists today.