Another explanation for that would be that the conditions under which life could arise were only temporary: it had to arise then, or it wouldn't arise at all. For example, possibly life arose in one of a very large number of wet planetesimals that were still warm enough (due to decay of short lived isotopes like 26Al) for the water to be liquid. Or perhaps phosphides in infalling meteorites were necessary to provide some necessary form of chemical energy.

Such a fascinating point that I did not think about, it seems this is related to competing theories what world was first; RNA world vs Protein World (may be even both).

Another possibility would be that panspermia could be possible in densely packed newly forming star clusters, like the one our solar system was born in.

These clusters can be very dense (10,000 stars per cubic parsec, perhaps). With such closely spaced stars, and with residual gas around the stars, it might be much easier for material ejected from one system to be captured in another.

So, IF life arose very early in one such system, it might spread to all the others. The statistical weight of "early OoL" events would be amplified, vs. OoL events that occurred later after the cluster had spread out and dissipated. Observers would tend to derive from these prolific spreading events, just because they'd seed so many systems.

This is a nice scenario for science fiction, since it would allow thousands of life bearing systems in our galaxy (with compatible biosystems!), while evading much of the bite of the Fermi argument. In this scenario, SETI should look for stars with compositions very similar to the Sun, spread on an arc ahead/behind our system on its orbit around the center of the galaxy (the stars would have spread to about 180 degrees along this orbit since their formation).