Even then, if we're talking panspermia we're talking something with an astronomically (ha!) low chance of happening if it's from outside the solar system. There'd need to be some seriously insane number of coincidences.
1) A rock hits a life bearing planet. Not unlikely, it's happened a fair bit on Earth.
2) The rock needs to eject material with enough force to escape not only the planet's gravity well, but the star's orbit or, failing that, get ejected without smacking into a planet.
3) The rock needs to have been life-bearing
4) The life needs to have survived several millennia at an absolute minimum while exposed to an environment that, while not exactly like the vacuum of space, it still hostile due to e.g. the lack of a magnetosphere
5) The rock needs to have been on an intercept course with the solar system and subsequently Earth, even if it was a number of gravity assists later (Which would add however many more years to the voyage)
6) It needs to be at a stage where the Earth is habitable. Given just how early life appeared on Earth it'd need to be very soon after the crust solidifies.
7) The organisms need to survive the impact
The conditions need to be survivable for the organisms
It's just... It's not going to happen in a realistic timescale of the universe. It's a nice idea and maybe even plausible in-system. We've got chunks of Mars on Earth after all, but it's like trying to use a molecule to not only hit a molecule, but a specific neutron in the nucleus while you're wearing beer goggles and you've got nothing better than a spray-n-pray method.