The general reason is that though there is no standard man, with a particular level of biological activity across various markers, they do tend to be consistent within themselves so confounding factors like testosterone levels can be evened out without too much trouble if they're interacting. OTOH, female-specific biology virtually runs on ever-varying levels of hormones, to put it crudely. If it's not something specifically meant to work within/upon that system, it's so much easier to not try to rule out the changing effects of the cycle from the changing effects of the candidate drug. Though this obviously leads to under-testing oestragen/lack-of-androgen interactions, clinical interactions with standard contraceptive medications, different ranges of BMI, etc.
(It is not helped that far more men tend to put themselves forward for Phase 1 than women. Or so it used to be, for several easily explainable reasons, and it probably has not changed. So, even if curating your random pool of volunteers to get things as balanced as you want, you're going to get an imbalance. Especially when you know you need to assign a disproportionately greater number of the relevent subset to the female-only cohort for those occasions that this is the entire point...)
Plus (post-thalidomide, especially, though that was a different type of failure) precautionarily stating "not to be taken during pregnancy/possibly pregnancy" all the way to the consumer-packaging level, without necessarily having any reason to believe it's an issue.
It's a bit of a mess, but born of over-caution rather than oherwise. And I'd never suggest under-caution, but then I'm not in that business at the moment (and never was anywhere near where I'd influence these things) so you don't need to rely on me to circle that particular square.