So we know (duh) that the XX or XY chromosomes make the biggest difference in testosterone exposure, much bigger than any observed environmental factors. Yes they're reactive, but they're also highly correlated with gender. Otherwise you'd see a lot more people born with the wrong, or intermediate, genitals. That's pretty damn rare however, which shows that accidental crossover from male/female levels of testosterone are equally rare. Testosterone levels are strongly clustered by gender, therefore any variation that they cause is also strongly clustered by gender.
The most likely answer is that testosterone is just a signal, and the things it turns on and off are the evolved traits. e.g. penises and vaginas are fairly arbitrary and complex organs. Testosterone does a good job of signaling which set to create (the vast majority of the time). Clearly genitals are complex and evolved systems, and there's nothing inherent in testosterone that makes them that way.
So we have other things that are evolved traits too, and some of those have different settings for different testosterone levels as well. Evolution can and would change how much those traits are receptive to testosterone. I mean, sexual attraction is a pretty arbitrary and complex trait, that is clearly evolved (otherwise how would you recognize a mate), and which one you are attracted to seems to correlate to testosterone exposure, which is a fairly arbitrary chemical. So the evolution happens in the systems which interact with testosterone. Basically there's nothing inherently special about the molecule that makes it do all those different things we know about: they're things for which testosterone receptors were evolved. And evolution has pretty free reign for what "low testosterone" means compared to "high testosterone": e.g. the dolls vs trucks thing. It's a completely arbitrary choice that low-T means liking dolls/babies.