Right now there's no gene crossover at all. Someone in another thread confirmed that traits are only inherited from the mother. Genetic variation seems to be crudely simulated by adding some randomness to numerical traits of the mother for her children. This of course wouldn't work for things like hair color, which aren't a numerical slider. The population model, single-chromosome mother-only genetics, and model of spread from one site to another all exacerbate the issue of genetic bottleneck.
I don't like the idea of spawning different castes for every combination of gender, hair and eye color. This could easily be done programmatically however, and not by hand. But there might be a better way, which involves far fewer castes, and is much less random, while still providing for diversity. You can take advantage of the fact that children will inherit the eye and hair color of their parents only if it's not over-ridden by the caste system. So a "normal" caste child should inherit a caste-specific coloration if their mother has it.
So, what could be done to inject just enough variation into the gene pool is to make extra female-only castes: one extra for each hair color, and one extra for each eye color. These children should inherit the other (hair or eye) color from their own mothers, and will pass on both hair and eye color onto their own children. So, if you have 20 eye colors and 20 hair colors, you can get away with creating 40 extra female castes rather than 2 (genders) * 20 * 20 = 800 new castes. I'd make the new castes have low pop ratios compared to the normal castes. e.g. if male pop_ratio = 100, then normal female pop_ratio would = 60, and each of the ~40 (assuming 20 of each eye and hair color) would have pop_ratio=1
these rarer castes would act to inject diversity into the ongoing gene pool, without having to totally randomize things at every generation, and without needing hundreds of extra castes: just one per hair color and one per eye color.