We're already working people to death for the heavy metals we're using now. You don't want to get into solar being dominant. Natural gas is probably cleaner, anyway, after cleaning up from solar panel production. Not to mention that the sun only shines on about half of the earth at a time, so you'd need worldwide transmission and production. Solar is the "look we're doing something" of power production. At least argue for wind.
I'm not sure that mining is the key thing to argue about against solar. There are a few grams of lead and cadmium in some types of solar panels, but they last 25 years. Smartphones are more of a concern there. We currently pull out 67,000 metric tons of uranium per year. So, are they treating the uranium miners particularly well? Also you bring up wind, but people say the same thing about rare metals in wind.
https://www.instituteforenergyresearch.org/renewable/wind/big-winds-dirty-little-secret-rare-earth-minerals/Estimates of the exact amount of rare earth minerals in wind turbines vary, but in any case the numbers are staggering. According to the Bulletin of Atomic Sciences, a 2 megawatt (MW) wind turbine contains about 800 pounds of neodymium and 130 pounds of dysprosium. The MIT study cited above estimates that a 2 MW wind turbine contains about 752 pounds of rare earth minerals.
As for solar:
https://en.wikipedia.org/wiki/Cadmium_telluride_photovoltaics#MaterialsCadmium (Cd), a toxic heavy metal considered a hazardous substance, is a waste byproduct of mining, smelting and refining sulfidic ores of zinc during zinc refining, and therefore its production does not depend on PV market demand.
We're using it because it's a waste product from other mining we were doing already. This is what makes the cadmium cells cheaper than silicon-based cells currently. But the cadmium is only a concern in cadmium-based cells, not all solar cells. Also:
Tellurium (Te) production and reserves estimates are subject to uncertainty and vary considerably. Tellurium is a rare, mildly toxic metalloid that is primarily used as a machining additive to steel. Te is almost exclusively obtained as a by-product of copper refining, with smaller amounts from lead and gold production. Only a small amount, estimated to be about 800 metric tons[53] per year, is available.
So, this is something we have a strict cap on how much we can use. It limits how many cadmium-telluride based solar panels we can make per year. So, scaling up solar panel production beyond that limit won't increase the amount of cadmium people are exposed to, nor put more strain on miners of rare metals, since further increases in solar production above these limits will be using other technologies such as traditional silicon based cells. Thus it would be factually incorrect to just scale up the current cadmium / tellurium usage to the whole industry as it grows. We're using those cells because they're currently the cheapest because they use what are industrial waste products from other types of production.
EDIT: also, for the "nuclear buys us 20 more years" argument, with current technology. That's wrong. Nuclear plants create CO2 emissions when you're building them, but less emissions when they're running. They have to run for long enough to make up for the pollution released during construction, which includes from concrete production and from the vehicles needed to mine, move, build everything.