You don't put solar power on farmland. You put it in the middle of the desert or on otherwise unused rooftops. You could meet more 100% of the human populations energy needs with solar power taking up a tiny percentage of the most desolate deserts on earth.
To put that in perspective: At an efficiency of 20%, you'd need the area of
Spain to power the world (which is approximately 1/40th of the uninhabited area of the Sahara). The best way to create so much surface area of solar cells is unfortunately not silicon, but
polymer solar cells, which have a projected maximum efficiency of 12% (but they're better with indirect sunlight and suffer less from cloud cover then silicon or thin-film ceramic solar cells). The big advantage of polymer solar cells is that they can be printed, much in the same way as newspapers. I can't find any figures right now, but one my professors calculated that if the production of solar cells would match the production of newspapers, it'd take less then two decades to match the world's energy demand.
Also, on the problem of storage and distribution,
There are numerous technologies being developed to turn carbon dioxide into fuel. There are also
numerous ways to store electrical energy generated at peak generation times for later use. Especially for a solar powered grid that's distributed amongst several production centers in deserts, a slight overcapacity combined with short-term storage, such as
SMES or
fly-wheel energy storage could be a real possibility.