Legalizing pot would definitely cut a huge hole in the black market. I'm lead to believe the majority of prisoners are on drug charges, and the majority of those are for marijuana charges, so that's maybe an effective 25-30% drop in the prison population for a start.
There's also the claims of pot as a "gateway" drug, but the only sensible reason to claim this as opposed to alcohol as the "gateway" is that both pot and other drugs are illegal and the supply networks interact, so someone (or a location) who can get you pot is also likely to be able to hook you up with other illegal drugs. If it was sold in shops, there would be no "networking" effect automatically exposing other drugs to users, no more likely than going out for a beer exposes you to crack and LSD.
Many drug dealers sell a variety of drugs, so taking out pot cuts into the profits of many sellers of other drugs, which would put a fair chunk of them out of business - lose the pot seller, you've lost a guy who also sold many other types of drugs - this guy was the classic "gateway" dude.
Also, "total ban" and "fully legalize" are NOT the only options for harder drugs, especially opiate-based drugs like heroin. There's another model of handling heroin which is very effective - the Swiss model pioneered in the 1990s. In that model, they prescribe heroin to long-term addicts. This is a control, clean supply which is consumed with supervision, and they proactively find work for the participants in the program.
It was extremely successful, the initial group were 99% homeless junkies, and 85% unemployment. After 2 years or so, everyone had a house, and their unemployment levels were around 10%. All this put the heroin trade out of business (thus making it harder for non-addicts to find as well). On top of that, there was a 40% drop in property crimes in the region, plus they estimate the government saved 7 Swiss Francs for every 1 Swiss Franc they spent.