A big part of it is that the US commanders in charge of the original Afghanistan operation (all the way up to Rumsfeld) took the wrong lessons from Germany and (especially) Japan. In both those nations, it really was as simple as "crush their military, then sit on them for a few years while you get a democratic government going". This lead US leadership to think they could easily do the same in Afghanistan, especially since the Afgani in the streets wasn't an enemy of the US in the first place.
The fundamental problem there was that both Germany and Japan had centralized governments and had some pre-WWII democracy attempts. Installing democratic governments to replace the fascist ones became largely a matter of keeping things quiet while you got the people on-board with the program.
Afghanistan didn't have that. The government system before the Soviets was fundamentally a tribal one, and they've never had real centralization of power. To accomplish the intended goals, the US occupation (yes, other countries were involved, but it was more a US operation than anybody else) would have had to construct a centralized government structure ex nihilo before any democratic government could take control. Instead, the US tried to simply restore the old mujahedeen factions to power, leading to a corrupt and ineffective government and military.
Afghanistan would have turned out better if it had been an old-school imperialist landgrab ending in annexation. Any case where this is a true statement means you have one monstrous clusterfuck on your hands.