Ah, the bronze colossi. The loss of my last fort.
They have this problem in that they never get tired, never feel pain, and, even though their eyes seem to get poked out all the time, blindness doesn't even slightly alter their lethality.
These features, combined with their size difference, damage blocking, and trap avoiding means that only a few very lucky shots will actually land damage on them -- or shots with incredibly crafted weapons. Meanwhile, they can lay down many dwarves in a single blow.
Unprepared, at least as I was, the best defense against them was a sheer mob. Although a colossus can strike down a single dwarf if he gets his hands free to do so, enough random wrestlers can normally make it so he only kills about a dwarf or dog every day or two. My suggestion is you assign war dogs to every newly enlisted peasant militiaman as he scurries towards his doom. The dogs will die, but the sheer mass of them will normally save dwarven lives.
I believe that the truly best defense against a colossus is a raised drawbridge and a ballista behind some fortifications.
On a last odd note, I found that the colossus, even though they are able to open doors, will often choose to beat down doors that aren't even locked. I guess beating doors is just a good time for your average shiny golem.