I've read of someone who did testing in arena mode and found that half a dozen rabbits can consistently kill an elephant. I don't remember who it was, but he/she sounded like they had actually done the test.
On the other hand, I've also read Girlinhat (whom I consider to be a reliable reporter) reporting about having a single legendary soldier take out entire sieges singlehandedly. Come to think of it, ThatAussieGuy also had one or two legendaries in his checkerboard fort who also took out sieges singlehandedly.
So numbers and skills seem to both matter. I suspect in the legendary vs. siege case, the single fighter was able to move about and only engage one or two goblins at a time, since sieges usually appear fairly spread out around the map and seldom arrive at the front door en masse. A skilled fighter with an axe or sword can neutralize an opponent in one or two swings, and keep from getting ganged up on. While the bunnies vs. Jumbo case probably had them all come together at once so the elephant had all the rabbits on him at once.
I remember one fight where numbers beat skill. I had a squad of mounted seigers stuck at the edge of my map because their squad leader was on a flying mount. I sent two legendaries out to get rid of them. One headed straight there, but the second decided a wandering wolf was a grater threat to the fortress and started chasing it. The first guy engaged the goblins, and immediately got knocked down by a voracious cave crawler. Then several goblins would take swings, which he would dodge/block/deflect, then he'd struggle back to his feet, only to be immediately knocked down by the crawler again, repeat ad nauseum. In the mean time, Urist McBadPriorities killed the wolf and started over to help, but then saw another wolf and started after it. This went on for several minutes (and wolves) with the poor dwarf facing the goblins never getting a chance to even take a swing. Eventually one of the goblins got a lucky hit and did damage, which slowed our hero down enough for other hits to start landing, and things went downhill quickly from there. By the time Urist McBadPriorities finally eliminated the wolf threat and got over there, Urist McHero was dead and another squad of dwarves had arrived to deal with the goblins.
The key here was that Urist McHero engaged a group that was bunched together and could all attack him at once, plus the large mounts were able to knock him down every time they tried. He survived a long time solely due to his skills, but was so overwhelmed defending that he was never able to do any attacking.