1: You swing your blade. At this point you are imparting kinetic energy into your strike. When using a heavy blade, you build up momentum during the swing, adding force by continued effort and essentially storing energy into the blade itself. With a wiffle bat, you end up with almost exactly as much energy as your arm produces. The weight of the wiffle can't really sustain any energy in itself.
2: The strike connects, and the energy is imparted. At this point your sharpness matters, as that determines how deep your blade will bite through the armor. Most kinetic energy will be delivered here.
3: The strike must carry through to do any damage to the person in the armor. To do so, you don't need a sharp blade, you need more energy. The blade edge has already passed through, and while it's still cutting the edges of the armor, the majority of it has pushed through the armor and is rubbing against the person inside. Due to adamantine's sharpness, we can temporarily ignore the flesh. The issue now, is that your axe has become a wedge. A wedge relies on pure force to literally shove something open. Adamantine has no weight to speak of, it's terrible at storing force, and when it gets to the "wedging" part, it would be rather poor.
It can be assumed that an adamantine battle axe looks more like a large record - perfectly flat along the blade with just a slim edge. It wouldn't be made to open armor, it would be made to slice through it, more akin to a rotary saw. A piece of goblin equipment would probably resemble a
CD stand with many thin grooves in it after a a battle.
And yes, you could solve all this by using a lead core, but currently DF utilizes one material per item, so we're assuming pure candy.