If you've got a garbage dump zone (press i to make a zone, g when done) next to a tile that is either a ramp or open space (a channeled hole in the ground, for example) the dwarf will throw the dumped object into that tile from the dump zone.
Combine this with a tile below filled with magma, and your dumped items (that aren't magma-safe) will vanish very quickly.
In addition, you can fix the refuse stockpiling by pressing o then r. Here you can switch if the refuse items will be saved (moved to refuse stockpile) or dumped. For example, I dump everything except shells and bones (note that if Other is selected for dumping, then your cagetraps will be dumped if they catch anything). This means that I can effectively have my refuse stockpile indoors (and avoid losing things over the seasons!), since everything hazardous gets dumped instead (into magma, in my case).
One of the biggest problems is waste management after a siege. There will literally be up to a thousand items - body parts, weapons, leather armor the goblins wore, and if you are unfortunate dwarf corpses. To quickly handle this problem, I have made sure that my dump zone (into magma) is close to the battlefield where I choose to engage my enemy. After a battle, when the dwarf corpses have been carried away to their coffins, I press d-b-c and claim the whole battlefield. Then I do d-b-d and select the whole area for dumping. Then I do d-b-m and select the whole area for melting. This means that all the useless stuff will get dumped, and all the metal items will get melted (at a forge designated to melt objects). Be careful though, so that you don't melt items dropped by your dead dwarves. These items are already claimed, so if you want to avoid melting your own stuff, you can just wait for your haulers to clear those things away before you select the battlefield for dumping/melting.
Bonus tip: A hunter is the best way to get an early food supply. But it produces a lot of refuse. Make sure that you have a dump zone nearby, and your haulers will quickly clear the butcher's shop. Meat or edible parts from a butcher shop must be moved to a food stockpile quickly, or they will rot!