Bottom line, you don't want anyone with an "important" or "urgent" job, or a localized job, to do hauling. Hauling can take a long time, and/or can take them far from where they started, often many times in a row, delaying that key job. No sense for a Legendary Mason to waste time doing a job that any peasant can do.
So, any dwarf with an important "production" job never hauls in my fort. This usually includes my masons, mechanic, and carpenter, at least (who "always" seem to be busy producing one thing or another). Also any metal smiths (armour/weapon, but also black/crafters) who are currently producing. And miners - they're busy elsewhere, no time to walk back to where hauling happens.
If you have a peasant-type with a key job or job combo, he may be turned off too - my Brewer/Herbalist has only enough time to do one or the other, no time for hauling. My Farmer(s) - depends how many I have; if only 1, then he's full-time, but if I get some high-Grower-level migrants, they might also do Food hauling (only).
Similarly, if it's time to milk/make cheese, or gather cave spider silk and spin it, or cook meals, or make soap (if the hospital is running low), or mill dye clothing, or similar "now is the time" jobs, then those specific peasants get all hauling turned off, and they concentrate on getting their job(s) done NOW, tyvm.
Anyone and everyone else, including any of the above that is NOT time-critical, have most hauling enabled all the time - a few non-critical soap jobs via the Manager will get done when they get done - if it's no rush, it's no problem.
That still works out to be well over 50% - maybe 60-70%?
The only "specialized" haulers I have are for Stone and Animal hauling. For these, I go into DFHack and look at the Attributes of my peasants* - for Animals, I want only higher Agility for speed to go find them asap, and for hauling (often quite heavy) Cages they also need high Strength - or, at least, not "low" in either. Similarly for stones (that occasionally do not have wheelbarrows) I want high Strength and and again "not low" Agility, as they are often walking loooong distances down z-levels.
(* I give all my "peasants" a nickname, preceded by the lone letter "p", so I can group them alphabetically in DFHack. True no-skill peasants get "p" (possibly w/ some other info), while my Master Threshers get a "P" w/ their skill(s) in their nickname. Works for me, even if same skill often has same "first name". ymmv.)
Almost everyone has Lever Pulling (except Miners), and most peasants have Refuse hauling for QSP duty. No one ever has Wood Hauling (that's QSP only). Burial, Trade Depot and Water Hauling are individually selected only when needed.
I'll also have "assistant" Masons and/or Mechanics, to build bridges and make blocks and load traps, so my main dwarves can concentrate on their workshops.