If you give your dwarves the Hunting labor and they will wear leather armor as civilians all the time. Boots, Helm, Armor and Leggings. They will also equip sheilds or bucklers. You can also equip and use weapons you assign them as civilian hunters. The downside is that hunting-enabled dwarves will go out and hunt. They will also have other odd side effects, like they will sleep wherever they are standing when they are tired and will never use their beds again. So expect to see Urists taking naps in their workshops, ect.
If you enable hunting for everyone, make sure to build a couple extra butcher shops and tanneries and enable the tanning and butchery professions with all these dwarves. If you have a fortress of hunting dwarves you will be drowning in a sea of dead animals and you need to be prepared to process them before they rot.
Also, making normal clothing out of leather doesn't effect its effectiveness as armor at all. Silk, cloth, leather all have a material rating of 50% So save your leather for making leather Armor instead. Getting your dwarves to wear clothing takes time also since they won't replace their clothing untill its rotted off them.
Also, if its only carp killing your dudes, armor isn't going to help. Carp kill by "dragging" the dwarf into the water where they drown. Well, actually its the Dwarf leaping into the water to tackle and wrestle the carp to death. Best solution to a carp problem is a big squad of marksdwarves.