Because they are not only protectors and guardians of the forest; they are also part of the natural ecological system.
Trees die naturally. Trees fall. Twigs and other items gather up in huge amounts. The elves do not spoil anything, they see to it that fallen beasts are consumed and fallen trees are put to good use. Yet they don't disrupt the balance of other scavangers in the ecosystem because they are so gentle and noble in their work of the forest. The life of an elf is spent tipping on this thin, proper line of balance of harmony.
Which sounds all well and good, until you recognize that even sentient creatures have a place in the cycle of life and death.
Honestly, after watching Avatar, I could not get over how similar the Pandorans were to the stereotypical "wood" elf that we see in DF. (Including the crying over having to kill wild animals that were attacking people.)
Anyway, I'd say that trees are not like animals.
Consider naval oranges: They are oranges that, thanks to a mutation, do not grow seeds. The only way to make more naval orange trees is to cut off a limb, plant it in the ground, and wait for it to turn into a new tree. Naval orange trees do not breed in any way other than by "cloning" them through snippets of previous naval orange trees. All naval orange trees are just clones of the original mutant, and have never changed (outside of potential mutations that occured within a living tree) since the first one was discovered.
Now then, as for how elves use this, consider this:
However, if elf-towns just spawned unbroken wooden houses with, I dunno, maybe some bushes (maybe trees or wooden protrusions for branches) hanging off into space for leaves, you could say they're not natural trees, and shaped that way the same way elves can somehow reach into a tree and harmlessly pull out wood from it.
EDIT: Sort-of like this: http://www.popsci.com/arbona/article/2006-11/grow-your-second-home
Basically, elves just manipulate trees into growing whatever they want, and then pull it out of the tree in a way that "does not hurt the tree". The "not hurting it" part seems initially impossible, but the only living part of a tree is the outer ring, the trunk is basically just dead wood. If you could prevent the growth of the outer ring in a section of the tree trunk after it has grown to the shape you want, you would not be harming the tree while exposing a dead piece of it that you could remove...