One of the main reason to swap branches out for a different item at the start is because there are four other items that offer slightly better stats in the long run:
Circlet: Two branches for the price of three, but it also builds into three very useful items for the laning stage (Which is two more than branches traditionally build into).
Gauntlet of Strength: If you feel you need the extra HP provided by this and the Bracer that it quickly builds into (Say, for example, if you're a support with low HP or you're up against a hard-hitter). Combines with Circlet.
Mantle of Intelligence: The Gauntlet of Strength for Intelligence. Similarly, this builds with Circlet into Null Talisman which is the intelligence hero analogue to the Bracer. This is useful for intelligence heroes who want to make an impact early by dealing out larger amounts of damage.
Slippers of Agility: The Agility equivalent to the other two items, builds with Circlet into Wraith Band (Built mainly for the same reason intelligence heroes build Null Talisman, you hit harder), or combines with itself again and a Stout Shield to create Poor Man's Shield, which is an excellent method of tanking damage early on in a lane where you might be getting hit a lot (Mainly by creeps, but also by heroes).
There is also the
Quelling Blade, which is a good buy on melee heroes (Less so on ranged) that need last hits in a lane but don't deal a large amount of damage on their own, so they need this item to help them.
Stout Shield: Has a chance to passively block damage, excellent if you have to go into a lane alone or if you're a melee hero laning against two ranged heroes who can deal out a lot of damage when you meander up for a last hit.
Ring of Protection: If your hero later has to build Tranquil Boots, buying this is very useful. Tranquil Boots' other two components (Boots of Speed and Ring of Regen) can be purchased at the side shop at the top and bottom lanes, however Ring of Protection can't, so it helps to have bought it earlier to save time waiting for the courier to bring your boots to you.
Sage's Mask: A decent amount of mana regeneration that later builds into Ring of Basilius (Free armor for allied heroes around you, and can be toggled on and off to offer armor to creeps).
Soul Ring recipe: Soul Ring lets you exchange health for mana, useful on heroes that have a healing spell (e.g. Omniknight or Bane). People buy recipe at the start because the other two component of Soul Ring can be bought at the Side Shop.
The main reason to go for other, less effective items over branches is because, after earning some money, those items quickly build into much more useful things.