The samurai summon has a bit of a weird power curve. He's tougher than average for a starting summon, and the first few skill-ups you spend on him will have quick returns in power. Once you have him to about five in every skill (around the time you hit level 7). he'll only be average compared to other summons, and by level 10 he'll be below par. I didn't keep him because the skill investments weren't worthwhile compared to increasing my mass follower limit. A lot of people also drop him to free up the 'lieutenant' slot he occupies and replace him with another henchman summon.
However, take a closer look at the skill tree. When you hit level 60+ in his skills, (which won't be until you're past level 50 yourself, mind) they start getting synergy bonuses. Actually, they have synergy bonuses from the start, but they're miniscule. Every level you put in a skill above 60, though, increases the bonus by 2%. All his skills synergise against each other, so if he's well rounded you'll see him catapult forward in real power. By the time you've got all his skills to 65, his real skill level is around 101 in each skill. This bonus keeps climbing, exponentially if you keep levelling up his skills in a rounded fashion. Since most summons will be averaging around 90 in their top skill, the balanced samurai peels ahead of them.
If you're not a summoner or breeder class (like my character), the samurai is worth sticking with. You might feel like you're carrying him at first, but I'm assured that when you tag-team a Glass Titan you'll be damn glad you kept him around.