Terminology wise, I think of Serfdom as a system which has the protection of a social contract. Land to rent and farm, in exchange for taxes, labour, the draft, or any combination of the above.
A manor might be able to support a few hundred Serf families... but there would be thousands of labourers -daytalers, farmhands and the like who don't even have the protection of a Serfdom arrangement. More often the Proletariat are the ones shouldering all the real work. And their numbers were easily in the millions, in a country like Britain in the Early Middle ages of around 700 AD.
Comparatively, a warband of 80 warriors during that time was considered an army. If the Fyrd is raised, that would be your serfs with feudal obligations, which add a few hundreds to thousands more men. The folks with no feudal obligations to stand in the battle line are those people doing the actual work - the day labourers and the like who mostly rely on the Serf class for their employment and food. (And it might be fair to say that the proles also didn't give a rats arse what happened to the politics, unless they really loved their masters)
In terms of raw wealth... a Villeinage (In the doomsday book system) is about 80 acres - the size of some commercial farms today. A half Villeinage, 40 acres. To qualify as a Yeoman, one needed at least 160 acres or equivalent wealth, privately owned and not held in fief to the local lord.
In terms of feudal obligations: the richer you are, the more well equipped you are expected to be when you turn up for duty.
Those of the serf class who were destitute would be the Cottars who could have anything from 5-20 Acres, which earns a subsistence. They were more prone to paying their taxes and rents in labour. But at least they had land and a more or less secure way to eat. Unlike the Proles, who were truly miserable.
How we get proles... the same way we get commoners. The parents of Serfs can't keep dividing up land for their children. Second sons and the like get disenfranchised among the lower classes, just as they do in the nobility.
A serf might live like a petty manager compared to the people who work for the serf, which are the ones living in hovels. But the serf has to maintain arms and profits for his Lord.
Unlike what mount & blade might depict, you wouldn't have serfs running away from serfdom. That's probably a confusion of terms, just like how the status of villain is looked down upon. There would be plenty of people who would want to take a serf's place. But maybe a serf might have indentured labourers who feel tempted to run, and the serf might pay some silver to a passing mercenary to catch them.
A serf isn't "free" - but what that meant socially is that he had obligations and duties to his liege Lord. But by extension we can also say that the little Lords aren't "free" either. Both Serfs and Lords were tied to their lands and overlords. A Prole is free. A prole has no liege and owes no one any loyalty. But a prole existed in the same kind of wage slavery that workers today face, if he wasn't outright indentured. And to the Doomsday book of William the Conqueror, they aren't even worth counting as people. (There's too many of em!)
((sorry about the obligatory historical terminology sperg))