Hm, I have about 5 villages and 1 castle, spread throughout Greater Vaegir Land (including Shapeshit as flipping always, heh). I think I'd much prefer to have a city, ideally with one castle and a few of the surrounding towns.
I'm pretty sure I have Diplomacy as part of this Freelancer mod. Any particular advice for consolidating? I do get the dialogue option to suggest trading holdings, but I'm sure it's hard to trade for a town.
I don't think you'll ever be able to trade up any holdings. It's only for trading same level holdings (castle for castle, village for village, etc), though you may also be able to trade down.
Where you go from there really depends on what you're looking to do. Starting your own faction is always fun, but with only one castle it wouldn't be easy (IIRC you'll lose all your villages if you break with your liege, except maybe if you own a village as well as its corresponding castle). You could also stay with your liege, build relations with him, and conquer loads of land for yourself, hoping he gives it to you. Getting at least one town is really useful, since they give insane amounts of money compared to other fiefs.
Some things to consider would be putting some money into productive enterprises for an independent stream of income, and working on building your right to rule and honor/dishonor up. If you want to start your own faction, you'll also need to work on getting a large army together in preparation (and the money to support it).