I lately really started to prefer taking a closer look into who my dwarves are, focussing more on their personalities and preferences. The more dwarves come in the migrant waves, the less I can do that - and the more stressful it gets, so I highly prefer smaller numbers. Sadly, having only 20 dwarves for example limits what you can do, my military already takes up 10-15 dwarves most of the time, and when their discipline skill gets very high they will never ever do anything but train (unless I missed something?! ... or unless I disable training in the barracks for the inactive squad every month...). Not to mention until shortly ago I didn't know sieges were set to start only at 80 dwarves due to the entity raws - of course now I set it to 50, but I feel like setting it to less would just be a bit too crazy for me.
I love the depth the personalities offer, the only thing that annoyed me a bit in my last fortress was that pretty much EVERYONE was a pessismist and "never happy". I actually had one dwarf who was "never the slightest bit cheerful about anything" - best thing was to keep her away from any stress at all, even though she luckily wasn't also very vulnerable to stress; still, even the high value living quarter, dining room and constantly creating masterfully prepared food (she was one of my cooks) didn't make her very happy. Quite frustrating when you always see her go "didn't feel anything about [the most amazing thing ever]".
On the other hand, I had a "perfectionist" dwarf in my previous fortress, who I unknowingly of that made an engraver - I can tell you, I loved that "engraved a masterwork"-spam he created. Didn't pay much attention to personalities in that fortress due to having more dwarves around; this dwarf was pretty much who got me interested in it.
Overall I like the balance. Neither having too many nor too few dwarves appeals to me for a standard fortress, though I got some ideas for "more challenging" fortresses with really, really few dwarves. I also love seeing a fortress grow through births over time, starting with fewer dwarves and slowly seeing the numbers rise, looking forward to every child becoming and adult at some point (still don't know why I need so many legendary planters though...).
Too many dwarves is also a bit annoying due to the low FPS, things might get done "faster" in ingame time, but they surely don't get done faster in reallife time anymore. (Then again, normal people don't stockpile thousands of ...everything.)
I like getting more "attached" to my dwarves, even though it then pains me when they ... suffer an absolutly unexpected and unforseeable accident.
So yeah, depending on what I want my fortress to be - more focussed on the single dwarf or more focussed on the whole thing - high numbers of dwarves can be stressful. But actually, for me, not having stuff get done is even more stressful and annoying. (I just wish the game could somehow handle more dwarves more easily, but then I'd not be able to keep track of their personalities and all anymore...I really enjoy advancing through the ingame years. Smaller populations make that easier, even though the caravan/diplomat spam, having 3 entities of each species around, can get quite ... taxing. Ugh...)