Channeling the newbie in me would say:
1) The UI: Its ridiculous, don't get me wrong once you get used to it you can speed through things quickly with hot keys but its a lot to get used to in an already complex game.
2) The graphics: These days I only use ASCII (or rather a tileset that is ASCII but squares it) but when I started it was like looking at a garble of random symbols and letters. Considering the already mentioned complex UI and the complexity of the game itself (didn't get farming right in your first year.. You probably died) it was hard to deal with.
My current things that bug me:
1) FPS of course: Granted these days I play the game on what may as well be considered 'hard mode'* and frequently die before FPS ever becomes a significant issue (plus I keep my base-FPS locked at 60, I Find that makes the game flow better for me and thus I won't notice any change in FPS until it slows enough to drop below that) however knowing that trying to make a long lasting fort is a lost cause does suck.
2) As others mentioned, the lists. While I've gotten over the UI the lists of things can be a pain to sort through. Names get cut off if they are too long, they are often not in any particular order, you cannot sort them or filter them or search them (with a few exceptions like in the managers table). There is so much I'd change with that (it would be nice, for example, if you could have a 'favorites' section where you could select specific things to be able to view quickly, like "drinks, food, wood, metal" or whatever so I don't have to check all the different sections when I want to know how much stock I have. That said alphabetizing the lists or better yet allowing you to filter or sort them would be good too, kind of like you can in the manager window (again).
3) More minor but not being able to choose specific materials to make certain things out of, like making a door or a floodgate out of a specific material without having to jump through hoops. You can do this with some things (like armor/weapons) I just wish you could do it with everything. I know its in the works though so that's good.
I am aware that the "FPS" issue runs deep, but to me (knowing nothing about programming) it wouldn't be that hard to take care of the other two and honestly that really were the only two notable things I could come up with. None of the above really put me off the game, but if I could pick two (realistic) things to change it would be the latter two.