Here are the rules I use on a regular basis (unless I'm trying to do something specific) in no particular order:
1) No walling yourself off, must have at least a 3 tile wide entrance to your base.
2) No using drawbridges (just normal bridges, no pulling them up), the only things that can be used as doors are actual doors and floodgates.
3) No magma for anything, including smelting. If you want to use a forge/oven you'll need to start chopping wood and making charcoal.. Or trading..
4) No danger rooms
5) No robbing the caravan
6) No traps (save perhaps cage traps for hunting purposes, not for defense), your only defense can be soldiers
7) No using steel (or only allow steel weapons or armor, but not both)
No using caverns (removes easy-ish sources of fresh water and materials)
9) Set starting metal to low.
Other than that some other ideas are mods which can add a lot of challenge, for example I always play with Fortress Defense II mod which adds a whole mess of enemies to attack your fort.
Another good idea is to pick some specific goal to accomplish that requires you learn something new. Like breach an aquifer, or set up an artificial waterfall, or keep a functional fort without using farms for food (only booze), or breeding animals, or whatever else. Not only can the goal be challenging in a new way but you'll come away knowing how to do something new.
Of course picking a difficult area is a good general challenge, a freezing or desert biome is a good start (though caverns have made both a lot easier than they were, for additional challenge don't allow yourself to use caverns).
Honestly, once you get over the learning curve, DF is an easy game. Unless you deliberately handicap yourself you can acquire near infinite food and supplies with minimal effort, you can arm your dwarves well with metal armor and weapons, and you can just about completely defend your fort from nearly anything with a good mix of traps, walls, and some soldiers for the few creatures that get through. Add to that the exploits that exist such as robbing caravans blind with little if any drawback and of course danger rooms and you can be just about invincible (short of succumbing to the dreaded FPS curse).
Think of the above as playing the game on 'easy', you give yourself all the advantages and take no drawbacks. No shame in it, and depending on what you are doing it may make it more fun, but if you find yourself feeling unchallenged simply start to give yourself some limitations to what you can do or what you can start with, pick more difficult starting locations, or set up some ground rules on what you are allowed to do in-game. You can turn 'easy' into 'impossible' pretty quickly..
As mentioned I play with all those above rules on plus the Fort Defense Mod. Its extremely difficult, I cannot wall myself off and cannot use traps, when the various sieges show up all I can do is respond with my soldiers. That said, for me, that is far more fun. I love trying to train up soldiers, trying to hammer out some weapons, and trying different strategies for defense.