What you do is drop cave-in dirt to plug certain holes. If the waterlogged area extends all the way to the very absolute edge of the map, then it counts as sourced water, which means that water will flow in from off-site. However, if you plug that up, you can then drain the water with pumps and careful evaporation, and then install a floodgate (and by that I mean drawbridge) near the cave-in plug, and then channel that back out. This lets you add water whenever you need it, but gives you the option to turn it off at will.
Be warned that:
you need to understand how cave-ins work!
you need to be dropping at least 2 full, unmined z-levels of dirt into place or it won't actually plug it up (and depending on how deep the water actually goes, you might need more)
you need to plug EVERY SINGLE HOLE before you bother to start pumping
you can't dig out the very last tile of the map, so you'll need to pick spots 2-4 tiles in from the edge as your plugs.
you need to watch out for cavern wildlife, as they can and will get in your way during construction--especially aquatic wildlife. Cage traps are great as if you can get the guys caged, new people won't wander in from off-site until you butcher the caged ones, but caging cavern creatures can be haphazard at best, unless you plant cages ALL across the map edge or otherwise have a way to funnel them through a caging zone. Also, flying creatures.
If all you want is a "large, wealthy fortress" then you don't actually need to ever go into the caves. Once you've cracked them, you can seal them back up, and you'll still be getting fungus and shrubs and saplings in your soil layers. It's also entirely possible, if you just want to dig deeper, to just dig staircases through the walls--as long as they're surrounded on all 8 adjacent tiles with stone, you won't get water in your staircase. Or horrible abominations, for that matter.