The caves were easily my favorite part of the Avernum/Exile series. There was always an ever present feeling that you were delving deep into ancient mysteries, and seeing sights that mortal eyes hadn't seen for millenia. Uncovering ancient horrors was pretty much par for the course.
I own all of the Avernum games, easily one of the best purchases I ever made. I've played most of the Exile games in trial mode, but I prefer the Avernum series.
I'd probably rank the series 2, 3, 1, 6, 5, 4, in order of most to least favorite.
The 'Geneforge-ness' of the second trilogy puts me off a little, as well, and there's a reason that 4 is at the bottom of my list. But, 5 is quite decent; you have a whole new network of caves to explore, and more deep, ancient horrors makes me a happy guy. 6 was an excellent game in its own right, easily worth the money. It isn't Avernum 2 good, but Jeff's excellent writing and storytelling really shine in 6.
FWIW, my biggest pet peeve about the second trilogy's engine is the lack of world map. The fact that the Great Cave seemingly measures a few thousand feet wide just trivializes much of the series wonder for me. I've always enjoyed imagining the massive size of the Great Cave and the Eastern Gallery, and the second trilogy, especially Avernum 4, made it look more like a few holes someone's backyard. Nevermind the fact that there, seemingly, are only a few hundred people living in all of Avernum, and 95% of them are only there for you to kill.
I don't know how many people here know this, but Jeff released the Blades of Exile source a few years ago. You can download full versions from the
Google Code page, and there are lots of fun downloaded adventures on the
forum.