Caves are the only natural way to enter into the cavern system, and the default for map generation is set to put most of them on mountain terrain, though some non-mountain caves can be found.
So, open the "Q"uest screen for the map and move the cursor to what looks like a rough stone. If you see "Cave" in the desciption, head to that location, > to get out of travel mode, and walk up to it. You should get a message saying you found a cave entrance, at which point climb on top of the mound of stone and work your way down level by level looking at the entire pile. Eventually you'll find the way in and a series of downward ramps will take you into the cavern system.
Of course Kobolds are programmed to make caves their home, so unless you want to kill the entire kobold population, you'll probably have to find a different cave to use as your entrance.
As well, the only way to move between cavern levels in a non-lethal non-"flight required" manner is to find "downward passages" which are basically stupidly difficult to find given that the underground has no map or travel system, and their number is probably limited per world map tile; and the game will not remember the path you took from the original cave entrance to where you currently are or where you traveled... So basically, if you enter the cave system to do some serious exploring, you WILL get lost and NEVER FIND YOU WAY OUT, EVER.
Caves are where adventurers go to die. No exceptions.
Only enter a cave if you're as prepared as you feel you can be; this includes having the swimming skill at "adequate" because there will be water you need to cross, and things in the water that will cause you to drown when they attack you, as having below adequate swimming skill causes you to "panic" when attacked in water and never recover from that panic, and thus drown.