I retired the adventurer, then started a new one to get the time to progress two weeks, retired them and re-exported the legends data for legendsviewer. As it is now the population of the cave is a bunch of bats and cave swallows, 1 kobold, 2 gremlins outcasts, and 1 orc outcast. So as far as legends goes I'm considered a bandit harassing the local site population of 1 kobold. I guess the other two gremlins left because there was a huge bandit in full iron armor harassing them. Immediately after typing this I moved my adventurer slightly and found a kobold sitting on top of a tree. I tried talking, but since kobolds don't talk I just claimed the site and cut his head off. I'm still not counted as the leader of the site though, so I'll continue my investigations.
UPDATE:I finally decided to abandon the idea of ruling the cave, it seems like it's probably impossible. Instead I've decided to conquer a city. First though, I need to make these two gremlins a real army. Unfortunately while I was investigating I got attacked by a giant bat and lost one of my gremlins. Not lost as in he died, just, I can't find him. I searched everywhere I just can't find anything. As a result I decided to search legendsviewer for any caves that contain gremlins. In the whol medium sized world with (I think) more caves than usual, there were three more caves with gremlins in them. The farthest away had two, the nearest one had four, and one that's pretty far away, but closer to the nearest one had five gremlins. So if I go to those two caves I can get a total of ten gremlins, and if I really want more I could go to the opposite side of the world to get two more. My only other option is to search the caverns. I'll probably have to search the caverns for blowguns eventually if I want any ranged gremlins (though I'd have trouble getting ammo so it might not be worth it), but it'll be very dangerous and i'm not guaranteed to find gremlins or animal people. I'll start on my journey to the two nearer caves and wait for any responses.