I play Adventure mode constantly, in my experience, i'd say one of three things happened:
You got lucky ( Or unlucky if you dislike Elves ) and got alot of Elves living in a human town ( Which happens, you can get Elven shopkeepers in Human towns...I have no idea how, but they do. )
Or, you got a very rare glimpse at a Goblin ruled human town. You may think "Wtf" but i'll explain: Goblins like to take prisoners of every race plausible, however, they CAN ( Rarely ) have Human/Elf even DWARF leaders, and in this case probably caught a shiz load of Elves ( Since they don't protect themselves well at ALL, and get pwned now and again in world gen ) as prisoners, and "Imprisoned" them there. It's HIGHLY, unlikely, but i believe it may have happened.
Or the third and most logical one. Elves took a Human town and you just so happened to bump into a Human peasent. If Elves ruled the place, they would never have Mayors, only druids, and if Humans owned it, you would quite easily notice the leader was a weaponmaster ( They glow constantly and are in the mayors house - the place you always start at ) though i do believe it is possible to get a peasent as a leader ( And it has happened, after a weapon master dies they randomly declare a leader - Peasents sometimes. ).
EDIT:
I doubt the Elves ( If they ruled the place ) would stay there, either demolish it ( Since it's made of wood ), or simply leave the place to ruin.
If the Peasent wasn't purple. He wasn't the Mayor. ( I think )
Any NPC who can talk will announce any Notable Kill he has made, that includes Notable Animals
I don't think the Peasent was the mayor, however, if he HAS made a Notable kill, i'd check it out, he could of killed a Megabeast (Though probably just a wolf ), in which case. He DAMN WELL SHOULD BE the Mayor!
And the tower ruled by a Human happens, i suppose he was a prisoner who earned his keep and eventually the awe of his Goblin captors, a bit like the film Gladiator. That or the Human prisoners over-powered the goblins, which, recently i believe possible due to the Civ bug. ( The goblins tried to kill the Human Prisoners mother for example, therefore fought the Goblins, who had their friends join in etc, those two goblins might have been exclusively friends to the Humans only. Gotta remember, goblins are pussies without their weapons and armor. )