I wanted to make a bug report, but it's been several version i ceased to climb inside of goblin twoers, too unplayable slow there and if something changed i may have missed it.
In 40.13 i noticed near a dwarf hamlet a goblin site , that on the world map was made of a single goblin tower glyph instead of many of them + dark fortress.
So i thought it would be much less laggy than usual goblin sites, and indeed it was less laggy but i ran into something odd and i can't say if it's a bug, but i really don't remember it being like that before.
In the couple of towers of that site, when i move up following the ramps, i can't reach the top floor by moving up as usual, to reach the top floor of a goblin tower i need to press ALT+the direction and select "Move to fire clay (North/Above)" (north or the material can vary depending on the goblin tower and the ramps orientation.
That seems odd because i really don't remember being forced to ALT+direction to reach the last floor, just direction was enough in my blurry memory.
I visited one goblin site and did not have to do any ALT movement to reach the top floor. But due to the lag you mentioned I have only tried this once or twice. It sounds like you encountered some towers that didn't generate properly, but I can't be sure. I also can't say how common this might be, as I've only visited a few goblin sites. I can only say this doesn't happen all the time, at least in (I think) 40.08 when I tried it.
I've spent some time wandering through goblin sites in adventure mode, and in all of my wanderings, I encountered one goblin.
Said goblin, a wrestler, stoically stood still until I lopped its head off with a great axe.
The aesthetics of the broken land with occasional towers is kind of creepy, but I was hoping for some goblins.
I've similarly wandered into towns that were undergoing invasion, but the goblins and humans were all standing around. When night fell, they even slept in the same houses. I wasn't entirely sure whether I was killing an invading army or massacring an immigrant enclave. It was very confusing, and perhaps distressing from an RP standpoint.
Then there was the time I wandered into a goblin camp and half of them were killing the other half (it was hard to say where it began, but there were more corpses than goblins.) This, at least, can be logically explained (Merry and Pippen being carried off in The Two Towers) and it's an improvement on the "stand around passively and be murdered" attitude of other bandits I've encountered...
Then I hiked half a continent, partly through vacated goblin towns (that I could not fast-travel through) and then through miles and miles of swamp to reach a fortress I was rather proud of...
And the game crashed the moment I came out of fast travel mode just outside the fort.
Adventure mode does seem to be quite broken.