Are you sure your tower is actually occupied? If it is are you sure it's not by someone that just isn't going to care about you? Like a cougar man or something? Just because you make a civ hostile towards you doesn't magically give every civilization (like kobolds, creature men, etc.) the power to siege you. A lot of towers, dark pits, forts, and towns are ruins/abandoned and no one actually lives there. This is especially true if you have a world with a lot of history or a very savage world with a high large creature count. This means more sites get taken over, destroyed by monsters/beasts/other civs, and otherwise have disasters that cause the population to either die or flee.
I once put my fort at a river fork with 3 goblin sites and a tower nearby and never got attacked. So I went into legends mode, and found out all those structures were ruins. The goblins pissed off the dwarves and the humans at about the same time and were stomped in about a 20 year war. So there were like 2 goblin sites left in my large world and they were in the middle of nowhere. Apparently there was a necromancer in the tower once, but some human killed him about 100 years before my dwarves showed up near his site. All this was within normal world parameters. So I basically settled in the ruins of some old civilization with no one around. I didn't even use advanced features, I just had a super savage world with tons of big monsters.