Humans and elves will come some time after they are angry enough at you.
Kobolds will start sending thieves once your pop hits 20 or you've traded a bit (value 500) or you've produced a bit more (value 5000). If they successfully steal objects, they'll come back with more thieves, and if they continue to be successful, they'll bring (buggy) archers. The power of the next attack is based only on how many objects were stolen during the last raid. They'll start sending archers if three or more objects are stolen. Assuming all the dice rolls go completely against you, and ten or more objects are stolen in the previous raid, they can send as many as 17 thieves and 36 archers, but this is incredibly unlikely. If the stolen object number is five or more, they'll start sending important historical figures.
Goblins start sending kidnappers when your population has hit 50 or you have traded a bit or produced a bit more (5x the kobold numbers). Once your population hits 80, they'll start to send more serious attacks instead of kidnappers on occasion. The numbers sent during the serious attacks depend only on how many attacks you've been through. If they have mounts or monsters, they'll start using them on the second attack, with multiple monster packs on the third. Important historical figures also come with the third attack, but master weapon users can come on the second. Important kidnappers come on the second kidnapping attempt.
All of this will be obsolete once I get the armies working properly on the world map, though there will still have to be some artificial mechanism in place to stop early forts from being wiped out.