Well, if you want to get excessive about it you can make a 'Forced Entryway'. Use raised bridges to create a walled off area right on the edge that has a single path to the fortress. Every invader and caravan will arrive at that spot, since that's the only way they can actually get to you. You can then make that area as heavily fortified as possible, including setting up special defense towers that can be opened/closed remotely to allow your warriors to enter and fight off any ambushes that show up. Or have tons of traps, chained animals, archer towers, etc.
You might still lose some caravans to ambushes showing up on top of them, but this gives you the most control.