I have found conversation triangulation to be the soundest way--I haven't chased down a bro like the one you're describing, but as a general rule:
--:(When asked for whereabouts of person): "Why, we're in X right now; he must be around here somewhere"
Means they're in the same building, or live there and are not present--time traveling a few tiles away and coming back.
--:(When asked for whereabouts of person): "We're in X, try first the (cottage\tavern\whatever) to the [CardinaLDirection]."
This is your main means of triangulation; figure out the fastest way to bring up this cause you'll need to use it like three times at least.
Fivegive me if this is ignorance on my part, but I believe unless he's in the Keep proper cause he's taken control he'll be out in a lair--again haven't taken down a bro quite like this before. If so, if you can find out the name of the Site hit Q(Log) and find it under the Sites tab.
If you're in the street and they're giving you response 1, it probably means he hangs out in the streets... I'd say kill his men until he comes to challenge you, and it seems like bandits own towns etc so maybe even though they're programmed to want the removal of bandits the townspeople are chummy with them\have intermarried over the decades so you're killing both their oppressors and their cousins.
EDIT: to clarify, don't use the rumors\incidents tree to do this--that'll give you their map location, not local coordinates. You're going to want to ask for directions to that person, which can be done via the directions menu or asking after troubles and managing to bring him up under bandits\criminals.