Except you don't control the Jarldom and you could be sent here to get you out of the way...Your just the diplomat, hell they may want you to be killed.
Plus strategically, attacking this city is folly.... You have to cross the bay of storms or whatever and since that seems to be the fact that it most likely has hurricanes every year, it would take roughly two seasons worth of work to muster an marine force big enough to attack this city and another few seasons to muster enough food and drink, crossing a northern ocean in winter is stupid.
Also considering that the ships that you would most likely use are not better in naval battles then the ships that the people of the Storm Coast currently use.
Plus you have several countries on your flank that would most likely like to tear a piece of land or two off of you.
Plus the people of the Storm Coast are natural sailors, your people are not.
Plus your up against me, if I get what I need.
Plus you most likely bought your weapons and armor from Lonemine, or at least the iron for them.
Plus your not using steel but iron.
Walking all the way, you have to cross through several countries of power and wealth, which means that they can raise forces against you. Plus it would take longer to march to this place.
Plus the only thing that this city is likely to provide is trading.
Plus you don't seem to be so good at tatics, you more of the charge and kill type.
Your in checkmate, mate.