Turning back without striking as firm a blow as possible is simply not viable as a matter of morale. We're here, and a certain inertia--a flood tide in the affairs of men, if you will--demands that we try our luck. We cannot catch up to the Duke. This is the only way we have a chance to make a difference now.
We knew there would be a large village. We knew there would be a few fisherman, a few patrol boats, a few crude towers. Nothing is new or unexpected. Palisades are not stone walls with parapets to hold archers. We have to keep archers off the crude towers, that's all. And when in our battles with these people have they skilfully used bows? Never. They favor axes, melee and thrown.
...I wasn't serious...
Should have been, since that ratio sounds good for most medieval towns. How many guards do we have in Curbiston? Torchester? We're bringing most our fighting men to the front, and they will too, especially since they did the exact same thing elsewhere, and Gorgal views us as weak. We knew this settlement would have thousands in it. That was the point. It was also the point that there is no reason known to Gorgal that he should guard it well. The sea raiders don't view the coast as a frontline in this war, and suddenly a bunch of enemies are storming out of it with flaming arrows in the air will break them.
How else can we make a difference? Slink back to Curbiston and await our conquerors?
Also, newlyness of the town reduces firehazard, as they haven't yet developed the continues mass of buildings common to so many cities, which made them so flammable.
Economy of movement guarantees that people will build close to the shoreline where they work and receive supplies. If the town is crowded with thousands of huts, the most crowded areas will already be near the shoreline on the most convenient parcels farthest from the landward areas where they expect attack. I think the housing near the harbour gate will be dense enough to get a very merry bonfire roaring.
But actually, since we don't want to necessarily burn the whole town, just cause panic, I wouldn't be sad if we burnt only parts of the town. The key with the fires is that it will restrict movement and ability for all of them to assemble, and pre-occupy the inhabitants by its own urgency of being put out, as well. Not necessarily that it will burn everything.