I managed to get a castle up with level 8 barracks. I'm pretty certain building skill stacks as well. I have 2 level 10 construction characters, and when they were both at a location, it was quicker. It didn't seem to matter whether they were in the garrison or party as far as I can tell, as long as they were at the site. I'm going to train up a few more construction characters next time. For people who don't know what skill to give a fighter after Berserker Rage, Construction is a good choice, since fighters have great encumberance and Construction isn't tied to a stat, by level 20 they make a great materials mule and builder.
I guess a party of 8 builders would be good, but since the other guy seemed to help while in the garrison (need to double-check this, build times were better with two, but seemed the same whether one was in the garrison or in the party), the potential limit might be 16 Construction skills stacking at once. Since it's 100's of turns to completely build out a castle with a level 10 barracks/market this might actually be worthwhile to pursue. So I think I'm going to churn a generation of fighters with Berserker Rage/Construction, since people report Shield Bash and Eagle Eye aren't really that great.