I had 15 follower at that point, and i still have a save where they were with me. Just tested it, after about 6 tries only one time got to the point of them actually attacking following companion. But I have a theory how to reproduce it. It begins with the fact that if you start a conversation with somebody who's in combat (chasing wildlife for example), you'll have same options as with hostile creatures:
And for some reason when your following companion talks to your builders, while any of them are in combat, they seem to have same options according to what they say:
And i think they use a random option
But the only option that
may cause fighting is demanding yield. They consider such demand as a form of bullying, and everybody become aggressive. (In couple of tries they demanded to yield without further escalation though)
In this case Ona started the conflict, but first attack came from Gib
Icop yielded and Ona asked him to drop his shield and weapon while everybody else were spitting at Icop (most of them are from the same hamlet as her, I recruited them as temporary builders of my adventurer's fort)
"There's nothing admirable about bullying others" I found this moment hilarious. Poor Icop.
In other cases there were times when some asked to cease hostility, to yield, or bragged about defeating many fearsome opponents, but it never escalated further. In my opinion this issue has nothing to do with civ/group logic, I think it's caused by hostility while in combat and weird dialogs because of that. Previously i experienced the same behavior with two other companions from this group.
While writing this I tried it again, but this time kept half of them with me and another half building. After some time and some fighting with ostriches it turned into bloodbath! Three dead bodies and few wounded. And this never happens if everybody assigned to build or to follow me at the same time.
I hope i didn't make a lot of mistakes, it's to late right now for proper English.
P.S. Oh, and nice art, saw it in the video before and thought that its really good!