It is really the combination of the prefix "extra", meaning "Above, beyond, outside of, exceeding", and "marital", --of or relating to marriage.
The implication that one MUUUUUUST be married to have "OK" sex, is where the stigma comes from. It is a very puritanical thing, and very abundantly manifest in the US. Since there is no marriage AT ALL, any sex that occurs, is immediately "Above, beyond, outside of, or exceeding" the non-existent marriage, and is thus "Extramarital."
The usual "Dog-whistle" phrase is "Out of wedlock!"
Eg, "She shacked up with billy joe and had that bastard child of hers out of wedlock!!"
The sentence is ambiguous.
It applies equally to--
She (subject) shacked up with Billy Joe (without her husband knowing) and had that bastard (illegitmate, concieved outside of a marrriage-- archaic, and stupid term that means nothing to anyone except 'church ladies') child of hers out of wedlock! (outside the GOD APPROVED sex with her GOD APPROVED marriage with her husband.)
and to
She (subject) Shacked up with Billy Joe (To whom she is not married-- even though neither she nor he are married, either to each other, or to anyone else) and had that bastard child of hers out of wedlock (DARED to be having sex without a GOD APPROVED marriage!!)
In both cases, the people using the phrase are making very asburd assertions about what is right and normal behavior, (and most importantly, the universality of their god, and the moralistic pablum that is associated), making aspersions against a child that has done nothing wrong, and generally being unpleasant and idiotic.
"out of wedlock" CAN be used in a more clinical fashion, as an archaic expression for "Outside marriage"-- and thus, "Extramarital."