I still cannot believe people aren't free to rule their political parties as they wish. But I'm starting to suspect parties in the US aren't the same as here. If you register as a Democrat or Republican or Green or whatever, do you need to register at the party's office to? And pay a contribution or something?
As I understand it, America's political parties are very different from how Europe tends to treat parties. I'm not even sure how to answer your question without just saying how it works altogether. Basically, for a candidate to be on a ballot, you have to register with the state's or locality's election commission (depending on the level of the office you're running for), and almost always pay a filing fee (anywhere from $10 to a couple hundred, depending again, and widely varying between states) and usually provide some number of signatures affirming that you have a significant enough base of popular support that keeping track of your candidacy won't be a waste of the government's time (usually a few thousand names, again ditto). Parties have nothing to do with this, and technically you don't even "need" to file to win for some offices, if you can run a huge enough publicity campaign to get people to write in your name on the ballot (this isn't always an option).
Where parties come in is that they make it a lot easier to run for office, with cash and people and recognition, especially in "down ballot" races where most people will have no idea who's actually running, so having the party label is a good leg up. Primary elections are managed by the state governments, but they're not... I guess "connected" to them is a good word. This is basically just because states all one by one took over the voting process as they all passed laws demanding transparency and accountability in party structures, and is actually fairly "recent" occurrence in American politics (mid 20th century). Running in a party primary is its own registration process with the party, and only affirms you as the party's "official" candidate, entitling you to the label on the ballot.
Party registration is a little funny, in that it is not tracked by the government, but is legally allowed to be stamped on your Voter ID card, because every state in the union rules that you can only vote in one party's primaries. This is always free, but you can't vote if you're not a registered voter in the first place. The specific rules on which registered voters are allowed to vote in a party's primaries differs widely,
widely between states - the "open" and "closed" labels mostly refer to whether you can mark your party registration when you show up to vote, or have to register in advance with a party to participate.
The only state that's significantly different from this is Louisiana, which places
all primary vote nominees on a single ballot (distinct to each district), with no party labels, and the top two winners are the ones listed on the general election. This is largely responsible for Louisiana's long history of hideous political corruption, because it relies entirely on name recognition, i.e. campaign spending; so it was (and still is) frequently the case where the general election candidates will both be from the same party.
In short, nothing about the primary process itself has anything to do with the two-party domination of American politics. That's much more just a particular evolution of history, wherein the parties have always sort of gravitated around a metric of the "in power" and "out of power" parties, massive separation of powers between offices notwithstanding.