In complete and total honesty, if you wanted to implement a paid mod system, it would be possible to do so in a way which would be constructive rather than destructive. Here's how I'd set it up:
You pay for downloading permission on the platform. Something pretty cheap. Say, $5-10 and you can download as many mods as you want for 30 days. When an author publishes a mod, they are able to associate other members of the platform with the mod as contributors. If a player downloads a mod, the author and all listed contributors are added as recipients to their payment. If the player uninstalls the mod before the end of the time period, they're removed. At the end of the download window, the fee is split evenly between every contributor and author (individuals who contribute to multiple mods receive payment for each). The platform provider, dev, and publisher each receive payment equal to one contributor share. Obviously you need very strong oversight and material motivation to not be a dickhead. Not really a good method for the latter that I can think of, it's the core problem with free markets, greedy shitheads are always waiting in the wings to game the system.
This particularly incentivizes the creation of large, "essential" mods, because they will be downloaded by virtually all mod users, giving them much larger shares of income overall than people who make small bit-work mods.
Now that's still complete shit compared to free modding, but it interferes much less with cooperative development (even encouraging it, someone who contributes to a lot of projects will make a lot). It removes the risk for users by disassociating costs from individual mods, and the lengthy trial period makes it difficult for cloned and shit mods to generate income-taken together they also eliminate the need for a refund system.
But something like that will never be implemented because at least two of the three corporate elements (platform host i.e. Steam and publisher) are always going to be greedy fuckers trying to make a dime on other peoples' work and give them shavings from a penny in return. Not to mention that both publishers and devs will likely be more interested in it as a way to exercise greater creative control and steal good ideas to pass off as their own without credit (not that that doesn't already happen, it's damn rare to have a case like nuCOM where the devs acknowledge the modding talent).
tl;dr it's possible to have semi-healthy semi-reasonable paid mods, but it will never happen because there are too many parties who want to exploit the system for their own gain rather than build something which is equitable and viable in the long term. Welcome to the glorious capitalist paradise.