There are basically three classes of people that don't vote in the US.
1. People who can't vote for some reason - those unable to get to the polling place due to work or transportation, those who can't get or refuse to get an ID (in states that require such), or some other reason I can't think of for some reason.
2. People that are sitting out because they don't like any of the options, or are refusing to vote as a "protest" against the system.
3. People who don't give the slightest trace of a shit what the outcome is.
These are roughly in order of the size of the groups.
Mandatory voting would probably largely eliminate group 1, because if it becomes required than employers will
have to make accommodations just like they are required by law to give you time off for jury duty.
For group 2 all it would do is cause a massive surge of write in votes. Somebody who hated Clinton, Trump, Stein, and Johnson isn't going to stop hating them just because they'll be punished for not voting, and if you're sitting out as a "protest" you are not going to abandon that to avoid a fine. Instead, they'll write in Mickey Mouse, or Donald Duck, or Beelzebub, or something like that.
For group 3 they're just going to hit buttons at random to get out of there as fast as they can. If you don't give a shit, you don't give a shit.
EDIT
it doesnt currently cost anything to register and vote
Are you suuuure? That's not what I remember from all the discussions going on around election time. Voter ID is not free. At least not in all states (maybe even none, don't remember). Registration might be free, but you still need a non-free voted ID to be allowed to register.
Not to mention missing out on a day of work.
Most states have no ID requirement whatsoever. Mandatory ID of any kind usually evokes images of the Gestapo in Americans, and they're only really used as transparent attempts at voter suppression.