Seriously - other than 'Lost to Reagan' what the hell did he do wrong? Everything I've read, seen, and heard indicates he avoided screwing things up and making things worse, and did at least a bit to make things better in many ways.
Carter receives such hate because he was unpopular and he was liberal. The conservatives mindset is that they are the only legitimate rulers of the country. That necessitates that any liberal president be an unworthy usurper. The problem is that with so many liberal president examples (liberal being defined as "not a member of the conservative movement"), you need a punching bag to bring up or the point grows awkward. Carter is all they have. Anyone who knows who FDR is loves him except for the libertarian fringe. Truman's domestic policy was too obscure and non-controversial and his foreign policy was too successful and ballsy to go after. JFK died a martyr in the public mythos. LBJ's policies are the exact things that liberals want people to remember and conservatives don't (domestically ramming medicare and civil rights down conservative throats, internationally, going into Vietnam with their full support). After all this we finally get to Carter. He comes at the end of 40 years of democratic domination of politics and liberal presidents who were fine with telling the conservatives to shove it. So 40 years of pent up rage ends up on him and they hate him even though he didn't do that much to hate.
Then of course comes the timing of his legacy. The previous liberal presidents all came from a time when the "real america" was seen as being democratic due to democratic dominance at the ballot box. So when conservatives spit rage, they weren't speaking for everyone in the popular mythos and we tend not to remember their complaints. But Jimmy retired into a country where conservatives were hitting their high water mark of the century. For a 12 year period, conservatives could claim a popular mandate and they used that mandate to heap it on Carter. 12 years of that was plenty of time to get from "conservatives hate Carter" to "Everybody knows Carter sucks."
They tried to same thing with Clinton and failed just reinforcing the "Slick Willie" myth. It's not hard to understand why they failed though if you aren't a Republican, the Clinton years are good and the GOP made an ass of itself over lewinsky. It's because of this failure that Clinton is described as possessing Godlike powers in political discourse. It's stupid, Clinton's o'shucks accent didn't give him mind control powers. But people believe it because they need some explanation for why Clinton wasn't Carter and can't see the obvious, not even Carter was ever the Carter as we remember him.
And if you look at Obama, it's more of the same. Fast and Furious! Bengazi! IRS prosecutions! Solyndra! Birth Certificate! It's all pretty similar to the stuff that was heaped on Truman, LBJ and JFK. Carter got that treatment too. It's just that when Carter did, more people voted against him and that was seen as an endorsement of the hatred of him.