interacting with
This is the important part for me. I think it's an incredibly significant difference. The majority of people I know who aren't gamers are potatos. They go to work/school and then spend the rest of their time vegetating in front of a tv. I think there's a core element of humanity that atrophies from being so passive.
...Did you just say tv-watchers are "passive" and "potatoes" compared to gamers, of all people? That's just hilariously ironic. Surely you realise this has everything to do with what you are watching and nothing to do with the medium. Would you really say that watching tv-documentaries about the state of the world is more "passive" and atrophying then when you start up your 137th game of Cataclysm?
Pah. "Interactivity". I'll just leave you with the comment that literature is the least form of interactive media, yet the one that'll atrophise you the least.
I think literature is very interactive. It gives you nothing more than an idea, and every other sense and understanding is left up to you to develop. The only reason games have the potential to be more interactive (psychologically) is because of the ability to be a part of that idea which literature only allows you to interpret. Although, as noted by others, many games don't take advantage of this, and become only exercises in mindless repetition.
I've put plenty of thought into this. My capstone project in college involved figuring out how to introduce multimedia and interactivity into comics, while retaining a literary experience instead of turning it into a gaming experience. I learned a lot about the comparative nature of the two media. It was pretty interesting how subtle things have a major impact on the audience's relationship with the work. For instance, I found that the reason voice-overs can never work in a multimedia comic is because they take away control over pacing. When they lose that control, they switch into the same passive mode as when watching tv. When they're expected to resume control, it's jarring. If you give too much control over certain things, people go into a "game" mode of interactivity, and get upset when the work doesn't offer them interaction with things that a game would normally offer. I didn't take it to a scientific level or anything, but I did try various techniques and observed how people reacted to them. I can say pretty confidently that differences between media run deeper than just how intelligent the content is.
Regardless of that, I do agree that not all tv is equal. Some shows are learning oriented or more actively engaging. The common factor I've noticed among people who spend the majority of their free time watching tv is they don't watch those things. They watch passive programming that lulls them into a mentally comatose state, and they languish there until something forces them to move again. I see this mostly among my parent's generation. I'm glad younger generations are moving away from it. Even the petty narcissism over social media that people like to criticize so much is a healthier use of time, in my opinion.