Why does non-RPG player automatically equate with casual? There are all kinds of genres of games and playing RPGs doesn't necessarily make you any more serious a gamer than anyone else, except for maybe people who play stuff like Brain Age and Bejeweled, but even then, there are some people who are hardcore about Bejeweled. And just because someone is a serious fan of RPGs doesn't mean they don't also enjoy other genres. Being an RPG player and an FPS player aren't mutually exclusive.
Furthermore, I don't think the TES games have ever really excelled as RPGs beyond the admittedly awesome skill system. Oblivion certainly wasn't a good RPG because it took nearly every RPG element from the previous games and threw them out for a more action/sandbox oriented experience which to me seems to fit better than the awkward hybrid system they had going before. And beyond that, it's always been especially hard to roleplay in Elder Scrolls games. Personally, I like to craft elaborate backstories for my characters and give them complex personalities, but playing those out in an Elder Scrolls game is difficult because there aren't many choices beyond which quests to take and how to murder this particular enemy. Admittedly Morrowind and especially Daggerfall were better at it but I always found it hard to think of my character as a separate entity rather than just an avatar for myself.
And frankly, beyond the Shivering Isles I really don't get why anyone would prefer Oblivion to Skyrim, the world was bland, the dungeons were cut and pasted, it had that awkward meta-gaming level system that was completely counter-intuitive to the concept of roleplaying, the voice acting and dialogue were even worse (I mean, come on, what the hell did they do to Patrick Stewart?), and melee combat was 99% backpedaling, destruction was ridiculously overpowered to the same extent that its underpowered in Skyrim, and the world leveling system was more prevalent and could completely ruin an inexperienced player's character.
Skyrim isn't perfect, or even amazing, but it's certainly a polishing of the ideas that Bethsoft was going for in Oblivion. To me, Oblivion is like an awkward adolescent, halfway between Morrowind and Skyrim and encompassing many of the worst parts of both games and lacking the sharpness of the things that the other two do well.