I don't think it's a question of being fun or not. What matters is if the gameplay carries the atmosphere of the game well. That's the point of a game. If you want to be all bullshitty, you can make an "interactive fiction" but it won't be considered a game by me. Having a game with only story is like having a film where there is no visuals, only dialog.
For example, a game I played recently (And am playing through again) is a prime example of the "it doesn't need to be fun, it needs to be good" idea. The game, named Knock Knock, is basically a game where you walk around a house turning on lights and avoiding monsters you can't fight against, then going for walks in forests looking for a girl who shows you a quick, cryptic video when you find her then disappears entirely. Sounds like the horrible lovechild of Dear Esther and the latest youtube personality's horror LP, right? Well, I still consider it one of the best experiences I've ever played on the virtue of the fact that the developers (Ice Pick Lodge, the same people who made the Void and Pathologic) managed to make it an engaging trip into the mind of a severely disturbed man haunted by his failures and stuck in a rut of superstitious rituals and childhood games.
You don't need a fun game, but you can't have a walking simulator and still call it an art game. I find the notion of art game silly anyway.