I'm gonna go with Advent Rising. It had tons of problems, but it was really cool and had a neat story.
Let's see... Die By the Sword was really hard to control at times, but the basic idea for the control system where you moved your mouse to move the sword the way you wanted was really cool, and the main character was really funny. Also, dismembering enemies and beating them to death with their own limbs was just keen!
Metal Fatigue wasn't really bad or anything, it just seems like it didn't have any real success and it's a shame because it was actually really good if a bit generic and unstable. But it had giant customizable mecha and you could steal enemy parts then research and reverse engineer them to use as your own.
Trespasser the physics based Jurassic Park FPS was really badly broken, but the idea was really cool.
Oh yeah, the Carmageddon series was a really cool combat racing game. It lacked mass appeal due to the whole pedestrian slaughtering thing I think, I played the first two but never got to play the third. The second one has an absolutely awesome damage system for it's time. The car would crumple, and parts would fall off, and you could even get ripped clean in half if you got his hard enough. The controls were kind of stiff though and it was often hard to tell where you were supposed to go... And I think it was a bit unstable too.