I liked the magic system (or rather, the rough idea) in Steamband, which is a bit obscure Angband variant which has been out of development for a few years, I think. As the name implies, it's a steampunk game. It does feature magic though, and in many forms. Some classes have their special powers: naturalists are able to use the forces of nature such as lightning and blizzards, mediums can use the force and try to seize enemy minds, aesthetes can enchant items and if you happen to be a hulking mecha or a more modest automaton, you can use a variety of your inbuilt tools that function as weapons, scanners, mining equipment and such.
Then, there's the traditional Angband spell book system, although it's been spiced up by making the "books" fit in three different categories. There's "black magic" which is pretty much what it says on the tin - you'll suffer nasty long-term damage from casting spells, especially miscasting them. The spells are powerful, though. Prayer books are safer to use, but miscasting spells will cause extremely nasty side effects. The spells are too specialized to be the sole means of survival in the early game, as the first direct damage spells one gets only works on automata and undead. Finally, engineering allows one to manipulate powerful machines that range from utility belts to powerful weapons. Miscasting is relatively safe here, but the best effects are unlocked only at a quite high level.
However, the system could've been much more. I really liked the idea of black magic actually causing the loss of something else than easy-to-regenerate mana.
Another Angband variant with a nice system is Hengband which, I think, is even more obscure. I don't know about their magic expect that they have a lot of specialized magic classes (mirror mages, blue mages, red mages, high mages, priests etc...) but the do pack perhaps the most interesting class in all Angband variants - the Ninja. You can deal powerful backstabs, vanish in darkness, hide in a fireball and perform the kawarimi no jutsu. Stealth action FTW!