Just a quick progress update today.
Most of my time since the last post has been taken up with ripping out my old UI controls and replacing them with a new UI library that has a lot more functionality. Now that all that is done, and I'm just about back to where I was before, new features should start coming quickly. Other than that I've created a program for editing token and argument definitions. I've added "sanity checking" to the raws, so you can scan for any conflicts of errors, like conflicting tokens, tokens with bad data, duplicate objects, and other stuff like that.
Here is a quick screenshot of the program with the new UI library.
That nested grid you can see there for the color will eventually list out the arguments as name value pairs that comprise the token's value. In the case of COLOR, it would be the "foreground", "background", and "bright". This should give some guidance as to what the arguments for a token are, and the ability for the program to restrict them to valid values.
The program will eventually support pluggable editors that can enhance it's functionality. There are 3 "levels" of editor that can be made. "Object" editors will allow you an easy way to edit an object as a whole without directly messing with the tokens. Token editors will let you edit token values without getting down to the argument level. And finally argument editors, which give the finest level of control. Editor plugins can be created by anyone with .NET experience.
I'm planning a public web based app for maintaining a "dictionary" of tokens, their arguments and other meta data such as conflicting tags. Anyone will be able to edit tag definitions through the site as new uses for tokens are discovered by modders, or new tokens are added by Toady. The dictionary will be freely available for use by any other program or website that needs information about raws in xml format. Thinks of it like the token section of the wiki, but more programmatic and automated. Raw explorer will be able to automatically update it's definitions from the site so it will always be up to date.