Just to give some history, Windows originally used a 16 bit unicode encoding for it's first foray into "unicode the everything". There are still people who feel that this encoding should be/is the standard for Windows. However, it's important to note that the 16 bit unicode Windows used was UCS2, not UTF-16. The former can only handle 16 bits of unicode and can *not* do any of the extended characters. There are a lot of advantages to doing this, the main one being that every character is *exactly* 16 bits, while both UTF-8 and UTF-16 can have arbitrarily long characters. The downside is that UCS2 (and especially the Windows version of UCS2 -- because MS is MS and they need their special spice on everything) isn't really used by anybody else. Also, there are things you can't encode in UCS2 (I *think* things like skin tones on emojis and sanskrit are some examples, but it's be a *very* long time since I really was in the middle of this stuff, so I can't remember).
My personal feeling is that switching to UCS2 would probably be a lot easier for DF because you can guarantee character indexing. You just make every characters 2 bytes instead of 1. The downside is that there are some edge cases for fonts, on other systems who don't tend to operate with this encoding. UTF-8 is basically guaranteed to work everywhere. It will be a much bigger job probably -- I'm thinking especially for tools like DFHack or Dwarf Therapist because they likely make some assumptions about string length and number of bytes per character. Having said that, if you restrict the character range to the characters already in use, it makes it a lot easier -- at the expense of making it a bit of a no-op for use. If you don't want to use the range of UTF-8, then you might as well stick to CP437. It's only slightly annoying as it is.
Author
Topic: Unicode-16 support for language raw files. (Read 470 times)