Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  

Author Topic: Token Streamlining  (Read 1324 times)

Terratoch

  • Bay Watcher
    • View Profile
Token Streamlining
« on: June 24, 2011, 10:01:08 am »

Have not looked, may already have been suggested, sorry if it was.

In the case where something may have many of the same entries (for example, [ETHIC:blabla:blabla]),
would it be reasonable to allow for enclosing multiple ethics in a single [ETHIC] tag:

So instead of
[ETHIC:blabla1:blabla1]
[ETHIC:blabla2:blabal2]

you would have
[ETHIC:blabla1:blabla1
blabla2:blabla2]

Just using ethic as an example, but allowing for everything like that to function like this may make things faster and easier.
Logged
The only thing better than dining on a fine goblin is using said goblins tallow for soap to fix up my soldiers.

IT 000

  • Bay Watcher
  • Strange Mood
    • View Profile
Re: Token Streamlining
« Reply #1 on: June 24, 2011, 11:54:51 am »

I like the current set tup, it's easier to read then a long line of text, I also fail to see it being useful in other places, could you give more examples?

I would be fine if both set ups were available, but if yours was to replace the original I would raise a few flags.
Logged

***CORROSION v2.14***
<<<More Than Just Zombies>>>
Back from the Dead!

LordZorintrhox

  • Bay Watcher
    • View Profile
Re: Token Streamlining
« Reply #2 on: June 24, 2011, 03:23:40 pm »

Well, the tag layout thus far is really more like XML with different delimiters.  For instance:

Code: [Select]
[ETHIC:ASSAULT:PUNISH_SERIOUS]
[ETHIC:MURDER:PUNISH_CAPITAL]

vs.

Code: [Select]
<Ethic type="ASSAULT" value="PUNISH_SERIOUS"></Ethic>
<Ethic type="MURDER" value="PUNISH_CAPITAL"><\Ethic>

I tend to treat XML as a good litmus test for describing structures, and honestly the way it is now is pretty close to how you would do it in XML.  The other way would be:

Code: [Select]
<Ethics>
    <Assault value="PUNISH_SERIOUS"></Assault>
    <Murder value="PUNISH_CAPITAL"></Murder>
</Ethics>

Which in DF tags would be:

Code: [Select]
[ETHICS:
    ASSAULT:PUNISH_SERIOUS:
    MURDER:PUNISH_CAPITAL]

Which we already knew, but for the sake of clearer discussion.  I personally favor the way it is now because of the ease of parsing and extensibility.

For instance, what if ethics are expanded to include optional tags?  What if at some point this:

Code: [Select]
[ETHIC:ASSAULT:PUNISH_SERIOUS]
[ETHIC:LYING:PUNISH_SERIOUS:EQUITY]
[ETHIC:MURDER:PUNISH_CAPITAL]

(where "EQUITY" means that the punishment must roughly match the severity of the lie) becomes valid tag code?  HOW do I write a parser for it if it could work like what you suggest:

Code: [Select]
[ETHICS:
    ASSAULT:PUNISH_SERIOUS:
    LYING:PUNISH_SERIOUS:EQUITY:
    MURDER:PUNISH_CAPITAL]

It can be done, certainly, but it is more complicated and requires more loops and checks inside the parser to pull it off, and the code itself has to be more aware of the possible valid chains of tags.  I have to know that any ethic can be followed by an optional modifier and what those modifiers are ahead of time, rather than sending tags off to some encapsulated parser that knows what to do with a list of tokens.  In short, the parser has to have parts of the interpreter in it, it has to know more about the structure of the tokens than just where they start and end.  That makes hard to maintain and more bug prone code.  It also increases load times and the like.

Much of the difficulty stems from the sorta odd way DF tags handle the start and end of blocks of things.  You can't put tags inside other tags and, unlike XML, there is no way to signal the end of a block except with an end of the file or the beginning of another block.  For instance, this:

Code: [Select]
[ETHICS:START]
    [ASSAULT:PUNISH_SERIOUS]
    [LYING:PUNISH_SERIOUS:EQUITY]
    [MURDER:PUNISH_CAPITAL]
[ETHICS:END]

would do what you'd like, allow the possibility of future optional tags with no headaches, and also make the parser easy to write and maintain.  Its also much easier to follow.
Logged
...but their muscles would also end up looking like someone wrapped pink steel bridge-cables around a fire hydrant and then shrink-wrapped it in a bearskin.

HEY, you should try my Dwarfletter tileset...it's pretty.
I make games, too

IT 000

  • Bay Watcher
  • Strange Mood
    • View Profile
Re: Token Streamlining
« Reply #3 on: June 24, 2011, 03:56:10 pm »

Hm... now that I can see an actual example it seems to be a lot easier. Don't know how easy it would be for Toady to code or transfer all documents into. But I for one would love to see this.
Logged

***CORROSION v2.14***
<<<More Than Just Zombies>>>
Back from the Dead!

Artanis00

  • Bay Watcher
    • View Profile
Re: Token Streamlining
« Reply #4 on: June 24, 2011, 04:09:57 pm »

I've noted the start and ending blocks thing before (in the modding forum). It's pretty non-intuitive.

I'd like to gently nudge people away from XML for things like this, as it is intended primarily for machine consumption, rather than human. Something like YAML would be better, as it is less verbose and easier to understand and edit by hand.

Code: [Select]
---
entity: # mapping civilization name to civilization values
    MOUNTAIN:
        creature: DWARF
        # skipping stuff...
        ethic: # mapping actions to societal responses.
            ASSAULT: PUNISH_SERIOUS
            MURDER: PUNISH_CAPITAL
            LYING: [PUNISH_SERIOUS, EQUITY] # a list of responses
        # skipping remainder...
    FOREST:
        creature: ELF
        # skipping remainder...
...

Unfortunately, if you drop into creature raws for a bit, there are non-data tags--doing things like tree traversal and queries against already processed data--which makes representing the data in a purely hierarchical format not impossible, but not simple.
Logged
Git - fast, efficient, distributed version control system
Github - Free public repositories, issue tracking, wikis, downloads...

Terratoch

  • Bay Watcher
    • View Profile
Re: Token Streamlining
« Reply #5 on: June 24, 2011, 04:16:23 pm »

Code: [Select]
[ETHICS:START]
    [ASSAULT:PUNISH_SERIOUS]
    [LYING:PUNISH_SERIOUS:EQUITY]
    [MURDER:PUNISH_CAPITAL]
[ETHICS:END]

Yes, that is what I had in mind.
Logged
The only thing better than dining on a fine goblin is using said goblins tallow for soap to fix up my soldiers.

LordZorintrhox

  • Bay Watcher
    • View Profile
Re: Token Streamlining
« Reply #6 on: June 24, 2011, 05:37:38 pm »

I've noted the start and ending blocks thing before (in the modding forum). It's pretty non-intuitive.

I'd like to gently nudge people away from XML for things like this, as it is intended primarily for machine consumption, rather than human. Something like YAML would be better, as it is less verbose and easier to understand and edit by hand.

Code: [Select]
---
entity: # mapping civilization name to civilization values
    MOUNTAIN:
        creature: DWARF
        # skipping stuff...
        ethic: # mapping actions to societal responses.
            ASSAULT: PUNISH_SERIOUS
            MURDER: PUNISH_CAPITAL
            LYING: [PUNISH_SERIOUS, EQUITY] # a list of responses
        # skipping remainder...
    FOREST:
        creature: ELF
        # skipping remainder...
...

Unfortunately, if you drop into creature raws for a bit, there are non-data tags--doing things like tree traversal and queries against already processed data--which makes representing the data in a purely hierarchical format not impossible, but not simple.

Mmm, that solves it nicely I think, and definitely easier to read for mere mortals.  Has my vote.
Logged
...but their muscles would also end up looking like someone wrapped pink steel bridge-cables around a fire hydrant and then shrink-wrapped it in a bearskin.

HEY, you should try my Dwarfletter tileset...it's pretty.
I make games, too

Artanis00

  • Bay Watcher
    • View Profile
Re: Token Streamlining
« Reply #7 on: June 24, 2011, 11:12:25 pm »

Mmm, that solves it nicely I think, and definitely easier to read for mere mortals.  Has my vote.

Another thing to keep in mind: there are tons of YAML parsers/emitters out there, for just about every language you can think of, and the in-memory representations are structures of basic types like strings, integers, floats, mappings, and lists, so programmatic manipulation of raws in this format would be a breeze.
Logged
Git - fast, efficient, distributed version control system
Github - Free public repositories, issue tracking, wikis, downloads...

612DwarfAvenue

  • Bay Watcher
  • Voice actor.
    • View Profile
    • TESnexus profile, has my voice acting portfolio.
Re: Token Streamlining
« Reply #8 on: June 25, 2011, 11:44:47 pm »

I personally think it's fine as it is. Why waste time on what is basically a reduntant change?
Logged
My voice acting portfolio.
Centration. Similar to Spacestation 13, but in 3D and first-person. Sounds damn awesome.
NanoTrasen Exploratory Team: SS13 in DF.

Artanis00

  • Bay Watcher
    • View Profile
Re: Token Streamlining
« Reply #9 on: June 27, 2011, 01:07:37 pm »

I personally think it's fine as it is. Why waste time on what is basically a reduntant change?

If you mean Terratoch's suggestion: multi-line tags and allowing repeated tags to be merged into one argument list, I agree that that is a change that doesn't need to be made (though marking the start and end of nested blocks explicitly would be great).

On the other hand, if you mean switching to YAML or XML, I'm afraid I must disagree. The current raw format is terrible: tag nesting is not marked in syntax and instead done by magic, it is extremely verbose, and is not pure data (we're just lucky it isn't Turing complete... yet).
Logged
Git - fast, efficient, distributed version control system
Github - Free public repositories, issue tracking, wikis, downloads...