Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 [2] 3

Author Topic: dfWorldDat : the Dwarf Fortress World.dat File Format Project  (Read 19113 times)

zilpin

  • Bay Watcher
  • 437 forever!
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #15 on: September 25, 2011, 05:11:39 pm »

The template file provided is proprietary to 010.
I use it because it's the best I've found.

Hex Editors are surprisingly scarce.  Hex Editors with capable template scripts, even more rare.
Logged

Thundercraft

  • Bay Watcher
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #16 on: September 26, 2011, 10:01:23 pm »

Have you considered that, perhaps, the fact that you use a template file for this project that is proprietary to an obscure, rather expensive software package may put off some folks who might otherwise consider joining this project? It may or may not be the Royals Royce of hex editors, but $50 seems a bit high for a single home license.

At least the use of proprietary Micro$oft software is understandable, because it's so popular and common anyway that one probably already owns it and there'd be obstacles to using an alternative (assuming it exists). But there are numerous free and relatively inexpensive hex editors available with respectable features, if one searches enough. It's just that commercial software is advertised a lot more and non-free hex editors are easier to find.

For example:

Frhed - Free Hex Editor - Free, open-source software with many advanced features

Free Hex Editor Neo - Free, lots of features

HexEdit - Free hex editor with numerous features, including a powerful expression calculator

Advanced Hex Editor (A.X.E.)  3.4
Shareware - Lots and lots of advanced features.
Cheap OEM Software > Advanced Hex Editor 3.4 - Same as above, but can purchase at a much cheaper price

Of the ones mentioned above, I've tried them all with the exception of Free Hex Editor Neo. And I've found each one to be capable, depending on one's needs and usage. Usually, however, I use either Frhed or AXE.

Edit: I found one more, and it looks pretty nice:

WinHex - Computer Forensics & Data Recovery Software, Hex Editor & Disk Editor. The variety of features seems downright enormous. Website states, "Try before you buy, as long as you need, for free." (Back in 2002, a much older version of WinHex placed first on Harrys Windows Hex Editor Review in terms of features.)

BTW, a lot of software available for "free download" nowadays have either very crippled features or are simply demoware that stops working after a trial period. But sometimes one can find earlier versions of the same program (particularly Shareware) that are either completely free or do not have such limited functionality.

Oh, and the following Wikipedia comparison of various hex editing software seems very thorough. I never knew there were so many different hex editors under so many difference licenses under different platforms!
Comparison of hex editors
« Last Edit: September 26, 2011, 11:11:51 pm by Thundercraft »
Logged

Ghills

  • Bay Watcher
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #17 on: September 27, 2011, 08:08:54 pm »

That is awesome.  I had no idea Wikipedia had a page of hex editors!

It looks like there are several with similar features to 010.  We should be able to find one that does what we need.  AXE looks like it does template scripting, for example.

EDIT for spelling failure.
« Last Edit: September 28, 2011, 07:21:34 pm by Ghills »
Logged
I AM POINTY DEATH INCARNATE
Ye know, being an usurper overseer gone mad with power isn't too bad. It's honestly not that different from being a normal overseer.
To summarize:
They do an epic face. If that fails, they beat said object to death with their beard.

Aerval

  • Bay Watcher
    • View Profile
Sandboxie
« Reply #18 on: September 28, 2011, 03:56:45 pm »

In theory a 30day limit is no real problem with a program like sandboxie. You could just delete the sandbox and reinstal (or we could look how the limit works, but that might be illegal while sandboxie is no problem)
Another way are programs like Runasdate ...
Logged

Thundercraft

  • Bay Watcher
    • View Profile
Re: Sandboxie
« Reply #19 on: September 28, 2011, 07:26:26 pm »

...Another way are programs like Runasdate ...

I've never tried sandboxie before. But I could see how a program could monitor changes to the registry and then undue them after the program is "uninstalled". However, I have doubts about using something like Runasdate. It might work in some cases. But software developers are getting smarter. More and more, merely rolling back the date will not work.

But regardless of the method, it's basically trying to cheat to get around a software protection scheme. The honest thing to do is to find and use a free (or cheaper) alternative.
Logged

Ghills

  • Bay Watcher
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #20 on: September 28, 2011, 07:50:56 pm »

In theory a 30day limit is no real problem with a program like sandboxie. You could just delete the sandbox and reinstal (or we could look how the limit works, but that might be illegal while sandboxie is no problem)
Another way are programs like Runasdate ...

1) Illegal and unethical.  They aren't always the same thing, but in this case I think they are.

2) Frankly that's a whole lot of work for not much return. At least from my perspective. And relying on an unstable foundation like that isn't always the best plan.  It's probably contributed to the almost total lack of interest in this project.

Fake edit: NINJA'D!
Logged
I AM POINTY DEATH INCARNATE
Ye know, being an usurper overseer gone mad with power isn't too bad. It's honestly not that different from being a normal overseer.
To summarize:
They do an epic face. If that fails, they beat said object to death with their beard.

Pseudo

  • Bay Watcher
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #21 on: September 28, 2011, 08:14:01 pm »

I would be interested in helping, assuming that you find another hexeditor...
Logged
The lady the dog the paper the boy writ hit bit knit.

English is weird.

zilpin

  • Bay Watcher
  • 437 forever!
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #22 on: September 29, 2011, 12:47:26 pm »

The nice thing about 010 templates is that the script syntax is so close to C that creating C structures from it is very fast.
Also means that the learning curve is easy.

WinHex appears to have a capable template, but it is a proprietary syntax.  It's scripting is very weak.  The UI seems very clunky.

A.X.E. has a nicer interface, and the data visualization feature is nice, but their website is down so I haven't seen what their template/scripting looks like.
I'd consider trying it, if they get their web site back up.

HexWorkshop is very nice, but way too expensive.
« Last Edit: September 29, 2011, 01:05:57 pm by zilpin »
Logged

zilpin

  • Bay Watcher
  • 437 forever!
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #23 on: October 04, 2011, 08:42:06 pm »


To be clear, the first to translate the template provided to their preferred free hex editor (and it works, and others can read it), that editor will be adopted.
Anyone should be able to read the 010 template, since it is C-style syntax.

I don't care what it is, if we get some momentum here.
Logged

Eric Blank

  • Bay Watcher
  • *Remain calm*
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #24 on: November 25, 2011, 02:50:33 am »

I'd add to the momentum of support, but unfortunately I'm useless and have no time to make myself useful. I would like to see it finished, though.
Logged
I make Spellcrafts!
I have no idea where anything is. I have no idea what anything does. This is not merely a madhouse designed by a madman, but a madhouse designed by many madmen, each with an intense hatred for the previous madman's unique flavour of madness.

zilpin

  • Bay Watcher
  • 437 forever!
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #25 on: November 28, 2011, 04:36:03 pm »

Thanks for the moral support anyways, Eric.

Anyone who is interested, take a look at the template.
Again, the structures should be readable to anyone familiar with C.
Anyone who wants to convert the template to a free hex editor, I'm game.

I'm just burnt out on all the time consuming trial-and-error that I've put into it.
Don't want to be alone, here.
Logged

Makbeth

  • Bay Watcher
  • His lower body is melted.
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #26 on: November 29, 2011, 01:03:04 pm »

I've wanted a tool that could show geology and underground features pre-embark for a long time.  I remember your deramp, pit, and tinker tools, and with this you are continuing your pattern of making the mods I most want to use.  Unfortunately, I don't have the know-how to be useful in this (Had a lot of trouble just setting up DFhack for compiling my own modules), but I want to say that I am very excited to see you working on this, and if there was any way I could help make it happen, I would.
Logged
Diso Faintpuzzles was born in 120.  Although accounts vary it is universally agreed that Diso was chosen by fate as the vanguard of destiny.

In the early spring of 143 Diso began wandering the wilds.

In the early spring of 143 Diso starved to death in the Horn of Striking.

Keshire

  • Bay Watcher
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #27 on: November 30, 2011, 07:20:26 pm »

I'm game. Have done a few file specs in the past, about time for something different. I wouldn't get too caught up on the hex editor. The important thing is that the spec is readable by a coder.

Here's what I got from a quick look
//from hex workshop
Code: [Select]
typedef struct dfworldheader
{
WORD Version;
//byte Unknown[144];

//Offloaded units somewhere in here??
//Also looks like two animalpeopleblock counts are listed here
LONG Unknown[10];

//These are sometimes off by a small bit from what I've seen ingame, maybe includes the culled numbers.
LONG HistFigs;
LONG Events;
LONG Dead;

LONG Unknown[7];
byte Unknown[1];
LONG Unknown[8];

byte Unknown[31];
}dfworldheader;
Code: [Select]
////////////////////////////////////////////////////////
//Animal People
//Task 1: Figure out Animal People
////////////////////////////////////////////////////////
typedef struct dfsubarray2{
DWORD count;
DWORD Element[count];
} dfsubarray2;

typedef struct dfanimalpeopleBlock{
DWORD count;
dfstring Element[count];
DWORD Element[count];
dfsubarray2 Element[count];
byte Unknown[26];
} dfanimalpeopleblock;

typedef struct subanimalpeopleArray{
DWORD count1;
WORD Element[count1];
DWORD count2;
DWORD Element[count2];
} subanimalpeopleArray;

//Seems to be 13 of these??
typedef struct dfsubanimalpeopleBlock{
//header stuff... Looks like Number of Beasts/savagery isn't a part of this.
DWORD BlockID;
dfstring Element;
WORD Unknown;
WORD Unknown;
byte Unknown[13]; //There's a dword that's BlockID+1, Also some other count/size it looks like

//extra data, bunch of subanimalpeopleArray, probably based on one of the unknowns.
} dfsubanimalpeopleblock;
« Last Edit: November 30, 2011, 11:30:47 pm by Keshire »
Logged

zilpin

  • Bay Watcher
  • 437 forever!
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #28 on: December 05, 2011, 10:04:18 am »

+1 to Keshire for the most useful first post from an Escaped Lunatic ever.

With all the changes made for the next release, I expect that many new blocks will be added to the file.  The more we understand about 31.25 before then, the less will be a mystery.
Good to have some activity.

Back before farmland & forts were added, I had a passable understanding of the region formats (at least to know where they begin and end).  I'm having a very difficult time figuring them out now, and imagine the next release will only further complicate them.
Logged

Keshire

  • Bay Watcher
    • View Profile
Re: dfWorldDat : the Dwarf Fortress World.dat File Format Project
« Reply #29 on: December 05, 2011, 05:57:30 pm »

+1 to Keshire for the most useful first post from an Escaped Lunatic ever.

With all the changes made for the next release, I expect that many new blocks will be added to the file.  The more we understand about 31.25 before then, the less will be a mystery.
Good to have some activity.

Back before farmland & forts were added, I had a passable understanding of the region formats (at least to know where they begin and end).  I'm having a very difficult time figuring them out now, and imagine the next release will only further complicate them.

Right. This would be a lot quicker if we could see more ingame numbers. Such as the events, hist figures and such. Doing a dump of the worldgen info didn't help me too much. Plus I'm at a loss on what I should be expecting to be stored in the world file. Picking out arrays and such shouldn't be too difficult at least.
Logged
Pages: 1 [2] 3