Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 [2] 3 4

Author Topic: Making a DF clone prototype with multiplayer. What do you want/expect ?  (Read 6243 times)

Purple Gorilla

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #15 on: November 04, 2015, 04:08:49 pm »

@miauw62
You misunderstood me, the problem is not the performance, the problem are solely the crashes and broken safefiles. A mature fortress will crash around twice per real-life hour, and every few days a safefile gets corrupted, and if you safe it may crash too. So the most important thing in your mind while playing is typically, "Should I safe and risk a crash on saving, or should I play on, and risk a random crash", and you have to back up safefiles everyday, and run "tweak fixmigrants" and "clean all" every five minutes. I is also odd, that DFHack is absolutely mandatory to really play the game, and Toady even banned the guy who created DFHack (hireing him would have been better).

Videogames are always more than just art. If you make a graphics heavy game, like a first.-preson-shooter, you should spend a lot of time on performance, writing graphics functions in assembly, and when you make a deep game like dwarf fortress, you should spend a lot of time on stability, writing failsafe funtions in C.

Verifying, if a pointer existst is usually done with NULL pointers, but if a pointer contains garbage, it will still crash. Also I read every single dumplog from Dr. Watson when DF crashes, and most crashes are actual access to NULL pointers. (If you see something like wrtiting to adress 50 it are usually instructions like MOV EBX,[EBP+10], MOV EAX,[EBX+50], there 50 is the offset inside a NULL structure, and EBX the NULL pointer, read from a parameter on the stack, acessed with EBP.) So appearently some functions assume to always get valid pointers.

It is possible to intercept garbage pointers with arrays :
Code: [Select]
struct dwarf{
bla; bla; bla;
};

struct dwarf *dwarves; /* Array allocated at load time */
int max_dwarves;         /* Maximum number of Dwarves, set at load time */

void build_consturction( struct job *j, blabla)
{
 struct dwarf *worker;
 blabla;
 blabla;
 worker=job->worker;  /* load a pointer from the job */
 if(worker<dwarves) return; /* pointer too small */
 if(worker>=(dwarves+sizeof(struct dwarf)*max_dwarves)) return; /* pointer too large*/
 blablabla(worker);
 blabla;
}

If job was thrashed by a growing tree, or watever, we load an invalid pointer in worker. Accessing this pointer with blablabla() will cause a segfault or damage data, but the two lines before using it check if the pointer is out of range, so you wont get a segfault.
For extra proofing, you can even check, if the pointer is misaligned with "if((worker-dwarves)%sizeof(struct dwarf)) return;"

If you use a storage-class from the C++ library istead of an array, you don't have acces at this low level, and the library functions will happyly crash. The only workaround is, to rewrite all the strorage classes yourself, and even then the functions using the classes must handle the error, so you have even more work, than in standart-C.

The Key is defensive programming. If you look at the source code at NetHack, you see haw it works. Problems that can be fixed, call the function impossible() that displays a "Programe in disorder" message, and if it can't be fixed, it is at least anounced with panic(), before trying to safe, and aborting. This shows, that the Dev Team thought about the problem.
---
Limits don't mean, what the limits are fixed for everybody. You just set the number in an init file.
Lets say your session has slots for 200 dwarves, and you have 200 dwarves. A birth times out, you get a Message :

THE CHEESE MAKER URIST BLABLA IS PREGNANT : ABORT (G)AME  ABORT (B)ABY

You pick ABORT (G)AME, and your game IS SAFED normally. You edit the init file and set the Dwarf limit to 201. You restart. During loading, the game shows, that it is actually loading in the moment (instead of freezing with a black screen), and suddendly you see the message OUT OF MEMORY, LOADING FAILED (good programmers actually check if alloc() returns a NULL pointer).  You now see, that the game can't handle the baby, set the limit back to 200 and choose (A)BORT BABY.
Another player in the same situation might have enough memory and can continue with the limit of 201.

There is nothing bad abouts limits. Other videogames have limits, CAD programes have limits and file systems have limits. The only important thing is, to know the limits and safely fail if the limit is hit, instead of crashing for the sake of freedom.

miauw62

  • Bay Watcher
  • Every time you get ahead / it's just another hit
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #16 on: November 04, 2015, 07:51:10 pm »

Such programming would, however, hide the faulty code that allowed null or garbage pointers to remain in important lists in the first place. It is important to know that there are bad pointers floating around and where they came from.

And when you said "array of fixed size", I thought you did mean an array, and the size of arrays is of course determined at compiletime, which is why I thought you ment hard-coded limits.
Logged

Quote from: NW_Kohaku
they wouldn't be able to tell the difference between the raving confessions of a mass murdering cannibal from a recipe to bake a pie.
Knowing Belgium, everyone will vote for themselves out of mistrust for anyone else, and some kind of weird direct democracy coalition will need to be formed from 11 million or so individuals.

omega_dwarf

  • Bay Watcher
  • Adequate Architect, Dabbling Modder
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #17 on: November 04, 2015, 10:36:15 pm »

I wasn't aware that 0.40.24 crashed so often. I've never had that problem. Possibly my forts aren't mature/large enough to encounter memory-related problems.

Purple Gorilla

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #18 on: November 05, 2015, 02:46:53 pm »

@miauw62:
Yes, fixed size was indeed misleading, sorry. I meant, allocating a block of a user defined size then loading a game.

If the programe finds a bad pointer, there are of course better ways to handle it than simply a return, like displaying a message or writing to the errorlog.

@omega_dwarf: Running out of memory is only one problem. It is however easy to track, as you get a different exeception code, and it crashes much faster as no core dump is created. Toady's C++ library actually detects the problem, then a method can't allocate memory, and calls the processor interupt 03h, which is actually the breakpoint interrupt. If you look at the disassemled code, you see that the part of the code consists of a large block of breakpoints, so this is probably not an actual breakpoint, but the libraries way to display custom errors. If you would run it from an IDE, there would probably be a meaningful error message (derived from the IP of the breakpoint), and installing a custom exception handler would replace the breakpoint.

While my Athlon XP with 1GB of memory is slightly outdated, it could easily handle 4x4 embraks or even larger if this probles would be handled:
* unused memory isn't freed
* Contaminants multiply in water
* Dwarves never clean up
* Every bodypart and item gets its own contaminant
As quitting and restarting, and running clean all, lets the memory drop from 500MB to 50MB, 90% of the resources are appearently used by bugs.

Most problems seam to be actually dangling pointers and memory leaks. If you save the game, and load it again without terminating Dwarf Fortress, loading takes significant longer, so there must be a lot of memory leaks. It is in fact faster to quit after every safe and start it again, instead of loading immedeatly.

Telltale signs of that are also spooky things happening.
In my current fortress, that crashes often, all that did happen:
* weird symbols as quality modifiers (derefered quality level ? Artifact, those name was lost ?)
* The stock screen shows two different types of copper bars (different isotopes ? :D )
* I have rock boulders (the rocktype rock from arena mode)
* I have rock bars (not blocks, "metal" bars)
* The rock bars are stuck in unmined rock. I was able to snatch one, by deconstructing a floor in the same 16x16 block. (fighting bugs with bugs)
* Logs from unknown frozen plant substance (fairly dense, makes good crosbow bolts) This is usually related to snow, but the site never freezes.
* Stuff stuck in mid air (cave in is on)
* The key configuration changes during the game for no obvious reason. After quitting and reloading it returns to normal.
* Moody Dwarves produce items of iron without ever picking up any iron.

In adventure mode, I had also multiple times the situation, where items (like my sword) exploded for no obvious reason.

OK, this is in fact in 34.11, but 40.24 is even worse. Multitile trees and background history often produce deterministic crashes, there you end up with a savefile that crashes a few seconds after unpausing, and there is nothing you can do about it. The only crash that can be prevented is building new buildings in trees, but that is extreme crippling. Lets say, you tap the caverns from above and want to build a platform above the water, but there is a mushroom. You have to build a second stairway to the floor, let some forgotten beasts in, pump the water out (as you can only chop the botommost part of a tree) and chop it. 40.23 (there trees vanish) seems to produce far more corrupted safefiles that 34.11. and 40.24, so there is in fact no version of current release that is playable. In 34.11. it is possible to play epic fortresses were children born here grow up and dwarfs die of old age, with enough patience and 2x2 embraks, but in 40.24 I never got past 5 years or so.

Snaake

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #19 on: November 05, 2015, 03:59:00 pm »

I wasn't aware that 0.40.24 crashed so often. I've never had that problem. Possibly my forts aren't mature/large enough to encounter memory-related problems.

I haven't played 0.40.24, but I'd like to 2nd this by saying that 0.34.xx didn't crash much at all for me, or for most people on the forums. Also, your (Purple Gorilla) problems regarding 2 different types of copper bars, rock boulders & bars etc. are for the most part stuff I've never heard about, tbh. The single exception is producing "iron" artifacts without picking up iron, that was because iron is the default material if something goes wrong with setting the main material for the artifact IIRC.

So to me, it just sounds like your install (maybe mods?) has had some major problems. 2 types of copper bars etc. might be caused by save corruption? As for your 4-point list of "problems", I'd say that
  • unused memory not being freed definitely is an optimisation issue
  • contaminants multiplying is probably Working As Intended, but this should/could/will probably be revisited at some point so they do get washed completely away at *some* point, both for gameplay and optimisation reasons
  • yea, dwarves never cleaning up is a gameplay issue, but would also help optimisation, much like the previous
  • Every bodypart and item getting it's own contaminant is... basically how DF does things ^^. And this sort of thing is why a lot of us here on the forums love it. Cleaning working better would also pretty much remove any long-term fps impacts from this.
Logged

puke

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #20 on: November 05, 2015, 06:40:50 pm »

DFHack is absolutely mandatory to really play the game, and Toady even banned the guy who created DFHack (hireing him would have been better).

Wha?  Thats not true.  peterix has not posted in a year, but he was logged on to the forum just today:  http://www.bay12forums.com/smf/index.php?action=profile;area=summary;u=16347

expwnent is currently carrying the torch, and he is an active participant on the forums.

I shouldn't even be arguing this, but neither should people read crazy things like this and think that Toady is banning people for writing utilities.  That's just silly.
Logged

omega_dwarf

  • Bay Watcher
  • Adequate Architect, Dabbling Modder
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #21 on: November 05, 2015, 08:10:45 pm »

Yeah, this sounds like the kind of thing that comes from an already-corrupted save, or even raw duplication. I know that versions after 34.11 and prior to 40.24 had serious issues with save corruption and crashes, but I really haven't heard much about any crashes in 40.24. Your 1GB of memory sounds like the obvious problem. I mean, yes, DF sounds like it has some memory problems, I don't doubt that; but these extremely weird scenarios and frequent crashes don't seem to be something most people encounter, and your setup has a clear weak spot.

I have a couple of ten-year-old forts in the current version which have zero problems. I've kept pop to 80 or below to preserve FPS, and I think I did the 4GB memory fix, so that's probably why. But I call that completely playable. I had a bad computer for a long time, so I know it's tough to just try to enjoy a game from a bad platform, but..it really sounds like that's the problem that so cripples the game for you - not the game itself, whatever issues it does have; it's meant to run with 2-4GB of memory available, and when it has that, it does work. It's just not made for your system, which is why (I think) the problems it does have are manifesting themselves so strongly on your computer.

So yeah, ideally, those problems should be fixed; there would probably be benefits for everyone if they were; but for now, the game is completely operable for a lot of people. If you can track down exactly what's going wrong, then by all means, let Toady know, and he'll probably fix it! Otherwise, I bet it would take months for just about anyone to solve the problem.

Anyway, yeah, back on topic, for a clone, fewer glitches would be nice :P

clockwork

  • Bay Watcher
  • Dabbling Performer
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #22 on: November 05, 2015, 09:26:31 pm »

Shameless but relevant post- follow the links, opinions, etc -

http://www.bay12forums.com/smf/index.php?topic=145323.msg5778407#msg5778407
Logged

Bumber

  • Bay Watcher
  • REMOVE KOBOLD
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #23 on: November 06, 2015, 09:13:28 am »

DFHack is absolutely mandatory to really play the game, and Toady even banned the guy who created DFHack (hireing him would have been better).

Wha?  Thats not true.  peterix has not posted in a year, but he was logged on to the forum just today:  http://www.bay12forums.com/smf/index.php?action=profile;area=summary;u=16347

expwnent is currently carrying the torch, and he is an active participant on the forums.

I shouldn't even be arguing this, but neither should people read crazy things like this and think that Toady is banning people for writing utilities.  That's just silly.
peterix was banned temporarily, until the issue was cleared up and he was found innocent of any wrongdoing.

(I had thought this was the thread Flying Dice was linking to, but apparently I was mistaken. Fortunately my previous response was generic enough to include Kiskara's brand of dickery as well.)
« Last Edit: November 07, 2015, 08:50:51 am by Bumber »
Logged
Reading his name would trigger it. Thinking of him would trigger it. No other circumstances would trigger it- it was strictly related to the concept of Bill Clinton entering the conscious mind.

THE xTROLL FUR SOCKx RUSE WAS A........... DISTACTION        the carp HAVE the wagon

A wizard has turned you into a wagon. This was inevitable (Y/y)?

puke

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #24 on: November 06, 2015, 10:44:02 am »

Following the deeper links, I learned two things: 

1 -Toady publicly apologized and welcomed peterix to continue further dfhack threads.

2 -if you knew about this incident beforehand, then you also knew not to advertise clones on the forums here.

Soooo...
Logged

omega_dwarf

  • Bay Watcher
  • Adequate Architect, Dabbling Modder
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #25 on: November 06, 2015, 11:24:20 am »

Side note, since I didn't know about this before, it looks like Impaler has kept Khazad going, and early in 2015 began advertising on the Gnomoria forums. So what happened? Was the stolen code in Khazad successfully taken down, and is this Impaler's own code now? Or is he still passing it off as his own, open-source, etc.?

Purple Gorilla

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #26 on: November 06, 2015, 03:13:42 pm »

I didn't advertize it and am not even inwolved in Thorlak's project, I was just pointing out the importance of stability.

OK, peterix was indeed unbanned. I wasn't playing dwarf fortress back then, but then I started, I was brownsing loosly through the forums for the legends of the forum (Khazad,<toad-edit: item removed>,Mermaid breeding and buthering,...) and found the thread. No I read it, carefully and did some research on google. The following things seemed to were going on :
* Toady says he released some symbols. This seems to be an *.EXE file for a debugger, so the names of functions and variables can be seen in a debugger. You can gardly call variable names intelectual property, but Toady was enraged about this.
* Peterix made DFHack. He also saw the symbols, but hadn't much use for them. Toady nowadays hasn't anymore problems with DfHack
* Impaler made Khazad, initially a visualizer.
* Peterix ported Khazad to Linux
* Impaler turned Khazad into a clone of Fortress mode. It seems to be written in Java, so there can't be any stolen code even if he had acces to it.
* Impaler encouraged people to stop donating to Toady. Toady looked throught Khazad's history on a code sharing platform and banned both contributors.
* Peterix was unbaned, because he was inwolved before impaler turned against Toady. Toady seems also not to be too enraged at clones in general, only to those that can damage his buisiness model.
---
More memory might not fix the problem in general. When the memory is filled with garbage, it will just take longer to finally crash. And again, such crashes are rare. I tried both vanilla, and vanilla with manually installed Dfhack, but it is if fact more stable with Dfhack, as "clean all" and "tweak fixmigrants" help against most problems. 
« Last Edit: November 06, 2015, 04:50:41 pm by Toady One »
Logged

puke

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #27 on: November 06, 2015, 04:27:08 pm »

well, lets move on without further comment on that topic..

I always believe that competing games in the same genre benefit the genre as a whole, so a MP 'clone' isnt necessarily damaging. 

I don't think 'cloning' is a great plan though, maybe rather Thorlak can find a unique hook and produce something different. 

Multiplayer construction management / farming / logistics sounds like a fun idea.  If the project is intended to be a short one, try to keep the feature set small and tight without copying everything in DF Fort Mode
« Last Edit: November 06, 2015, 05:55:37 pm by puke »
Logged

Chimerat

  • Bay Watcher
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #28 on: November 06, 2015, 06:44:59 pm »

Maybe a little late to get in here, but I'd like a way to either have a Fortress shared with someone else (I do Farming and Gathering, they deal with the Military, for example.), or a way to have a Fortress that's vital to trade. Or maybe even play a Trader, and hire other players to protect me as I go about the land.

Not sure how engaging Guard Duty would be, though.

What I wouldn't want is for multiplayer to just have all my fun ruined by constantly having my progress destroyed by other player raiding my base all the time. That the main reason I don't generally play multiplayer anything... :-\
Logged

DgN13

  • Escaped Lunatic
    • View Profile
Re: Making a DF clone prototype with multiplayer. What do you want/expect ?
« Reply #29 on: November 07, 2015, 12:12:56 am »

@Gorilla -- Seems a little silly to start writing the book on "how not to have memory management errors in C++" (by writing C, apparently) when it hasn't even been established what technologies Thorlak's game will use. It's not like there aren't plenty of languages in the same speed-order-of-magnitude of C++ that don't let you commit those memory management errors, and most likely the OP already has entrenched opinions on what technologies to use and how he likes them.

Maybe he's going to do the whole thing in APL -- won't you feel silly then!
Logged
Pages: 1 [2] 3 4