Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Greiger]

Just posting to report another happy test of 64 bit.  Ran a unmodded 'medium region' world for a bit over 100 years, and didn't notice much loss of speed if any.  Saving the map did take a bit longer but not significant enough for me to say definitely that the 64 bit was the cause instead of history differences.

Started up a max embark size map with a mountain, volcano and brook.  Took about 5 minutes to load the map.  And there was some slowdown for the first 10 or 15 seconds after unpausing but after that things seem to be going great.  Currently running that map at 50 FPS shortly after embark which is my comfortable fps cap.  I kinda didn't expect it to run that well or I would have set the fps cap higher.

So pretty much just a preliminary report that nothing seems horribly borked right out the bat.  Probably gunna need others to fully test max size maps like this for the longplay though.  While I'm not likely going to play a full size map like this for realz, it does mean that I'll probably be extending my embark size out to 6x6 or 8x8 instead of my current 4x4 when I play 64 bit.  Instability is really the only thing keeping me from doing it in 32 bit now.

Spoiler: Specs (click to show/hide)

OS: Windows 8.1
CPU: Intel Core i7-4790
RAM: 8 GB
Running drive: Samsung SSD EVO 500GB
Graphics: NVIDIA GeForce GTX 760

[lethosor]

Did hear that in the suggestion that just moving to 64 bit could end up with pathfinding being 5 times faster, though that might have been before running and jumping, its also early in so there might be more gains to come.

That's not true at all (see above). Where'd you hear that?

Edit: "above" being the previous few pages, I guess.

With 64-bit, memory ceases to be an issue, so paths can be cached to hell and back. It won't come immediately with the upgrade, but it is something to keep in mind.

Just because DF can use 8 GB of memory (for example) for paths if it wants doesn't mean everyone suddenly has 8 GB of memory to throw at DF. If DF does start requiring huge amounts of memory on a normal basis, it will start swapping and become really slow for a lot of players. The main advantage of a 64-bit build is that people who do have large amounts of memory can play really large worlds (or do other memory-intensive things) without DF crashing because it's incapable of using that much memory.

[Random_Dragon]

I'm still wanting to know if you eliminated orographic precipitation as a cause. D:

Why should that have anything to do with deterministic world generation?

If it affects the eventual layout of the map's terrain, affecting where sites get placed, affecting whether a town is eligible to get placed, etc. Either way, Zorbeltuss seems to have determined that history results vary even in 32-bit, so this question was a moot point.

[waveclaw]

The modernization of the Linux distribution also impacts choice of compilers and libraries.  GCC and LLVM can use different C++ libraries. (http://libcxx.llvm.org/) binaries differ as LLVM is much more strict about the new standard and doesn't use GCC extensions. 

There is much to consider.  C++ 0x11 changes in GCC 5 mean that software compiled with newer systems won't run on older systems.  The newer systems have a dual ABI (Application Binary Interface) to handle this but really older systems are out of luck See the GCC manual on Dual ABI.

lsb_release -a: Ubuntu 8.10, codename intrepid

The real worry with moving from such an ancient system is crossing the glibc 2.1 symbol versioning threshold.   After 2.1 the GNU standard C library started versioning symbols - that is the name of functions and variables exported for use.  Software written with newer GCC (>=3.4) and older GCC (<3.4) have all sorts of problems.

I don't currently know how to decide between 10 all the way up to 15.  Would using 14.04/15.10 cut anybody out?

The issue to watch for is the C++ 0x11 updates, This was a very big deal in the 2011-2012 time frame.  Almost as big at the 2014 work on making /usr and /lib into links to /usr/lib or /usr/bin.  Since you have to cross that huge gap for the symbol issue why not go with the latest version?

Linux users tend to be a very diverse bunch but general end-users follow the normal OS adoption curve. Perhaps take a user poll?  Examine the bug tracker for OS releases on recent or active tickets?

At the worst the binaries could be statically linked (SDL.a included),  The binaries would be huge and people use alternatives like uclib.a to help reduce the code bloat.  And it would piss off people who make native packages since statically linked binaries are strongly disliked by Linux distribution makers.  With 32-bit versions a static binary is like to be too big to load into memory.  For 64-bit a statically linked game might load but just be ridiculously slow.

Using a newer GCC might break compatibility for some people, but that should be relatively easy to fix by distributing the newer libstdc++ that comes with it. (

In Windows it is seen frequently that developers ship their standard C library or an installer package for it with an application for just that application's use. 

In Linux this is usually not done.  An installation of Linux or BSD is a distribution of software after all.  Anything missing should be added to the installed system for everyone to use.  Linux development workflows and release paradigms are all designed around this idea.

On top of this the glibc and libstdc++ libraries are special on Linux.  The glibc library is the ABI (Application Binary Interface).  That is your software's interface to the kernel and thus the world.  So glibc and libstdc++ are not just a grab bag of standard C/C++ stuff but are 'the computer' for your native application.  Yes, you can build software directly against kernel APIs, et cetera and throw away any pretense of running on anything else.

Plus you can't use tricks like LIBPATH and LD_LIBRARY_PATH to override glibc.  You have to mess with the application loader's RPATH on compile time and LD_PRELOAD custom libraries at run time.  Even then your success will vary. See http://www.lightofdawn.org/wiki/wiki.cgi/NewAppsOnOldGlibc for some of the stuff involved.

This ABI not only has to match between your application and libstdc++ but also for every library and kernel function you want to load.  Problems with the ABI cause 'Symbol not Found' errors when compiling and linker errors from ld.so when trying to run applications such as "version `GLIBC_x.y.z` not found."

So to ship libstdc++.so you'd also have to ship matching libraries for everything that the application needs.   Custom SDL.  Custom TTY.  Custom sound and OpenGL libraries for SDL.  Matching libpthread even though you never directly use threads. Everything.  This runs into 100s of libraries for some applications.  This is usually where a company resorts to static compiles.

Oh, you can forget about DFHack or gdb working unless they ship the same pile of libraries as Dwarf Fortress, too.  Those need to match close enough to inspect the running game and inject data into it.

well, I would use ubuntu 16.04 LTS x64 for compiling. If older versions of linux are required, then I would use docker.

Even if you ship the universe of "shared" libraries there's also the risk that your custom world of stuff you are running work NOT work on the Linux kernel you have installed.  Containers like docker are popular because they manage all of this and attempt to resolve that issue. But the ability to run a container implies you are running a 2014 or later Linux distribution probably with systemd.

Should Dwarf Fortress start trying to ship glibc.so or libstdc++.so I'd have to strip it our when packaging it. Even then basic checks on the package (rpmlint, deblint, etc) would scream about shipping any standard library.  Services like the Open Build Server would just refuse to build the package.

This is probably not a big loss.  It appears that most people who want something as convenient as packaged software use one of the big install kits.  Those kits come with custom tile sets, DFHack and the other quality of life improvement tools.  (Plus I never got a clear license answer to my question so I cannot Legally provide binary packages, either.)

Also, it is definitely a good idea to produce several versions of binaries with different cpu instructions support (vanilla, SSE2, AVX, AVX2 - the only difference for you is just a compiler flag), because it does not take any (!) additional time, but may yield tens of percents of execution speed. This is valid for both Windows, Linux and OSX

As eternaleye pointed out, GCC and Linux support function multi-versioning since 4.8.   There is no need to generate different binaries.  This is another reason to switch to the latest GCC for full support of the range of optimizations available. 

Do note that some kinds of optimization will influence repeat-ability of simulations like Dwarf Fortress world generation.  Lookup the scholarly articles on optimization robustness for some cool math on stability, sensitivity and limit theorems of multidimensional optimization.

[Cexp]

I got the impression real machines or multi boot are used for builds. One thing I have not seen mentioned in this thread is the possibility of using virtual machines.

Advantages:
* You can easily have many more configurations (i.e. keeping an ancient x86 distro for maximum compatibility - also less likely to break due to HW incompatibilities - and then a modern one for x64)
* You can move and backup them, they are not tied to a certain host machine
* Nowadays performance is great (for compiling, I measured <5% slowdown compared to native; depends on hardware)
* Testing upgrades / new configurations is less risky (you can test in a different VM)
* More developer comfort (depending on the current way of doing things)

There are some idiosyncrasies to work out but in my opinion it sure beats having a separate dedicated build machine. Testing might be a bit trickier, but I've run DF within a VM in the past without issues (granted getting video acceleration to work WAS an issue; maybe things have improved since then).

[Toady One]

Went with 14.04 LTS.  So far, the 32 bit build is having trouble finding zconf.h, which I just found various chatter about but no clear solution that worked for me.  The 64 bit build is entering compile without errors, but it locked up my mouse and the clock only updates every five minutes, and it isn't advancing through the source files...  the machine is running loud, but I have no idea if it making progress or if it is just spinning its wheels.  Going to leave it for a bit and try again.  No idea what's up there.

[Admiral Obvious]

Well, the current 64 bit test worked perfectly for me.

Went and stress tested worldgen, with maximum everything, and it went fine for me, on a 64 bit Windows 7, Ultimate (if OS type matters).

Clocked the game using up to the magical 4 GB number, and it didn't crash, on an evil ocean island, with a brook with a 16x16 embark, in a 1050 year old world. Framerate was awful, even for my tank of a PC, but by armok, it ran.

[lethosor]

waveclaw:  DF has been shipping libstdc++.so for about six years now (in the libs folder). The only major issue with it that I recall is that DFHack has to delete it in order for DFHack builds built with newer GCC versions to work.

There's also a way to disable the newer GCC 5 ABI (-D_GLIBCXX_USE_CXX11_ABI=0), if that's what you were referring to.

[Toady One]

Nope, it doesn't appear to have been creating object files.  Just sitting there working on...  something.  Same thing happened on a repeat attempt.

[lethosor]

What does gcc --version give you? Not sure why it's hanging, but I think 14.04 comes with GCC 4.8. (That's probably a decent middle ground between too-new and too-old when it comes to GCC, assuming it works.) Can it compile a simple test file? What compiler flags are you using? Some people have reported issues with -O2 or -O3, although I don't know if it's related at all (and you probably don't want to get rid of those for releases).

Regarding zconf.h, you might try installing zlib1g-dev:i386, or maybe lib32z1-dev.

[Toady One]

Getting zlib1g-dev:i386 removed gcc from my computer, apparently...

edit: so I installed it again...  and now it looks like 32 bit doesn't have the error message, but it's crapping out the same way 64 bit does.  I'll see if I can get a smaller file built.

(gcc is 4.8.4)

[mifki]

Getting zlib1g-dev:i386 removed gcc from my computer, apparently...

edit: so I installed it again...  and now it looks like 32 bit doesn't have the error message, but it's crapping out the same way 64 bit does.  I'll see if I can get a smaller file built.

(gcc is 4.8.4)

sudo apt-get install gcc-multilib

And then again

sudo apt-get install zlib1g-dev:i386

and

sudo apt-get install zlib1g-dev:amd64

[lethosor]

Also g++-multilib. (I don't think that's the issue with the build, though - it should fail pretty quickly if gcc-multilib or g++-multilib aren't installed.)

(edit: clarified that missing packages probably aren't the cause of the build hanging)

[Toady One]

A "Hello, world" printf main() with stdio.h worked fine with g++.  The 32/64 continue to not work in the same way (that is, the zconf thing seems to be resolved, but the compile lags the machine to death).  I'll play around with more compiler options.

edit:

g++ file.cpp -m32 -O3 -o test

and

g++ file.cpp -m64 -O3 -o test

both produce a working hello world "test" executable.

[lethosor]

Any idea how much memory G++ is using? "top -a", "top -o mem", or "top -o %MEM" might list processes sorted by memory usage (or it could be another flag; top can apparently be really inconsistent).