Who here has tried making a similar game? Who has been thinking about it, but never quite done anything? And perhaps most interestingly, what would be the most important things you would do differently, both in terms of architecture and game design?
So, I'm a programmer (and I like to think of myself as a good one), and I've tried to make a similar game before (back in the downtime between graduating and finding a job), and I have to say that while if I were starting to make Dwarf Fortress today I'd do some things differently (particularly around handling multiple cores and what I suspect is a task-creation system that doesn't scale well with number of taskable items on the map, in terms of committing excessive memory access), but I'd be at least five years behind. As I said, I've tried making a game like dwarf fortress (though less ambitious, it still wanted to have a randomly-generated world + significant history), and as with any software engineering project there are tons of unexpected things that come up and need to be solved, and even having seen Dwarf Fortress I've be surprised if I could get a Dwarf Fortress clone to where Dwarf Fortress is now in five years. I deeply admire Tarn (and Zach) for their collective determination, creativity, and programming skill.
2: He also lacks the rigorous and formal knowledge of computers and programming skills that I possess. While he is a capable learner, there really is no replacement for the fundamentals and he would be much farther along if he had taken a year or two to study this at the beginning of the project.
That doesn't matter in my opinion. Anyone with great interest in a field will quickly assume a level of competence that far exceeds the "average" graduate. Not that I want to downplay the value of a proper CS education - a highly motivated student in this field can surely make tremendous strides. All I am saying is that that you can easily learn as much by yourself, and more, than your average student mediocre CS student.
Somewhat contradicting myself from above, I have to agree with Nadaka that a 'proper' CS background is something that I think would really help Dwarf Fortress now. I find that while anyone with a great interest in the field can definitely exceed an average graduate, in order to truly excel you need to both have a great interest
and the theoretical background. I'm sure that Tarn's math background helps immensely here (Because CS is essentially applied math), but at the same time it seems like there are certain things that show up as DF grows in complexity (like multithreading, or efficient memory access) that are artifacts that don't show up in the math, and Tarn has to learn about them as they become problems, rather than designing for them from the start. With that said, we're now thinking of things that are relevant now, rather than when DF was first released, much less first designed. You said that you spent a few years working on the infrastructure to generate a scalable simulation game, an effort that I'm sure a) didn't seem appropriate when DF was first designed and multiple processors were few and far between and b) is directly opposed to the spirit of releasing bits of the game as they are playable. You've seen the list of planned features for DF: Can you imagine writing the infrastructure to support everything on that list, and only then starting to write the actual game?
Kind of on that, I suspect that the inside of Dwarf Fortress is about as messy as you'd expect a constantly evolving five-year-old program to be, and so while we can look from the outside and say "It would be nice if it supported multiple cores" or "it would be nice if it supported multiplayer", adding things like that might well involve a fairly drastic rewrite of what's been done already. We've seen that before, of course, with the OpenGL rendering rewrite that we got with 40d and I suspect that many things are being rewritten under the hood with the current town/world changes, but adding something like support for multithreading at this point would almost certainly involve a total rewrite. I think it's not just that Tarn didn't plan for multithreading or fortresses with hundreds of thousands of items, but that they weren't appropriate to think about when he started writing DF.
So, in summary: Have I thought about making a game like DF before? Yes. If I was writing a game that I knew would turn into what DF is now, would I do things differently from what Tarn has done? Yes. Do I think that having had a 'real' CS education might have helped DF's development? Yes. At this point, though, looking at where DF was, what it has turned into, and what it is trying to be, I'd say that I can't see myself doing things as well as Tarn has thus far.