Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[chmod]

So I skipped the internet this weekend and missed a really fun thread that seemed to be mostly about cheese, but was also asking why anyone would donate to a project that hasn't borne fruit for over a year.

First my qualifications: I've been writing code since I was 11. I am now 31. That is 20 years of continuous experience. 10 of those were as a professional, where peers with big budgets gave me actual cash money to deliver working software to them. 7 of those years have been creating and contributing to open source projects.

That said, I am a bad programmer. All of us are. There is a measure of every program called Cyclomatic Complexity. This is the raw number of realities that can exist in a program. Loops, conditionals, input, time, and state all MULTIPLY complexity of a program. A program with as many possible conditions as DF has a CC score in the millions if not billions. Humans are incapable of storing a logical graph of that size in their heads. Which is why we put it into a computer in the first place.

As features increase linearly, complexity increases exponentially. Which is why we don't have things like SkyNet. This is a fundamental problem with programming, and you can read about for the next several months if you're interested.

So you may wonder, why is this guy going on about complexity? Well, because high complexity leads to regression traps. A regression is when a coder changes a feature with a certain intent and inadvertently breaks an existing feature. As your complexity score climbs in to the millions and billions of possible state combinations these become harder and harder to find or prevent. This is why MS Office has a QA department of over 500 people. Several of them are there to make sure you can still open an Excel spreadsheet from 10 years ago.

DF has gotten complex enough to be a regression mine-field. Where every good-intentioned "easy" problem, can cause the breakage of an unknown number of other items. For this reason alone, feature-addition speed is reduced, and will keep declining. But there is another reason: Labor of Love.


<<INTERMISSION>>
(please visit the snack bar)

Most readers have come into contact with Labor of Love software at one point or another, and unfortunately judge it unfairly against a corporate software bar. Some folks hit on this in the thread I was originally referring to. But with no QA department, or accounts receivable, or shareholders driving the agenda, Toady has to be all of those people. He is simultaneously thinking about high-quality, broad platform support, performance, increased revenue, advertising, support, let alone game-play. And do you know what wins in all of those internal conflicts? "What is fun?"

You may think you're oppossed to this method of development, but I'd bet even money that is why you're here. Because what is fun to Toady lines up with what you think is fun. I sometimes log into my development account for Dwarf Therapist, see 160 bugs, decide I only care about 3 of them, still decide not to fix a single one, and then go add a scripting engine for filtering. See what I did there? I have a public record of what people want, I even have donations from some of them. I have bugs that I agree should be fixed, and you know what I do? Ignore all of it and do what makes me happy.

Now I do fix those bugs eventually, but I didn't start my project to make everyone else happy. I did it because I wanted the fruit from that tree. I can't speak for Toady, but I know that's why he works on DF: HE WANTS A GAME LIKE DF TO BE IN THE WORLD.

I gave him some cash frankly because I had some, and I respect the amount of love he's baked into this game. I see him as a fellow craftsman, and I know a lot of his pain and joy, although I've never undertook something of this magnitude.

To sum it up, I will steal (selectively) from ESR in his paper "The Cathedral and the Bazaar"

 * Every good work of software starts by scratching a developer's personal itch.
 * Good programmers know what to write. Great ones know what to rewrite (and reuse).
 * If you have the right attitude, interesting problems will find you.
 * When you lose interest in a program, your last duty to it is to hand it off to a competent successor.
 * Release early. Release often. And listen to your customers.
 * Smart data structures and dumb code works a lot better than the other way around.
 * If you treat your beta-testers as if they're your most valuable resource, they will respond by becoming your most valuable resource.
 * Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.

And from E.W. Dijkstra (my personal hero)

"We understand walls in terms of bricks, bricks in terms of crystals, crystals in terms of molecules etc. As a result the number of levels that can be distinguished meaningfully in a hierarchical system is kind of proportional to the logarithm of the ratio between the largest and the smallest grain, and therefore, unless this ratio is very large, we cannot expect many levels. In computer programming our basic building block has an associated time grain of less than a microsecond, but our program may take hours of computation time. I do not know of any other technology covering a ratio of 10¹⁰ or more: the computer, by virtue of its fantastic speed, seems to be the first to provide us with an environment where highly hierarchical artefacts are both possible and necessary. This challenge, viz. the confrontation with the programming task, is so unique that this novel experience can teach us a lot about ourselves. It should deepen our understanding of the processes of design and creation, it should give us better control over the task of organizing our thoughts. If it did not do so, to my taste we should not deserve the computer at all!"

tl;dr Take your Ritalin and try again

[qwertyuiopas]

From my limited experience(Less then ten years, I think), I can see that DF does not nessecarily have a complexity in the millions.

Consider many of the standard functions: They usually do wwork and then return a much simplified result. malloc() either returns a failure or a pointer(success), leaving two states, and one of them is an error.

If, after any sufficiently important function fails, the calling function immediately quits, then you can reduce the complexity massively.

Example: I have a function that does a bunch of error handling by |ing a bit into the error code. as long as at least one of the 18 20(just noticed two I missed) conditions are true, the error code is nonzero, and the function quits before any real code occurs. Technically, the same can be accomplished by reducing it to a single massive error test, but that itself would add complexity beyond what it does, split into individual tests.

Additionally, if all error checks are made initially, and each one quits on failure, each one only adds one possible path. If a failure in one of those causes the calling function to also quit early, it has again greatly reduced the complexity.


So, if it is structured properly, DF's complexity can be much lower that you might assume. Though I don't think I ever wrote anything above or likely even close to ten thousand lines(probably not much past two thousand, and plenty of that whitespace or copied code), so I might be underestimating the possibilities.

[Mike Mayday]

Look what you've made me do:
http://www.bay12games.com/forum/index.php?topic=44585.0

[Urist McDepravity]

From my limited experience(Less then ten years, I think), I can see that DF does not nessecarily have a complexity in the millions.

Consider many of the standard functions: They usually do wwork and then return a much simplified result. malloc() either returns a failure or a pointer(success), leaving two states, and one of them is an error.

A game like DF has ALOT of rand() usage. Random world, random characters, random events. There's a reason why most of games stick with some predesigned world and predesigned scenarios/characters.

[Riemann]

Hi, how’s it going chmod? Feeling ok lately? Been off the dried frog pills?

See, the thing is that I've been coding for about the same time as you posted above - though not as a freelancer, have tended to work as a full timer at a couple startups and lately a very large company - and I have no idea what point you are trying to communicate here.

When you got to the "tl;dr" you should have spent a little less time on the stale attempt at wit and a little more time thinking about why you are posting this at all.

[Footkerchief]

Hi, how’s it going chmod? Feeling ok lately? Been off the dried frog pills?

See, the thing is that I've been coding for about the same time as you posted above - though not as a freelancer, have tended to work as a full timer at a couple startups and lately a very large company - and I have no idea what point you are trying to communicate here.

When you got to the "tl;dr" you should have spent a little less time on the stale attempt at wit and a little more time thinking about why you are posting this at all.

I can't tell whether you've seen the thread that this one is a response to: http://www.bay12games.com/forum/index.php?topic=44484.0

[Jifodus]

I don't trust the usefulness of Cyclomatic Complexity, so you lost me there.

Also, humans can store the logic graph in their heads, it's just compressed using an imperfect compressing algorithm.

To me, I see DF more as a continuous function, a fractal, or a recurrence relation that has outliers that don't fit the normal graph of the function.  Each of those outliers are the bugs, and should probably be handled or they can be left in for "personality".  Since there's a functional way to represent the logic graph, I'd easily be able to keep track of logic graph.

Of course, this may be just me looking at DF as a bottom up simulation, not a top down simulation.

[Warlord255]

From my limited experience(Less then ten years, I think), I can see that DF does not nessecarily have a complexity in the millions.

Consider many of the standard functions: They usually do wwork and then return a much simplified result. malloc() either returns a failure or a pointer(success), leaving two states, and one of them is an error.

A game like DF has ALOT of rand() usage. Random world, random characters, random events. There's a reason why most of games stick with some predesigned world and predesigned scenarios/characters.

While the complexity ramping does loom rather dangerously (just look at how INSANE the bugs in the game are), I'm inclined to agree on this point; DF's reliance on random generators to produce its complexity helps alleviate that problem immensely.

It is ostensibly easier to fix the algorithm that produces mountains than to manually fix each offending mountain.

[chmod]

Hey look! Replies!

I wrote it because I wanted to. I think therefore I am etc... etc... It wasn't an attempt at wit, it was trying to give people some insight into what a labor of love software project is, and a contrast against traditional corporate development.

As far as complexity goes, you can either "believe" in it or not, but that's like saying you don't believe in summation, or the cosine function. It just doesn't make sense.

I was simply pointing out that if you take of any one of our paused forts in the fourth year as a snapshot of a state-machine, then you realize no human could run from 1 frame to the next purely in their head without making mistakes, you would have to write something down at some point, thus proving my point. This doesn't have anything to do with cyclomatic complexity I realize. However the cyclomatic complexity is obscenely high to run a simulation that we can't hold in our heads. No matter what language it was written in, you can't have that many decisions going on without tons of loops, variables, states, structures etc...

Even if the code is elegant, it is still complex. I was just reminding people that when we flirt with complexity of this magnitude, regression risk is high. And at the same time, telling people not to worry because that complexity is what is so appealing.

[zwei]

Behold, glory of modularization! Awe at automated testing! Detest elegant code! Burn clever tricks, burn them all with fire! Embrace analysis! Write design documents!

It is not that making complex software is hard, it just incorporates many things that coders hate, so they do it their way. And it ends up ugly.

[DeathOfRats]

Y'know, Zwei, I sort of agree with you, I sort of don't.

Yes, making good complex software incorporates many things that coders hate, and that means often they don't get done and, as you say, it gets ugly.

However, complex software is, by it's very nature, well... complex. And that means that, yes, it is hard. It is less hard if you do follow proper procedures to help you deal with all that complexity, but that doesn't make it easy.

Also, as an aside, we must have different definitions of what elegant code is, because I find it doesn't at all fit with the spirit of the rest of the things you mentioned.

[Footkerchief]

Also, as an aside, we must have different definitions of what elegant code is, because I find it doesn't at all fit with the spirit of the rest of the things you mentioned.

Yeah, I was wondering about that too.  Elegant for its own sake, maybe, but usually when I see code get called elegant, it's because it's concise, clear, and logically intuitive, not because the programmer spent hours reducing it to its most arcane essence or something.

[zwei]

Also, as an aside, we must have different definitions of what elegant code is, because I find it doesn't at all fit with the spirit of the rest of the things you mentioned.

Yeah, I was wondering about that too.  Elegant for its own sake, maybe, but usually when I see code get called elegant, it's because it's concise, clear, and logically intuitive, not because the programmer spent hours reducing it to its most arcane essence or something.

It is like marrying framework, at first it makes everything better. Everything gets perfectly structured, fit in logically, adding more of same is easy and simple. Neat and cool.

But then you want to to something a bit different. Something that requires you to bend rules a bit. So oppression starts. It becomes unwieldy, starts to get in the way. It can make you life hell if you want as little as extra variable passed through.

Soon you will realize that each change will affect pretty much everything connected. Getting rid of it means that you will have to rewrite damn lot of code.

So there comes despair:
 * You continue adding new features to core of your clever code as needed in order to be able to use it for new features, coping with fallout it produces, adding to mess.
 * Or you can set up new miniframework for special cases, adding to mess.
 * Or you can try to rewrite from scratch, with everything you learned in last iteration.
 * Or all of this at same time.

You weep.

Simple clever idea to, for example, maximize reuse, can make development hell on earth.

[Shades]

Behold, glory of modularization! Awe at automated testing! Detest elegant code!

To me elegant code is all part and parcel of sensibly modularised and unit-tested code.

Also, as an aside, we must have different definitions of what elegant code is, because I find it doesn't at all fit with the spirit of the rest of the things you mentioned.

Yeah, I was wondering about that too.  Elegant for its own sake, maybe, but usually when I see code get called elegant, it's because it's concise, clear, and logically intuitive, not because the programmer spent hours reducing it to its most arcane essence or something.

Again I find concise, clear and logically intuitive code is reduced to it's most arcane essence. Which ties in with this:

Make everything as simple as possible, but not simpler.

Even if he never said it it is a good view to have.

[Nexii Malthus]

Humans are incapable of storing a logical graph of that size in their heads.

Not entirely true. We are just beginning to stare at the tip of an iceberg in understanding our human brains. Take Savants, they are capable of extremely rare abilities, such as one person who had a record to remember the digits of PI down to the last 45,000th number precisely. Or twins who had the ability to count the number of items thrown chaotically. Say dropping a box of 142 matchsticks violently, they could take a nearly instantenous count of how many matchsticks were lying on the ground.

Our brain is indeed our most extremely powerful tool. We just need to find ways to unlock the keys of that potential. Also, the savants I described weren't just freaks of nature or abnormal retards, no, they were simply normal healthy beings.

Ultimately it is up to you how you find those keys, or if you ever will in your lifetime (To say that if you even bother to do empirical tests).