Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Michael]

One popular request is for an officially supported API to allow scripting or communication between DF and another program.  It was put on Eternal Suggestions almost a year ago and is sitting at 5 votes.  While that is not high in general, it's one vote more than the most popular of my own suggestions, and despite the fact that Toady One has more or less condemned the idea.

The problem is easy to understand.  There is pent-up demand for a game that is DF at the back end, but with a completely re-done user interface, and an API would allow creation of such a beast.  Toady fears this would allow someone to effectively become a co-author, stealing all new users who would otherwise grudgingly learn DF on it's own terms.

One simple way to allow some API benefit while minimizing the risk, is to limit the API in a simple way.  Plugin access to game data would normally be blocked, but the plugin could at any time prod DF into making the Manager do some office work.  This work would have no direct effect on the game, but once the job is finished, the game would be frozen and the plugin would be allowed most of the read and write access the player has.  Only once the plugin yields will time continue in the game.

Since an office needs to be built before the Manager can do anything, a player would need to learn basic skills in raw DF (how to create furniture and indoor space) before he can turn over control to the plugin of his choice.  And the time lag would make a full wrapper impractical even late game. 

But it should be good enough to allow for private implementations of "Standing Production Orders", the current #2 suggestion.

[sockless]

The reason is about risk, but not for the reason you say.

Toady doesn't want to implement an API for this reason:
Since the game is in early alpha, he is bound to make changes to almost everything that's in the game right now. If he makes an API, the chances are that he's going to have to change it quite a bit in the future, this would break all current APIs, which would annoy a lot of people and many would just stick with the old version. It's already a problem and we don't even have an API, this problem is in the form of DFhack and co., these all broke at 31.13 when Toady changed compiler, Runesmith hasn't even been fixed yet.

[Michael]

If he makes an API, the chances are that he's going to have to change it quite a bit in the future, this would break all current APIs, which would annoy a lot of people and many would just stick with the old version.

Only if the API is designed lazily.  An API that simply gave out the pointers to major data regions and let the plugin self-serve would of course pin Toady down severely.  (It also would make it trivial for the plug-in to break the rules I've proposed to prevent a DF-wrapper).

But a higher-level API, where information is given as text strings rather than internal code, could be stable so long as the game maintains a fortress mode.

ie:

Code: [Select]

    ws = find_a_workshop("carpenter");
    if (ws)
    {
          add_workshop_job(ws,"larch","bed");
    }

Note that so long as the wrapper is prevented, there is no need to rush and provide API access to everything.  Toady's fear, as I understand it, is that new users will abstain from trying new features if they cannot access them through their favorite wrapper -- thus pressuring him to rush out new APIs and then lobby wrapper authors to use them.

[NW_Kohaku]

Well, let me start by throwing this in here:

Anyway, yes, this has been suggested before and Toady has made a couple direct responses.  Any supporters of the idea should definitely read these:

Spoiler: response (click to show/hide)

Somebody reported a post in this thread...  it didn't really matter which one.  A bit of chilling would be good.  There are a few people that have failed to chill.  There are quite a few statements here I'd respond to in person myself, and some very sharply, but internet discussions are more fragile and more prone to derailment.  So I'll try to stick to what seems to be the core of the matter.

Which for me (and granted, I'm picking a recent post -- I've read the thread, but don't remember when this came up first in this thread) is along the lines of what Jamuk said.  I'm being asked to assume risk.  I don't accept assessments of the risk as extraordinarily slight -- I've seen no sound basis for those assessments here.  I also don't think the currently proposed licensing/obfuscation method of putting the cat back into the bag would work -- with a principal downside being that it requires me to turn into an asshole who, after initially sanctioning the process, breaks everything that people have been working on.  Publishing an API and then trying to reel it back in just seems completely unworkable and irresponsible to me.  So the point is to assess risk upfront and then make a decision I can live with.

And right now, I have no idea what would happen.  This isn't the same as leaving my math job or adding the Z coordinate.  The first was a decision I had to make personally for my quality of life, and the second was a calculated risk which I didn't really have strong misgivings about (though some people are still pro-2D version, I think I can get most of them what they want over time in terms of gameplay difficulty, etc., though it'll never feel like it's quite on that rail again most likely).  Do I accept that third parties could improve the interface?  Of course.  Do I think I could do better than a third party myself, even over time?  No, not at all, though I can certainly improve on what I've got now to a large degree even without external input (which I, of course, love to receive).  So, initially, there's some attraction to the third party interface idea.

However, reflecting on the position I'd be in, there are things not to like about it.  How many threads were there about broken utilities when this version came out?  If more than half the player base comes in off a third party interface (and given how much the current interface sucks, and how much it is a source of first time downloaders dropping the game, this is not only imaginable, it is very, very likely), how would it be if it broke at each release?  There's no way to mitigate that without my direct involvement -- imagine a release down the line where you can suddenly move dwarven armies around on the world map, with a tactical view and various options.  That interface can't write itself, and it wouldn't be a quick patch, though certainly dedicated people, assuming that about whoever is maintaining the front-end at that time, can pull things together rapidly.  The pressure on me to work directly with them to get the interface out at the same time as the game itself would likely be immense and disruptive, given what little evidence we have from broken utilities.  That's not to say that I often get requests to work with utility writers (other than from the writers themselves, who I generally accommodate), but this would be at a different order of magnitude.

So what's the exact risk there then?  First, I don't want to work with other people.  So, assuming I don't do that, there's now a constant amount of pressure on me and a general disruption in the forums.  The latter could be mitigated with some announcements/guidelines, etc., but the pressure wouldn't go away.  Monetarily, it's impossible to say what on earth would happen, but if I supported the third party interface directly pre-release, I'd likely make more money, but I'd be unhappy.  If I didn't support it directly, but it was there, I'd still likely make more money, but I'd be unhappy.  That's not to say that I don't want to grow the audience and allow more people play the game, but I want to do it in a way with which I feel satisfied, even if that ends up being slower or just plain worse than a hypothetical third party alternative.  If someone, as a potential donor, thinks that's unreasonably selfish, that person shouldn't send me any more money than he or she thinks I've deserved for his or her enjoyment, just like everybody else.  Despite my dedication to this project, I'm unwilling to sacrifice my enjoyment of working on it for anything, including its quality or even its future release if it comes down to that.  That should be plainly obvious.  I'm not a slave.  Of course I do some things I don't like working on, with the interface and more.  I fulfill requests for features I'm not going to use myself.  However, again, the current proposal is of a different order of magnitude, with the potential of ruining it for me, and the end result is completely uncertain.

Given what I've seen here and there, it seems like a full third party interface might develop even without my involvement (rather than the various utilities we have now), but in that case, despite the same issues that might come up, at least it won't be a situation of my own creation that I feel a strong obligation to deal with, although the pressure would still be there.

Incidentally, in the proposed model, I still don't think I'd be able to bring code back in to the core project -- the first Kobold Quest port attempt showed that people bringing in code don't necessarily check or care about the licenses of the code they are swiping, and I can't assume that responsibility for them (it has stuff from Wine which shouldn't have been there).  Problems come up here with tilesets and so on too -- I know there've been a few cases where a contributor used somebody else's art and then it was put in a game, causing some legal trouble, though I don't have them at my immediate disposal.

I may have missed a lot, both problems I'm not thinking of and ways to handle them, since this gets complicated and is, due to lack of information, somewhat unspecified, but that's what I've got right now.

Spoiler: followup (click to show/hide)

I tried hard to put myself in your shoes and see things from your point of view, but apparently I failed to understand one thing (correct me if I'm wrong): the general atmosphere of the forums and feedback you receive from players is an integral part of your happiness. I was imagining that you could just let the mess settle down by itself after every release, but since I am not this forum's moderator, I can't really tell how it feels to have a big mess of threads about stuff not working.

It generally sucks to have a bunch of people complaining that things don't work, especially when there's something I can do about, that is, work with the interface modders more closely, having in this scenario sanctioned third party interfaces and released a public API.  It's the pressure that would make me unhappy.  It's good when the forum people are happy, but that wasn't what I was getting at so much.

When you speak of pressure, are you picturing direct pressure? I have a hard time imagining people screaming at you on the forums, except maybe for the occasional moron that can be ignored. Wouldn't people rather be putting pressure on the third-party developers to update their interface ASAP? But maybe you'd feel pressure nevertheless because you dislike seeing that people cannot play the game.

There would be direct pressure on me to work with third party developers -- this isn't about screaming.  This thread itself is direct pressure to work with third party developers (or at least get the ball rolling).  That isn't to say I disagree about you posting it.  I'm simply stating what it is, and I think that yes, there'd be a lot more of this, a lot more I'd have to reply to, and a lot of...  pressure.  Additionally, I'm not just talking about external pressure, in fact, principally not -- having created the situation, I would feel some obligation to make it run as smoothly as possible, and I would rapidly become irritated by this, because it's some of my least favorite kinds of work.

One thing I'd like to point out, though, is that you seem to think that a single interface, "The front-end", would come out of third-party development. Even though that's possible, open-source usually seems to bring diversity (look at the number of linux distros), especially in a single-player game where people don't have to use compatible clients, so they can play together. So I think we'd likely see an improved ascii client, a 2d tiles client, an isometric client, a 3d client, and maybe several of each one. In my view it would distribute the uproar, so that at least not everybody would be complaining about the same thing  . But I don't know if you see that as a relief.
[emphasis added]

He he, of course not.  It's worse actually -- more parties to work with.

Lastly, I may have a good way of mitigating the breakage of third-party clients at every release, without requiring your direct involvement. Assuming that you develop your own open-source interface at the same time as your closed-source server, you would likely publish your interface changes to a public SVN server. You would probably do it incrementally, at the rate of one or several commits a day - akin to backing up your work whenever you've made a step. In that case, you can bet that enthusiastic developers will watch those changes keenly, to see what's going on with the interface. This means that third-party client makers will already know that moving "dwarven armies around on the world map, with a tactical view and various options" is coming for the next version (from your very detailed dev notes), so they'll have time to plan which kind of interface they want to provide for that. And since they'll see the necessary code changes evolve gradually, because you'll always need an up-to-date interface for bugfixing, they'll be able to follow closely your development. They might have to adapt to your last minute changes, but otherwise I believe that the days of bugfixing prior to release could be enough for some of them to finalize their clients on time.
Now of course, you have to feel comfortable with that way of working.

Yeah, this somewhat mitigates breakage with those third parties that are active and approaching the situation with some dedication, but the trade-off of working with my day-to-day changes under that sort of scrutiny is definitely a minus for me, and I don't think it could really happen in a vacuum without questions about changes and so on, or without me feeling like I have to provide some sort of documentation as I add things.  I don't know if that sounds like quibbling, but I can't easily overstate how much I hate managing code or collaborating on code or anything along those lines.

Additionally, I think breakages would still happen, especially when staff leave projects hanging (work obligations, got tired of it, vacations, etc.), and it wouldn't change the fundamental situation I'm imagining, although in the case of project abandonment or delays with staffing, where there isn't pressure to work with the (missing) people in question, it's more the pressure to incorporate changes back into the base API, which as I stated in my previous long post, I wouldn't be able to do without potential legal problems.


I don't mean to write as if I'm sure about any of this, but once I drop this bomb in my project, I can see every day becoming "oh man, what a goddamn hassle" instead of what I've got now, which has less of that.  Again, I think the OP is justified in starting this thread, so I'm not raising that as an issue, but I've also spent some hours in here, reading posts and composing these replies.  That stuff just starts to magnify, and I'll either end up ignoring people I don't feel like I should be ignoring, or not ignoring people to the detriment of the core game itself and my own well-being.

What I would like to hear from Toady is what kind of graphics he has in mind for when he gets to the interface overhaul. That way we could at least prepare some sprites ahead of time and have nearly complete graphics sets as soon as the version with the new interface comes out. I personally waste a lot of my free time doodling in GIMP, so I might as well do something useful.

Depending on the stage of the interface overhaul, ultimately I'm going to be support 2D tilesets (probably in dimensions of multiples of 4 because I'm lazy with image file headers).  So if you want to draw up some 32x32s or something, you won't be wasting your time, I think.  It should be fairly straightforward to support single z-slice isometric stuff as well, once I get that going, since I'd just have to change the print locations and print order, though transparency decisions are probably annoying, and it's slightly more annoying to get multilayer isometric stuff going, since people are going to want various options about display there, so I don't really have a clear opinion on the future of isometric.  The support for a resizeable viewport/window is definite, but nothing has been decided on layering there (for instance, critters walking over grass tiles, that kind of thing -- people will want more and more out of this system, such as inventory and wound displays, and I'm not sure where lines will be drawn, or where it will bog down, anyway).

Also some earlier stuff:

Spoiler: part 1 (click to show/hide)

This has come up before, but I don't remember if I've actually entered into a discussion on it.  As it stands, without knowing much about it, my issues are as follows:

It seems like I'd have to publish all my class definitions to make this practical.  I don't anything about reverse engineering, but it doesn't seem far from opening up the source at that point.

The class definitions would also not document and otherwise support themselves, and there would be pressure on me to do so, which is not a one-time cost.

In fact, if a third party front-end becomes popular, I think I'd be forced into relationship or even a state of dependence on the third party that I'm not comfortable with.  And I don't think we can just gloss over version compatibility.  Ignoring it or handling it before releases -- either approach has ramifications for me.

Certainly players would benefit from having third party front-ends.  They'd benefit from open source and third party back-ends too.  If it means cutting myself out of my own game, I'm not going to support it.

I also don't know anything about scripting, and it's tough for me to say whether time spent learning a scripting language, implementing scripts and supporting third party front-ends is better spent adding modestly to the interface myself.

Spoiler: part 2 (click to show/hide)

"You break everything, we fix it, it's all good."

I don't think it's that simple.  Even the utilities that people use cause some grief when they are broken -- it would be far worse for a popular front-end.  If the majority of the player base has to wait after each release for third party fixes to a broken front-end that may or may not even happen, it puts a lot of pressure on me to resolve the issue by working more closely with the third party, a completely unknown entity that happens to maintain the front-end most people use.  It would be difficult for me to retain my independence once in that position.

I think somebody mentioned in an email discussion I had that in that case I should just incorporate some of the more popular aspects of third party interfaces back into the main release in order to avoid forming that kind of tie and to avoid serious bumps in playability for front-end users after each release, but I don't know the legal ramifications of that, and my first impression is that I'd be setting myself up for trouble, though some major publishers seem to do this without issue from what little I know about it.

I can vaguely see how something could be set up to automatically prepare a lot of this now, but it still seems like the functions etc. that are created would need to be documented to be useful, especially if the front-end needs to be patched quickly.  If variables are removed or changed around, it seems like the incompatibility for front-ends relying on a previous version of the automatically generated system could be pretty severe.

If you want to make an argument for an API, this is the argument Toady has made against it that you have to overcome.

You have to convince him that the risks are mitigatable, and that it would be fairly simple to pull this sort of API off.  It probably wouldn't hurt to either create or link to instructions on how to create useful APIs, since Toady may not have any experience with creating an API.

It would probably also be best to couch your argument more in the terms of how people already wait upon programs like Dwarf Therapist to play DF, and cannot play DF without it, and how the API can be built without needing maintenance from Toady and makes the outside programs work with DF better, resulting in less of a headache for him, and not more of one.

Toady also likes organized threads.  Rather than a running argument back and forth in a thread, try to put everything you really need to know about the thread in the first post, and just link to the important arguments in the thread if need be.

Good luck, and godspeed.

[Michael]

Well, let me start by throwing this in here:

(... extended footkerchief quote elided by Michael ...)

If you want to make an argument for an API, this is the argument Toady has made against it that you have to overcome.

Actually, I read that before I even started this thread.  It's just that I think his point about being forced to keep adding API features at the same rate of new native-UI commands only applies if people use the plugin interface to create DF-wrappers -- and the time lag would prevent that.

It would probably also be best to couch your argument more in the terms of how people already wait upon programs like Dwarf Therapist to play DF, and cannot play DF without it, and how the API can be built without needing maintenance from Toady and makes the outside programs work with DF better, resulting in less of a headache for him, and not more of one.

I'd hesitate to use that line of argument.  A lot of the existing debugger-based tools do things that my API would never provide, such as outright cheating.

I don't think it possible to make an API complete enough to eliminate entirely the inclination to hack DF's address space without raising the specter of a wrapper (and then most of Toady's complaints would become quite valid).  So it won't make DFHack go away.

But I'm not looking for an API that complete.  A limited one, with the time lag and a few simple commands, would be still be quite useful.

[NW_Kohaku]

But I'm not looking for an API that complete.  A limited one, with the time lag and a few simple commands, would be still be quite useful.

In that case, you should edit your initial post to make that very explicit to the sort of person who only skims most threads (and change the subject line to match).  You have to make this argument convincing to Toady, now, not necessarily us, and he has a lot of things to read.

You're using terms like "API", which is colored by the preconceived notion of the arguments that have taken place before this one. 

No, it's not really fair, but it's true that people pass judgment fairly quickly on these things, especially when they see key terms that they have already loaded with emotions like "API" is because of previous arguments.

You should clear off that first paragraph of the original post, it doesn't have much to do with the argument.  Toady knows what's already been on the ESV.  The second paragraph doesn't need to be there - you're arguing to us in that paragraph, not Toady.

If you want to argue for this on the basis of it not being capable of breaking anything else, you should do a quick talk about the specific problem this specific solution will solve, and then go about talking about how it would be both easy to implement and would not require updating and would make Toady's maintenance of the game easier overall.

[Draco18s]

Only if the API is designed lazily.  An API that simply gave out the pointers to major data regions and let the plugin self-serve would of course pin Toady down severely.  (It also would make it trivial for the plug-in to break the rules I've proposed to prevent a DF-wrapper).

But a higher-level API, where information is given as text strings rather than internal code, could be stable so long as the game maintains a fortress mode.

ie:

Code: [Select]

    ws = find_a_workshop("carpenter");
    if (ws)
    {
          add_workshop_job(ws,"larch","bed");
    }


Just FYI:
A high-level API is harder to code than a low level API.  While more flexible, it takes a larger amount of code, with very broad definitions in the external interface portion that need to be narrowed down to figure out the specific object or task that the general socket applies to.

When you start taking input as textual strings, rather than as something concrete then you add in the additional overhead of having to parse and compare strings (string comparison is orders of magnitude slower than integer comparison; although not likely to be a factor for DF where the major bottleneck is pathfinding and object tracking*).  And remember, custom workshops are possible.  And custom items, reactions, creatures....

*If 100,000 stones isn't slowing down your game, I want to know what kind of computer you have.

[devek]

Only if the API is designed lazily.  An API that simply gave out the pointers to major data regions and let the plugin self-serve would of course pin Toady down severely.  (It also would make it trivial for the plug-in to break the rules I've proposed to prevent a DF-wrapper).

But a higher-level API, where information is given as text strings rather than internal code, could be stable so long as the game maintains a fortress mode.

ie:

Code: [Select]

    ws = find_a_workshop("carpenter");
    if (ws)
    {
          add_workshop_job(ws,"larch","bed");
    }


Wrong on a few points. DF is very alpha and many aspects about how it will work are undecided and will change. It is impossible to design an api for what you are talking about without limiting the creative development of DF.

It comes down to this... all third party utilities are already possible. If we spent the time doing it, everything you could imagine the hackers know how to do. The specific example you give is actually done by the program in my signature and is an open source project. The question is, should the burden of compatibility fall on Toady or the hackers?

With the system the way it is now, the hackers have no trouble interfacing with new versions of DF. What you're asking is for that burden placed on Toady instead, and for no real gain. The only reason there isn't a third party interface, is because no one has spent the time to write one.

If you let it up to the community to vote on, I bet you they would prefer Toady continue to make DF a better game and adding new features than waste time dealing with other programmers.

(Also, DF can't technically queue a job for a larch bed.. only jobs for wood beds will do.)