Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Bumber]

Notice: PyLNP does this in a better way. You'll probably want to use that instead, if you have it.

I finally finished a basic working keybind patcher. It seems to be working okay. Requires Python 3.

Code: (keybind_patcher.py) [Select]

#------------------------------------------------------------------------------------------------------------------------
#---------------------------------------- Bumber's Dwarf Fortress Keybind Patcher ---------------------------------------
#------------------------------------------------------------------------------------------------------------------------
# keybind_patcher.py <1> <2> (<3>)
# Parameters:
#   1: The location of the new interface.txt to patch.
#   2: The location of the old interface.txt to source from.
#   3: Optional string "REPEAT_FAST" or "REPEAT_SLOW", will overwrite all of the other before patch.
#
# Notes: Missing lines in old interface.txt do not cause unbinding of a key in the new.
#        To remove a binding, comment-out the token like so: (KEY:X) or (SYM:1:X)
#        The old interface.txt can be left mostly empty with only the desired changes.
#        The repeat status of bindings in old interface.txt overrules the optional repeat overwrites.
#        After the merge, the new interface.txt will be renamed to interface.txt.bak,
#        and the merged interface.txt will replace it. If this fails (i.e., when interface.txt.bak already exists),
#        the merged interface.txt will be left as interface_merged.txt.
#
# Contributors: Bumber
# Last Modified: 5/27/2016
#------------------------------------------------------------------------------------------------------------------------
#--------------------------- An example of a Windows batch script I use to streamline things: ---------------------------
#------------------------------------------------------------------------------------------------------------------------
# @rem This batch file and the interface changes file should sit outside the new Dwarf Fortress folder
# @set /p newfolder=Enter the name of the DF folder to patch (no quotes): %=%
# keybind_patcher.py "%~dp0%newfolder%\data\init\interface.txt" "%~dp0interface_changes.txt" "REPEAT_SLOW"
# @pause
#------------------------------------------------------------------------------------------------------------------------
import re #regex
from sys import argv #parameter list
from os import rename #for swapping the file names afterwards

def main():
#Regex patterns
reg_bind = re.compile(r"\[BIND:(\w+):(\w+)\]") #1:Command, 2:Repeat
reg_token = re.compile(r"(\[|\()((?:(?:SYM|BUTTON):.:.+)|(?:KEY:.+))(?:\]|\))") #1:Bracket ('[' else remove), 2:Interior
reg_simple = re.compile(r"\[((?:(?:SYM|BUTTON):.:.+)|(?:KEY:.+))\]") #1:Interior; new_config won't have commented binds

#---- Get all the parameters and file operations out of the way ------------------------------------------------------
force_repeat = "" #will replace all "REPEAT_SLOW" or "REPEAT_FAST"; old interface.txt overrules
if len(argv)<3:
exit("""ERROR: Insufficient number of arguments.
Syntax: keybind_patcher.py <new interface.txt> <old interface.txt> optional:<force_repeat string>""")
if len(argv)>=3:
force_repeat = argv[3]
if force_repeat!="REPEAT_FAST" and force_repeat!="REPEAT_SLOW":
exit('ERROR: force_repeat="{}"\nUse only "REPEAT_FAST" or "REPEAT_SLOW"'.format(force_repeat))
try:
old_config = open(argv[2],'r',encoding="cp437") #open old interface.txt for reading
except:
exit("ERROR: Unable to open old interface.txt at: {}".format(argv[2]))
try:
new_config = open(argv[1],'r',encoding="cp437") #open new interface.txt for reading
except:
old_config.close()
exit("ERROR: Unable to open new interface.txt at: {}".format(argv[1]))
try:
dir = re.search(r"(.*?)[^\\]*$",argv[1]).group(1) #the directory of new interface.txt
merge_config = open("{}interface_merge.txt".format(dir),'w',encoding="cp437") #open merged interface.txt for writing
except:
old_config.close()
new_config.close()
exit("ERROR: Unable to create interface.txt.merge at: {}\nMake sure directory isn't read-only.".format(dir))

#---- Build a dictionary from the old interface.txt: table[Command]=(Repeat,[(Bracket,Interior), ...]) ---------------
table = {} #bindings dictionary for old_config; key=Command, element=(Repeat,Entries)
entries = [] #token list; element=(Bracket,Interior)
command = "" #current binding; null string evaluates False
repeat = "" #repeat string of current binding

for line in old_config:
match = reg_bind.search(line)
if match: #if line starts new bind group
if command: #avoid first run or stray tokens
table[command] = (repeat,entries) #add completed bind group
entries = [] #clear list
command = match.group(1)
repeat = match.group(2)
else:
match = reg_token.search(line)
if match: #if line contains complete token
if match.group(1)=='[': #not a commented-out line
entries.append((True,match.group(2))) #add (Bracket,Interior) to the list
else: #commented-out to remove keybind
entries.append((False,match.group(2))) #add (Bracket,Interior) to the list
elif line.encode("cp437") not in [b'\n',b'\xef\xbb\xbf\n']: #LOG: ignore empty lines and encoding
print('Old Config: Invalid line: "{}"'.format(line)) #LOG: Line did not contain a token
if command: #final bind group, avoid stray tokens
table[command] = (repeat,entries) #add completed bind group
old_config.close() #done with old interface.txt

#---- Output the new_config group-by-group, filling in the extra binds as we go, ignoring as needed ------------------
entries = [] #token list; element=Interior
command = "" #current binding; null string evaluates False
command_in_table = False #True if Command exists as a dictionary key
repeat = "" #repeat string of current binding

print('',file=merge_config) #start with a newline
for line in new_config:
match = reg_bind.search(line)
if match: #if line starts new bind group
if command: #avoid first run or stray tokens
if command_in_table:
output_merge(entries,table[command][1],merge_config) #output the merged tokens
else:
output_merge(entries,[],merge_config) #just output the new_config tokens
entries = [] #clear list
command = match.group(1)
command_in_table = command in table #check if the bind group is in the table
repeat = match.group(2)
if force_repeat and repeat!="REPEAT_NOT":
repeat = force_repeat #replace "REPEAT_SLOW" or "REPEAT_FAST"
if command_in_table:
repeat = table[command][0] #prefer old Repeat, regardless of force_repeat
print("[BIND:{}:{}]".format(command,repeat),file=merge_config) #[BIND:Command:Repeat]\n
if repeat!=match.group(2): #LOG
print("[BIND:{}:{}] -> {}".format(command,match.group(2),repeat)) #LOG: BIND repeat was changed
elif command_in_table:
print("[BIND:{}:{}]".format(command,repeat)) #LOG: Beginning of new changed BIND group
else:
match = reg_simple.search(line)
if match and command: #if line contains complete token and not stray
entries.append(match.group(1)) #add token Interior to the list
elif line.encode("cp437") not in [b'\n',b'\xef\xbb\xbf\n']: #LOG: ignore empty lines and encoding
print('New Config: Invalid line: "{}"'.format(line)) #LOG: Probably a comment
if command: #final bind group, avoid stray tokens
if command_in_table:
output_merge(entries,table[command][1],merge_config) #output the merged tokens
else:
output_merge(entries,[],merge_config) #just output the new_config tokens
new_config.close() #done with new interface.txt
merge_config.close() #done with merge interface.txt
try: #replace the unmerged config with the new one
rename("{}interface.txt".format(dir),"{}interface.txt.bak".format(dir)) #backup the unmerged interface.txt
try:
rename("{}interface_merge.txt".format(dir),"{}interface.txt".format(dir)) #replace with merged interface.txt
except:
print("Warning: Failed to rename interface_merge.txt to interface.txt")
except:
print("Warning: Failed to rename interface.txt to interface.txt.bak\nDoes it already exist?")
re.purge() #done with regex patterns
return

#---- Helper functions ---------------------------------------------------------------------------------------------------
def output_merge(new_list,old_list,out_stream): #output the merged tokens
found = [] #intersected tuples of new_list and old_list
not_found = [] #tuples of new_list not in old_list
intersect = set() #set of Interiors used to ignore changed entries of new_list
for tuple in old_list: #tuple is (Bracket,Interior)
if tuple[1] in new_list: #if Interior exists in both new_list and old_list
found.append(tuple)
intersect.add(tuple[1]) #add Interior to the intersection
else:
not_found.append(tuple)
for interior in new_list: #print unchanged tokens first so checking file differences is more sane
if interior not in intersect:
print("[{}]".format(interior),file=out_stream) #"[Interior]\n"
for tuple in found: #print the changed tokens next
if tuple[0]: #token is valid
print("[{}]".format(tuple[1]),file=out_stream) #"[Interior]\n"
print("    [{}] Already exists".format(tuple[1])) #LOG: This line could be removed from old_config
else: #token is commented-out
print("    ({}) Keybind removed".format(tuple[1])) #LOG: Removed entry
for tuple in not_found: #lastly the tokens unique to old_list
if tuple[0]: #token is valid
print("[{}]".format(tuple[1]),file=out_stream) #"[Interior]\n"
print("    [{}] New entry".format(tuple[1])) #LOG: New entry
else: #token is commented-out
print("    ({}) Was not found".format(tuple[1])) #LOG: Bind did not exist to remove
return

if __name__ == "__main__": #just in case this program were to be called by another for whatever reason
main()
Future improvement: Work this into the output_merge function:

Code: [Select]

found = filter(lambda x: x[1] in new_list, old_list)
not_found = set(old_list).difference(found)
Original Post:

Sometimes when DF updates, new keybindings or settings are added to the init files. This makes using outdated custom inits risky, and making the new changes manually is bothersome.

The easy solution would be a utility that patches in your customizations to each new version. It could have a GUI (sort of like LNP has for some init settings) that lets you set keybindings and settings, and builds it into a custom settings file. The core part of the utility is that it can take said custom settings file and apply those changes to the various places they need to be. Bonus features would be to warn the user of conflicting or unset bindings, bad values, recent changes, etc.

If there's already a DF-specific utility like this, great, tell me. If there's a generic utility that could be used for the core purpose, okay. If someone wants to make it themselves (and possibly integrate it into LNP,) go ahead. Otherwise, I'll try to get something simple done (probably won't have a GUI) in Python, if ever I have the free time.

[Meph]

WinMerge or Beyond compare.

You open the new file, open the old file, and they show you the differences, making manual updates extremely easy. I use Beyond Compare to merge mods or update mods to new versions.

[Bumber]

WinMerge or Beyond compare.

You open the new file, open the old file, and they show you the differences, making manual updates extremely easy. I use Beyond Compare to merge mods or update mods to new versions.

I've been using diffchecker.com, but the keybindings move around in the file when you change them in-game. It makes things a bit complicated.

Ideally, I was thinking something like an automated search and replace with a whole bunch of terms at once. Maybe with regex's, if I could be bothered to figure them out.

Linux has Sed, but I think that does a full file pass each search term, which might not scale to our purposes. Then again, IDK, maybe it's actually optimized.
Edit:
Specifically, one of the issues of just using a merge is that it can't tell which in the old file are your changes, and which are outdated default values. In version 0.42.01 the artifacts menu binding changed from 'l' to 'L'. If you did any sort of automated merge, you'd have both the artifacts menu and the new locations menu bound to 'l', which is undesirable.

[klassekatze]

I don't think you need to worry about scalability here. I'd say it literally takes more time to render the sed output in your console than it does to search that file a hundred times. Not because sed is special (though I'm sure it is efficient) but because searching a text file is just that cheap at this file size.

Anyway something like this should help some. This is a node.js script, because I am not super familiar with python or the like. If you are/have used sed then you should be able to use this fine. Just install node, save it as dfinitmigrate.js in the init folder, and run 'node dfinitmigrate.js OLDINIT.txt NEWINIT.txt' in the console in that folder - for each setting in the new init, it will check the old init file and copy its line if it has a different value for it. It will not copy old settings if they don't even have a line in the new file, though.

Code: [Select]

var fs = require('fs');

//node dfinitmigrate.js OLDINIT.txt NEWINIT.txt

var len = process.argv.length;
var oldc = process.argv[len-2];//second to last argument
var newc = process.argv[len-1];//last argument

var OLD_CONF = fs.readFileSync(oldc).toString();
var NEW_CONF = fs.readFileSync(newc).toString();

var regex = /^\[([\w]*):([\w]*)\]/gm

//var arr = regex.exec(NEW_CONF);//.match(regex);

var OLD_CONF_TABLE = [];
var NEW_CONF_TABLE = [];

var myArray;
while ((myArray = regex.exec(OLD_CONF)) !== null) {
OLD_CONF_TABLE[OLD_CONF_TABLE.length] =
{
'full':myArray[0],//'[SOUND:NO]',
'key':myArray[1],//'SOUND',
'val':myArray[2],//'NO',
};
}
regex.lastIndex = 0;
while ((myArray = regex.exec(NEW_CONF)) !== null) {
NEW_CONF_TABLE[NEW_CONF_TABLE.length] =
{
'full':myArray[0],
'key':myArray[1],
'val':myArray[2],
};
}
console.log(OLD_CONF_TABLE.length);
console.log(NEW_CONF_TABLE.length);

function oldhaskey(key) {
    var length = OLD_CONF_TABLE.length;
    for(var i = 0; i < length; i++) {
        if(OLD_CONF_TABLE[i].key == key) return i;
    }
    return -1;
}

var new_copy = NEW_CONF;
for(var i = 0; i < NEW_CONF_TABLE.length; i++)
{
var old = oldhaskey(NEW_CONF_TABLE[i].key);
if(old != -1)
{
if(NEW_CONF_TABLE[i].full != OLD_CONF_TABLE[i].full)
{
new_copy = new_copy.replace(NEW_CONF_TABLE[i].full,OLD_CONF_TABLE[i].full);
console.log('migrating value of '+OLD_CONF_TABLE[i].full);
}
}
}
fs.writeFileSync('./MIGRATE.txt', new_copy);



[milo christiansen]

I have a Rubble addon that "prepares" a new DF install for use. It uses the Rubble raw merge engine to patch the init files with my favorite settings.

The best part? It's totally automatic, just install Rubble and apply the addon with independent apply mode.

I included the addon with the standard Rubble distribution, not because I expect others to like my settings, but because that way others have something to modify with their own settings instead of needing to write their own system. (see "Util/Milo's Settings")

[Bumber]

Anyway something like this should help some. This is a node.js script, because I am not super familiar with python or the like.

If I'm understanding the code correctly:

Code: [Select]

1. It loads the tokens from both files into arrays (new_conf and old_conf) with each entry consisting of [full,key,val].
2. Creates a copy (new_copy) of the new_conf. For each entry in new_conf:
    If a similar key from new_conf with different properties exists in old_conf, search and replace all instances of the new entry with the old one in new_copy.
3. Write new_copy to MIGRATE.txt.
Variable 'val' never seems to be used. The format of keybindings is:

Code: [Select]

[BIND:SEC_SELECT:REPEAT_NOT]
[SYM:1:Enter]
[SYM:1:Numpad Enter]
[BIND:SELECT_ALL:REPEAT_NOT]
[SYM:1:Enter]
[SYM:1:Numpad Enter]Which doesn't look like it's going to play well if you try to change just one of these bindings. It will replace all instances of [SYM:1:Enter] in the file.

I have a Rubble addon that "prepares" a new DF install for use. It uses the Rubble raw merge engine to patch the init files with my favorite settings.

The best part? It's totally automatic, just install Rubble and apply the addon with independent apply mode.

Any idea of how the merge handles keybindings? If I want to remove a keybinding, e.g.:
[BIND:A_MOVE_NE:REPEAT_FAST]
[SYM:0:9]
[SYM:0:Numpad 9]
[SYM:0:Page Up]Could I do so without it being ignored as a missing entry?

[milo christiansen]

Any idea of how the merge handles keybindings? If I want to remove a keybinding, e.g.:
[BIND:A_MOVE_NE:REPEAT_FAST]
[SYM:0:9]
[SYM:0:Numpad 9]
[SYM:0:Page Up]Could I do so without it being ignored as a missing entry?

No errors are produced for missing entries.

All the merger does it parse rules to make a tree, then it loads your "source" raws into a tree shaped by the rules, and finally it applies changes to tags matched by both the rules and the source tags. You cannot use the merger to add new tags or remove old tags, only change existing tags.

If you have more questions you can look at the Rubble documentation ("other/Rubble Basics.md" has a section on the rule format) or you can look at BAMM (Button's Automated Mod Merger). Rubble's merger is basically an extended re-implementation of the BAMM raw merge engine.

[Bumber]

I completed a basic program in Python. OP updated.

[PeridexisErrant]

I completed a basic program in Python. OP updated.

Making me late to the party with PyLNP

The save/load function for keybinds (here) only saves changed bindings, always in the same order as the vanilla file.  (Like this).

If you want to remove a key from some binding, just write the binding without that key.  Vanilla bindings can be reapplied by an empty file (very efficient!).  Also very easy to read, and unless an update changes one of the edited bindings it's entirely safe between DF versions.

[Bumber]

Making me late to the party with PyLNP

The save/load function for keybinds (here) only saves changed bindings, always in the same order as the vanilla file.  (Like this).

If you want to remove a key from some binding, just write the binding without that key.  Vanilla bindings can be reapplied by an empty file (very efficient!).  Also very easy to read, and unless an update changes one of the edited bindings it's entirely safe between DF versions.

Cool. It's certainly a lot less messy than mine. It probably wouldn't add new vanilla bindings to modified groups, but that doesn't happen very often.
The main idea behind my program was that bindings could be easily removed by commenting them out like you can with aquifers. Turns out interface.txt doesn't like that (it drops the remaining bindings in the group), meaning mine doesn't support a simple upgrade of an existing binding file (and thus in-game assignment) like yours might. At least I managed to teach myself regular expressions.