Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2 3

Author Topic: Announcement Window, a Python announcement filter.  (Read 46852 times)

Algee

  • Escaped Lunatic
    • View Profile
Announcement Window, a Python announcement filter.
« on: October 27, 2015, 04:28:47 pm »

I recently tried using the announcement filter utility that came with the Lazy Newb Pack and was constantly annoyed by its problems (Not scrolling for new announcements being the biggest). So I started writing a python program that fixed those issues, this is what I came up with.

Spoiler: Screenshot (click to show/hide)

The program will give you a live feed of your announcements and combat reports, so you don't need to pause the game to read them. It should work on any version of python above v2.3.x.

There are two windows you can configure filters for, allowing you to separate important announcements from the never ending combat reports (or the less important stuff: "Urist mcHauler cancels Store Item in Stockpile: Item inaccessible."). The filters that it matches are loaded from a text file and can be edited however you want, as long as you have a basic understanding of regular expression syntax.

You can find the project on github: https://github.com/NuAoA/AnnouncementWindow

Spoiler: Changelog (click to show/hide)


I compiled a .exe for windows that you can find on the releases page. If you are on OS X or Linux you will need python to run this program (Which should already be installed on your os). I can't test the program on those systems myself, so please let me know if you discover any bugs/crashes.
« Last Edit: November 09, 2015, 09:31:23 pm by Algee »
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #1 on: October 27, 2015, 07:22:18 pm »

Tix should be mentioned as a dependency. What exactly is it used for, by the way?
Edit: It seems that Tix isn't available on some platforms, and the sources I checked recommend Ttk as a more up-to-date alternative (which I can confirm is available).
Edit 2: I got this to launch on OS X and fixed the menu bar (see the pull request I just made). Not sure about Linux yet.
« Last Edit: October 27, 2015, 07:44:52 pm by lethosor »
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

Algee

  • Escaped Lunatic
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #2 on: October 28, 2015, 12:29:40 am »

Tix should be mentioned as a dependency. What exactly is it used for, by the way?
Edit: It seems that Tix isn't available on some platforms, and the sources I checked recommend Ttk as a more up-to-date alternative (which I can confirm is available).
Edit 2: I got this to launch on OS X and fixed the menu bar (see the pull request I just made). Not sure about Linux yet.

Thanks for the help with OS X, let me know if you run into any more problems.

I removed Tix and ttk is now only needed for the font dialog. I might try and boot this up on my raspberry pi tomorrow to see how it works in debian.
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #3 on: October 28, 2015, 06:25:20 pm »

A couple things I was looking at:
* Any thoughts on allowing the number of panels to be adjusted? It looks fairly simple (removing panels on the fly might be a bit harder), and the configuration format ought to be able to handle it, but I don't know if there's another obstacle.
* It might be nice to have a built-in editor for filters.txt, and it would pretty easy to implement (PyLNP does this, IIRC). I don't think xdg-open is available or works on all Linux distributions, at least.
* Following those ideas, would you be opposed to moving the current menu options to a menu (like the "Options" menu currently present on OS X)? The current menu bar on Windows does look nice, but I feel like additional menu options would run out of space pretty quickly. (You could always add buttons for a few commonly-used options instead.)

(I'm able to work on these, but I wanted some input first.)
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

Algee

  • Escaped Lunatic
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #4 on: October 28, 2015, 08:07:20 pm »

Quote from: lethosor
* Any thoughts on allowing the number of panels to be adjusted? It looks fairly simple (removing panels on the fly might be a bit harder), and the configuration format ought to be able to handle it, but I don't know if there's another obstacle.

Yes I am thinking of doing that. I've actually written most of the code with this in mind, including my data structure for the information about filter color & visibility. I think the only changes I would need to make is on the frontend along with the window specific settings in settings.cfg. Ideally I would like to have the panels be dockable, so you can put them side by side or up/down in whatever configuration you desire, but that will get pretty complicated. I don't think removing them will be much of a issue, since the program just iterates over each window and sends them the new announcements. Currently you can effectively hide one of the panes by moving the sash bar to the edge of the screen. it saves the sash position when you exit, so if you wanted to only use one window you don't have to move it every time you launch the program.   

Quote from: lethosor
* It might be nice to have a built-in editor for filters.txt, and it would pretty easy to implement (PyLNP does this, IIRC). I don't think xdg-open is available or works on all Linux distributions, at least.

Yea that would be nice, but the file is available in the program directory. If I were to build a editor, I would probably concentrate the visibility settings for each panel & the colors into a single window, so you can see the list of regular expressions along with how they are configured to display. Currently, the window that lets you configure colors is hideous, and could use a lot of work. This is my first time programming with a GUI library, so every time i add something new (like the font dialog) I have to pour over the doc's and figure out how that widget works. So I've just been making stuff functional rather than making it look pretty.

Quote from: lethosor
* Following those ideas, would you be opposed to moving the current menu options to a menu (like the "Options" menu currently present on OS X)? The current menu bar on Windows does look nice, but I feel like additional menu options would run out of space pretty quickly. (You could always add buttons for a few commonly-used options instead.)

If you are talking about a file menu kind of thing for the menu bar, that's actually the next thing I plan to work on. I initially avoided that because I only had a single option (set gamelog.txt) but I have a few ideas for stuff to add to the menu. Like duplicating the window pulldown options into sub menus on the menu bar, options to remove the border and force the window to stay on top (overridedirect), and probably move all the settings.cfg options over there. 

Logged

PeridexisErrant

  • Bay Watcher
  • Dai stihó, Hrasht.
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #5 on: October 31, 2015, 10:57:23 pm »

PTW - this is already great, and it looks like ongoing development will be more reliable than it's predecessors.

I tried adding my utility to the utility folder (the .exe i've compiled with py2exe), and I get the following error when I try and open it through the LNP.

Weird.  I also get an error, and launching it from a .bat script works... except through PyLNP.

I tried compiling through Nuitka with --standalone (which produces real native executables), but got a bunch of Py3 import errors.  Can you see if the output of "pip install nuitka && nuitka run.py --standalone" has the same problem?
Logged
I maintain the DF Starter Pack - over a million downloads and still counting!
 Donations here.

Pidgeot

  • Bay Watcher
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #6 on: November 01, 2015, 06:04:46 pm »

PTW - this is already great, and it looks like ongoing development will be more reliable than it's predecessors.

I tried adding my utility to the utility folder (the .exe i've compiled with py2exe), and I get the following error when I try and open it through the LNP.

Weird.  I also get an error, and launching it from a .bat script works... except through PyLNP.

This was only through the binary, right? It looks like it's catching the Tcl/Tk DLLs extracted by PyInstaller. Utilities are launched as child process of PyLNP, so you're probably seeing its environment variables... and PyInstaller sets some environment variables (TCL_LIBRARY and TK_LIBRARY) to have them loaded from its temporary directory.

What *exact* Python version are you using? I see Python updated Tcl/Tk to 8.5.15 in 2.7.7; I *think* my builds were using 2.7.5, so if you're on the other side of the split, that's probably the reason it's complaining...

I've uploaded a recompile with Python 2.7.10 (which happens to be the version on my laptop); see if that one makes a difference.

My first choice would be to try another EXE builder. PyInstaller would be a good candidate here, since it already knows to set those variables - you might have to compile in single-file mode, however.

However, this is probably something that should be fixed in PyLNP anyway... I'm not sure if PyInstaller is doing anything like this on Linux or OS X (I can't check since my VMs are on my main PC, which I still do not have access to); but I could at the very least prepare a Windows-only 0.10c which deletes those two environment variables for spawned child processes. If the recompiled PyLNP works, I'll get right on that, because that'll be a more reliable option anyway...

TheBloke

  • Bay Watcher
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #7 on: November 03, 2015, 01:33:43 pm »

Finally got around to trying this and really like it.  Great job, this is definitely going on my list of always-run exes for DF.

I'd also like to add a +1 to the idea of allowing the user to specify extra panes, allowing them to display messages in any number of customisable divisions.

PTW - this is already great, and it looks like ongoing development will be more reliable than it's predecessors.

I tried adding my utility to the utility folder (the .exe i've compiled with py2exe), and I get the following error when I try and open it through the LNP.

Weird.  I also get an error, and launching it from a .bat script works... except through PyLNP.

I have this problem from PyLNP as well.  I can run AnnouncementWindow.exe fine by double clicking on it (no need for a BAT), but executing it from PyLNP gives me a bunch of errors in the log about not being able to find TCL:


The errors all relate to a temporary directory, and I've looked in my Temp folder and see a couple named as above, eg _MEI81642, containing files relating to PyLNP.  For example, containing LNP.ICO, PyLNP.exe.manifest, and then a bunch of Python DLLs, including tcl85.dll.

So it seems that an option for packaging Python EXEs on Windows is as archives which expand library and other files to a temp folder and then execute from there.  At least, this is one way Python executables can run on Windows - not a method seemingly used by AW, but used by PyLNP and at least one other Python app on my system (I see I have another _MEI* folder from some other app, not PyLNP - possibly Google Drive in fact.) 

Both PyLNP and AnnouncementWindow use TCL: AW uses version 8.5.2 and PyLNP is 8.5.15.  So the problem is that when AW runs from PyLNP it seems to include PyLNP's temp folder in its search for tcl*.dll, and look there first - before its own folder.  This causes it to fail on the package requirements, because it finds PyLNP's 8.5.15 tcl85.dll, and its rules specify that it requires only version 8.5.2 (package require -exact Tcl 8.5.2)  It's probably an environment problem, inheriting PyLNP's environment which specifies PyLNP's Temp folder at the start.

Maybe there's a setting you can specify that can change where the Windows version looks for includes, such on Windows it only looks for TCL in the same directory where AW.exe is found?  Or looks there first at least; ignoring any environment PATH.

Or, perhaps you can make the EXE the same way that PyLNP is made, such that it has tcl85.dll bundled in the Exe and it expands this into a Temp folder before running?  If it had its own temp folder it would presumably look there before any other Python temp folder.  That would also mean you could distribute fewer files, in the same way that PyLNP distributes only an Exe: with the DLLs and other files baked into that Exe.

Or perhaps the quickest fix for now would be to just change AW's packaging requirement rules to specify that TCL can be any 8.5 version from 8.5.2 onwards.  The logs show "package require -exact Tcl 8.5.2", so if this could be changed to whatever syntax accepts 8.5.* then it could use either its own 8.5.2 or PyLNP's 8.5.15 version of tcl.dll.  (This is of course assuming that nothing breaks with a higher 8.5 version! Hopefully not as the versioning suggests it should only be bugfixes from 8.5.2 -> 8.5.15.)

I suppose that might not be the best long term solution, as it requires that AW can always work with every version of 8.5.* TCL that might be found on a system.  But it should work quickly for now, if the fundamental environment search path issue can't be fixed as quickly/easily.

Hope that helps and thanks again for the great tool, really like it.
« Last Edit: November 03, 2015, 01:56:26 pm by TheBloke »
Logged

Algee

  • Escaped Lunatic
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #8 on: November 04, 2015, 05:49:37 pm »

PTW - this is already great, and it looks like ongoing development will be more reliable than it's predecessors.

I tried adding my utility to the utility folder (the .exe i've compiled with py2exe), and I get the following error when I try and open it through the LNP.

Weird.  I also get an error, and launching it from a .bat script works... except through PyLNP.

I tried compiling through Nuitka with --standalone (which produces real native executables), but got a bunch of Py3 import errors.  Can you see if the output of "pip install nuitka && nuitka run.py --standalone" has the same problem?

I am away from my computer for the week so I can't test these things out. If you can get a compiled exe working I would go with that for the time being.

A far as I remember I installed a 32 bit version of python 2.7.10 that I use to compile the exe, but I can give pyinstaller a try when I get back. It would be nice to get everything bundled into a single exe like pyLNP, which I don't think py2exe can do.
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #9 on: November 04, 2015, 06:27:13 pm »

Or perhaps the quickest fix for now would be to just change AW's packaging requirement rules to specify that TCL can be any 8.5 version from 8.5.2 onwards.  The logs show "package require -exact Tcl 8.5.2", so if this could be changed to whatever syntax accepts 8.5.* then it could use either its own 8.5.2 or PyLNP's 8.5.15 version of tcl.dll.  (This is of course assuming that nothing breaks with a higher 8.5 version! Hopefully not as the versioning suggests it should only be bugfixes from 8.5.2 -> 8.5.15.)
I don't think that's possible - that line looks like a Tcl command to me, which means it's part of the bundled Tkinter/Tk/Tcl library (and could well be called from C). My guess is that the library is set up to require a specific version of Tcl to ensure that it's getting one compatible with the current Python installation.
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

TheBloke

  • Bay Watcher
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #10 on: November 05, 2015, 10:21:40 am »

Or perhaps the quickest fix for now would be to just change AW's packaging requirement rules to specify that TCL can be any 8.5 version from 8.5.2 onwards.  The logs show "package require -exact Tcl 8.5.2", so if this could be changed to whatever syntax accepts 8.5.* then it could use either its own 8.5.2 or PyLNP's 8.5.15 version of tcl.dll.  (This is of course assuming that nothing breaks with a higher 8.5 version! Hopefully not as the versioning suggests it should only be bugfixes from 8.5.2 -> 8.5.15.)
I don't think that's possible - that line looks like a Tcl command to me, which means it's part of the bundled Tkinter/Tk/Tcl library (and could well be called from C). My guess is that the library is set up to require a specific version of Tcl to ensure that it's getting one compatible with the current Python installation.

Ah yes you're right, it is TCL - I had thought at first it was the Python package manager.

A quick glance at the TCL docs suggests it's possible to specify a range of versions.  It looks like one can specify a min-max version to package require.  The specifying of requirements appears to be as per the rules for the package vsatisfies command.  TclCmd - Package.

For example, to cover both 8.5.2 and the 8.5.15 from PyLNP: package require Tcl 8.5.2-8.5.16 (not 8.5.15 as the max is matched exclusive), or to specify any version equal to or greater than 8.5.2: package require Tcl 8.5.2-.  Or simply package require Tcl 8.5.2 which should match from 8.5.2 up to "the next major version number" - which I suppose means 9.0 in this case, though perhaps is 8.6.  Either would be fine for this case.

However, as you suggest, this line might be being added automatically by Tcl and not something easily changeable by Algee.  Though perhaps there is a Python command/config that controls what Tcl version requirements are specified?

Anyway it's probably moot as Algee isn't around to try any quick fixes and sounds like he will try for the fully bundled EXE route when he's back, which should solve it in the cleanest way.
Logged

TheBloke

  • Bay Watcher
    • View Profile
Re: Announcement Window, a Python announcement filter.
« Reply #11 on: November 05, 2015, 10:43:46 am »

As a workaround for anyone wanting AW working from PyLNP before Algee can get it fixed properly:

  • Install AW into LNP/Utilities/AnnouncementWindow 1.1/ (or whatever folder name you want)
  • Create a file called AnnouncementWindow.bat in that folder, containing:
Code: [Select]
@SET TCL_LIBRARY=./tcl/tcl8.5

@start AnnouncementWindow.exe
  • Edit PyLNP's LNP\Utilities\utilities.txt and add the following lines:
Code: [Select]
[AnnouncementWindow.bat:Announcement Window]

then under the Exclusions section:

AnnouncementWindow
[AnnouncementWindow.exe:EXCLUDE]

    Now AW will run from PyLNP, either as a double-click or as an autorun. 
    « Last Edit: November 05, 2015, 10:50:50 am by TheBloke »
    Logged

    PeridexisErrant

    • Bay Watcher
    • Dai stihó, Hrasht.
      • View Profile
    Re: Announcement Window, a Python announcement filter.
    « Reply #12 on: November 05, 2015, 05:39:39 pm »

    Very nice, thanks!
    Logged
    I maintain the DF Starter Pack - over a million downloads and still counting!
     Donations here.

    Algee

    • Escaped Lunatic
      • View Profile
    Re: Announcement Window, a Python announcement filter.
    « Reply #13 on: November 09, 2015, 01:25:54 pm »

    I packaged the exe using pyinstaller instead of py2exe and it fixed the issue with the 0.10b pyLNP package. As a bonus everything is neatly packed into a few files. I've heard that the new LNP release by PeridexisErrant already fixed the issue, even when you bypass the .bat script. I assume it uses the 2.7.10 build Pidgeot posted here, along with TheBloke's .bat?

    Anyway, i'll probably test it out a bit more to make sure there's no new issues then upload a new release to github later today.
    Logged

    PeridexisErrant

    • Bay Watcher
    • Dai stihó, Hrasht.
      • View Profile
    Re: Announcement Window, a Python announcement filter.
    « Reply #14 on: November 09, 2015, 06:24:42 pm »

    Yep, it does. Nice work though, I'll be upgrading when it's ready :)
    Logged
    I maintain the DF Starter Pack - over a million downloads and still counting!
     Donations here.
    Pages: [1] 2 3