Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2 3 4

Author Topic: [UTIL] The LurkerTracker (version updated)  (Read 12223 times)

Zathras

  • Bay Watcher
  • Boogie thinks you being confoosed.
    • View Profile
[UTIL] The LurkerTracker (version updated)
« on: November 15, 2010, 07:24:16 pm »

And now, a brief word from our sponsors...
Tired of lurkers lurking like lurkscum? Annoyed that your little brain can't keep track of people's whereabouts?
Fingers cramped from too much clicking looking for that elusive scummy post?

Never fear, The LurkerTracker® is here!



As I've received a couple of requests, I open this topic to post the source of the LurkerTracker, and process. Hopefully people will find it useful and even grow it into something better.


I - What is it?

The LurkerTracker is a perl script that will take a file with one or more saved pages of forum posts, and will create a summary of who posted what when, with URLs, and how long since their last post. If you haven't seen it, as example here's one for the currently running VM4, so you know what it looks like:


LurkerTracker

GlyphGryph has posted 16 times; the last on: 2010/11/13 14:00, 52 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
IronyOwl has posted 17 times; the last on: 2010/11/13 18:26, 47 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17]
JanusTwoface has posted 21 times; the last on: 2010/11/13 17:05, 48 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21]
Ottofar has posted 3 times; the last on: 2010/11/13 03:01, 63 hours ago. Posts: [1][2][3]
Pandarsenic has posted 20 times; the last on: 2010/11/13 02:20, 63 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20]
Zathras has posted 2 times; the last on: 2010/11/13 01:50, 64 hours ago. Posts: [1][2]
webadict has posted 15 times; the last on: 2010/11/15 09:04, 8 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15]



*Updated* II - I can haz surz plz?

Sure. There are two parts: a small preprocessing script that works around my feeble regexp-fu by making post URLs easier to find, and the lurkertracker proper. They both read from standard input and write to standard output, so the typical use is to pipe them together like:

Code: [Select]
    $   ./posturlfix.pl < saved_pages | ./lurkertracker.pl > summary.txt
Then just copy the contents of summary.txt and paste it into your post.




Update: this code now reflects the latest version, with DreathplacementTracking, AutoProd, RedTextFinder, and sorting alive players first. See here for what the new version looks like.



III - How do I use it? What are the caveats?

It is as easy as 1-2-3, just thee little (cumbersome, but see point IV below for possible improvements) steps. At the moment, the process is as listed below. I'll use the running VM4 as an example, which will produce the summary at the start of this post.

1. Grab the pages.
Go to the topic, and for each page since the game proper started (page one, in this case, page four in P17, with standard 15-posts-per-page settings), right click on the page and select "save page as...", with the HTML-only option. It will default to "index.php.html"; to this, I append the page number. I end up with index.php.html.1 through index.php.html.8 (eight pages in that game as of this writing; if you use forum settings for more posts per page, this step will use fewer files; the script won't care either way).

Optional step: the game proper started with reply #13; if you don't want the "in" and other pre-game discussion to be included, simply edit the first file, search for "#13", and remove all text before the first "<h4><a href="http://[...]" title="View the profile of Ottofar">Ottofar[/url]</h4>" before that. It will get rid of about 850 lines of crap that the script doesn't care about. Be sure to leave the <h4> line, as that's the first thing we look for in the script.


2. Concatenate your pages into one large file.
How you do this depends on your operating system. In linux, one easy way is:
Code: [Select]
    $ for i in 1 2 3 4 5 6 7 8 ; do cat index.php.html.$i >> vm4_pages ; doneYou now have one file, vm4_pages, with all the posts in the game.


3. Now run the preprocessor, and feed the output to lurkertracker, and the output of that to your summary text file.
A simple way to do this is piping them together, but you can also use intermediate files.
Code: [Select]
    $   ./posturlfix.pl < vm4_pages | ./lurkertracker.pl > vm4_summary.txt...and that's it. vm4_summary.txt will look like the report at the top of the page. In fact, steps two and three could be combined, just feeding "index.html.*" to the preprocessor, but the intermediate file lets you pause and make sure you have your ducks in a row.



As for caveats or other points, a couple:

  • Beware of "Today": the forum will use "Today" instead of the current date for posts less than 24hrs old. The script recognises this and changes it to your current local date. The flipside of this is that if you run the report a couple of days later, it will change it to today's today, not to a couple of days ago's today. So if you wish to keep current, be sure to re-save those pages after their proper date shows instead of "Today."
  • Timezone: the posts time is shown in the forum's localtime. For the "last post was so and so ago" part, the script needs an offset between your local time and forum time, in seconds. The variable $TZOFF holds this; it has now 7200, as I'm two hours earlier than the forum, but be sure to change it to match your locale.
  • Replacements: the top section, where variables are declared, includes the death/replacement tracking. This should be updated to reflect your current game. The format is pretty straighforward: "oldplayer" => "newplayer" for replacements, "player" => "player: dead, roleflip" for deaths. If confusing, the whole hash can be removed with no impact to the result, but of course deaths/replacements won't be tracked then.
  • Prods: there's a variable called $prodtime now, which holds the time from last post needed for a prod to be printed for the player; it needs to be in seconds, and is currently set at 36 hours. Edit if your prodding rules are different, or just add a couple of zeroes if you want no prods issued.
  • Debug: the script includes a "debug print" mode. If the variable $DEBUG is changed to non-zero, it will produce a list of all posts it found, with author, reply number, forum post date and URL. This will make it easier for you to see if it's using the correct data. Sample debug output follows.



IV - It sucks! I can do better!

Please do! My perl is ten years rusty and my regexp-fu feeble; if people find it useful I'm sure  many regulars can improve or rewrite it to make it much, much better. It could probably grab the forum pages dynamically instead of having to save them to a file first, be cleaner/smarter/faster, or have a number of new features. Knock yourselves out.

« Last Edit: January 21, 2011, 01:21:05 am by Zathras »
Logged
My soul has been freed by the King of the Mafia.

IronyOwl

  • Bay Watcher
  • Nope~
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #1 on: November 15, 2010, 07:27:56 pm »

Ha, nice.
Logged
Quote from: Radio Controlled (Discord)
A hand, a hand, my kingdom for a hot hand!
The kitchenette mold free, you move on to the pantry. it's nasty in there. The bacon is grazing on the lettuce. The ham is having an illicit affair with the prime rib, The potatoes see all, know all. A rat in boxer shorts smoking a foul smelling cigar is banging on a cabinet shouting about rent money.

Zathras

  • Bay Watcher
  • Boogie thinks you being confoosed.
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #2 on: November 15, 2010, 07:54:38 pm »

[Reserved for updates, FAQs or other administrative stuff]

  • 2010/12/01: Updated source to latest version.
  • 2011/01/20: Fixed small platform-dependent bug on date finder; changed %replace example to more generic values.


Argembarger got it working for him! Kudos; he can now post trackers everywhere too!  8-)
« Last Edit: January 21, 2011, 01:25:56 am by Zathras »
Logged
My soul has been freed by the King of the Mafia.

webadict

  • Bay Watcher
  • Former King of the Mafia
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #3 on: November 16, 2010, 09:59:49 pm »

I think the real question we're all asking is... can it be in a fan-fiction?
Logged

Zai

  • Bay Watcher
  • Elmo? Is that a SIMPLE UTENSIL?
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #4 on: November 16, 2010, 10:18:01 pm »

Neat stuff!

I think the real question we're all asking is... can it be in a fan-fiction?
O God no. Keep that in the Fan-Fiction thread. (But I just realized calling stories about forum members "fan fiction" means the writer is a fan of the forum members)
Logged
DEATH has been waiting for you. He has poured you some TEA.

webadict

  • Bay Watcher
  • Former King of the Mafia
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #5 on: November 16, 2010, 10:18:54 pm »

Neat stuff!

I think the real question we're all asking is... can it be in a fan-fiction?
O God no. Keep that in the Fan-Fiction thread. (But I just realized calling stories about forum members "fan fiction" means the writer is a fan of the forum members)
I'm a fan of myself.
Logged

Zai

  • Bay Watcher
  • Elmo? Is that a SIMPLE UTENSIL?
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #6 on: November 16, 2010, 10:33:43 pm »

I'm a fan of myself.
We already knew this, Web.
Logged
DEATH has been waiting for you. He has poured you some TEA.

webadict

  • Bay Watcher
  • Former King of the Mafia
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #7 on: November 16, 2010, 11:06:41 pm »

I'm a fan of myself.
We already knew this, Web.
So... I should write stories about myself?
Logged

Criptfeind

  • Bay Watcher
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #8 on: November 17, 2010, 10:40:45 pm »

No.

but really this thing is cool.
Logged

Pandarsenic

  • Bay Watcher
  • FABULOUS Gunslinger
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #9 on: November 20, 2010, 04:28:14 am »

Logged
KARATE CHOP TO THE SOUL
Your bone is the best Pandar honey. The best.
YOUR BONE IS THE BEST PANDAR
[Cheeetar] Pandar doesn't have issues, he has style.
Fuck off, you fucking fucker-fuck :I

Mephansteras

  • Bay Watcher
  • Forger of Civilizations
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #10 on: November 20, 2010, 09:59:18 am »

Interesting. I'll have to look at the code you wrote.
Logged
Civilization Forge Mod v2.80: Adding in new races, equipment, animals, plants, metals, etc. Now with Alchemy and Libraries! Variety to spice up DF! (For DF 0.34.10)
Come play Mafia with us!
"Let us maintain our chill composure." - Toady One

Zathras

  • Bay Watcher
  • Boogie thinks you being confoosed.
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #11 on: November 22, 2010, 03:21:50 am »

Please do, it badly needs the attention of someone who knows what he's doing. In the mean time, I'm still tinkering with it.

Announcing the latest updates: DreathplacementTracking, and AutoProd:

I've added a hash to process a replacement of the player name with something else, like "dead, role" for corpses, "Nonplayer" for kibitzers, "Moderator" for mods, and "NewPlayer" for replaced dudes.

I've also added a $prodtime variable, players who have not posted in that many seconds will be appended to a $prod string, and the report footer will automatically request a prod for them.

Both the replacement and prod lists now have intelligence to avoid prodding/listing dead people or mods, as I'd think it rude to ask the mod to go prod himself.

Sample output (using day-old paranormal data, for sample purposes only, not current to the game):


LurkerTracker

Argembarger has posted 9 times; the last on: 2010/11/21 02:14, 23 hours ago. Posts: [1][2][3][4][5][6][7][8][9]
Criptfeind: dead, town medium has posted 28 times; the last on: 2010/11/17 18:26, 103 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]
JanusTwoface has posted 35 times; the last on: 2010/11/19 13:28, 60 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35]
[...]
Mish has posted 15 times; the last on: 2010/11/18 22:29, 75 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15]
Moderator has posted 28 times; the last on: 2010/11/20 15:44, 34 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]
Mr.Person has posted 16 times; the last on: 2010/11/21 00:28, 25 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
Mysteriousbluepuppet has posted 9 times; the last on: 2010/11/18 10:20, 87 hours ago. Posts: [1][2][3][4][5][6][7][8][9]
Nonplayer has posted 2 times; the last on: 2010/11/10 23:21, 266 hours ago. Posts: [1][2]
Org: dead, town medium has posted 18 times; the last on: 2010/11/12 20:21, 221 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18]
[...]
webadict has posted 22 times; the last on: 2010/11/18 20:01, 77 hours ago. Posts: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]

Replacements considered: Eduren => Argembarger; Solifuge => Mish.
The Morgue: Criptfeind: dead, town medium; Org: dead, town medium; SHAD0Wdump: dead, town telepath.

Moderator: please prod JanusTwoface, Mish, Mysteriousbluepuppet, webadict.


Notes: "Moderator" includes posts by both Meph and Jokerman; the Argembarger posts list includes Eduren's posts, likewise for Mish and Solifuge. The prod list skips over corpses, kibitzers and mods. Nonplayer has "ToonyMan" only at the moment, but any number of intruders can be mapped to it and will show on a single line.

Neat, huh?   8-)

Too bad it's too soon to post a new one to Paranormal to show it out... tomorrow before end of day maybe. If there's interest, I'll update the source in the original post, otherwise I'll wait until I've tinkered with it more.

ETA: Also, the replacements, morgue and prod lists will only print if there are any in the list. In a fresh game with no deaths and so forth, the footer will simply be empty.
« Last Edit: November 22, 2010, 04:24:18 am by Zathras »
Logged
My soul has been freed by the King of the Mafia.

Zathras

  • Bay Watcher
  • Boogie thinks you being confoosed.
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #12 on: November 22, 2010, 06:47:31 am »


One more, not fully working yet, but posted here so people can give ideas on how to make it work.
The LurkerTracker's Experimental RedTextFinder!


First, sample output, which will tell you pretty much all you need to know:


LurkerTracker

Argembarger has posted 9 times; the last on: 2010/11/22 02:14, 3 hours ago. Latest red text: MBP
Criptfeind: dead, town medium has posted 28 times; the last on: 2010/11/17 18:26, 107 hours ago. Latest red text: SHADOWdump
JanusTwoface has posted 35 times; the last on: 2010/11/19 13:28, 64 hours ago. Latest red text: vote Pandarsenic
Jim Groovester has posted 70 times; the last on: 2010/11/22 04:01, 1 hours ago. Latest red text: Pandarsenic
KaminaSquirtle has posted 60 times; the last on: 2010/11/20 23:39, 30 hours ago. Latest red text: Unvote.&nbsp; Vote Pandarsenic.
Leafsnail has posted 20 times; the last on: 2010/11/20 23:23, 30 hours ago. Latest red text: Unvote.&nbsp; Vote Pandarsenic.
Mish has posted 15 times; the last on: 2010/11/18 22:29, 79 hours ago. Latest red text: MBP
Moderator has posted 28 times; the last on: 2010/11/20 15:44, 37 hours ago. Latest red text: ToonyMan
Mr.Person has posted 16 times; the last on: 2010/11/22 00:28, 5 hours ago. Latest red text: Pandarsenic
Mysteriousbluepuppet has posted 9 times; the last on: 2010/11/18 10:20, 91 hours ago. Latest red text: Ottofar
Nonplayer has posted 2 times; the last on: 2010/11/10 23:21, 270 hours ago.
Org: dead, town medium has posted 18 times; the last on: 2010/11/12 20:21, 225 hours ago. Latest red text: Kaminasquirtle
Ottofar has posted 12 times; the last on: 2010/11/22 03:50, 1 hours ago. Latest red text: Kamina
Pandarsenic has posted 26 times; the last on: 2010/11/20 12:01, 41 hours ago. Latest red text: unvote KaminaSquirtle and vote Zathras
SHAD0Wdump: dead, town telepath has posted 48 times; the last on: 2010/11/17 18:15, 107 hours ago. Latest red text: Janus
Toaster has posted 23 times; the last on: 2010/11/20 22:27, 31 hours ago. Latest red text: Unvote.&nbsp; Vote Pandarsenic
Zathras has posted 32 times; the last on: 2010/11/20 03:55, 49 hours ago. Latest red text: <strong>Pandarsenic</strong>
webadict has posted 22 times; the last on: 2010/11/18 20:01, 81 hours ago. Latest red text: OFFLINE

Replacements considered: Eduren => Argembarger; Solifuge => Mish.
The Morgue: Criptfeind: dead, town medium; Org: dead, town medium; SHAD0Wdump: dead, town telepath.

Moderator: please prod JanusTwoface, Mish, Mysteriousbluepuppet, Zathras, webadict.



Notes:
1) Obviously, I removed the post lists for this to keep the list short. Once it works, it will say "Posts: [1][2]... Latest red text: Unvote or whatever" on the same line.

2) Good: the red text itself is a link to the post where it happened. Also good: the line won't appear if a player hasn't voted: see Nonplayer.

3) Bad: it considers any red text it finds, even if it's inside a quote (like Person's or Moderator's) or in a signature (like Webadict's).

4) Bad: it obviously retains some html garbage like &nbsp and <strong>.


I think it makes a nice addition, but I don't know how to get around the bad points mentioned above. Ideas are welcome.
Logged
My soul has been freed by the King of the Mafia.

Argembarger

  • Bay Watcher
    • View Profile
    • Not quite yet
Re: [UTIL] The LurkerTracker
« Reply #13 on: November 22, 2010, 10:02:15 am »

RedText finder? You've outdone yourself, this seems fantastic!

My only suggestion for the LurkerTracker in general is, would it be possible to tabulate the data to make it easier to read?
Logged
Quote from: penguinofhonor
Quote from: miauw62
This guy needs to write a biography about Columbus. I would totally buy it.
I can see it now.

trying to make a different's: the life of Columbus

Zathras

  • Bay Watcher
  • Boogie thinks you being confoosed.
    • View Profile
Re: [UTIL] The LurkerTracker
« Reply #14 on: November 22, 2010, 06:29:23 pm »

Thanks, I hope it's useful I think I figured out how to skip the quotes and signatures, I'll post a new one to the thread.

I don't follow what you mean by tabulated; do you mean to break it into columns? I'm not sure that'd make it more readable, as people use different screen resolutions and text wrapping would look pretty bad. I like to keep each dude to one line, and the whole report as compact as possible, but maybe if you make a mock-up of what you mean I'd understand you better.
Logged
My soul has been freed by the King of the Mafia.
Pages: [1] 2 3 4