Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 9 10 [11] 12

Author Topic: Lurker Tracker - Web Edition  (Read 22111 times)

lordnincompoop

  • Bay Watcher
  • Allusionist
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #150 on: April 25, 2013, 05:52:21 pm »

No, Tylui, the bug Dariush posted a pic of still hasn't been fixed.

I get the error very often and very unpredictably. Without any options and just the URL, it works most of the time, but if I dare select any option or use the substitution function, the program just spits that at me.
Logged

Tylui

  • Bay Watcher
  • O_o
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #151 on: April 25, 2013, 06:29:17 pm »

Same exact error? And thank you for doing my research for me, lncp :)
Logged

webadict

  • Bay Watcher
  • Former King of the Mafia
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #152 on: April 25, 2013, 08:25:19 pm »

No, Tylui, the bug Dariush posted a pic of still hasn't been fixed.

I get the error very often and very unpredictably. Without any options and just the URL, it works most of the time, but if I dare select any option or use the substitution function, the program just spits that at me.
Can I see the code? I'd like to try to find it.
Logged

zombie urist

  • Bay Watcher
  • [NOT_LIVING]
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #153 on: April 25, 2013, 09:23:45 pm »

The sources are in the first post.
Logged
The worst part of all of this is that Shakerag won.

Tylui

  • Bay Watcher
  • O_o
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #154 on: April 25, 2013, 10:41:58 pm »

hey webadict do you have webspace? i don't recall. If so, you got a spot for a python script? :P
Logged

webadict

  • Bay Watcher
  • Former King of the Mafia
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #155 on: April 26, 2013, 12:15:47 am »

hey webadict do you have webspace? i don't recall. If so, you got a spot for a python script? :P
I do not.
Logged

Tylui

  • Bay Watcher
  • O_o
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #156 on: April 26, 2013, 10:59:27 am »

'kay I'm getting something set up. :) Expect an update on this soon :O

EDIT: Looks like he uploaded a change I made last night. Captain Ford, your thread should work now.

http://think0028.com/lurkertracker.py?sort=post&url=http://www.bay12forums.com/smf/index.php?topic=124340.msg4162201#msg4162201
Logged

Captain Ford

  • Bay Watcher
  • Eating dragons has some side effects...
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #157 on: April 26, 2013, 12:00:22 pm »

Yay! Thank you for that.  :)
Logged
...Holy shit. Ford, you get the Official Medal of Epic Awesomeness.
Its official! Ford! You need to put it in your sig now! "Official Mafia Welcomer!"

Captain Ford

  • Bay Watcher
  • Eating dragons has some side effects...
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #158 on: April 26, 2013, 12:04:11 pm »

Hmm...actually, no, it's still broken. Check out Nerjin's vote log:

All of the following html code is contained in Nerjin's last vote. It's hiding all of his votes after the one he made in Reply #134.

Code: [Select]
<a style="color: red" href="http://www.bay12forums.com/smf/index.php?topic=124340.msg4164840#msg4164840">Reply #134 - Light Dariush's <span style="color: teal;" class="bbc_color">Teal<span style=" Candle</span><br /><br /><span style=" light="" vector&#039;s="" <span="" class="bbc_color">Green<span style=" candle</span> </span></a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4165537#msg4165537'>Reply #169 - candle<br /><br />Light Remuthra <span style=" color:="" teal;"="" class="bbc_color">Teal<span style=" candle.</span></a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4184004#msg4184004'>Reply #328 - Light Dariush</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4185667#msg4185667'>Reply #335 - Rescribe the tally sheet from what I can remember</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4185688#msg4185688'>Reply #338 - Light Solifuge white</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4187671#msg4187671'>Reply #361 - Light Solifuge</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4187895#msg4187895'>Reply #365 - Light Solifuge</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4202931#msg4202931'>Reply #462 - Light Dariush White</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4202939#msg4202939'>Reply #464 - Light Solifuge White</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4203190#msg4203190'>Reply #475 - Snuff Ford White</a></li>
<li><a style='color: blue' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4204311#msg4204311'>Reply #513 - Remuthra</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4204800#msg4204800'>Reply #528 - Light Dariush <span style=" color:="" teal;"="" class="bbc_color">Teal</span></span></span></span></a>
Logged
...Holy shit. Ford, you get the Official Medal of Epic Awesomeness.
Its official! Ford! You need to put it in your sig now! "Official Mafia Welcomer!"

Tylui

  • Bay Watcher
  • O_o
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #159 on: April 26, 2013, 01:50:41 pm »

Well as soon as I can find a host that doesn't block scraping, and is free, we'll be good to go. I'd like to get it off of think's server.
Logged

Captain Ford

  • Bay Watcher
  • Eating dragons has some side effects...
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #160 on: April 28, 2013, 12:44:47 pm »

Aha! I figured out what's causing my problem. (With the tracker for the Great Temple)

This is the html that's showing up in the lurkertracker:
Quote

Light Dariush's
  <span style="color: teal;" class="bbc_color">Teal
    <span style=" Candle</span>
    <span style="Light Vector's
      <span style="color: green;" class="bbc_color">Green
        <span style=" candle
    </span>
  </span>

This is the html from Nerjin's actual post, with the important parts highlighted in red:
Quote

Light Dariush's
  <span style="color: teal;" class="bbc_color">Teal
    <span style="color: red;" class="bbc_color"> Candle</span>
    <span style="color: red;" class="bbc_color">Light Vector's
      <span style="color: green;" class="bbc_color">Green
        <span style="color: red;" class="bbc_color"> candle</span>
      </span>
    </span>
  </span>


The problem happens when redtext is contained inside other redtext!
Logged
...Holy shit. Ford, you get the Official Medal of Epic Awesomeness.
Its official! Ford! You need to put it in your sig now! "Official Mafia Welcomer!"

Dariush

  • Bay Watcher
  • I don't think I !!am!!, therefore I !!am!! not
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #161 on: April 28, 2013, 01:08:16 pm »

The problem happens when redtext is contained inside other redtext!
Spoiler (click to show/hide)

More seriously though, why would you put redtext inside redtext? o_0

Captain Ford

  • Bay Watcher
  • Eating dragons has some side effects...
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #162 on: April 30, 2013, 03:42:39 pm »

You know, I'm building a tool similar to the LurkerTracker, and there are some technical questions I have.

First off, I feel like scraping at 15 posts per page is a really inefficient. Is there any way to get more posts per page without logging in? Does the <acronym="LurkerTracker">LT</acronym> handle that by logging into an account configured for 50 posts per page, or is there something I can add to the request to change the number of posts I'm getting back at a time?

I'm almost positive that the LT scrapes the entire thread every time it's refreshed. Because you can't really assume that posts weren't edited. I know from previous comments that it has a cache, but I'm not sure how it uses it.
Logged
...Holy shit. Ford, you get the Official Medal of Epic Awesomeness.
Its official! Ford! You need to put it in your sig now! "Official Mafia Welcomer!"

zombie urist

  • Bay Watcher
  • [NOT_LIVING]
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #163 on: April 30, 2013, 07:05:15 pm »

This code logs you in and then visits some topics. The resulting topics should be 50 posts/page if your settings are like that. If this works correctly, you should only need to login once and then you can scrape many pages. Also you can save the pages and cache them if you want.

Code: [Select]
import urllib
import urllib2
import cookielib

import re

def bay12login(username, password):
    cj = cookielib.LWPCookieJar("bay12forums.txt")
    cj.save()
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    urllib2.install_opener(opener)

    response_data = urllib.urlencode({'user':username, 'passwrd' : password, 'cookielength' : -1})
    response = urllib2.urlopen("http://www.bay12forums.com/smf/index.php?action=login2", response_data)
    cj.save();

def visitTopic(topic):
    response = urllib2.urlopen("http://www.bay12forums.com/smf/index.php?topic=" + topic + ".0")
    r = response.read()
    #doSomething(r)


#CHANGE THIS TO LOGIN
bay12login("USERNAME", "PASSWORD")
visitTopic("86473") #lurker tracker topic
visitTopic("88720") #new player's guide
Logged
The worst part of all of this is that Shakerag won.

Captain Ford

  • Bay Watcher
  • Eating dragons has some side effects...
    • View Profile
Re: Lurker Tracker - Web Edition
« Reply #164 on: April 30, 2013, 07:22:44 pm »

That was shockingly straightforward. I should be able to adapt that to my code.

Thanks for the help, ZU.

It would still be awesome to find a way to get more than 50 posts per page, but that's probably not possible.
Logged
...Holy shit. Ford, you get the Official Medal of Epic Awesomeness.
Its official! Ford! You need to put it in your sig now! "Official Mafia Welcomer!"
Pages: 1 ... 9 10 [11] 12