Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[lordnincompoop]

No, Tylui, the bug Dariush posted a pic of still hasn't been fixed.

I get the error very often and very unpredictably. Without any options and just the URL, it works most of the time, but if I dare select any option or use the substitution function, the program just spits that at me.

[Tylui]

Same exact error? And thank you for doing my research for me, lncp

[webadict]

No, Tylui, the bug Dariush posted a pic of still hasn't been fixed.

I get the error very often and very unpredictably. Without any options and just the URL, it works most of the time, but if I dare select any option or use the substitution function, the program just spits that at me.

Can I see the code? I'd like to try to find it.

[zombie urist]

The sources are in the first post.

[Tylui]

hey webadict do you have webspace? i don't recall. If so, you got a spot for a python script?

[webadict]

hey webadict do you have webspace? i don't recall. If so, you got a spot for a python script?

I do not.

[Tylui]

'kay I'm getting something set up. Expect an update on this soon :O

EDIT: Looks like he uploaded a change I made last night. Captain Ford, your thread should work now.

http://think0028.com/lurkertracker.py?sort=post&url=http://www.bay12forums.com/smf/index.php?topic=124340.msg4162201#msg4162201

[Captain Ford]

Yay! Thank you for that. 

[Captain Ford]

Hmm...actually, no, it's still broken. Check out Nerjin's vote log:

All of the following html code is contained in Nerjin's last vote. It's hiding all of his votes after the one he made in Reply #134.

Code: [Select]

<a style="color: red" href="http://www.bay12forums.com/smf/index.php?topic=124340.msg4164840#msg4164840">Reply #134 - Light Dariush's <span style="color: teal;" class="bbc_color">Teal<span style=" Candle</span><br /><br /><span style=" light="" vector&#039;s="" <span="" class="bbc_color">Green<span style=" candle</span> </span></a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4165537#msg4165537'>Reply #169 - candle<br /><br />Light Remuthra <span style=" color:="" teal;"="" class="bbc_color">Teal<span style=" candle.</span></a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4184004#msg4184004'>Reply #328 - Light Dariush</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4185667#msg4185667'>Reply #335 - Rescribe the tally sheet from what I can remember</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4185688#msg4185688'>Reply #338 - Light Solifuge white</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4187671#msg4187671'>Reply #361 - Light Solifuge</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4187895#msg4187895'>Reply #365 - Light Solifuge</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4202931#msg4202931'>Reply #462 - Light Dariush White</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4202939#msg4202939'>Reply #464 - Light Solifuge White</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4203190#msg4203190'>Reply #475 - Snuff Ford White</a></li>
<li><a style='color: blue' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4204311#msg4204311'>Reply #513 - Remuthra</a></li>
<li><a style='color: red' href='http://www.bay12forums.com/smf/index.php?topic=124340.msg4204800#msg4204800'>Reply #528 - Light Dariush <span style=" color:="" teal;"="" class="bbc_color">Teal</span></span></span></span></a>

[Tylui]

Well as soon as I can find a host that doesn't block scraping, and is free, we'll be good to go. I'd like to get it off of think's server.

[Captain Ford]

Aha! I figured out what's causing my problem. (With the tracker for the Great Temple)

This is the html that's showing up in the lurkertracker:

Light Dariush's
  <span style="color: teal;" class="bbc_color">Teal
    <span style=" Candle</span>
    <span style="Light Vector's
      <span style="color: green;" class="bbc_color">Green
        <span style=" candle
    </span>
  </span>

This is the html from Nerjin's actual post, with the important parts highlighted in red:

Light Dariush's
  <span style="color: teal;" class="bbc_color">Teal
    <span style="color: red;" class="bbc_color"> Candle</span>
    <span style="color: red;" class="bbc_color">Light Vector's
      <span style="color: green;" class="bbc_color">Green
        <span style="color: red;" class="bbc_color"> candle</span>
      </span>
    </span>
  </span>

The problem happens when redtext is contained inside other redtext!

[Dariush]

The problem happens when redtext is contained inside other redtext!

Spoiler (click to show/hide)

More seriously though, why would you put redtext inside redtext? o_0

[Captain Ford]

You know, I'm building a tool similar to the LurkerTracker, and there are some technical questions I have.

First off, I feel like scraping at 15 posts per page is a really inefficient. Is there any way to get more posts per page without logging in? Does the <acronym="LurkerTracker">LT</acronym> handle that by logging into an account configured for 50 posts per page, or is there something I can add to the request to change the number of posts I'm getting back at a time?

I'm almost positive that the LT scrapes the entire thread every time it's refreshed. Because you can't really assume that posts weren't edited. I know from previous comments that it has a cache, but I'm not sure how it uses it.

[zombie urist]

This code logs you in and then visits some topics. The resulting topics should be 50 posts/page if your settings are like that. If this works correctly, you should only need to login once and then you can scrape many pages. Also you can save the pages and cache them if you want.

Code: [Select]

import urllib
import urllib2
import cookielib

import re

def bay12login(username, password):
    cj = cookielib.LWPCookieJar("bay12forums.txt")
    cj.save()
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    urllib2.install_opener(opener)

    response_data = urllib.urlencode({'user':username, 'passwrd' : password, 'cookielength' : -1})
    response = urllib2.urlopen("http://www.bay12forums.com/smf/index.php?action=login2", response_data)
    cj.save();

def visitTopic(topic):
    response = urllib2.urlopen("http://www.bay12forums.com/smf/index.php?topic=" + topic + ".0")
    r = response.read()
    #doSomething(r)


#CHANGE THIS TO LOGIN
bay12login("USERNAME", "PASSWORD")
visitTopic("86473") #lurker tracker topic
visitTopic("88720") #new player's guide

[Captain Ford]

That was shockingly straightforward. I should be able to adapt that to my code.

Thanks for the help, ZU.

_{It would still be awesome to find a way to get more than 50 posts per page, but that's probably not possible.}