Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Armok]

Then I could give the script to someone with more bandwidth/space/time, and they could get the dump.

As I have said a few times in this thread, that'd be me.

[qwertyuiopas]

Got javascript loading the text from the bay12 main site.

Need to research spoiler format, and get it to run through all the pages of threads, and at least find the threads.

To do that, I must learn javascript string parsing.

[qwertyuiopas]

Update:

http://www.bay12games.com/forum/index.php?topic=39690",http://www.bay12games.com/forum/index.php?topic=24492.0",http://www.bay12games.com/forum/index.php?topic=27009.0",http://www.bay12games.com/forum/index.php?topic=38441.0",http://www.bay12games.com/forum/index.php?topic=40391.0",http://www.bay12games.com/forum/index.php?topic=39823.0",http://www.bay12games.com/forum/index.php?topic=39823.0",http://www.bay12games.com/forum/index.php?topic=39240.0",http://www.bay12games.com/forum/index.php?topic=40273.0",http://www.bay12games.com/forum/index.php?topic=39464.0",http://www.bay12games.com/forum/index.php?topic=39464.0",http://www.bay12games.com/forum/index.php?topic=38232.0",http://www.bay12games.com/forum/index.php?topic=38232.0",http://www.bay12games.com/forum/index.php?topic=38516.0",http://www.bay12games.com/forum/index.php?topic=38516.0",http://www.bay12games.com/forum/index.php?topic=39039.0",http://www.bay12games.com/forum/index.php?topic=40311.0",http://www.bay12games.com/forum/index.php?topic=39646.0",http://www.bay12games.com/forum/index.php?topic=40187.0",http://www.bay12games.com/forum/index.php?topic=31296.0",http://www.bay12games.com/forum/index.php?topic=31296.0",http://www.bay12games.com/forum/index.php?topic=37529.0",http://www.bay12games.com/forum/index.php?topic=37529.0",http://www.bay12games.com/forum/index.php?topic=39377.0",http://www.bay12games.com/forum/index.php?topic=39855.0",http://www.bay12games.com/forum/index.php?topic=39844.0",http://www.bay12games.com/forum/index.php?topic=39818.0",http://www.bay12games.com/forum/index.php?topic=39808.0",http://www.bay12games.com/forum/index.php?topic=38897.0",http://www.bay12games.com/forum/index.php?topic=38897.0",http://www.bay12games.com/forum/index.php?topic=36470.0",http://www.bay12games.com/forum/index.php?topic=36470.0",http://www.bay12games.com/forum/index.php?topic=39358.0",http://www.bay12games.com/forum/index.php?topic=38508.0",http://www.bay12games.com/forum/index.php?topic=33795.0",http://www.bay12games.com/forum/index.php?topic=33795.0",http://www.bay12games.com/forum/index.php?topic=38867.0",http://www.bay12games.com/forum/index.php?topic=38739.0",http://www.bay12games.com/forum/index.php?topic=39248.0",http://www.bay12games.com/forum/index.php?topic=36687.0",http://www.bay12games.com/forum/index.php?topic=39245.0",http://www.bay12games.com/forum/index.php?topic=29710.0",http://www.bay12games.com/forum/index.php?topic=38854.0",http://www.bay12games.com/forum/index.php?topic=38520.0",http://www.bay12games.com/forum/index.php?topic=32251.0",http://www.bay12games.com/forum/index.php?topic=39036.0",http://www.bay12games.com/forum/index.php?topic=31370.0",http://www.bay12games.com/forum/index.php?topic=32327.0",http://www.bay12games.com/forum/index.php?topic=32327.0",http://www.bay12games.com/forum/index.php?topic=38833.0",http://www.bay12games.com/forum/index.php?topic=38073.0",http://www.bay12games.com/forum/index.php?topic=31638.0",http://www.bay12games.com/forum/index.php?topic=31638.0",http://www.bay12games.com/forum/index.php?topic=38565.0",http://www.bay12games.com/forum/index.php?topic=38565.0",http://www.bay12games.com/forum/index.php?topic=31552.0",http://www.bay12games.com/forum/index.php?topic=36218.0",http://www.bay12games.com/forum/index.php?topic=38638.0",http://www.bay12games.com/forum/index.php?topic=38638.0",http://www.bay12games.com/forum/index.php?topic=38513.0",http://www.bay12games.com/forum/index.php?topic=31865.0",http://www.bay12games.com/forum/index.php?topic=31865.0",http://www.bay12games.com/forum/index.php?topic=38651.0",http://www.bay12games.com/forum/index.php?topic=38732.0",http://www.bay12games.com/forum/index.php?topic=37589.0",http://www.bay12games.com/forum/index.php?topic=38570.0",http://www.bay12games.com/forum/index.php?topic=38461.0"

[Aqizzar]

Hmm...  Now ah'm no expert in da ways o'dat dere computin', but ah suspec' there be a problem.

[qwertyuiopas]

This code, on IE6(you might need to slightly alter it for anything newer), will list every topic on every page of VN.

Next step: Making an array of them all.

Then a page parsing function, applied to just the first page(Because I don't have the bandwidth to do it all)

Spoiler (click to show/hide)

Code: [Select]

<script language = javascript>
var r;
var o;
function a(p)
{
    o="";
    for(var n=0;n<=800;n+=50)
    {
        o+="http://www.bay12games.com/forum/index.php?board=9."+n+"<br><br>";
        r=new ActiveXObject("Microsoft.XMLHTTP");
        r.onreadystatechange=bbb;
        r.open("GET","http://www.bay12games.com/forum/index.php?board=9."+n,false);
        r.send(null);
        o+="<hr>";
    }
    document.write(o);
}
function bbb()
{
    if(r.readyState == 4 && r.status == 200)
        VNparse(r.responseText);
}
function VNparse(s)
{
    var a=s.match(/"http:\/\/www.bay12games.com\/forum\/index.php\?topic=\d+.0"/g);
    for(var u=0;u<a.length;u++)
    {
        o+=a[u]+"<br>";
    }
}
</script>
<body onLoad=a("http://www.bay12games.com/forum/index.php?board=16.0")></body>

Edit:
Last version before it does any actual page parsing, this lists all pages though doesn't do anything with them.

Spoiler (click to show/hide)

Code: [Select]

<script language = javascript>
var r;
var o;
var aa = new Array();
function a()
{
    o="";
    aa[0]="http://www.bay12games.com/forum/index.php?topic=40391.0";
    if(1)
    {
        for(var n=0;n<=800;n+=50)
        {
            r=new ActiveXObject("Microsoft.XMLHTTP");
            r.onreadystatechange=bbb;
            r.open("GET","http://www.bay12games.com/forum/index.php?board=9."+n,false);
            r.send(null);
        }
        for(var p in aa)
            parsePage(aa[p].replace("\"",""));
    }
    else
        parsePage(aa[0].replace("\"",""));
    document.write(o)
}
function bbb()
{
    if(r.readyState == 4 && r.status == 200)
        VNparse(r.responseText);
}
function VNparse(s)
{
    var a=s.match(/"http:\/\/www.bay12games.com\/forum\/index.php\?topic=\d+.0"/g);
    aa=aa.concat(a);
}
function parsePage(page)
{
    o+="<hr>"+page+"<hr>";
    /*r=new ActiveXObject("Microsoft.XMLHTTP");
    r.onreadystatechange=parsePage2;
    r.open("GET",page,false);
    r.send(null);*/
    o+="<hr>";
}
function parsePage2()
{
    if(r.readyState == 4 && r.status == 200)
        VNpage(r.responseText);
}
function VNpage(s)
{
}
</script>
<body onLoad=a()></body>

[qwertyuiopas]

Quite a bit of success, it is now parsing down to a post level, although it won't go on to the second page or later.
(The lines of - were hr s, but were converted by the copy/paste)

(Edit:The spoiler/code combo was messed up. Unfortunately. Will be better with some effort)

Spoiler (click to show/hide)

[code]--------------------------------------------------------------------------------
http://www.bay12games.com/forum/index.php?topic=40391.0
--------------------------------------------------------------------------------
Topic: Javascript VN parser spoiler test topic.


Poster: qwertyuiopas
Javascript VN parser spoiler test topic.
[/a] [/div] [div class="smalltext"]« on: Today at 11:44:45 AM »[/div][/td] [td align="right" valign="bottom" height="20" style="font-size: smaller;"][div id="msg_706374_quick_mod"] [/div][/td] [/tr][/table] [hr width="100%" size="1" class="hrcolor" /] [div class="post" id="msg_706374"] This post is here to learn how spoilers work


[div class="spoiler"][div class="spoilerheader"][a href="javascript:void(0)" onClick="n = this.parentNode.parentNode.lastChild; if(n.style.display == 'none') { n.style.display = 'block'; } else { n.style.display = 'none'; } return false;"]Spoiler[/a] (click to show/hide)[/div][div class="spoilerbody" style="display: none"]test1[/div][/div][div class="spoiler"][div class="spoilerheader"][a href="javascript:void(0)" onClick="n = this.parentNode.parentNode.lastChild; if(n.style.display == 'none') { n.style.display = 'block'; } else { n.style.display = 'none'; } return false;"]Spoiler[/a] (click to show/hide)[/div][div class="spoilerbody" style="display: none"]test2[/div][/div][div class="spoiler"][div class="spoilerheader"][a href="javascript:void(0)" onClick="n = this.parentNode.parentNode.lastChild; if(n.style.display == 'none') { n.style.display = 'block'; } else { n.style.display = 'none'; } return false;"]Spoiler[/a] (click to show/hide)[/div][div class="spoilerbody" style="display: none"][spoiler]nested spoilers[/div][/div]

[/div] [/td] [/tr] [tr id="msg_706374_footer"] [td valign="bottom" class="smalltext" width="85%"] [table width="100%" border="0" style="table-layout: fixed;"][tr] [td colspan="2" class="smalltext" width="100%"] [/td] [/tr][tr] [td valign="bottom" class="smalltext" id="modified_706374"] [/td] [td align="right" valign="bottom" class="smalltext"] [a href="http://www.bay12games.com/forum/index.php?action=reporttm;topic=40391.0;msg=706374"]Report to moderator[/a]   [img src="http://www.bay12games.com/forum/Themes/darkling/images/ip.gif" alt="" border="0" /] [a href="http://www.bay12games.com/forum/index.php?action=helpadmin;help=see_member_ip" onclick="return reqWin(this.href);" class="help"]70.81.242.178[/a] [/td] [/tr][/table] [hr width="100%" size="1" class="hrcolor" /]
--------------------------------------------------------------------------------

Poster: qwertyuiopas
Re: Javascript VN parser spoiler test topic.
[/a] [/div] [div class="smalltext"]« Reply #1 on: Today at 01:55:29 PM »[/div][/td] [td align="right" valign="bottom" height="20" style="font-size: smaller;"][div id="msg_706520_quick_mod"] [/div][/td] [/tr][/table] [hr width="100%" size="1" class="hrcolor" /] [div class="post" id="msg_706520"] Second post test.
[table class="bbc_table"][tr][td]Table![/td][/tr][/table] [/div] [/td] [/tr] [tr id="msg_706520_footer"] [td valign="bottom" class="smalltext" width="85%"] [table width="100%" border="0" style="table-layout: fixed;"][tr] [td colspan="2" class="smalltext" width="100%"] [/td] [/tr][tr] [td valign="bottom" class="smalltext" id="modified_706520"] [/td] [td align="right" valign="bottom" class="smalltext"] [a href="http://www.bay12games.com/forum/index.php?action=reporttm;topic=40391.1;msg=706520"]Report to moderator[/a]   [img src="http://www.bay12games.com/forum/Themes/darkling/images/ip.gif" alt="" border="0" /] [a href="http://www.bay12games.com/forum/index.php?action=helpadmin;help=see_member_ip" onclick="return reqWin(this.href);" class="help"]70.81.242.178[/a] [/td] [/tr][/table] [hr width="100%" size="1" class="hrcolor" /]
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
[/code][/spoiler]

[Armok]

I'm going to bed quite soon, from the looks of it I might even have something to run once I wake up? if that's the case, make sure to leave some instruction on what I should do in order to run it and such.

[qwertyuiopas]

And here you have it, the DE-POSTER v.01.

It still can't to multi-page topics though.

Spoiler (click to show/hide)

Code: [Select]

<script language = javascript>
var r;
var o;
var aa = new Array();
function a()
{
    o="";
    aa[0]="http://www.bay12games.com/forum/index.php?topic=40391.0";
    if(0)
    {
        for(var n=0;n<=800;n+=50)
        {
            r=new ActiveXObject("Microsoft.XMLHTTP");
            r.onreadystatechange=bbb;
            r.open("GET","http://www.bay12games.com/forum/index.php?board=9."+n,false);
            r.send(null);
        }
        for(var p in aa)
            parsePage(aa[p].replace("\"",""));
    }
    else
        parsePage(aa[0].replace("\"",""));
    document.write(o)
}
function bbb()
{
    if(r.readyState == 4 && r.status == 200)
        VNparse(r.responseText);
}
function VNparse(s)
{
    var a=s.match(/"http:\/\/www.bay12games.com\/forum\/index.php\?topic=\d+.0"/g);
    aa=aa.concat(a);
}
function parsePage(page)
{
    o+="<hr>"+page+"<hr>";
    r=new ActiveXObject("Microsoft.XMLHTTP");
    r.onreadystatechange=parsePage2;
    r.open("GET",page,false);
    r.send(null);
    o+="<hr>";
}
function parsePage2()
{
    if(r.readyState == 4 && r.status == 200)
        VNpage(r.responseText);
}
function VNpage(s)
{
    var d=s.search("Topic:");
    s=s.substring(d+6);
    d=s.search("&nbsp;");
    o+="Topic: " + s.substring(0,d) + "<br><br>";
    d=s.search("Are you sure you wish to enable notification of new replies for this topic");
    s=s.substring(10,d);
    var loop = 0;
    while(loop<100)
    {
        if(s.search(/<div class="signature">/i)==-1)break;
        d=s.search(/<div class="signature">/i);
        parsePost(s.substring(0,d));
        o+="<hr>";
        s=s.substring(d+1);
        loop++;
    }
}
function parsePost(s)
{
    var d=s.search(/<a href="http:\/\/www.bay12games.com\/forum\/index.php.action=profile;u=\d+" title="View the profile of .+">.+<\/a>/i);
    s=s.substring(d);
    d=s.search("View the profile of ");
    s=s.substring(d+20);
    d=s.search("\">");
    o+="<br>Poster: "+s.substring(0,d);
    s=s.substring(d*2+6);
    d=s.search(/.msg\d+#msg\d+">/);
    s=s.substring(d);
    d=s.search("\">");
    s=s.substring(d+2);
    d=s.search(/<\/a>/i);
    o+="<br>"+s.substring(0,d)+"<br>";
    s=s.substring(d);
    d=s.search(/<div/i);
    s=s.substring(d);
    d=s.search(/<\/div>/i);
    o+=s.substring(0,d)+"<br>";
    s=s.substring(d+6);
    d=s.search(/msg_\d+"/i);
    s=s.substring(d);
    d=s.search(/>/i);
    s=s.substring(d+1);
    d=s.search(/<tr id.+msg_\d+_footer"/i);
    o+=s.substring(0,d);
}
</script>
<body onLoad=a()></body>

[qwertyuiopas]

Well, it worked for a single thread, but not the whole first page.

As it is, it can fully process it to the HTML for the contents of each post and arrange it in a simplified format.

Here is the output of this thread.

Technically, the result is a HTML page containing the HTML of the output, by swapping < and > for the &gt; and &lt; because I couldn't get page source or save as to work.

[ToonyMan]

Makes sense.

[qwertyuiopas]

Issue: Was using "are you sure you want to be notified of replies" dialog text to find the end.

Solution: switched to mark as read.

One more test: first 5 off first page.

If it passes, I will give the full script, and hope someone cares to sit through the whole thing. The first page alone is 2-5 minutes, maybe more, and there are 16 pages... If IE6 does not quit early...

Though a later browser might work faster or better, someone would have to upgrade the page grab 'new' line.

[qwertyuiopas]

Well,
HERE IT IS!

However, even the first ~5 topics took well over a minute, at least on IE6.

I give this to you, whoever you is, who has a browser that can support the code for the entire VN, and can wait the hour(s) for it to process.

Maybe a more modern browser won't have as much of a problem.

Spoiler (click to show/hide)

Code: [Select]

<script language = javascript>
var r;
var o;
var aa = new Array();
var pagenum;
var rr;
function a()
{
    o="";
    //aa[0]="http://www.bay12games.com/forum/index.php?topic=39823.0";//"http://www.bay12games.com/forum/index.php?topic=40391.0";
    if(1)
    {
        for(var n=0;n<=800;n+=50)
        {
            r=new ActiveXObject("Microsoft.XMLHTTP");
            r.onreadystatechange=bbb;
            r.open("GET","http://www.bay12games.com/forum/index.php?board=9."+n,false);
            r.send(null);
        }
        for(var p in aa)
        {
            pagenum=0;
            if(aa[p]!=aa[p-1])
            {
                parsePage(aa[p].replace(/"/g,""));
                o+="<hr><hr><hr><hr><hr>";
            }
            //if(p>10)break;
        }
    }
    else
    {
        pagenum=0;
        parsePage(aa[1].replace(/"/g,""));
    }
    document.write(o.replace(/</g,"&lt;").replace(/>/g,"&gt;"));
}
function bbb()
{
    if(r.readyState == 4 && r.status == 200)
        VNparse(r.responseText);
}
function VNparse(s)
{
    var a=s.match(/"http:\/\/www.bay12games.com\/forum\/index.php\?topic=\d+.0"/g);
    aa=aa.concat(a);
}
function parsePage(page)
{
    o+="<hr>"+page+"<hr>";
    rr="";
    r=new ActiveXObject("Microsoft.XMLHTTP");
    r.onreadystatechange=parsePage2;
    r.open("GET",page,false);
    r.send(null);
    var d=page.search(/\d\D\d+/);
    d+=page.substring(d+1).search(/\D\d+/);
    var dd=parseInt(page.substr(d+2));
    if(rr.indexOf(page.substr(0,d+2)+(dd+50))>=0)
    {
        parsePage(page.substr(0,d+2)+(dd+50));
    }
    else
        o+="<hr>";
}
function parsePage2()
{
    if(r.readyState == 4 && r.status == 200)
    {
        rr=r.responseText;
        VNpage(rr);
        fail=0;
    }
}
function VNpage(s)
{
    var d=s.search("Topic:");
    s=s.substring(d+6);
    d=s.search("&nbsp;");
    o+="Topic: " + s.substring(0,d) + "<br><br>";
    d=s.search("markasread");
    s=s.substring(10,d);
    var loop = 0;
    while(loop<100)
    {
        if(s.search(/<div class="signature">/i)==-1)break;
        d=s.search(/<div class="signature">/i);
        parsePost(s.substring(0,d));
        o+="<hr>";
        s=s.substring(d+1);
        loop++;
    }
}
function parsePost(s)
{
    var d=s.search(/<a href="http:\/\/www.bay12games.com\/forum\/index.php.action=profile;u=\d+" title="View the profile of .+">.+<\/a>/i);
    s=s.substring(d);
    d=s.search("View the profile of ");
    s=s.substring(d+20);
    d=s.search("\">");
    o+="<br>Poster: "+s.substring(0,d);
    s=s.substring(d*2+6);
    d=s.search(/.msg\d+#msg\d+">/);
    s=s.substring(d);
    d=s.search("\">");
    s=s.substring(d+2);
    d=s.search(/<\/a>/i);
    o+="<br>"+s.substring(0,d)+"<br>";
    s=s.substring(d);
    d=s.search(/<div/i);
    s=s.substring(d);
    d=s.search(/<\/div>/i);
    o+=s.substring(0,d)+"<br>";
    s=s.substring(d+6);
    d=s.search(/msg_\d+"/i);
    s=s.substring(d);
    d=s.search(/>/i);
    s=s.substring(d+1);
    d=s.search(/<tr id.+msg_\d+_footer"/i);
    o+=s.substring(0,d);
}
</script>
<body onLoad=a()></body>

[Heron TSG]

Send it to Armok, he offered to do it. Thanks, qwerty!

[qwertyuiopas]

Quick question though:

Why, when I tried to port it to more modern javascript, did firefox quit on the (unmodified) line aa[1].replace(/"/g,"")?

[Armok]

Ok, woke up, read new posts in this topic.
I'm a bit confused. If I run this script, will it take every post, from every page of every topic, from every  page of topics, so that once it has finished there is no information in the entire VN that it has not backed up completly, and save this information to my harddrive? or is that script not finished yet?
Secondly, how exactly do I run it? I'm not to familar with javascript (my knowlege of it euals abaut "it's a programing language that is often embeaded in weabpages, and has very litle to do with actual Java"), and I'm curently using firefox. I Do have exlprorer instaled so that shuldnt be a problem thou... still, I'm not sure how to run this javascript thing, and also do I have to do something like navigate to a spesific page before starting it or give it some input settings or anything like that?
Also, what is the bottleneack of its speed? Is it the CPU, the bandwidth, the bay12 response time, or what. ALso, how much memory does it take, I could store around 50 gigabytes or something equaly exesive if necesery, but if it's more than 2 gigabytes I shuld probably shufle around a few files first. Thou from the looks of it it's just text so it probably ends up something rediculusly tiny comared to any of those numbers, but it dosnt hert asking.