Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Parisbre56]

New version is up. It downloads images (both those posted by users and the images the forum theme uses), it can login, it's about 5 times faster than the old one and it has a better GUI.

Please tell me if it doesn't work for you or if you have any questions/suggestions.

EDIT: Turns out it doesn't work in Windows again. Probably screwed something up in the file names. I'll fix it tomorrow.

[Parisbre56]

Found the source of my Windows problems. Turns out java, for some strange reason, has decided that the 32-bit version of the JVM must have a standard heap size of 250MB. And guess which version of the JVM gets automatically installed in any Windows machine, both 32 and 64 bit? Yep, you guessed it, the 32-bit one.

This means that as soon as the program starts using more than 250 MB of memory, it crashes, unless you've downloaded and installed the 64-bit version for some reason. This means that you're fine as long as the pages of the thread are less than 1069 (in 15 posts per page mode).

Java itself gives you no way to alter the heap size from inside the program itself. I could tell you "You can use the command line to increase your heap size", but that reduces the ease of use, which was the whole reason for making a GUI. I can create different executable for each OS, but that defeats the purpose of using Java to make this cross platform. So it seems like the only thing I can do now is rewrite the program to download the data into temp files.

Yay, problems caused by arbitrary limits. At least I am learning stuff.

[Parisbre56]

Added the use of temp files to solve the memory problems. It should be working fine now.

[Xantalos]

Eeeexcellent. Time to read through ER.

[Parisbre56]

New version is up. It can save output as a "lightweight" .txt file, perfect for devices with low memory or e-book readers.
Either click "Browse" and select the "Plain Text" option or change the output file's extension from ".html" to ".txt" before pressing start.

I had some problems with quotes and spoilers, but I think I solved them now. Tell me if you notice anything wrong.
And tell me if you have any suggestions about the plain text output (more/less info, options you'd like to have, etc).

I've noticed that some output files are just too large, so I've added the ability to separate a file into multiple files to my long term goals.

[Person]

Are you still working on this?

[Parisbre56]

Are you still working on this?

Sometimes, although not as much as I'd like to. Got some work done with altering the page a bit (removing sigs and avatars and stuff) but I haven't finished it yet, that's why I haven't released an update.

Any specific feature you really want to see? If it's not something too complicated I could get it done in a week or two.

[Person]

I don't have anything in mind other than what you're already working on, but there might be a few people that would prefer not having the images, most likely for bandwidth reasons. Having a toggle for that in new versions might be nice.

Edit: Thought of something. When downloading larger threads, having to scroll down forever is somewhat of a problem, especially when it comes to remembering where you left off. Having a way to split the threads back into pages would be great if possible, ideally in a customizable way.

[Fayrik]

I remember reading this thread and pondering how useful this program could possibly be one day...

Here's the source code if anyone wants to mess with it. (10 MB, Eclipse project)

And now, I might just need this source code to make a branch program to spider search for rule violations on a forum I moderate.
If I succeed, I'll post up the branch here.

[Parisbre56]

I don't have anything in mind other than what you're already working on, but there might be a few people that would prefer not having the images, most likely for bandwidth reasons. Having a toggle for that in new versions might be nice.

Edit: Thought of something. When downloading larger threads, having to scroll down forever is somewhat of a problem, especially when it comes to remembering where you left off. Having a way to split the threads back into pages would be great if possible, ideally in a customizable way.

Yeah, the first one is easy, just changing one line of code into another. I'll add a button to do that in the next version.
The second one, I've been thinking about this myself. It's next on my list. I'll get working on it after the holidays.

I remember reading this thread and pondering how useful this program could possibly be one day...

Here's the source code if anyone wants to mess with it. (10 MB, Eclipse project)

And now, I might just need this source code to make a branch program to spider search for rule violations on a forum I moderate.
If I succeed, I'll post up the branch here.

Just want to warn you that at best, what I have here could be used as an inspiration, if that. I don't think it will be much help to you at its current state. I've written this with the philosophy of "get it working and get it out there as soon as possible" not "make the code easy to understand and extend".

If you have access to the forum's database, I suggest using that to do your search. If it uses something like SQL, it should be much much easier.
I can also give you a list of all of the tools and packages I used, if you're interested. If I had those tools from the start, I might had done a better job. I'll probably rewrite it completely if I find some time.

[Parisbre56]

New version can split the output into multiple files.

Simply click the "Stuff" menu and then the "Other Options" menu item. The posts per page is the option you want.
Option only works for html output. Plain text output is still one huge file. I should probably add an option to enable it or disable it.

You can also mess with the other options there if you want. Although the checkboxes there don't do anything yet.


Stopped the program from failing when downloading image links with no file type (like this one: https://i.chzbgr.com/maxW500/7047413504/h45184DA5/)
Fixed a divided by zero error that happened in some rare cases.

Spoiler: Minor bugs for me to fix later (click to show/hide)

Can't get page numbers to show properly when pressing the ... part of the change page. Even though all pages are shown correctly in the first second and middle points, I can't make the ... points to show the correct page numbers when they expand, even though though they point to the correct page files. Thus we are left with the following page number problem:

Not expanded:
0 ... 2 3 [4] 5 6 ... 8

Expanded (by clicking the ...):
0 2 2 3 [4] 5 6 8 8

Where the red 2 points to page 1 and the red 8 points to page 7.


There is sometimes an extra empty final page. Need to check my divisions, probably getting an extra page in there somewhere.


Add a check for <posts per page> equals 0 and throw an appropriate error. Right now it just stops working.

[Parisbre56]

New version can combine multiple files into one and order the posts based on time.
Unfortunately, it can only process times in the %a %d-%m-%Y, %H:%M:%S format.

So make sure to log in and change your date format to that if you want to use it until I get around to fixing it.
The time format setting is in Profile -> Look and Layout.

Here's the entire ER subforum, if you wanna see what the output looks like.
It's about 100 MBs and exactly 1000 pages long in 50 posts per page format.