Topic: Oh jeez, Quicktopic is shutting down (Read 63080 times)

Vector · « **Reply #15 on:** November 08, 2021, 12:42:35 pm »

Quote from: heydude6 on November 08, 2021, 02:15:08 am

Posting to show my interest in joining the archival efforts. Certain standout games are missing from the notable games archive, like Vector's masterpiece The Lonely Prince: He Who Shall Serve (also known as princess mafia). I swear it used to be there cause I can't imagine how else I could have discovered it.

My games used to be linked in my sig. I have three in that series: Ex Cathedra, The Lonely Prince, and Iron Diadem.

If you helped save my games it would really save my bacon. Thank you for helping.

heydude6 · « **Reply #16 on:** November 08, 2021, 02:28:56 pm »

Quote from: TolyK on November 08, 2021, 07:42:05 am

Wayback only shows the last page of comments, which isn't very long. Trying to click "all messages" doesn't work for me.

That's why you don't archive the quicktopic itself, but the print preview you get when you click the "print" button in the bottom left corner.

The button actually takes you to a dialogue box when you click it. You have to then click "Show printable page" to get to the actual webpage you need to save.

Important note: You have to immediately click "Show printable page", if you change any of the settings then wayback fails to archive the page for some reason. This poses a problem for multi-page quicktopics. The solution though is to simply click the "All messages" button before you click the "print" button. The immediately click "Show printable page" like usual.

Anyway, I thought you guys would like to see a practical example of what this looks like. I decided to archive Town of Salem mafia since it was hosted on traditional FG&RP rather than the mafia sub and thus it could be easily overlooked.

Spoiler: Let's save some screenspace (click to show/hide)

I hosted these links by editing my post, which happened to be the final one on that thread (thus avoiding necro). This the last bit of archiving I'm going to do for the moment. I have a University project I need to finish by Wednesday. Once I officially begin my efforts though, I'll only be saving the links on my computer, not hosting them (except perhaps on Github). Time is limited and writing posts like that wastes it. I'll focus on hosting after I've saved everything.

There are going to be issues though. Nobody posted the Town of Salem Deadchat link in the main thread. I had to archive and post my own link from the PMs. This wasn't publicly available info to begin with, but I think this is a good proof of the fact that not everything is going to be saved unfortunately. We must soldier on despite this though.

I'll report any issues like that in some sort of weekly report. Maybe some volunteers can contact former players?

Quote from: Skynet on November 08, 2021, 06:53:35 am

perhaps we could divide up the load so that Wayback sees this as two people uploading stuff instead one person uploading stuff?

I don't think Wayback is going to care about that. Their limit is something like 15 archives per minute per user which isn't something I think we're going to breach. Dividing up the load sounds good though, I was hoping to be able to do something like that. I'll be starting from the back of the subforum (ie. page 33), and you can start from the front (ie. page 1). I'll have to come up with a more sophisticated method of dividing up the work if we get more volunteers though.

Toaster · « **Reply #17 on:** November 08, 2021, 11:47:53 pm »

There's an option at the bottom to create a "Shared Topics" page; I've done this with my entire list of topics to try it out. It's 95% Mafia topics with a couple other forum games mixed in.

If that doesn't work, below is spoiled a complete list of mine:

Spoiler (click to show/hide)

Is this helpful? I'm pretty sure there's nothing in there that needed to be kept secret any more.

TolyK · « **Reply #18 on:** November 09, 2021, 05:19:58 am »

I get a 403 forbidden for all QuickTopic URLs. Maybe because I'm in Eastern Europe?

I can't even access my own created QuickTopic chats. :/

KYOSN on WayBack:

Dead Chat

Mafia Chat

Cult Chat

Shakerag Chat

Observer Chat

Mason Chat

The "print" trick shows this:

Code: [Select]

Hrm.

The Wayback Machine has not archived that URL.

heydude6 · « **Reply #19 on:** November 09, 2021, 05:27:21 am »

Quote from: TolyK on November 09, 2021, 05:19:58 am

The "print" trick shows this:

Code: [Select]
Hrm. The Wayback Machine has not archived that URL.

Had that issue too. That's why I posted this:

Quote from: heydude6 on November 08, 2021, 02:28:56 pm

Important note: You have to immediately click "Show printable page", if you change any of the settings then wayback fails to archive the page for some reason. This poses a problem for multi-page quicktopics. The solution though is to simply click the "All messages" button before you click the "print" button. Then immediately click "Show printable page" like usual.

Did you continue to get that error despite following this advice? Regardless, I should probably add this to my first post as well.

TolyK · « **Reply #20 on:** November 09, 2021, 07:32:34 am »

My issue was that QuickTopic was just not letting me access it from where I live. I made a VPN tunnel to the US and it started working. So, I guess I'll access that way.

Also, if we want to host the messages themselves, we could probably host them on some OneDrive or cloud storage somewhere (I already host a public Google Cloud blob, which is pennies every month, so...). We just need a specified format, or a script to scrape the chats from the topic (though I think QT has anti-bot measures...).

Skynet · « **Reply #21 on:** November 28, 2021, 09:44:11 am »

Just want to check to see how progress is going on saving Quicktopics. I've did some work trying to manually save stuff on Internet Archive, but it may be a bit rough, especially because there is a delay between Internet Archive saving a site and then actually making it accessible for people to look at. Currently focused on trying to save all Quicktopics on the Page 1 of the forum, and will probably find a way to automate the process if at all possible.

Quicktopic will shut down on Saturday December 11th, which is two weeks from now. We should probably try to save all Quicktopics by the end of this week, I guess?

heydude6 · « **Reply #22 on:** November 28, 2021, 10:25:11 am »

School ended up being more demanding than I expected. I was only able to find one day in my schedule where I could work on archival and in that time I managed to save the oldest 3 pages of the subforum. At that rate, everything will be archived within 10 more days which is pretty good I must say.

There are a few caveats though. The oldest page is only half as long as the rest, and quicktopic was not widespread back then. Only Meph's Paranormal games used it in that era, so I didn't have to spend as much time actually using the Wayback machine. It's unknown if I will be able to maintain this rate as bay12 mafia matures, but I also started in the late evening and wasn't working as efficiently as I could have, so it's a crapshoot.

Despite these challenges, I still see this as evidence of archival being doable. With two people working on the case, they only have to do 5 days of work each which is very achievable. At the end, I guess it just comes down whether I'm doing this alone or not...

Skynet · « **Reply #23 on:** November 28, 2021, 11:03:20 am »

Yeah, I think if we work together, we'll probably be able to submit all the links within ~5-10 days. Work's been hectic for me, so I've been pretty busy myself.

The first Bay12Forums page has been backed up, and I also archived the Skynet's Seduction quicktopics as well. (EDIT: Also, the first "U, Robot" game and the CYOM4 - Parakeetkinchat chat. There's some issue with 2uwv6EA2CNe, the quicktopic ID for CYOM4's mafia chat, though...it's still not accessible after 38 minutes. This is going to annoy me for a while.)

I also came up with an automated process as well for this situation. I installed this Python command-line tool (https://github.com/agude/wayback-machine-archiver ), and then writing a quick Ruby script to use this command-line tool:

Code: [Select]

quicktopic_ids = ["p65kRWLUC6a", "qH9b47QBY9u"] #These were ids for Skynet's Seduction quicktopics. Replace with them with the ids you want to backup.

quicktopic_ids.each do |id|
  full_url = "\"https://quicktopic.com/53/H/#{id}?o=0&range=all&print=1&submit=Show+printable+page\"" #necessary to add quotation marks so that bash can properly interpret this URL as a string
  #system("echo #{full_url}")
  puts "We're attempting an archive of #{full_url}"
  system("archiver #{full_url}")
  puts "Archive complete...moving onto new archive!"
  puts "..."
end

Note that for each ID, the script will pause for a bit (since it's submitting the link for you) before printing out an exception message like "ERROR:root:404 Client Error: NOT FOUND for url: https://web.archive.org/web/20211128153251/https://quicktopic.com/53/H/p65kRWLUC6a?o=0&range=all&print=1&submit=Show+printable+page" .

That exception message is due to the lag situation - the page has been successfully submitted and crawled by Internet Archive, but the page itself is not accessible on Internet Archive. I've seen this issue even when I was manually saving these links, so I don't think it's a big deal - but it is annoying, and I do have to later on manually confirm that these links actually do work, just to assure myself that the backup was successful.

Even if one prefers to use the manual process, the fact that the URL template:

Code: [Select]

https://quicktopic.com/53/H/[INSERT_ID_HERE]?o=0&range=all&print=1&submit=Show+printable+page

actually works means that you can skip the "click on the Show printable page button" step - all you really need now is the quicktopic ID.

heydude6 · « **Reply #24 on:** November 28, 2021, 11:25:03 am »

Knowing about the template definitely speeds things up. Actually, your script also looks like something I might use.

How would I go about configuring it to work on my own computer? I've never used a tool like this so you'll have to explain every step to me.

Skynet · « **Reply #25 on:** November 28, 2021, 01:37:18 pm »

Yeah, configuring stuff may be pretty hard. Ruby and Python is already preinstalled on Mac and Linux machines, but if you're using Windows, you'll have to first install those languages as well. And in any rate, no matter what OS you're on, you'll likely want to upgrade to the latest version. Then, you can:

Step 1: Install wayback-machine-archiver:

Code: [Select]

pip3 install wayback-machine-archiver
Step 2: Confirm that the tool is installed:

Code: [Select]

archiver --help
Step 3 (Optional): Check to see if the tool works by passing in the URL of a page you wanted archived...

Code: [Select]

archiver "https://quicktopic.com/53/H/[INSERT_ID_HERE]?o=0&range=all&print=1&submit=Show+printable+page"
Step 4: Copy and paste the code in a *.rb file (let's call it "file.rb") and then execute that code using

Code: [Select]

ruby file.rb***
Since configuration is a rather time-consuming and annoying process, I decided instead to upload the script onto repl.it, a code-sharing website that already had the configuration set up for you. You can see it here: https://replit.com/@tra38/ArchiverBackup#main.rb

Main difference is that instead of using a Python package, I instead used a native Ruby gem - "wayback_archiver" - so I wouldn't need to depend on Python, and could rely solely on Ruby. This was because I couldn't find a way to use both Ruby and Python at the same time...or rather, couldn't find a way to quickly do it. A short-and-dirty hack (using wayback_archiver) was good enough. I tested that script by archiving Gnosia's dead chat and scum chat.

To be able to use the repl.it's script, you just need to make an account on the site. Then you can "fork" the codebase, making the modifications as appropriate and then clicking on Run to execute the code.

P.S.: I also ran the repl.it's script with CYOMA's mafia chat, and it looks like that chat is now finally publicly accessible.

P.P.S: Catch with this new updated script is that you don't get the web.archive.org URL when the archive process is complete (though considering that URL is buried within a bunch of spurious error messages, maybe that's probably a benefit rather than a malus). You would likely need to do a manually confirmation on web.archive.org.

heydude6 · « **Reply #26 on:** November 28, 2021, 01:43:32 pm »

Where do the links get stored once you archive them? It doesn't do much good to save them on the Wayback machine if you can't reach them again afterwards.

Skynet · « **Reply #27 on:** November 28, 2021, 01:58:14 pm »

The links get stored on https://web.archive.org/web/*/https://quicktopic.com/53/H/[INSERT_ID_HERE]?o=0&range=all&print=1&submit=Show+printable+page .

Assuming that web.archive.org has a snapshot available for that page, it will let you know how many snapshots it has and gives you links each of them, organized by the month, day, and year of the snapshot. I was able to verify that the Gnosia's scumchat was saved, for instance, by visiting https://web.archive.org/web/*/https://quicktopic.com/53/H/dqvc7nwNYxHM?o=0&range=all&print=1&submit=Show+printable+page .

If web.archive.org doesn't have a snapshot available, it will give you the option of saving that page (though it may reject your ability to save if it already did a snapshot within the last 45 minutes).

heydude6 · « **Reply #28 on:** November 28, 2021, 02:14:31 pm »

Ok, I think I understand now.

Reverie · « **Reply #29 on:** November 28, 2021, 02:20:57 pm »

I agree with webadict, Discord is robust and ubiquitous enough to serve quicktopic's purpose, maybe except with one caveat. If we use a shared server with hidden channels for each game, there is still the question of its transparency of user activity. Simply being a member there could be a tell, for example.

Bay 12 Games Forum

News:

Author Topic: Oh jeez, Quicktopic is shutting down (Read 63080 times)

Vector

Re: Oh jeez, Quicktopic is shutting down

heydude6

Re: Oh jeez, Quicktopic is shutting down

Toaster

Re: Oh jeez, Quicktopic is shutting down

TolyK

Re: Oh jeez, Quicktopic is shutting down

heydude6

Re: Oh jeez, Quicktopic is shutting down

TolyK

Re: Oh jeez, Quicktopic is shutting down

Skynet

Re: Oh jeez, Quicktopic is shutting down

heydude6

Re: Oh jeez, Quicktopic is shutting down

Skynet

Re: Oh jeez, Quicktopic is shutting down

heydude6

Re: Oh jeez, Quicktopic is shutting down

Skynet

Re: Oh jeez, Quicktopic is shutting down

heydude6

Re: Oh jeez, Quicktopic is shutting down

Skynet

Re: Oh jeez, Quicktopic is shutting down

heydude6

Re: Oh jeez, Quicktopic is shutting down

Reverie

Re: Oh jeez, Quicktopic is shutting down