Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[S31-Syntax]

if you wanted you download the wiki you could use this
http://www.httrack.com/
ideally someone would do so every so often (say 2-3 months) zip it up and upload it somewhere else so the server bandwidth doesn't get taken by people just downloading the whole thing.

That... sounds like something I would be willing to do.

[S31-Syntax]

Wow, looks like i'm downloading everything. I see... every current version of DF there. OSX, Windows, Linux.

Some page relating to robots. Robots.txt??


10... 15... 30... so many instances of robots.txt!! WHY?!

[Max White]

Max White cancels post: Confused by Robots.txt.

[S31-Syntax]

5h 10m in, still rolling. its throttling itself to about 26kb/s... so this is going to take a while.
690megs so far. When its all said and done, i'll probably 7z it and belch it onto filefront or something.

If its under 2 gigs i'll see if Ubuntu One will take it. I don't use it for much anywho.

[Derekristow]

I suggest putting it in a torrent, that way people with fairly horrible connections can still access it. 
Can torrent files be updated?  That would make distributing updates incredibly easy.

[Dwarf]

I'd say torrents are a bad idea except for files which get a fair amount of interest, which wouldn't be the case here. I'd say it would run out of seeders one day and be stuck with a few leechers.
Just upload it somewhere.

[Spade]

Maybe for the people who only play DF when they lose internet connection, like me.

[G-Flex]

I honestly don't understand the purpose of this. How often are you going to be at a computer, playing DF, but without internet access?

[Dwarf]

I'd download it, already the archive-y value it has for me makes it attractive

[G-Flex]

"Archive-y value"? The wiki already archives everything.

I'm also a little concerned about how much of the wiki's bandwidth this kind of thing would use up if multiple people do it.

[Dwarf]

Arhive-y as in. having it right here, on my HD.

And I guess it'll be mirrored on another site so the wiki bandwith need not be strained.

[Jake]

I honestly don't understand the purpose of this. How often are you going to be at a computer, playing DF, but without internet access?

Long plane, railway or bus rides spring to mind, as do long holidays in the middle of nowhere. I'm also frightened to even speculate how my new Windows CE minibook's browser is going to handle the wiki now that someone had the inspired idea of dropping support for Internet Explorer.

[TheCze]

I wish I have had the wiki when I was in hospital for 2 months

[S31-Syntax]

I will archive it in a way that it will allow people who don't have constant internet connections to still have access to the (so far) most recent version of the wiki. As long as its only me sucking the bandwidth at its throttled 26kb/s then the wiki won't be hammered so much.

On top of that, its also downloading copies of every DF version available. we're down to 28_181_39b right now. Those are coming from mirrors. It also seems to be following ALL links outward and is trying to grab things from the pages at the end of those links... ads included. But those attempts seem to be stopped by this robots.txt file. I'm guessing a "no trespassing" sign for bots like this program i'm using.

I'm doing this for several reasons.
1) secondary archiving is nice
2) having the wiki on hand at any time evar is also nice
3) I really like having three things in a list... so here.

This thing looks like it will include things like...
All the DFtalks
All the DF versions
The entire wiki
robots!
All the DF mods that it can find
All the secrets of the universe

[Greiger]

Just for the hell of it, I'll confirm that robots.txt is indeed an anti-bot thing.  It mostly just tells bots that there isn't anything worth looking at beyond that point. 

As much for the bot's sake as the site from the way I understand it.  Since some site structures can confuse the hell out of them apperantly.