Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 6 7 [8] 9 10 11

Author Topic: Bay12_SS: The Shitpost Simulator.  (Read 21482 times)

Sensei

  • Bay Watcher
  • Haven't tried coffee crisps.
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #105 on: January 31, 2016, 08:02:14 pm »

Do we have an uncompiled form of the script you're using? It might only grab the first 10,000 words or something.
Logged
Let's Play: Automation! Bay 12 Motor Company Buy the 1950 Urist Wagon for just $4500! Safety features optional.
The Bay 12 & Mates Discord Join now! Voice/text chat and play games with other Bay12'ers!
Add me on Steam: [DFC] Sensei

O.Wilde

  • Bay Watcher
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #106 on: January 31, 2016, 08:03:30 pm »

It's the code in the first post.
Code: (This) [Select]
import random

class Markov(object):

def __init__(self, open_file, chain_size=3):
self.chain_size = chain_size
self.cache = {}
self.open_file = open_file
self.words = self.file_to_words()
self.word_size = len(self.words)
self.database()

def file_to_words(self):
self.open_file.seek(0)
data = self.open_file.read()
words = data.split()
return words

def words_at_position(self, i):
"""Uses the chain size to find a list of the words at an index."""
chain = []
for chain_index in range(0, self.chain_size):
chain.append(self.words[i + chain_index])
return chain

def chains(self):
"""Generates chains from the given data string based on passed chain size.

So if our string were:

"What a lovely day"

With a chain size of 3, we'd generate:

(What, a, lovely)

and

(a, lovely, day)
"""

if len(self.words) < self.chain_size:
return

for i in range(len(self.words) - self.chain_size - 1):
yield tuple(self.words_at_position(i))

def database(self):
for chain_set in self.chains():
key = chain_set[:self.chain_size - 1]
next_word = chain_set[-1]
if key in self.cache:
self.cache[key].append(next_word)
else:
self.cache[key] = [next_word]

def generate_markov_text(self, size=25):
seed = random.randint(0, self.word_size - 3)
gen_words = []
seed_words = self.words_at_position(seed)[:-1]
gen_words.extend(seed_words)
for i in xrange(size):
last_word_len = self.chain_size - 1
last_words = gen_words[-1 * last_word_len:]
next_word = random.choice(self.cache[tuple(last_words)])
gen_words.append(next_word)
return ' '.join(gen_words)
Logged
What could pre-industrial societies do, run a bunch of cattle off a cliff? Boo fucking hoo I'll be crying for them while I just dump these litres of acidic chemicals into this river. Scrubs.

SirQuiamus

  • Bay Watcher
  • Keine Experimente!
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #107 on: February 01, 2016, 08:25:54 am »

The markovgen script works as it should. I tried feeding it the entire King James Bible as a .txt file (788031 words without verse numbers and PG preambles), and this is what came out:

Spoiler (click to show/hide)

As you can see, the names in that extract are from all over the Bible, not just the first few books of the Old Testament.

EDIT: Another way to test the script is to call its file_to_words function directly and write the output into a file:

Code: [Select]
import markovgen
file_ = open('/your/folder/of/choice/textfile.txt')
markov = markovgen.Markov(file_)
output = markov.file_to_words()
file = open('/your/folder/of/choice/outputfile.txt', 'w')
file.write(str(output))
file.close()

This produces a comma-separated list of all the words in the textfile, and if the output file is longer, you can be sure it works.
« Last Edit: February 01, 2016, 08:43:07 am by SirQuiamus »
Logged

Avis-Mergulus

  • Bay Watcher
  • This adorable animal can't work.
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #108 on: February 01, 2016, 08:43:41 am »

I'd add a method to check that the number of opening parentheses matches the number of closing parentheses.
Logged
“See this Payam!” cried the gods, “He deceives us! He cruelly abuses our lustful hearts!”

WealthyRadish

  • Bay Watcher
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #109 on: February 01, 2016, 11:34:23 am »

I kind of like it leaving parenthesis hanging. It's that extra layer of irritating as you're parsing it (with it sitting in the back of your mind, like those people who put multiple sentences in the same set of parenthesis. You start wondering after the first full stop if they're really keeping track or just forgot, and by the end you have to reread it to make sure you didn't completely miss the meaning because of horde of college age warriors take to the best thing about it in my opinion. I can't stand that Alaska border, being supreme dictator of the oven, what amazing spat did I just miss?
Logged

TheBiggerFish

  • Bay Watcher
  • Somewhere around here.
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #110 on: February 01, 2016, 11:37:32 am »

...Very funny (not
Logged
Sigtext

It has been determined that Trump is an average unladen swallow travelling northbound at his maximum sustainable speed of -3 Obama-cubits per second in the middle of a class 3 hurricane.

Graknorke

  • Bay Watcher
  • A bomb's a bad choice for close-range combat.
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #111 on: February 01, 2016, 12:33:51 pm »

We need to get a job to help Toady with work on Armok: Adams said that the text-based graphics forces players to get a job in order to finish his previous work, Armok.
Got me. Fuck.
Logged
Cultural status:
Depleted          ☐
Enriched          ☑

Xantalos

  • Bay Watcher
  • Your Friendly Salvation
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #112 on: February 02, 2016, 02:22:26 am »

Well then. I'm rather disinclined to actually putting myself through this thing, but if anyone wants to try, feel free.

*postcount 35,309*

Should be an entertaining mix.
Logged
Sig! Onol
Quote from: BFEL
XANTALOS, THE KARATEBOMINATION
Quote from: Toaster
((The Xantalos Die: [1, 1, 1, 6, 6, 6]))

TheBiggerFish

  • Bay Watcher
  • Somewhere around here.
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #113 on: February 02, 2016, 06:49:06 am »

Hoo boy.
Logged
Sigtext

It has been determined that Trump is an average unladen swallow travelling northbound at his maximum sustainable speed of -3 Obama-cubits per second in the middle of a class 3 hurricane.

SirQuiamus

  • Bay Watcher
  • Keine Experimente!
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #114 on: February 05, 2016, 05:36:28 pm »

I've re-written O.Wilde's scraper script so that it removes the contents of unwanted tags, rather than just the tags themselves.

Code: [Select]
from bs4 import BeautifulSoup
import requests
import html
import re
n = 0
postcount = 9769
user_id = '21412'
url_ = 'http://www.bay12forums.com/smf/index.php?action=profile;u=' + user_id + ';area=showposts;start='
file = open('\your\folder\LSP.txt', 'a')
while n <= postcount:
    url = url_ + str(n)
    page = requests.get(url)
    print(page.url)
    soup = BeautifulSoup(page.text, 'lxml')
    list_ = soup.find_all('div', class_='quoteheader')
    list_.extend(soup.find_all('div', class_='quotefooter'))
    list_.extend(soup.find_all('div', class_='spoiler'))
    list_.extend(soup.find_all('div', class_='codeheader'))
    list_.extend(soup.find_all('blockquote'))
    list_.extend(soup.find_all('code'))
    for tag in list_:
        tag.decompose()
    soup2 = soup.select('div.list_posts')
    soup3 = html.unescape(str(soup2))
    soup4 = re.sub('(\s{2,}|\s,\s)', ' ', soup3)
    soup5 = re.sub(r'[^\x00-\x7F]', '', soup4)
    soup6 = re.sub('<div class="list_posts">', '\n', soup5)
    soup7 = re.sub('<br/>', '\n', soup6)
    soup8 = re.sub('(<.*?>|[\[\]])', '', soup7)
    file.write(soup8)
    n = n + 15
file.close()

Here's a complete dump of all LSP posts, by the way.

Quote from: LSP_SS
it as if ukraine will pay the bill that in attempt or someone of his way in, steal an anvil inside her.

give us your votes we'll kill more of a strutball, either. 900 of heat. you're attempting to ruse that one nerd claims

if you need more oxygen by percentage if there's something you haven't already given

Quote from: LSP_SS
to cut a bitch catherine the great things were sent or where I'm going?) allowing yours to ensure historical accuracycitation needed HUZZAH
* LordSlowpoke stands up on a map.

Buddy from the wall. I just bundle.

Quote from: LSP_SS
this is better - it's either not report at all find free nethost toss tinyboard onto nethost problem seems solved save the you

your pipes can handle a bit so the third bush gee, time to earn moneyi have to do more things thrown in than just a bit too high us

increase the amount of comparing things do
« Last Edit: February 05, 2016, 05:38:17 pm by SirQuiamus »
Logged

TheBiggerFish

  • Bay Watcher
  • Somewhere around here.
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #115 on: February 05, 2016, 05:38:04 pm »

SirQiuamus:Spoiler tags include responses though?
I don't know your code though.
Logged
Sigtext

It has been determined that Trump is an average unladen swallow travelling northbound at his maximum sustainable speed of -3 Obama-cubits per second in the middle of a class 3 hurricane.

SirQuiamus

  • Bay Watcher
  • Keine Experimente!
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #116 on: February 05, 2016, 05:49:03 pm »

SirQiuamus:Spoiler tags include responses though?
I don't know your code though.
Yeah, that's one drawback. You should really try your hand at this if you have an idea about how to sift the relevant post content from all the other weird shit that goes into the spoiler tags.

I'm just an ignorant n00b with too much "free time."
Logged

chaotic skies

  • Bay Watcher
  • Vibing in anti-space
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #117 on: February 05, 2016, 08:37:45 pm »

Grr...

LXML won't install. This thing's been sitting here saying "Installing lxml" for the past 4 days.
Logged
Don't let me start a forum game, smack me with a paper towel roll if needed

Professional Thread Necromancer

WealthyRadish

  • Bay Watcher
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #118 on: February 06, 2016, 02:44:28 am »

The scrapers are nice, but I think more carefully crafted files are the way to go. Here are a few generated from only LW sentences containing the word "remove":

Quote from: LW_remove
That struggle is much like the struggle against temptations, do you remove their clothing they burn to death very quickly. That's why I remove mud from the premises! Really I am not most worried about that, but am most worried about the requirement that operators be required to maintain the technical capabilities to remove the communists and then die off once their use was over [and of course, things do

Quote from: LW_remove
she didn't even remove evidence of her pre-blackfacing it up on the internet, they'll destroy your files. Or that irregardless of state law, police will go as far as to plant or remove evidence of her pre-blackfacing it up HNNNNNGGGGGGGGG AVAST BEST WAIFU SAVE ALL MIDGET DWARF AND CALL PUTIN BLUFF BEST DAY OF LIFE REMOVE CRYPTMAN FROM PREMISES GIVE BACK YOG MERKEL CLAY If you remove yourself from them, or do you overcome them?

Quote from: LW_remove
I am not most worried about the requirement that operators be required to maintain the technical capabilities to REMOVE KRAUT you are the STINK MERKEL go back to YOG-SOHOTH To be honest in the unlikely event I was offered com mod I'd turn it down simply because it'd mean I wouldn't be able to remove a part of their body they won't realize the effects of. Should also remember to bring fire starters, to remove Andals with.

Quote from: LW_remove
It's important because 'remove kebab' is a socialist and the circumstances around the board members leaving e.t.c. I don't think Russia desperately trying remove American shitlords from the premises first and concern their livelihood second. And to remove encryption. AND REMOVE ISRAELIS FOR A FEE God wills beard, remove beardless from the world stage. Remove statist from the world stage. Remove statist from the premises! To be honest in the observation alone there was discussion, we talked about the likelihood


Logged

O.Wilde

  • Bay Watcher
    • View Profile
Re: Bay12_SS: The Shitpost Simulator.
« Reply #119 on: February 06, 2016, 02:47:59 am »

I'm just an ignorant n00b with too much "free time."
Who happens to write some gorgeous code. <3
Logged
What could pre-industrial societies do, run a bunch of cattle off a cliff? Boo fucking hoo I'll be crying for them while I just dump these litres of acidic chemicals into this river. Scrubs.
Pages: 1 ... 6 7 [8] 9 10 11