Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[ashton1993]

Hello, I'm an AI researcher. I have built a chatbot, and created this game: a real-life turing test.

You will be randomly connected to either another human player or robot. You will chat for 30 seconds, and then try to predict whether you were talking to a human or robot.

The points system:

* Correctly guess who you're talking to: 1 point
* Convince a human they're talking to a robot: 2 points
* You guess wrong: -1 point
* Stranger guesses correctly: -1 point

I'll use the chat recordings and predictions to improve my AI.

Let me know what you guys think!

https://turing-game.tech

[Trekkin]

Pretty sure you're a robot.

Do I win?

[ChairmanPoo]

Apparently I'm pretty good at impersonating chatbots

[Reelya]

It's a nice idea, but I think a different system would work better to collect data.

The total time-limit on the conversation isn't good. A possible alternative is to limit the number of responses on each side, and have a time out on typing per-side. It's just a typing race right now and there's very little talking to the other person going on. For example, if the conversation length is 30 seconds, each side would receive 15 seconds of typing time.

"turns" of typing rather than typing at once would also give you data in the format of "statement => response" which would be much more useful as training data for some sort of AI. You'd record what was said as well as timing data (the time spent typing, how long before typing began etc), and this would allow you to mimic the user's input with your AI in a more convincing way.

Second, once a conversation ends, you can in fact just automatically start another one, and shunt the form to fill out about the previous conversation to the side.

Rather than having the conversation start when you click, have it only start on say 30-second boundaries, and you're automatically logged into a new conversation at the 30-second mark, while the form for the previous round gets shunted to the side. This would facilitate things so that humans keep chatting for longer. If the players have to click "play again" then this is a break-point at which people have to actively decide to keep playing. By assuming they're going to keep playing by default, and rolling that directly into a new chat, you'll get much higher retention.

Also, you should just let people keep playing even if no humans are available. The human in question still won't know whether they're talking to a bot vs a human, and the extra responses will be useful to train your bot. This should only matter if there's literally only one human on your site, so you should still do it.

Also other small things: a bit of CSS to make things look nicer. Horizontally center the div, because it looks much more professional than something shunted into the corner. Take the clickable options out of the dialogue box, and make them into a toolbar along the top, and add a matching footer-bar. A couple of simple things will make it look like a proper website.

[Trekkin]

It's also more than a little suspicious that there's no indication whatsoever that any legitimate research organization has anything to do with this, which is not helped by the questionable experimental design.

So, ashton1993, does anyone with, say, a professorship in computer science know you're an "AI researcher?"

[Reelya]

Looks like a student project but not an especially well thought-out one.

He could do some research into user-retention first to get it off the ground better. It become boring after about 2 conversations.

[ashton1993]

It's also more than a little suspicious that there's no indication whatsoever that any legitimate research organization has anything to do with this, which is not helped by the questionable experimental design.

So, ashton1993, does anyone with, say, a professorship in computer science know you're an "AI researcher?"

OK, aspiring researcher. Here's a video series I made with some science experiments: https://www.youtube.com/watch?v=0Z0N0mlCHbc&list=PL5EzazM9HRyVocL4HlQ0zE1rJnOEjOnsv

Looks like a student project but not an especially well thought-out one.

He could do some research into user-retention first to get it off the ground better. It become boring after about 2 conversations.

I built the interface pretty fast, it could definitely be better. Still recording hundreds of conversations though, so I guess it's good enough?

[Reelya]

If you streamline it into a "continuous mode", where each conversation happens automatically at 30-second intervals (e.g. 19:00:00, 19:00:30) etc you could get a lot more data then. Doing it according to a shared clock will also maximize your chance of pairing two humans, eliminating the "there are no humans to talk to" effect, which i experienced multiple times in a few minutes, between separate play sessions, leading me to quitting completely. Sure, I don't mind putting time into something, but I highly object to having my time completely wasted in that fashion.

"game over" screens are also a natural break-point in gameplay, at which people stop and go do something else. The frustration over being told there are no pairings possible also sent me away at that point. Have it by default just start another conversation at the 30 second mark instead of being given a choice.

I'd also do what I suggested, and have turns with a Chess-like 15-second counter per "side". The data will end up more orderly (statement->response pairs) and less frustrating for the user when the other person isn't replying.

[Trekkin]

OK, aspiring researcher.

So what's your end goal here, then? You're going to go apply to graduate school and stick on your CV that you halfway-plausibly lied about doing research already, so hey, fake it till you make it, right? Or maybe just take whatever machine learning thing you've put together to Google and hope one guy with no qualifications can make something to impress actual software engineers? Maybe try to Kickstart your way into a startup with a Youtube series and basic web design skills?

Garage science is by far the worst way to get to do real science. 

EDIT: Also, hundreds of conversations is a laughably small training set. So would thousands be.

[Reelya]

The "I guess it's good enough" comment is especially worrying in retrospect. It's not good enough, it's horrible. A few hundred data points of low quality isn't good enough. And that's on the back of initial interest, which trails off quickly.

Assuming the average person does 20 conversations, that's only like 5 people or something accounting for a hundred data points. It's doing crap.

All the advice I've given is entry-level no brainers such as centering the goddamn div. It's not hard, but it add 100% more to the look and feel. "It's good enough" to ignore basic common sense design principles like having a non-awful layout.

[ChairmanPoo]

Why give points for pretending to be a robot? It's useless for your purposes.

Are there any bots in that chat even?

[ashton1993]

The "I guess it's good enough" comment is especially worrying in retrospect. It's not good enough, it's horrible. A few hundred data points of low quality isn't good enough.

Assuming the average person does 20 conversations, that's only like 5 people or something accounting for a hundred data points. It's doing crap.

It's thousands now, & it's actually fine. The chatbot has already been trained on millions of conversations, just not chats it's participated in itself. This dataset will help fine-tune it

Why give points for pretending to be a robot? It's useless for your purposes

Not all humans will pretend to be robots & it's pretty easy to filter (based on predictions over multiple conversations). Think they'll be interesting data points to analyse, haven't figured out what to do with these ones yet.

[ashton1993]

If you streamline it into a "continuous mode", where each conversation happens automatically at 30-second intervals (e.g. 19:00:00, 19:00:30) etc you could get a lot more data then. Doing it according to a shared clock will also maximize your chance of pairing two humans, eliminating the "there are no humans to talk to" effect, which i experienced multiple times in a few minutes, between separate play sessions, leading me to quitting completely. Sure, I don't mind putting time into something, but I highly object to having my time completely wasted in that fashion.

"game over" screens are also a natural break-point in gameplay, at which people stop and go do something else. The frustration over being told there are no pairings possible also sent me away at that point. Have it by default just start another conversation at the 30 second mark instead of being given a choice.

I'd also do what I suggested, and have turns with a Chess-like 15-second counter per "side". The data will end up more orderly (statement->response pairs) and less frustrating for the user when the other person isn't replying.

Good ideas, I'll try to get around to them next week 

[ChairmanPoo]

Can you tell a robot apart from a human?

What is a human?



A miserable little pile of secrets! But enough talk… Have at you!

[Trekkin]

Think they'll be interesting data points to analyse, haven't figured out what to do with these ones yet.

So apart from everything else, you designed the experiment backwards. That is not a good look. Neither is changing your data collection in the middle of the run.

This project will not help you get anywhere you want to go, and you would be much better served accumulating resume fodder by working with someone who does this professionally in order to actually get into a program that will teach you the right way to do what you've tried to do here.