Johnathan Zittrain “Minds for Sale”
This video was brought to my attention by a high school friend, Fred Benenson. Fred works simultaneously for Creative Commons and Kickstarter and is also the guy behind the Emoji Dick Kickstarter project which aims to use Mechanical Turk to translate Moby Dick into Emoji. I like to plug my friends.
Anyway, here is the video he shared. Some of my thoughts are below.
John starts with a friendly introduction and quickly jumps into a story about Tweenbots. The idea is that there is a cardboard robot with boring capabilities (can only walk in a forward straight-line) that seeks to get to location x. It asks people nearby for help and manages to find it’s way to x with the help of 40 people. John points out that this is a neat metaphor for Internet routing but also points out that the real meat of the experiment is the ability to bank on trusting the world to help get a task completed.
After a story about reaching out to the Internet to put help together a book cover for one of his new books he tells a story about LiveOps, a site that creates operators (as in call center operators) out of anyone on the Internet. LiveOps helped replace the overwhelmed phone operators during Katrina and managed to successfully put together the operator network in around 7 hours. Not exactly the fastest turn-around time, but certainly impresive given the resources Katrina’s disaster likely required.
The economics of MTurk are fascinating. Chris’s paper discusses the considerably high quality and incredibly cheap work that can be done via MTurk. John discusses some attempts at getting work for free by creating games around the hits (mturk tasks, for the uninitiated). One such game involved making the components of electronics components more compact. The makers of the game don’t know the answer to the games, but as people play the game they are contributing to new designs and design automation is taking place.
When I was first playing with MTurk, I posted some hits asking users “What do you think of Obama?” with a freeform text field and offered 10 cents per answer. All of the answers were quite good and some were very long. The long answer was intriguing because it told me that some people have a lot to say on this topic. So I thought I’d try posting the 10 more hits but offering 5 cents per answer this time. There seemed to be no difference in quality. Fascinating. I tried posting 10 more hits, this time offering no reward at all. I received only one bad answer and the other 9 were still great. It seems there is a loose relationship between the reward and quality of answers. Generally, you can get free work if the hits are interesting, fun or, as in my example, provocative. I consider this somewhat analogous to the way Wikipedia is built. The 1,400 primary people behind Wikipedia are the type of people who enjoy writing encyclopedia’s for fun, after all.
Since we’re on the topic of economics, it’s worth mentioning that MTurk has been referred to, somewhat cynically, as an electronic sweatshop where people do work for pennys. John tackles this topic by bringing up the concerns that brought about modern labor laws and then contrasts them with what’s offered by MTurk. For example, if we turn one of those MTurk games into something embedded in a friendly screen with Sesame Street type modeling for the UI, you could create a child labor market. Also along the lines of ethically dubious behavior: I have found people gaming review based sites by paying turkers to post positive reviews! John found this too. I didn’t realize knowledge of the review gaming was so widespread, but I’m glad to see it. It’d be nice if Amazon would do something to prevent this!
It seems like the worst example of using Mechanical Turk purely for it’s do-this-for-me abilities is a hit that wants someone to review something positively and then never mention that they did this on behalf of another group. The name hit seems perfect, because it’s like posting a job offer for a do-this-for-me hitman. A mercennary of sorts for minimally time consuming tasks. This is compelling enough to warrant some action to filter these things the way we filter spam.
John’s objections to ignorance are interesting to me. They come across, in ways, as being a bit more paranoid than I am. I’m not too concerned by the tasking because I believe a spam handling approach can generally solve the issues. At the very least, you can reduce the markets in such a way that it’s very difficult to get these kinds of tasks out. Filter the list of tasks. Some will slide through, so a flagging system would also have to exist enabling users to police the system too.
Open Mind was brought up. I hadn’t seen Open Mind until just now and am amazed to see it is a Django site much like the Wikitrans system I’m building. It also uses Django and Pinax for the foundation. Good choice!
Open Mind is a system from MIT for interacting with Mechanical Turk for farming out tasks that are useful for researching patterns. My new friends at University of Maryland introduced this to us. Instead of actually checking out the website, I went to the code to see how they were interacting with Mechanical Turk. I didn’t think it could have been boto, because that funcitonality wasn’t working properly for a while.
Open Mind has solved a problem I have been working for, but they did it in a centralized control kind of way. The tasks management is built inside Django such that a task goes to Mechanical Turk and builds the hit answer management system out of their website instead of Amazon’s by embedding the content used for the hit inside a frame. This essentially reduces Amazon to just being a market place system instead of a markplace with a data warehousing system. When using Python boto, you are putting all of the data inside Amazon’s database and simply querying that database over time or after receiving a notification. If you insert some hit data, the database uses some of that data for the display methodologies, like the url for the contents of a frame, and sends the hit to http://ourexperiment.com. The data comes directly into ourexperiment.com and you send whether or not you like the answer, the reward, any bonus information, etc, to Amazon for handling the distribution.
I am not totally sold on whether this is the right way to do it but I think management through Amazon has value. It’s exactly what they offer through their API after all. Creating a marketplace should be the priority and offering data storage, via creating your data through API calls, should make that easier. Therefore, just choose whether you want the data visible to Amazon or you want to keep it all locally. Perhaps a library should be born.
I’m working on boto as need arises for certain functionality. Next up is a workflow management system. It’s going to be a django app with no url’s behind it. Any logic that is mechanical turk related will be in a library. I am very interested in building out boto and Mitch Garnaat (maintainer of boto) has made me a committer. I want to build boto into a full representation of Amazon’s mturk API and introduce the dev and users mailing lists so everyone can help. The boto development is mature but the mechanical turk stuff is fairly new. Meaning, I’d love help.
I believe the research benefits of this tool are numerous and exciting. So much progress has been made in analyzing statistics and then using that data to train computer intelligence. I agree with Andrew Ng’s statement that machine learning is one of the most exciting, if not the most exciting, fields that exist for study. It’s the ultimate in science because we reduce the interpretation of correctness and efficiency, with extensive use of mathematics and philosophy, into very small questions about the world with a very small scope. Fans of Edmund Burke or Michael Oakeshott will be proud.
And so my view of mturk’s unethical potential is different from John’s because I believe in the openness of information and also the ability of society to act responsibly as new information is created. This is the essence of law, isn’t it? We can simultaneously have this research system and public contribution to knowledge while also promising to catch the bad guys. MIT and Stanford are my favorite examples of good guys in their distribution of lectures, assignments, answers and so on for people to teach themselves the information these groups (MIT and Stanford) are generating. This is new knowledge, in many cases, so let’s agree on what we consider harmful. I think a good place to start is simply with detecting people looking to spread misinformation and keep the explanations/definitions of misinformation public and available for everyone to discuss.
On top of that, there is a question regarding the digitization of mechanisms that already exist. It seems too similar to an argument for accepting whatever defficiencies currently exist. I don’t see the point of not at least asking the questions that could provide answers. “Please upload a picture of a red balloon.” OK, here is one. Someone can audit this answer. In fact, that’s a typical response to solving the crimes of misinformation, like the hits that introduce false opinions to review sites. I’m willing to guess that almost everyone cringed a little bit when they first heard about that. But policing the communications sure sounds ominous, doesn’t it? Like something out of a sci-fi novel. But that’s where we’re at as a society in terms of progress. What was once sci-fi is now current. We must figure out good answers to the hard social questions and keep trying to improve. We can be great, if want to be. It doesn’t have to be 1984.
Towards the end, John drops a bombshell idea. He says he wouldn’t mind having younger children spending time communicating about data on wikipedia, but he isn’t sure how he’d feel about farming out tasks that children can use to earn money for their school. I agree that these are different! Kids working on knowledge processing systems are learning by improving data and inspecting it for inaccuracies. It’s an education right there in itself along with also being a positive contribution to society’s ability to learn. Letting a school turn it’s children into workers, however, is the same as creating a child labor market. But the purpose of the school is the child’s education, not their contribution to the schools funds so we must filter these tasks accordingly. Explaining this to our kids will educate them about social responsibility and allowing them to correct us will engage them, and all of society, in the discussion. Score!
Where do world.all_people() want to go tomorrow?
—
