Software that delegates tricky problems to human workers is changing the nature of crowdsourcing
"I'D RATHER have a computer as my boss than a jerk," says Daniel Barowy. To that end he has created AutoMan, the first fully automatic system that can delegate tasks to human workers via crowdsourcing platforms such as Amazon's Mechanical Turk.
Artificial intelligence is improving all the time, but computers still struggle to complete certain tasks that are easy for us, such as quickly reading a car's license plate or translating a joke. To get round this, people can post such tasks on platforms like Mechanical Turk for others to complete. Barowy wanted to automate this process - and AutoMan was born.
"We think of it as a new kind of computing," says Barowy, a computer scientist at the University of Massachusetts, Amherst. "It changes the kind of things you can do."
Barowy and colleagues designed AutoMan to send out jobs, manage workers, accept or reject work and make payments. "You're replacing people's bosses with a computer," he says.
The quality guarantee is the most important contribution of the work, says Barowy. "Without a mechanism for addressing the quality of worker output, full automation is not possible."
Unlike existing crowdsourcing platforms, AutoMan doesn't attempt to predict the reliability of its workers based on their previous performance. Instead, if it is not sure it has the correct answer, it keeps on posting the same job, upping the fee each time, until it is confident that it does.
"One way to think about it is that it saves the interesting parts, the creative parts, or the fun parts for people," says Barowy. "It's really the best of both worlds. You have the computer doing the grunt work."
AutoMan could be used by developers of apps like VizWiz, in which blind people take a photo of their surroundings and receive a description of the scene. The algorithm could be incorporated into the app, sending the photos to crowdworkers, choosing the correct descriptions and sending them back to the app's user.
Of course, human labour doesn't come free. AutoMan will be given a budget by the app developer and be programmed to keep costs down. Quicker - or higher quality - responses will cost more but AutoMan will manage all of this automatically. Anyone using such hybrid software wouldn't know whether they were interacting with a machine or a crowd of humans - or both.
So how do Mechanical Turk workers feel about being directly employed by a computer? Barowy has received positive feedback so far. When a human boss rejects your work, it can feel personal or unfair. But that's not the case with AutoMan. "People ended up liking the system because it's impartial," he says. The team presented the work at the OOPSLA conference in Tucson, Arizona, last month.
"Any programmer could pick this up and use it," says Michael Bernstein of Stanford University in California. "That's a really powerful thing." Bernstein has developed hybrid computational systems himself, such as Soylent, a word processor that uses crowd workers to edit text.
Barowy's team hopes that their system will make crowdsourcing mainstream, with software delegating tasks to human workers around the globe. "AutoMan might even help grow a new class of jobs that could become a new sector of the world economy," says team member Emery Berger, also at the University of Massachusetts.
People power makes Google work
Google likes to give the impression that it organises the world's information using algorithms alone, but the manual for its human raters tells the true story. Google's small army of home workers have a big say in what sites we are offered when we type in a search term.
The manual, revealed by technology website The Register, gives instructions on how raters should judge whether a set of search results matches a user's intention. They are also asked to make calls on a website's "relevance" - something that popular myth suggests is handled by the PageRank algorithm alone - and "quality". Raters are told to look for websites with content that is less than four months old.
If you would like to reuse any content from New Scientist, either in print or online, please contact the syndication department first for permission. New Scientist does not own rights to photos, but there are a variety of licensing options available for use of articles and graphics we own the copyright to.
All comments should respect the New Scientist House Rules. If you think a particular comment breaks these rules then please use the "Report" link in that comment to report it to us.
If you are having a technical problem posting a comment, please contact technical support.
