Real world computer vision systems typically have some intrinsic value in their underlying business use. Serving the the right image in a search result ad might be worth $0.001 and counting nuclear particles in material images might be worth $10,000. In general we want to build systems which produce sufficiently accurate results within the budget. Although human interaction can improve accuracy in many algorithms, it also costs money. Most computer vision research has focused on purely automated algorithms, arguing that human labor is much too expensive to include in operational algorithms. This is equivalent to finding solutions for the $0.00 value case. We should explicitly investigate joint algorithms using computers and human labor, and report accuracy as a Cost vs Accuracy curve as additional human labor is inserted. We investigate some representative computer vision tasks, e.g. Object Detection, Image Matching, and X. We introduce several general strategies for combining existing vision algorithms with human labor, e.g. PruneFalsePositives, Y, Z. These strategies provide for increased accuracy in those cases when the task has positive value. Learn more.
Whether you need help gathering data, labeling machine learning training examples, running experiments, or transcribing audio, we turn today to crowdsourcing platforms such as Amazon Mechanical Turk (www.mturk.com). However, these platforms are notoriously bad at ensuring work quality, producing fair wages for workers, and making it easy to author effective tasks. It's not hard to imagine that we could do better.
This research will be a complete design, implementation, launch, and evaluation of a new crowdsourcing platform. What would it take to create an effective marketplace? One where workers have more power in the employment relationship, or could take additional responsibility for the result quality? How might we design such a market? Could we launch it and become the new standard? This research in human-computer interaction will involve a combination of design thinking, web development, and experimental design.
At a 1906 county fair, the statistician Francis Galton watched as eight hundred people competed to guess the weight of an ox. He famously observed that the median of the guesses, 1207 pounds, was within 1% of the true weight. Since then, this "wisdom of the crowd" effect has been documented in a variety of domains, ranging from trivia questions to combinatorial optimization. Yet, perhaps surprisingly, we have little understanding of why, or even when, the phenomenon holds.
In this project, we will leverage the wisdom of the crowd (you!) to systematically investigate the wisdom of crowds. We will develop a large, online experiment, in which each team creates a module to study the wisdom of the crowd in a specific domain. Ideally, each team will collectively have some experience in web development and statistical analysis. At the end of the project, we aim to submit our results to a top computer science conference, such as ICWSM, EC, or WWW.
CrowdFlow: Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility. Alex Quinn, et. al. U. Maryland TR 2010. [pdf]
Crowdsourcing Annotations for Visual Object Detection. Hao Su, et. al. AAAI HCOMP 2012. [pdf]
The HPU. James Davis, et. al. IEEE CVPR 2010. [pdf]
Soylent: A Word Processor with a Crowd Inside. Michael S. Bernstein, et. al. ACM UIST 2010. [pdf]
Distributed Human Computation. Serge Belongie, et. al. [link]
