PhD Defense: Toward Theoretical Measures for Systems Involving Human Computation

Abstract

As we enter an age of increasingly larger and noisier data, the dynamic interplay between human and machine analysis grows ever more important. At present, balancing the cost of building and deploying a collaborative system with the benefits afforded by its use is precarious at best. We rely heavily on researcher intuition and current field-wide trends to decide which problems to approach using collaborative techniques. While this has led to many successes, it may also lead to the investment of significant time and energy into collaborative solutions for problems that might better have been (or have already been) solved by human or machine alone. In the absence of a secret formula to prescribe this interplay, how do we balance the expected contributions of human and machine during the design process?

Can we describe the high-level complexity of these systems with the same robust language as we use to describe the complexity of an algorithmic system? In this work, we investigate the complementary nature of human and machine computation as used in visual analytics and human computation systems, and present a theoretical model to quantify and compare the algorithms that leverage this interaction.