Reply: Crowdsourcing as a Novel Method to Evaluate Aesthetic Outcomes of Treatment for Unilateral Cleft Lip
The opportunity that crowdsourcing provides is a mechanism for rapidly obtaining large volumes of unbiased responses from lay public so that we may be able to produce a more objective assessment of outcome. As with any new approach, there are limitations, and we agree with Bechar et al. that the results we obtained may not be immediately comparable to previous studies. Although our use of a ranking system is not an absolute rating scale, it still can be used for comparative study. Further work and application of the approach will provide opportunities for calibration and creation of sets of standardized control “yardstick” images so that outcomes among subject groups or different studies can be compared between centers.
Although the term crowdsourcing evokes a focus on contemporary technology, its application simply involves a survey that is administered to obtain public opinion. With the use of the Internet, and an engaged group of crowd workers, we can now rapidly obtain large volumes of responses. How we harness that potential, and refine it, is the exciting challenge that is now before us.