| 来源类型 | Article
|
| 规范类型 | 其他
|
| DOI | 10.1111/tgis.12194
|
| Limitations of Majority Agreement in Crowdsourced Image Interpretation. |
| Salk C; Sturn T; See L; Fritz S
|
| 发表日期 | 2016
|
| 出处 | Transactions in GIS : 1-17
|
| 出版年 | 2016
|
| 语种 | 英语
|
| 摘要 | Crowdsourcing can efficiently complete tasks that are difficult to automate, but the quality of crowdsourced data is tricky to evaluate. Algorithms to grade volunteer work often assume that all tasks are similarly difficult, an assumption that is frequently false. We use a cropland identification game with over 2,600 participants and 165,000 unique tasks to investigate how best to evaluate the difficulty of crowdsourced tasks and to what extent this is possible based on volunteer responses alone. Inter-volunteer agreement exceeded 90% for about 80% of the images and was negatively correlated with volunteer-expressed uncertainty about image classification. A total of 343 relatively difficult images were independently classified as cropland, non-cropland or impossible by two experts. The experts disagreed weakly (one said impossible while the other rated as cropland or non-cropland) on 27% of the images, but disagreed strongly (cropland vs. non-cropland) on only 7%. Inter-volunteer disagreement increased significantly with inter-expert disagreement. While volunteers agreed with expert classifications for most images, over 20% would have been mis-categorized if only the volunteers’ majority vote was used. We end with a series of recommendations for managing the challenges posed by heterogeneous tasks in crowdsourcing campaigns. |
| 主题 | Ecosystems Services and Management (ESM)
|
| URL | http://pure.iiasa.ac.at/id/eprint/12947/
|
| 来源智库 | International Institute for Applied Systems Analysis (Austria)
|
| 引用统计 |
|
| 资源类型 | 智库出版物
|
| 条目标识符 | http://119.78.100.153/handle/2XGU8XDN/130689
|
推荐引用方式
GB/T 7714 |
Salk C,Sturn T,See L,et al. Limitations of Majority Agreement in Crowdsourced Image Interpretation.. 2016.
|
|
文件名：
|
Task%20Difficulty%20Rerevised%20CLEAN.pdf
|
|
格式：
|
Adobe PDF
|
此文件暂不支持浏览
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。
