“Bias in Human-in-the-loop Artificial Intelligence”
Paid micro-task crowdsourcing has gained popularity also thanks to the rise of AI because of the convenient way to generate large-scale manually annotated corpora and because of the possibility to create human-in-the-loop systems. However, when using crowdsourcing platforms for data gathering purposes, human factors need to be taken into account as humans now become part of (and are not just users of) the system. In this talk I will discuss our recent research in the area of micro-task crowdsourcing with a focus on understanding crowd worker behaviors and their implications on the quality of the collected data and the bias in it. I will first discuss open challenges in the crowdsourcing ecosystem including issues caused by adversarial approaches that may disrupt the crowdsourcing model as we know it. I will then discuss how human bias is reflected in the data which is being collected by means of crowdsourcing. Finally, I will present our work making use of fine-grained behavioral logs.
Date and Time: 04/12/2020, 11:00 – 12:00 EET
Speaker: Dr. Gianluca Demartini
Dr. Gianluca Demartini is an Associate Professor in Data Science at the University of Queensland, School of Information Technology and Electrical Engineering. His main research interests are Information Retrieval, Semantic Web, and Human Computation. His research has been supported by the Australian Research Council (ARC), the EU H2020 framework program, the UK Engineering and Physical Sciences Research Council (EPSRC), and by Facebook. He received Best Paper awards at the AAAI Conference on Human Computation and Crowdsourcing (HCOMP) in 2018 and at the European Conference on Information Retrieval (ECIR) in 2016 and in 2020 and the Best Demo Award at the International Semantic Web Conference (ISWC) in 2011. He has published more than 100 peer-reviewed scientific publications at venues such as WWW, ACM SIGIR, VLDBJ, ISWC, and ACM CHI.