“Bias in Human-in-the-loop Artificial Intelligence”
Abstract:
Paid micro-task crowdsourcing has gained popularity also thanks to the rise of AI because of
the convenient way to generate large-scale manually annotated corpora and because of the
possibility to create human-in-the-loop systems. However, when using crowdsourcing
platforms for data gathering purposes, human factors need to be taken into account as
humans now become part of (and are not just users of) the system.
In this talk I will discuss our recent research in the area of micro-task crowdsourcing
with a focus on understanding crowd worker behaviors and their implications on the quality
of the collected data and the bias in it. I will first discuss open challenges in the
crowdsourcing ecosystem including issues caused by adversarial approaches that may disrupt
the crowdsourcing model as we know it. I will then discuss how human bias is reflected in
the data which is being collected by means of crowdsourcing. Finally, I will present our
work making use of fine-grained behavioral logs.
Location: Online
Link:
Date and Time: 04/12/2020, 11:00 – 12:00 EET
Speaker: Dr. Gianluca Demartini
Bio.:
Dr. Gianluca Demartini is an Associate Professor in Data Science at the University of Queensland, School of Information Technology and Electrical Engineering. His main research interests are Information Retrieval, Semantic Web, and Human Computation. His research has been supported by the Australian Research Council (ARC), the EU H2020 framework program, the UK Engineering and Physical Sciences Research Council (EPSRC), and by Facebook. He received Best Paper awards at the AAAI Conference on Human Computation and Crowdsourcing (HCOMP) in 2018 and at the European Conference on Information Retrieval (ECIR) in 2016 and in 2020 and the Best Demo Award at the International Semantic Web Conference (ISWC) in 2011. He has published more than 100 peer-reviewed scientific publications at venues such as WWW, ACM SIGIR, VLDBJ, ISWC, and ACM CHI.