Abstract
Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not account for the errors that speech recognition models might introduce, nor do they consider the language variations (dialects) of the users. To address this gap, we augment an existing QA dataset to construct a multi-dialect, spoken QA benchmark on four languages (Arabic, Bengali, English, Kiswahili) with more than 68k audio prompts in 22 dialects from 245 speakers. We provide baseline results showcasing the real-world performance of QA systems and analyze the effect of language variety and other sensitive speaker attributes on downstream performance. Last, we study the fairness of the ASR and QA models with respect to the underlying user populations. The dataset and code for reproducing our experiments are available here: https://github.com/ffaisal93/SD-QA.
Fahim Faisal
PhD Student
My name is Fahim Faisal. My academic interest involves learning different aspects of computational linguistics and natural language processing (eg. machine translation). Currently, I am working on a project related to semi-supervised learning of morphological process of language.
Sharlina Keshava
CS MSc → Amazon
Sharlina Keshava is a part-time masters student in the Computer Science department at George Mason University. She is interested in various aspects of multilingual Natural Language Processing and Machine Learning and applying language technologies to social responsibility domains. Her current research is on fairness in language tools.
Antonios Anastasopoulos
Assistant Professor
I work on multilingual models, machine translation, speech recognition, and NLP for under-served languages.