Cooperative and transparent machine learning for the context-sensitive analysis of social interactions

  • The research area of Social Signal Processing paves the way for conversational companions, such as virtual agents or social robots, to become aware of nuances in our behaviours and implicit messages that come along with them. For machines to understand and interpret such behavioural cues, the state-of-the-art procedure is the application of various machine learning techniques. In many ML tasks, statistical models are trained on a large amount of annotated samples and an algorithm aims to match patterns that represent specific classes or values. ML tehniques, such as artificial neural networks, nowadays do pretty well in mapping and even identifying low level features to a specific recognition problem. A large drawback here is that the decisions they are making are not comprehensible and understandable to humans and that their assumptions are often wrong in changing contexts. Therefore a new research direction -"eXplainable Artificial Intelligence" (XAI)- identified the need of AIThe research area of Social Signal Processing paves the way for conversational companions, such as virtual agents or social robots, to become aware of nuances in our behaviours and implicit messages that come along with them. For machines to understand and interpret such behavioural cues, the state-of-the-art procedure is the application of various machine learning techniques. In many ML tasks, statistical models are trained on a large amount of annotated samples and an algorithm aims to match patterns that represent specific classes or values. ML tehniques, such as artificial neural networks, nowadays do pretty well in mapping and even identifying low level features to a specific recognition problem. A large drawback here is that the decisions they are making are not comprehensible and understandable to humans and that their assumptions are often wrong in changing contexts. Therefore a new research direction -"eXplainable Artificial Intelligence" (XAI)- identified the need of AI systems to be able to explain their decisions. In this thesis we investigate strategies to make the recognition and interpretation of complex social signals more transparent and explore ways to empower the human in the machine learning loop. To gain a better understanding of how humans interpret social cues, we first introduce an overview on results of behavioural psychology. We then describe the creation of various multi-person and muli-modal corpora in varying contexts that aim to induce multiple aspects of such behaviours. Next, we briefly introduce common techniques used in the area of social signal processing and machine learning. To successfully annotate and manage large continuous databases, a novel tool, named NOVA is presented. It allows to distribute the annotation task on multiple labellers and supports various types of annotations. NOVA further allows to take advantage of ML techniques already during the annotation process (a concept named cooperative machine learning). By employing CML, data is annotated simultaneously with the machine, which speeds up the annotation process and gives a more transparent idea of a machine's decisions. For inferring more complex behaviours, such as a person's conversational engagement or emotion regulation strategies, an approach is introduced that considers the predictions of multiple social cue recognisers and various types of context information. Finally, an outlook on future research directions is given.show moreshow less

Download full text files

Export metadata

Statistics

Number of document requests

Additional Services

Share in Twitter Search Google Scholar
Metadaten
Author:Tobias BaurORCiDGND
URN:urn:nbn:de:bvb:384-opus4-401533
Frontdoor URLhttps://opus.bibliothek.uni-augsburg.de/opus4/40153
Advisor:Elisabeth André
Type:Doctoral Thesis
Language:English
Year of first Publication:2018
Publishing Institution:Universität Augsburg
Granting Institution:Universität Augsburg, Fakultät für Angewandte Informatik
Date of final exam:2018/07/24
Release Date:2018/10/29
GND-Keyword:Maschinelles Lernen; Mensch-Maschine-Kommunikation; Mensch-Maschine-Schnittstelle; Benutzeroberfläche; Multimodales System
Pagenumber:255
Institutes:Fakultät für Angewandte Informatik
Fakultät für Angewandte Informatik / Institut für Informatik
Fakultät für Angewandte Informatik / Institut für Informatik / Lehrstuhl für Multimodale Mensch-Technik Interaktion (Human Centered Multimedia)
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
Licence (German):Deutsches Urheberrecht mit Print on Demand