Decision-theoretic, personality-based management of turn-taking conflicts in multimodal dialogue systems

  • Humans tend to interpret the behavior of robots and virtual characters in human terms. Therefore, interaction designers need to ensure that the agent’s behaviors align with the personality that users are meant to associate with it. One such behavior is the turn-taking in conversations with the user. In particular, overlaps and interruptions are loaded with stereotypes about dominance, but also positive phenomena such as shared enthusiasm. Silence can be awkward or a sign of patient listening. In order to generate consistent behavior for a wide range of conversational agents - from obedient home assistants to antagonists in training simulations - a psychologically sound model is required. This thesis therefore reviewed existing theories about how personality and interpersonal attitude are reflected in turn-taking behavior, specifically the timing of speech activity and the accompanying gaze signals. A decision-theoretic approach was then chosen to model idealized human-like reasoningHumans tend to interpret the behavior of robots and virtual characters in human terms. Therefore, interaction designers need to ensure that the agent’s behaviors align with the personality that users are meant to associate with it. One such behavior is the turn-taking in conversations with the user. In particular, overlaps and interruptions are loaded with stereotypes about dominance, but also positive phenomena such as shared enthusiasm. Silence can be awkward or a sign of patient listening. In order to generate consistent behavior for a wide range of conversational agents - from obedient home assistants to antagonists in training simulations - a psychologically sound model is required. This thesis therefore reviewed existing theories about how personality and interpersonal attitude are reflected in turn-taking behavior, specifically the timing of speech activity and the accompanying gaze signals. A decision-theoretic approach was then chosen to model idealized human-like reasoning and thereby strike a balance between generating more natural agent behavior and meeting the heightened expectations that humans have towards a rational machine. The concept presented here consists of three parts. The influence diagram chooses the turn-taking behavior, such as starting to speak or averting the gaze, that best fulfills the agent’s goals. The Participant Framework connects this behavior model to the dialogue manager, providing context information for the influence diagram’s decisions and using said decision to regulate the dialogue flow. Finally, the RobotEngine Framework connects the dialogue application to different virtual and robotic agents in a way that keeps the turn-taking behavior separate from the agents’ implementation. Finally, the developed behavior model was tested in two different example setups. A non-interactive prototype with a simplified behavior model had two virtual characters talking to each other, implemented as separate processes and limited to explicit verbal communication. This limitation was meant to simulate the incomplete knowledge that a conversational agent could obtain from a human user. An online study confirmed that the generated behaviors led to personality judgments in line with theory and related works. Afterward, an interactive prototype was set up with incremental speech recognition and gaze detection. A preliminary evaluation was conducted by analyzing recordings of human-agent conversations. While the generated behavior variations were in line with the expected patterns, the interactivity introduced numerous challenges that will require a more thorough analysis in the future. Nevertheless, important lessons were learned and summarized as recommendations for evaluating such a system.show moreshow less

Download full text files

Export metadata

Statistics

Number of document requests

Additional Services

Share in Twitter Search Google Scholar
Metadaten
Author:Kathrin JanowskiORCiDGND
URN:urn:nbn:de:bvb:384-opus4-1150427
Frontdoor URLhttps://opus.bibliothek.uni-augsburg.de/opus4/115042
Advisor:Elisabeth André
Type:Doctoral Thesis
Language:English
Year of first Publication:2024
Publishing Institution:Universität Augsburg
Granting Institution:Universität Augsburg, Fakultät für Angewandte Informatik
Date of final exam:2024/04/08
Release Date:2024/11/08
Tag:turn-taking conflict; personality simulation; embodied conversational agent; dialogue system; decision theory
GND-Keyword:Mensch-Maschine-Kommunikation; Dialogsystem; Entscheidungstheorie; Persönlichkeit; Unterbrechung
Pagenumber:xxi, 355
Institutes:Fakultät für Angewandte Informatik
Fakultät für Angewandte Informatik / Institut für Informatik
Fakultät für Angewandte Informatik / Institut für Informatik / Lehrstuhl für Menschzentrierte Künstliche Intelligenz
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 000 Informatik, Informationswissenschaft, allgemeine Werke
Licence (German):Deutsches Urheberrecht mit Print on Demand