Open Access

Johannes Wagner

• In recent years, Social Signal Processing (SSP) has gradually emerged as a new research field in human-computer interaction aiming to build a new generation of "social computers", which will be perceived as more natural, efficacious and trustworthy. So far, most studies have been conducted offline and have been investigated social interaction under laboratory conditions. The problem of plain offline studies is that they tend to convey a too optimistic picture of what can actually be achieved since they avoid problems which occur only when a system is tested in the "open world". Hence, despite a good deal of work carried out in the field of SSP, we have not seen many online systems. It is the goal of this thesis to encourage developers to put more effort into building online systems instead of confining their work to pure offline studies. Thus, this study examines the limitations of previous studies and proposes methodological and technical solutions, which pave the path towards aIn recent years, Social Signal Processing (SSP) has gradually emerged as a new research field in human-computer interaction aiming to build a new generation of "social computers", which will be perceived as more natural, efficacious and trustworthy. So far, most studies have been conducted offline and have been investigated social interaction under laboratory conditions. The problem of plain offline studies is that they tend to convey a too optimistic picture of what can actually be achieved since they avoid problems which occur only when a system is tested in the "open world". Hence, despite a good deal of work carried out in the field of SSP, we have not seen many online systems. It is the goal of this thesis to encourage developers to put more effort into building online systems instead of confining their work to pure offline studies. Thus, this study examines the limitations of previous studies and proposes methodological and technical solutions, which pave the path towards a more realistic and application-oriented development. Today, most available SSP corpora contain audiovisual material composed of discrete and exaggerated samples. To ease data collection and enrich corpora with additional measurements, a mechanism will be proposed that allows for the synchronisation of audiovisual signals with other signals such as motion, eye gaze, and physiological feedback. To combine information from such diverse sources intelligent fusion strategies are needed. Unfortunately, conventional approaches have shown poor results in realistic scenarios. Thus, this study describes a novel fusion algorithm which allows for a better modelling of the temporal dependencies between modalities. Finally, it is common practice in offline studies to tweak the database to suit the classification process, for instance by removing parts with sparse interaction and non-prototypical behaviour. This can lead to suboptimal results if the learning process does not fit the final application. Therefore, a more application-related methodology will be proposed making it more likely that a system will perform satisfactory when transformed into an online approach. In order to meet the increased implementation issues of developing real-time systems, tools must be provided that take as much work off the hands of the developers as possible. With this in mind, a new open-source framework called Social Signal Interpretation will be introduced. It supports the complete process of the machine learning pipeline and provides a software architecture to accomplish complex processing pipelines from single, reusable units. It implements the discussed synchronisation and fusion mechanisms and systems developed with SSI can be immediately tested using live input.…