Open Access

Ionuț Damian

• Recent technological advancements have enabled the miniaturization of electronics to a degree that powerful mobile computers and versatile sensor arrays can be integrated in phones, watches or even glasses. This dissertation looks at how such devices, in particular smartphones, smart watches and smart glasses, can be used to help users participating in social interactions improve the quality of their behaviour, and thus better the outcome of the interaction. To this end, the concept of social augmentation is introduced. Social augmentation makes use of behavioural feedback loops to analyse the behaviour of the user in realtime and, based on the outcome of this analysis, provide live feedback to the user on how to improve their behaviour. The concept is positioned as an evolution of classical social skills training methods, capable of augmenting the social skills of the users in realtime during actual social interactions. It is targeted at users who wish to perform better duringRecent technological advancements have enabled the miniaturization of electronics to a degree that powerful mobile computers and versatile sensor arrays can be integrated in phones, watches or even glasses. This dissertation looks at how such devices, in particular smartphones, smart watches and smart glasses, can be used to help users participating in social interactions improve the quality of their behaviour, and thus better the outcome of the interaction. To this end, the concept of social augmentation is introduced. Social augmentation makes use of behavioural feedback loops to analyse the behaviour of the user in realtime and, based on the outcome of this analysis, provide live feedback to the user on how to improve their behaviour. The concept is positioned as an evolution of classical social skills training methods, capable of augmenting the social skills of the users in realtime during actual social interactions. It is targeted at users who wish to perform better during critical social interactions, such as job interview or speaking in public. However, social augmentation can also help persons who suffer from various disabilities better regulate their behaviour to avoid misunderstandings and generally increase their functional independence. The main challenge of this approach lies in designing the augmentation to function alongside social interactions without disrupting them. To address this, the thesis introduces a conceptual framework for social augmentation, which has been informed by empirical and theoretical findings from the fields of cognitive psychology, human-computer interaction and digital signal processing. With the help of three user studies spanning two scenarios (public speaking and group discussions), the social augmentation concept has been tested for effectiveness in improving social behaviour without disrupting the social interaction. To foster further research in this area, the SSJ open source software framework for social augmentation has been implemented. Through a combination of state-of-the-art behaviour analysis and live feedback techniques, SSJ supports the behavioural feedback loop in its entirety. First, it allows the processing and classification of social signals extracted from various sensors locally on mobile devices and in realtime. Using the results of the behaviour analysis, SSJ can deliver unobtrusive, multimodal live feedback using various output devices including head-mounted displays, vibro-actuators and headphones. Thus, SSJ enables the creation of powerful and versatile social augmentation systems. Moreover, thanks to a modular and flexible design, the augmentation is not restricted to any particular scenario, but can be targeted at manifold social situations.…