- Background: Measurement-based and data-informed recommendations can assist therapists in clinical decision making to improve treatment effectiveness. However, there is promise for improving predictive accuracy and robustness by integrating multimodal and multimethod assessments, since an important factor limiting the performance of predictive models may be the data sources. New data layers that have recently been introduced into psychotherapy research and shown to carry meaningful information include verbal, paraverbal, and nonverbal features extracted from videos. This complex data might better capture the complexity of psychological problems and treatment processes. To assess it automatically in therapy videos, a psychotherapy research lab (clinical psychology) cooperates closely with a department for human-centered artificial intelligence (computer science) in a project that is funded by the German Research Foundation (DFG). Leveraging current knowledge and applications fromBackground: Measurement-based and data-informed recommendations can assist therapists in clinical decision making to improve treatment effectiveness. However, there is promise for improving predictive accuracy and robustness by integrating multimodal and multimethod assessments, since an important factor limiting the performance of predictive models may be the data sources. New data layers that have recently been introduced into psychotherapy research and shown to carry meaningful information include verbal, paraverbal, and nonverbal features extracted from videos. This complex data might better capture the complexity of psychological problems and treatment processes. To assess it automatically in therapy videos, a psychotherapy research lab (clinical psychology) cooperates closely with a department for human-centered artificial intelligence (computer science) in a project that is funded by the German Research Foundation (DFG). Leveraging current knowledge and applications from computer science to improve outcome prediction in psychotherapy, this study aims to conduct a multimodal and multimethod data assessment of psychotherapy outpatients using video recordings of therapy sessions.
Methods: For n = 355 outpatients (N = 1,900 sessions) with psychological disorders, several novel data layers of video-recorded sessions are examined as predictors of treatment outcome. The video recordings of on average ~5.5 sessions per patient with the treating therapist will be used to assess nonverbal (emotional valence and arousal, smiles) and paraverbal features (vocal valence, arousal, and dominance) using the Nonverbal-Behavior Analyzer (NOVA). This tool is developed by the computer science team based on the clinical-psychological feedback. Furthermore, videos will be automatically transcribed in NOVA and natural language processing (NLP) tools are used for sentiment analysis (positive, neutral, and negative sentiments) and the assessment of speech pauses. Machine learning approaches will select important person-level (i.e., pre-treatment assessments) and session-level (i.e., video, audio, and text features) predictors of outcome. Outcome will be predicted from the selected variables with a linear mixed model and evaluations will be based on the explained variance.
Results: We expect the machine learning algorithms to select significant predictors for outcome from different data layers. The prediction model including multimodal and multimethod assessments will be compared to a model based on self-report measures only in terms of explained variance in outcome to evaluate the incremental predictive power of video-based information. Additionally, we present NOVA as the result of a multidisciplinary endeavor to automatically assess and integrate different data layers in psychotherapy outcome prediction.
Conclusions: The results will be discussed against the background of the recent literature on multimodal and multimethod assessments in psychotherapy and beyond the field. Findings may advance measurement-based and data-informed psychological therapies. Implemented in a comprehensive decision-support system, the predictions could support scientifically trained therapists in their decision-making by providing broad information on the patient going beyond classical psychological psychometrics. Because several sessions per patient will be analyzed, predictions can be updated for new sessions allowing a process evaluation that supports treatment adaptation. Furthermore, the software NOVA, which is described in this manuscript, provides an easy to set up and easy to use interface coming from computer science for implementation in clinical practice and emotion research.…
