Feeling machines: ethics, culture, and the rise of emotional AI

  • This paper explores the growing presence of emotionally responsive artificial intelligence through a critical and interdisciplinary lens. Bringing together the voices of early-career researchers from multiple fields, it explores how AI systems that simulate or interpret human emotions are reshaping our interactions in areas such as education, healthcare, mental health, caregiving, and digital life. The analysis is structured around four central themes: the ethical implications of emotional AI, the cultural dynamics of human-machine interaction, the risks and opportunities for vulnerable populations, and the emerging regulatory, design, and technical considerations. The authors highlight the potential of affective AI to support mental well-being, enhance learning, and reduce loneliness, as well as the risks of emotional manipulation, over-reliance, misrepresentation, and cultural bias. Key challenges include simulating empathy without genuine understanding, encoding dominantThis paper explores the growing presence of emotionally responsive artificial intelligence through a critical and interdisciplinary lens. Bringing together the voices of early-career researchers from multiple fields, it explores how AI systems that simulate or interpret human emotions are reshaping our interactions in areas such as education, healthcare, mental health, caregiving, and digital life. The analysis is structured around four central themes: the ethical implications of emotional AI, the cultural dynamics of human-machine interaction, the risks and opportunities for vulnerable populations, and the emerging regulatory, design, and technical considerations. The authors highlight the potential of affective AI to support mental well-being, enhance learning, and reduce loneliness, as well as the risks of emotional manipulation, over-reliance, misrepresentation, and cultural bias. Key challenges include simulating empathy without genuine understanding, encoding dominant sociocultural norms into AI systems, and insufficient safeguards for individuals in sensitive or high-risk contexts. Special attention is given to children, elderly users, and individuals with mental health challenges, who may interact with AI in emotionally significant ways. However, there remains a lack of cognitive or legal protections which are necessary to navigate such engagements safely. The report concludes with ten recommendations, including the need for transparency, certification frameworks, region-specific fine-tuning, human oversight, and longitudinal research. A curated supplementary section provides practical tools, models, and datasets to support further work in this domain.show moreshow less

Download full text files

Export metadata

Statistics

Number of document requests

Additional Services

Share in Twitter Search Google Scholar
Metadaten
Author:Vivek Chavan, Arsen Cenaj, Shuyuan ShenORCiD, Ariane Bar, Srishti Binwani, Tommaso Del Becaro, Marius FunkORCiDGND, Lynn Greschner, Roberto Hung, Stina KleinORCiDGND, Romina Kleiner, Stefanie Krause, Sylwia Olbrych, Vishvapalsinhji Parmar, Jaleh Sarafraz, Daria Soroko, Daksitha Withanage DonGND, Chang ZhouORCiD, Hoang Thuy Duong Vu, Parastoo Semnani, Daniel Weinhardt, Elisabeth AndréORCiDGND, Jörg Krüger, Xavier Fresquet
URN:urn:nbn:de:bvb:384-opus4-1300854
Frontdoor URLhttps://opus.bibliothek.uni-augsburg.de/opus4/130085
Parent Title (English):AI Grid and SCAI Spring School 2025, Sorbonne University, Paris, France, 7-11 April 2025
Publisher:arxiv, Cornell University
Place of publication:Ithaka, NY
Type:Conference Proceeding
Language:English
Date of Publication (online):2026/05/04
Year of first Publication:2025
Publishing Institution:Universität Augsburg
Release Date:2026/05/04
First Page:arXiv:2506.12437
Note:
This paper was collaboratively developed during the Spring School “How can emotionally intelligent AI transform society?”, held in English at Sorbonne University (SCAI, 4 Place Jussieu, Paris) from April 7 to April 11, 2025. The Spring School was co-organized by the Sorbonne Center for Artificial Intelligence (SCAI) and AI Grid, as part of a Franco-German initiative to foster interdisciplinary dialogue on the ethical, cultural, and societal dimensions of emotionally intelligent AI. The document reflects the insights and collaborative reflections of participating researchers, doctoral students, and invited experts from various disciplines, generated through keynotes, panel discussions, World Café sessions, and hands-on workshops.
DOI:https://doi.org/10.48550/arXiv.2506.12437
Institutes:Fakultät für Angewandte Informatik
Fakultät für Angewandte Informatik / Institut für Informatik
Fakultät für Angewandte Informatik / Institut für Informatik / Lehrstuhl für Menschzentrierte Künstliche Intelligenz
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
Licence (German):CC-BY 4.0: Creative Commons: Namensnennung