Feeling machines: ethics, culture, and the rise of emotional AI
- This paper explores the growing presence of emotionally responsive artificial intelligence through a critical and interdisciplinary lens. Bringing together the voices of early-career researchers from multiple fields, it explores how AI systems that simulate or interpret human emotions are reshaping our interactions in areas such as education, healthcare, mental health, caregiving, and digital life. The analysis is structured around four central themes: the ethical implications of emotional AI, the cultural dynamics of human-machine interaction, the risks and opportunities for vulnerable populations, and the emerging regulatory, design, and technical considerations. The authors highlight the potential of affective AI to support mental well-being, enhance learning, and reduce loneliness, as well as the risks of emotional manipulation, over-reliance, misrepresentation, and cultural bias. Key challenges include simulating empathy without genuine understanding, encoding dominantThis paper explores the growing presence of emotionally responsive artificial intelligence through a critical and interdisciplinary lens. Bringing together the voices of early-career researchers from multiple fields, it explores how AI systems that simulate or interpret human emotions are reshaping our interactions in areas such as education, healthcare, mental health, caregiving, and digital life. The analysis is structured around four central themes: the ethical implications of emotional AI, the cultural dynamics of human-machine interaction, the risks and opportunities for vulnerable populations, and the emerging regulatory, design, and technical considerations. The authors highlight the potential of affective AI to support mental well-being, enhance learning, and reduce loneliness, as well as the risks of emotional manipulation, over-reliance, misrepresentation, and cultural bias. Key challenges include simulating empathy without genuine understanding, encoding dominant sociocultural norms into AI systems, and insufficient safeguards for individuals in sensitive or high-risk contexts. Special attention is given to children, elderly users, and individuals with mental health challenges, who may interact with AI in emotionally significant ways. However, there remains a lack of cognitive or legal protections which are necessary to navigate such engagements safely. The report concludes with ten recommendations, including the need for transparency, certification frameworks, region-specific fine-tuning, human oversight, and longitudinal research. A curated supplementary section provides practical tools, models, and datasets to support further work in this domain.…


| Author: | Vivek Chavan, Arsen Cenaj, Shuyuan ShenORCiD, Ariane Bar, Srishti Binwani, Tommaso Del Becaro, Marius FunkORCiDGND, Lynn Greschner, Roberto Hung, Stina KleinORCiDGND, Romina Kleiner, Stefanie Krause, Sylwia Olbrych, Vishvapalsinhji Parmar, Jaleh Sarafraz, Daria Soroko, Daksitha Withanage DonGND, Chang ZhouORCiD, Hoang Thuy Duong Vu, Parastoo Semnani, Daniel Weinhardt, Elisabeth AndréORCiDGND, Jörg Krüger, Xavier Fresquet |
|---|---|
| URN: | urn:nbn:de:bvb:384-opus4-1300854 |
| Frontdoor URL | https://opus.bibliothek.uni-augsburg.de/opus4/130085 |
| Parent Title (English): | AI Grid and SCAI Spring School 2025, Sorbonne University, Paris, France, 7-11 April 2025 |
| Publisher: | arxiv, Cornell University |
| Place of publication: | Ithaka, NY |
| Type: | Conference Proceeding |
| Language: | English |
| Date of Publication (online): | 2026/05/04 |
| Year of first Publication: | 2025 |
| Publishing Institution: | Universität Augsburg |
| Release Date: | 2026/05/04 |
| First Page: | arXiv:2506.12437 |
| Note: | This paper was collaboratively developed during the Spring School “How can emotionally intelligent AI transform society?”, held in English at Sorbonne University (SCAI, 4 Place Jussieu, Paris) from April 7 to April 11, 2025. The Spring School was co-organized by the Sorbonne Center for Artificial Intelligence (SCAI) and AI Grid, as part of a Franco-German initiative to foster interdisciplinary dialogue on the ethical, cultural, and societal dimensions of emotionally intelligent AI. The document reflects the insights and collaborative reflections of participating researchers, doctoral students, and invited experts from various disciplines, generated through keynotes, panel discussions, World Café sessions, and hands-on workshops. |
| DOI: | https://doi.org/10.48550/arXiv.2506.12437 |
| Institutes: | Fakultät für Angewandte Informatik |
| Fakultät für Angewandte Informatik / Institut für Informatik | |
| Fakultät für Angewandte Informatik / Institut für Informatik / Lehrstuhl für Menschzentrierte Künstliche Intelligenz | |
| Dewey Decimal Classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik |
| Licence (German): | CC-BY 4.0: Creative Commons: Namensnennung |



