MMMS: Multi-Modal Multi-Surface Interactive Segmentation
- In this paper, we present a method to interactively create segmentation masks on the basis of user clicks. We pay particular attention to the segmentation of multiple surfaces that are simultaneously present in the same image. Since these surfaces may be heavily entangled and adjacent, we also present a novel extended evaluation metric that accounts for the challenges of this scenario. Additionally, the presented method is able to use multi-modal inputs to facilitate the segmentation task. At the center of this method is a network architecture which takes as input an RGB image, a number of non-RGB modalities, an erroneous mask, and encoded clicks. Based on this input, the network predicts an improved segmentation mask. We design our architecture such that it adheres to two conditions: (1) The RGB backbone is only available as a black-box. (2) To reduce the response time, we want our model to integrate the interaction-specific information after the image feature extraction and theIn this paper, we present a method to interactively create segmentation masks on the basis of user clicks. We pay particular attention to the segmentation of multiple surfaces that are simultaneously present in the same image. Since these surfaces may be heavily entangled and adjacent, we also present a novel extended evaluation metric that accounts for the challenges of this scenario. Additionally, the presented method is able to use multi-modal inputs to facilitate the segmentation task. At the center of this method is a network architecture which takes as input an RGB image, a number of non-RGB modalities, an erroneous mask, and encoded clicks. Based on this input, the network predicts an improved segmentation mask. We design our architecture such that it adheres to two conditions: (1) The RGB backbone is only available as a black-box. (2) To reduce the response time, we want our model to integrate the interaction-specific information after the image feature extraction and the multi-modal fusion. We refer to the overall task as Multi-Modal Multi-Surface interactive segmentation (MMMS). We are able to show the effectiveness of our multi-modal fusion strategy. Using additional modalities, our system reduces the NoC@90 by up to 1.28 clicks per surface on average on DeLiVER and up to 1.19 on MFNet. On top of this, we are able to show that our RGB-only baseline achieves competitive, and in some cases even superior performance when tested in a classical, single-mask interactive segmentation scenario.…


| Author: | Robin SchönORCiDGND, Julian LorenzGND, Katja LudwigGND, Daniel KienzleORCiDGND, Rainer LienhartORCiDGND |
|---|---|
| URN: | urn:nbn:de:bvb:384-opus4-1254957 |
| Frontdoor URL | https://opus.bibliothek.uni-augsburg.de/opus4/125495 |
| ISBN: | 979-8-3315-5500-9OPAC |
| Parent Title (English): | 2025 IEEE International Conference on Content-Based Multimedia Indexing (IEEE CBMI), 22-24 October 2025, Dublin, Ireland |
| Publisher: | IEEE |
| Place of publication: | Piscataway, NJ |
| Editor: | Liting Zhou, Klaus Schoeffmann, Duc Tien Dang Nguyen, Dragi Kimovski |
| Type: | Conference Proceeding |
| Language: | English |
| Date of Publication (online): | 2025/09/25 |
| Year of first Publication: | 2025 |
| Publishing Institution: | Universität Augsburg |
| Release Date: | 2025/09/25 |
| First Page: | 1 |
| Last Page: | 8 |
| DOI: | https://doi.org/10.1109/CBMI66578.2025.11339342 |
| Institutes: | Fakultät für Angewandte Informatik |
| Fakultät für Angewandte Informatik / Institut für Informatik | |
| Fakultät für Angewandte Informatik / Institut für Informatik / Lehrstuhl für Maschinelles Lernen und Maschinelles Sehen | |
| Dewey Decimal Classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik |
| Licence (German): | Deutsches Urheberrecht |



