Segformer++: efficient token-merging strategies for high-resolution semantic segmentation
- Utilizing transformer architectures for semantic segmentation of high-resolution images is hindered by the attention’s quadratic computational complexity in the number of tokens. A solution to this challenge involves decreasing the number of tokens through token merging, which has exhibited remarkable enhancements in inference speed, training efficiency, and memory utilization for image classification tasks. In this paper, we explore various token merging strategies within the framework of the Segformer architecture and perform experiments on multiple semantic segmentation and human pose estimation datasets. Notably, without model re-training, we, for example, achieve an inference acceleration of 61% on the Cityscapes dataset while maintaining the mIoU performance. Consequently, this paper facilitates the deployment of transformer-based architectures on resource-constrained devices and in real-time applications.
Download full text files
Postprint. © 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Statistics
Additional Services
| Author: | Daniel KienzleGND, Marco Kantonis, Robin SchönORCiDGND, Rainer LienhartORCiDGND |
|---|---|
| URN: | urn:nbn:de:bvb:384-opus4-1131572 |
| Frontdoor URL | https://opus.bibliothek.uni-augsburg.de/opus4/113157 |
| ISBN: | 979-8-3503-5142-2OPAC |
| ISSN: | 2770-4319OPAC |
| Parent Title (English): | IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) 2024, San Jose, CA, USA, August 7-9, 2024 |
| Publisher: | IEEE |
| Place of publication: | Piscataway, NJ |
| Editor: | Guan-Ming Su, Homer Chen, Ming-Ching Chang, Mohan Kankanhalli, Prabhakaran Balakrishnan, Dae Yeol Lee |
| Type: | Conference Proceeding |
| Language: | English |
| Date of Publication (online): | 2024/05/23 |
| Year of first Publication: | 2024 |
| Publishing Institution: | Universität Augsburg |
| Release Date: | 2024/05/23 |
| First Page: | 75 |
| Last Page: | 81 |
| DOI: | https://doi.org/10.1109/MIPR62202.2024.00019 |
| Institutes: | Fakultät für Angewandte Informatik |
| Fakultät für Angewandte Informatik / Institut für Informatik | |
| Fakultät für Angewandte Informatik / Institut für Informatik / Lehrstuhl für Maschinelles Lernen und Maschinelles Sehen | |
| Dewey Decimal Classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik |
| Licence (German): |  Deutsches Urheberrecht |
