ClusterDataSplit: exploring challenging clustering-based data splits for model performance evaluation

  • This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set. Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm. It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions. Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data. In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance. We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.

Download full text files

Export metadata

Statistics

Number of document requests

Additional Services

Share in Twitter Search Google Scholar
Metadaten
Author:Hanna Wecker, Annemarie FriedrichORCiDGND, Heike Adel
URN:urn:nbn:de:bvb:384-opus4-1056625
Frontdoor URLhttps://opus.bibliothek.uni-augsburg.de/opus4/105662
ISBN:978-1-952148-82-8OPAC
Parent Title (English):Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, November 20, 2022, online
Publisher:Association for Computational Linguistics
Place of publication:Stroudsburg, PA
Editor:Steffen Eger, Yang Gao, Maxime Peyrad, Wei Zhao, Eduard Hovy
Type:Conference Proceeding
Language:English
Year of first Publication:2020
Publishing Institution:Universität Augsburg
Release Date:2023/07/11
First Page:155
Last Page:163
DOI:https://doi.org/10.18653/v1/2020.eval4nlp-1.15
Institutes:Fakultät für Angewandte Informatik
Fakultät für Angewandte Informatik / Institut für Informatik
Fakultät für Angewandte Informatik / Institut für Informatik / Professur für Sprachverstehen mit der Anwendung Digital Humanities
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
Licence (German):CC-BY 4.0: Creative Commons: Namensnennung (mit Print on Demand)

OPUS4 Logo