Linguistic frameworks go toe-to-toe at neuro-symbolic language modeling

  • We examine the extent to which, in principle, different syntactic and semantic graph representations can complement and improve neural language modeling. Specifically, by conditioning on a subgraph encapsulating the locally relevant sentence history, can a model make better next-word predictions than a pretrained sequential language model alone? With an ensemble setup consisting of GPT-2 and ground-truth graphs from one of 7 different formalisms, we find that the graph information indeed improves perplexity and other metrics. Moreover, this architecture provides a new way to compare different frameworks of linguistic representation. In our oracle graph setup, training and evaluating on English WSJ, semantic constituency structures prove most useful to language modeling performance—outpacing syntactic constituency structures as well as syntactic and semantic dependency structures.

Download full text files

Export metadata

Statistics

Number of document requests

Additional Services

Share in Twitter Search Google Scholar
Metadaten
Author:Jakob PrangeGND, Nathan Schneider, Lingpeng Kong
URN:urn:nbn:de:bvb:384-opus4-1178284
Frontdoor URLhttps://opus.bibliothek.uni-augsburg.de/opus4/117828
ISBN:978-1-955917-71-1OPAC
Parent Title (English):NAACL 2022: proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, July 10-15, 2022, Seattle, WA, USA
Publisher:Association for Computational Linguistics (ACL)
Place of publication:Stroudsburg, PA
Editor:Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Type:Conference Proceeding
Language:English
Year of first Publication:2022
Publishing Institution:Universität Augsburg
Release Date:2025/01/07
First Page:4375
Last Page:4391
DOI:https://doi.org/10.18653/v1/2022.naacl-main.325
Institutes:Fakultät für Angewandte Informatik
Fakultät für Angewandte Informatik / Institut für Informatik
Fakultät für Angewandte Informatik / Institut für Informatik / Lehrstuhl für Computerlinguistik
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
Licence (German):CC-BY 4.0: Creative Commons: Namensnennung

OPUS4 Logo