Media / Publication
November 04, 2025

Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments

This study argues that reducing multiple annotations to one “ground truth” can obscure meaningful disagreement in subjective tasks, such as identifying the human values behind arguments. It finds that models perform best when trained to detect subjectivity directly, helping flag arguments that different people may reasonably interpret in different ways.

Abstract

Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.

To continue reading please visit: https://doi.org/10.18653/v1/2025.findings-emnlp.824

This open access article was published in Findings of the Association for Computational Linguistics: EMNLP 2025 on 4 November 2025.

Homayounirad, A., Liscio, E., Wang, T., Jonker, C. M., & Siebert, L. C. (2025). Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments. In Findings of the Association for Computational Linguistics: EMNLP 2025 (pp. 15237–15252). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.824

Keywords: Annotations, Subjectivity