Human Label Variation in Discourse and Pragmatic Phenomena

Over the past 20 years, the NLP community has given steadily-increasing attention to the idea of regarding human annotation disagreement not as a nuisance but as a potential asset that can be exploited for understanding the respective task better. In practice, this means that multiple annotations are not just adjudicated or averaged (and the individual annotations then thrown away) but taken as a spectrum that constitutes a "complex ground truth", for instance in the form of probability distributions over labels attached to items.

While some of the pioneering work on appreciating human label variation (HLV) came from research in discourse tasks such as anaphoric reference and discourse structure, the great majority of the recent work in the HLV community has concentrated on tasks in which variation arises from subjective differences, such as humour or offensive language detection, whereas the study of variation in pragmatics and discourse has seen relatively little progress. This is on the one hand surprising - pragmatics and discourse are highly prone to interesting variation in human judgement - but may well be an effect of the relative complexity of the annotation tasks, which make treatments of HLV difficult.

In this workshop, we aim to bring together experts on HLV and on discourse phenomena, and to identify ways forward. We welcome contributions on the topics listed below (the list is not meant to be exhaustive).

Submissions are now closed. Notifications of acceptance have been sent, and we thank everyone who submitted an abstract. We look forward to the accepted contributions and the discussions at the workshop.

We primarily aim to promote participation and active discussion, and no proceedings will be published. Therefore, workshop submissions are not limited to unpublished work.

Accepted contributions include long (30 min. + discussion) and short (15 min. + discussion) presentations as well as poster presentations.

Subject to discussion at the end of the workshop, we envisage the possibility of a follow-up publication in the form of a journal special issue (with a new call for papers and reviewing process).