Beyond Human Judgment: A Bayesian Evaluation of LLMs' Moral Values Understanding

How do Large Language Models understand moral dimensions compared to humans?

This first comprehensive large-scale Bayesian evaluation of leading language models provides the answer. In contrast to prior approaches based on deterministic ground truth (obtained via majority or inclusion consensus), we obtain the labels by modelling annotators’ disagreement to capture both aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty (model domain sensitivity).

We evaluated Claude Sonnet 4, DeepSeek-V3, and Llama 4 Maverick across 250K+ annotations from nearly 700 annotators in 100K+ texts spanning social networks, news, and discussion forums. Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing that AI models generally rank among the top 25% of annotators in terms of balanced accuracy, substantially better than average humans.

Importantly, we find that AI produces far fewer false negatives than humans, highlighting their sensitive moral detection capabilities.

Keywords: Computational Ethics, Large Language Models, Moral Foundation Theory, Bayesian modeling, soft labels

Extended version: arXiv:2508.13804 Supplementary materials: osf.io/tpzau

1 Introduction

1.1 Background

Moral Foundations Theory (MFT) provides a comprehensive framework for understanding human moral reasoning across cultures, identifying core dimensions typically expressed as virtue/vice pairs: Care vs. Harm, Fairness vs. Cheating, Loyalty vs. Betrayal, Authority vs. Subversion, and Sanctity vs. Degradation (Graham et al., 2013; Haidt, 2012). These foundations shape individual and collective decision-making, from political preferences to social behavior (Feinberg and Willer, 2013; Graham et al., 2009; Roy and Goldwasser, 2021; Nguyen et al., 2022), making their computational detection crucial for understanding discourse dynamics and developing ethically-aligned AI systems.

Text Foundation

"My heart breaks seeing children separated from families at the border" Care

"Everyone deserves equal access to healthcare regardless of income" Fairness

"Respect your elders and follow traditional values that built this nation" Authority

"Stand with our troops - they sacrifice everything for our freedom" Loyalty

"Marriage is sacred and should be protected from secular corruption" Sanctity

Table 1: Posts and Associated Moral Foundation

The computational linguistics community has successfully fine-tuned pre-trained language models to predict moral values (Nguyen et al., 2024; Zangari et al., 2025a; Preniqi et al., 2024), achieving good alignment with human judgment when domain similarity and sufficient training data are available. However, systematic evaluation of large language models (LLMs) remains limited despite their rapid advances and potential as a compelling alternative that should suffer less from poor generalization and distribution shift.

This paper addresses these limitations through rigorous large-scale evaluation of state-of-the-art language models across established moral reasoning corpora, employing Bayesian methods to resolve disagreeing annotations.

1.2 Contribution

Bayesian uncertainty modelling of moral annotations. We introduce Bayesian modelling of annotator disagreements for moral foundation evaluation, moving beyond simple deterministic ground-truth assumptions. This captures both aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty (model sensitivity across domains and foundations).

Large-scale evaluation. We analyse market-leading large language models with 1M+ queries across 100K+ texts and 250K+ annotations from diverse sources, providing the most comprehensive moral reasoning evaluation to date.

Statistical analysis of Type I/II errors. We demonstrate that AI performs comparably to top annotators in balanced accuracy, considerably improving false negatives at the price of slightly increased false positive rates - contrary to fears that AI may underpredict moral values.

Novel GPU-optimized implementation of Bayesian labels. We developed a TensorFlow framework using sparse operations for scalable Bayesian inference on soft labels, of interest to the computational linguistics community.

1.3 Related Work

Dictionary methods for predicting moral values were initially developed Hopp et al. (2021) and remain widely used in applied studies, but demonstrate poor precision compared to pre-trained language models Nguyen et al. (2024).

Pretrained language models show strong alignment with human moral judgments given sufficient training data Nguyen et al. (2024); Zangari et al. (2025a); Preniqi et al. (2024), though they suffer from distribution shift and poor cross-domain generalization. This makes large language models a compelling alternative.

Large language models were recently compared to human performance Bulla et al. (2025), concluding LLMs superiority. However, their strict majority voting for ground truth excluded nuanced moral content, retaining only less ambiguous cases and departing from the inclusive consensus practices (marked when flagged by at least one annotator) Nguyen et al. (2024); Zangari et al. (2025a); Preniqi et al. (2024). Additionally, leave-one-out estimation of annotator ranking lacked statistical robustness. Another recent work Skorski and Landowska (2025) found LLMs showing imbalance - lower than anticipated recall - under inclusive annotator consensus.

Our Bayesian method resolves these problems by modeling annotator disagreement, striking the balance between inclusive and overly exclusive consensus rules, while our large-scale evaluation across multiple datasets and modern models ensures robustness of findings.

More on moral foundation theory. For comprehensive coverage of (computational) moral foundation theory challenges, see Zangari et al. (2025b).

2 Data and Methods

2.1 Datasets

Our robust evaluation utilizes three established and diverse corpora totaling 250K+ annotations of moral values from hundreds of annotators with diverse expertise (from experts to crowd-workers) across 100K+ texts spanning social media, news, and forum discussions.

Moral Foundations Twitter Corpus (MFTC) Hoover et al. (2020): 128,454 annotations from 23 trained annotators across 33,686 tweets from seven discourse domains.

Extended Moral Foundations Dictionary (eMFD) Hopp et al. (2021): 73,001 crowd-sourced annotations from 654 contributors on 54,867 text segments extracted from approximately 8,000 news documents by major outlets, including The Washington Post, CNN, Fox News selected via GDELT.

Moral Foundations Reddit Corpus (MFRC) Trager et al. (2022): 61,226 annotations from 6 trained coders across 17,885 Reddit comments from 12 subreddits covering politics and everyday moral discourse.

2.2 Bayesian Annotation Competence Model

As in the related work, we frame the problem as binary prediction tasks for individual moral foundations (abbreviated to care, fairness, authority, loyalty, sanctity). Optionally, we consider the aggregated "any moral content" category derived from positive labels across foundations. We note that this aggregated category is inherently unreliable when only positive labels are provided (as in eMFD), since true negatives cannot be distinguished from unlabeled moral content, creating systematic classification ambiguity.

Given substantial inter-annotator disagreement demonsrated by PABAK scores in Table 2, we resort to probabilistic (Bayesian) methods of obtaining ground-truth labels Paun and Simpson (2021).

Foundation MFTC MFRC eMFD

Care 0.71 0.67 0.33

Fairness 0.63 0.64 0.36

Loyalty 0.62 0.82 0.38

Authority 0.52 0.78 0.41

Sanctity 0.58 0.83 0.44

Any 0.34 0.38 1.00

Table 2: Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) scores measuring inter-annotator agreement across datasets and moral foundations

We model annotator disagreements using a variant of Dawid-Skene’s model Dawid and Skene (1979) with weak Dirichlet priors to estimate ground-truth labels and annotator reliability:

Model Specification. We assume N texts, J annotators, and K categories. True category prevalences follow 𝝅∼Dir​(𝜶) where 𝜶=(1,1,…,1) provides uniform priors. Each annotator j has a confusion matrix 𝚯j with rows 𝜽j​k∼Dir​(𝜷k), where 𝜷k is a K-dimensional vector with βk​k=2 (diagonal) and βk​l=0.5 for l≠k, encoding weak belief that annotators correctly identify majority of categories. For text i with true category zi and annotations 𝐲i=(yi​1,…,yi​J) we have:

Pr{zi=k∣𝐲i,𝝅,𝚯}∝πk∏j=1Jθj​k,yi​j,

This posterior accounts for varying annotator reliability while estimating both confusion matrices and ground-truth labels. For computational stability and reliable convergence, all calculations are performed in the log-domain using logits.

Figure 1: Graphical model representation of the model for multi-annotator classification. Light gray circles represent latent variables, dark gray rectangles represent observed variables, white circles represent parameters, and blue rectangles represent hyperparameters. Plates indicate replication over items (N) and annotators (J).

Implementation. We developed a GPU-optimized implementation in TensorFlow Abadi et al. (2015) using custom graphs and sparse tensor operations for computational efficiency. The framework supports MAP estimation and Monte Carlo sampling with optional fixed effects modeling (Appendix B). Testing on A100 GPUs achieved 1,000 optimization steps per second for 100k annotations, enabling rapid convergence across large-scale datasets.

2.3 Large Language Models

Three recent advanced large language models were analysed for moral foundations classification: (1) Claude Sonnet 4 (Anthropic, 2025) with a 200,000 token context window; (2) DeepSeek-V3 (DeepSeek, 2024) with 671 billion total parameters, 37 billion active parameters, and a 128,000 token context window; and (3) Llama 4 Maverick (Meta, 2025) with 402 billion total and 17 billion active parameters, and a 256,000 token context window.

All models were queried using the temperature setting of 0.30 to balance deterministic responses with modest variability. Each text was analysed individually (no batching of inputs) to ensure focused classification results. The prompt used was:

Experiments were tracked using W&B11 1 https://wandb.ai/ and repeated to ensure stability and reproducibility.

Appendix A provides details and rationale about the prompt selection.

2.4 Metrics

We apply standard classification metrics to Bayesian-estimated confusion matrices: Balanced Accuracy 12​(T​PT​P+F​N+T​NT​N+F​P), Precision T​PT​P+F​P, Recall T​PT​P+F​N, False Positive Rate F​PF​P+T​N, and False Negative Rate F​NF​N+T​P.

2.5 Supplementary Materials

Code, data preprocessing scripts, experimental results, and additional evaluation results are available from the project repository Skórski (2025).

3 Results

Bayesian models were fitted for each language model and dataset, the model considered a one additional annotator each time, and confusion matrices obtained through MAP were used to calculate classification metrics and percentile ranks.

3.1 Accuracy Analysis

Balanced accuracy results appear in Table 3 and Figures 2, 3 and 4.

Moral Foundations

Model Metric Any Authority Care Fairness Loyalty Sanctity

MFTC Dataset

Claude Sonnet 4 Acc% 75 80 78 82 81 90

Pct 50 83 75 71 79 100

DeepSeek-V3 Acc% 77 82 78 81 80 85

Pct 54 88 71 71 79 92

Llama 4 Maverick Acc% 68 79 76 83 80 87

Pct 42 79 63 71 75 96

Human Avg% 72 67 71 75 72 67

eMFD Dataset

Claude Sonnet 4 Acc% 85 74 88 83 72 80

Pct 95 75 97 91 78 84

DeepSeek-V3 Acc% 93 81 88 82 85 75

Pct 100 89 98 85 93 77

Llama 4 Maverick Acc% 95 83 89 84 83 82

Pct 100 92 98 88 91 86

GPT-5 mini Acc% 82 64 78 77 62 66

Pct 92 46 81 76 45 49

Human Avg Acc% 63 64 62 64 62 63

MFRC Dataset

Claude Sonnet 4 Acc% 74 83 90 80 86 91

Pct 57 100 100 86 86 100

DeepSeek-V3 Acc% 68 83 90 73 84 82

Pct 14 86 100 43 86 100

Llama 4 Maverick Acc% 62 83 89 72 83 87

Pct 14 100 100 43 86 100

Human Avg% 75 69 76 75 71 70

Table 3: Model performance on moral foundation classification across datasets. Acc% shows balanced accuracy and Pct shows the corresponding percentile.

The key insights are:

AI Superiority over Humans. AI models consistently outperformed humans across datasets, typically ranking in the top 25% of annotators. Human performance averaged 67-76%, while AI achieved 62-95% depending on dataset.

Similar Dataset Difficulty. Average AI performance was similar across datasets: MFRC (83.7%), eMFD (81.9%), and MFTC (81.5%), with negligible difference between the highest and lowest performing datasets.

Model Strengths. While all models outperformed human annotators, there is no strong winner among them. Claude appears to be a slight overall leader, scoring high consistently and achieving first or second-best results most of the time, with particular excellence in nuanced Care and Sanctity foundations. DeepSeek and Llama 4 perform similarly most of the time, but show a visible gap of 8-9% behind Claude on MFRC in Sanctity and Fairness.

Consistent Accuracy across Foundations. All moral foundations achieved strong performance across datasets, with overall average accuracy exceeding 80%: Care (85.1%), Sanctity (84.3%), Loyalty (81.6%), Authority (80.9%), and Fairness (80.0%). Particularly high results for "Sanctity" are notable given its known difficulty to classify due to cultural contexts.

Figure 2: DeepSeek-V3 vs human accuracy (MFTC).

Figure 3: Claude Sonnet 4 vs human accuracy (MFRC).

Figure 4: Llama 4 Maverick vs humans (eMFD).

3.2 False Positive vs. False Negative Analysis

Figure 5 and Table 4 compare false positive and false negative rates across models and foundations. The key findings are:

AI-Human Error Trade-offs. Across all foundations and datasets, AI models achieve 2−4× lower false negative rates compared to humans (19.4% vs 52.7% on average), at the expense of slightly higher average false positive rates (16.1% vs 10.1%), overall remaining more balanced (as visually revealed in Figure 5). This reveals opposing strategies: AI provides balanced detection, while humans classify more conservatively, systematically underdetect genuine moral signals.

Foundation-Specific Patterns. On average, the largest improvement in FNR appears in Care (40.8 point FNR versus humans), followed by Authority (35.6 points), Loyalty (34.5 points) and Sanctity (34.0 points). Fairness remains more challenging, with an average improvement of 24.2 points.

Human Conservatism Impact. On MFRC and MFTC, annotated by a pool of experts, AI appears more balanced (average FNR-FPR AI gap of 1.4pp and 1.6pp, respectively) than in eMFD (gap of 8.7pp) crowd-annotated. This can be explained by the Bayesian method that estimates AI performance by seeking a consensus with a much higher number of conservative / biased annotators.

Moral Dimensions

Authority Care Fairness Loyalty Sanctity

Model FNR FPR FNR FPR FNR FPR FNR FPR FNR FPR

MFRC Dataset

Claude Sonnet 4 16.5 17.2 5.3 15.2 12.3 27.2 8.7 19.5 7.9 9.5

DeepSeek-V3 18.7 14.4 7.3 13.7 36.4 18.0 9.6 21.4 31.3 4.5

Llama 4 Maverick 14.6 20.0 10.8 11.4 28.1 28.8 9.7 24.8 15.2 10.5

Human Baseline 56.5 5.3 42.4 5.3 40.9 9.4 52.4 5.4 55.4 3.7

MFTC Dataset

Claude Sonnet 4 14.7 25.3 7.2 35.8 9.3 25.8 20.1 18.7 7.8 12.2

DeepSeek-V3 24.6 11.1 15.2 28.5 28.6 8.7 22.9 16.7 26.7 3.8

Llama 4 Maverick 23.9 18.0 19.9 28.2 22.8 11.5 13.3 27.7 16.1 9.2

Human Baseline 53.1 13.4 50.3 6.9 43.5 6.2 46.9 8.6 59.1 7.3

eMFD Dataset

Claude Sonnet 4 33.6 18.4 9.1 15.2 19.3 15.0 44.9 10.6 36.1 4.1

DeepSeek-V3 18.2 19.0 7.5 16.0 25.0 11.5 16.8 13.0 47.8 2.4

Llama 4 Maverick 12.6 20.7 7.9 14.1 18.2 13.4 19.9 14.8 32.2 3.8

Human Baseline 56.5 16.0 59.8 15.2 54.9 16.2 59.4 16.4 61.2 13.5

Table 4: False positive and negative rates (%) by model across moral foundations and datasets. Human baseline represents average performance across all annotators.

Figure 5: Error trade-offs in moral foundation detection. AI models (shapes) vs human baselines (circles) across datasets with colors denoting moral foundations. Diagonal lines indicate error balance (FPR = FNR).

4 Case Study

Human annotation, while considered the gold standard for supervised learning, may systematically underestimate moral content prevalence in text. As noted in annotator training guidelines, "Ambiguities present considerable challenges for human annotators who must strike an acceptable balance between exploiting often weak signals of moral sentiment while also avoiding unfounded speculation about authorial intent." Our results suggest AI models navigate this balance more effectively, achieving both higher recall rates and superior balanced accuracy compared to human annotators.

To verify this finding, we evaluated a selection of posts where AI predicted substantially more moral content than human annotators. An independent domain expert confirmed that AI was correct in the majority of cases, suggesting that human annotators systematically miss moral content that AI systems correctly identify. Full details of this validation study are provided in the supplementary material and code repository. The examples below represent cases where no human annotator detected moral language, false negatives in human annotation that AI correctly identified, demonstrating AI’s superior sensitivity to subtle moral cues in text.

Conclusion

This work presents the most comprehensive evaluation to date of moral foundation detection capabilities in state-of-the-art large language models, analyzing 1M+ queries across 250K+ annotations from three established corpora. By employing Bayesian annotation modeling, we moved beyond deterministic ground-truth assumptions to explicitly account for annotator disagreement in moral foundation classification.

Our key findings demonstrate that AI models consistently outperform human annotators in moral foundation detection, typically ranking in the top 25% across all datasets. Critically, AI models achieve 2−4× lower false negative rates, while accepting slightly higher false positive rates, revealing fundamentally different error strategies. Although humans often miss genuine moral foundations due to biases, AI models provide more balanced detection of moral content.

These findings have important implications for deploying LLMs in moral foundation analysis. The superior recall capabilities make AI models valuable for detecting moral foundations that humans might overlook, though slightly higher false positive rates require careful calibration for specific applications. Our uncertainty-aware evaluation framework provides a foundation for more nuanced assessment of AI moral foundation detection that accounts for inherent subjectivity rather than treating classification as deterministic.

Future work should focus on improving model calibration for moral foundation detection and extending evaluation to other moral frameworks, as well as exploring connections to other datasets such as anticipatory discourses Landowska et al. (2023); Skórski et al. (2025).

Acknowledgements

The authors thank the people and institutions who supported this research: Murmuras GmbH22 2 https://murmuras.com/ for providing access to state-of-the-art large language models’ APIs, University of Luxembourg for financial support and computing infrastructure, Kozminski University and Polish Ministry of Science for supporting participation in the "Top 1000 Innovators of Poland in Silicon Valley" program, Pierre Pailler from Virgule for the opportunity to present this research to the general public Pailler (2025), and the anonymous reviewers of the UncertainNLP workshop for their valuable feedback.

Limitations

Fixed Effects Modeling Limitations.

While our Bayesian framework supports demographic fixed effects to account for systematic annotator differences, we did not employ these due to limited demographic metadata (only the crowd-sourced eMFD corpus contains sufficient demographic data). This somewhat constraints our ability to model known sources of variation in moral foundation detection across annotator groups.

Content Moderation Limitations.

Content moderation systems may introduce systematic bias by refusing to process morally relevant content. Azure OpenAI rejected 10% of Twitter data citing improper content, while Anthropic’s model questioned the factual accuracy of referring to an "Algerian dictator" in our Macron example. After excluding OpenAI, content moderation affected less than 0.5% of our final evaluation dataset.

Data Availability Constraints.

The three corpora span different periods (2016-2022) and textual domains, reflecting constraints of available annotated datasets rather than ideal experimental design. These temporal and domain variations may somewhat limit the generalizability of the findings.

Annotation Protocol Heterogeneity.

The three corpora employed different annotation protocols—varying in annotator training, guidelines, and expertise levels—which may introduce inconsistencies in the ground truth labels. The MFTC used trained experts while eMFD relied on crowd workers, potentially creating systematic differences in annotation quality and interpretation that our Bayesian model may not fully reconcile.

Prompt Sensitivity Limitations.

While we deliberately avoided extensive prompt engineering to reflect typical usage conditions, LLM performance on moral classification tasks is known to be sensitive to prompt phrasing and structure (see Skorski and Landowska (2025)). Our single-prompt approach yields balanced results suitable for general-purpose evaluation, though alternative prompt strategies (e.g., chain-of-thought, few-shot examples, explicit instructions emphasizing sensitivity or specificity) could shift the precision-recall tradeoff. Such variations might improve performance on specific moral concepts or optimize for particular error patterns, but would deviate from typical deployment conditions we aimed to assess.

Low-Resource Language and Cultural Bias Limitations.

Our evaluation is limited to English-language datasets, as annotated moral reasoning datasets in low-resource languages are not yet available. This gap somewhat limits cross-cultural validation of our findings, though annotator diversity—from trained experts to crowdsourced contributors—helps mitigate individual cultural biases within English contexts. As the field develops multilingual moral foundation datasets, our framework can readily extend to cross-linguistic evaluation.

Theoretical Scope Limitations.

Our exclusive focus on Moral Foundation Theory, while practical and widely-used for large-scale analysis, represents a significant theoretical limitation. Alternative moral and ethical theories (e.g., virtue ethics Culham et al. (2024), deontological ethics Gawronski and Beer (2016) or care ethics Deak and Saroglou (2016)) offer different perspectives on morals that may capture dimensions not encompassed by MFT’s five foundations. Furthermore, the binary classification framework fails to capture the intensity, valence, or contextual nuance of moral expressions. However, MFT remains best suited for large-scale computational analysis given its established usage, data availability, and continued popularity in computational linguistics research. Future work should explore hybrid approaches that combine MFT with complementary frameworks like virtue ethics or care ethics to provide more comprehensive moral analysis.

References

Abadi et al. (2015) M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng TensorFlow: large-scale machine learning on heterogeneous systems. Note: Software available from tensorflow.org External Links: Link Cited by: §2.2.

Bulla et al. (2025) L. Bulla, S. De Giorgis, M. Mongiovì, and A. Gangemi Large Language Models meet moral values: A comprehensive assessment of moral abilities. Computers in Human Behavior Reports 17, pp. 100609. External Links: ISSN 24519588, Document, Link Cited by: §1.3.

Culham et al. (2024) T. E. Culham, R. J. Major, and N. Shivhare Virtue ethics and moral foundation theory applied to business ethics education. International Journal of Ethics Education 9 (1), pp. 139–176. External Links: ISSN 2363-9997, 2364-0006, Document, Link Cited by: Theoretical Scope Limitations..

Dawid and Skene (1979) A. P. Dawid and A. M. Skene Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics 28 (1), pp. 20. External Links: 10.2307/2346806, ISSN 00359254, Document, Link Cited by: §2.2.

Deak and Saroglou (2016) C. Deak and V. Saroglou Valuing Care Protects Religiosity from the Antisocial Consequences of Impersonal Deontology. Journal of Empirical Theology 29 (2), pp. 171–189. External Links: ISSN 0922-2936, 1570-9256, Document, Link Cited by: Theoretical Scope Limitations..

Feinberg and Willer (2013) M. Feinberg and R. Willer The moral roots of environmental attitudes. Psychological Science 24 (1), pp. 56–62. Cited by: §1.1.

Gawronski and Beer (2016) B. Gawronski and J. S. Beer What makes moral dilemma judgments “utilitarian” or “deontological”?. Social Neuroscience, pp. 1–7. External Links: ISSN 1747-0919, 1747-0927, Document, Link Cited by: Theoretical Scope Limitations..

Graham et al. (2013) J. Graham, J. Haidt, S. Koleva, M. Motyl, R. Iyer, S. P. Wojcik, and P. H. Ditto Moral foundations theory: the pragmatic validity of moral pluralism. Advances in Experimental Social Psychology 47, pp. 55–130. Cited by: §1.1.

Graham et al. (2009) J. Graham, J. Haidt, and B. A. Nosek Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology 96 (5), pp. 1029–1046. External Links: Document Cited by: §1.1.

Haidt (2012) J. Haidt The Righteous Mind: Why Good People Are Divided by Politics and Religion. Knopf Doubleday Publishing Group. External Links: ISBN 978-0-307-90703-5 Cited by: §1.1.

Hoover et al. (2020) J. Hoover, G. Portillo-Wightman, L. Yeh, S. Havaldar, A. M. Davani, Y. Lin, B. Kennedy, M. Atari, Z. Kamel, M. Mendlen, G. Moreno, C. Park, T. E. Chang, J. Chin, C. Leong, J. Y. Leung, A. Mirinjian, and M. Dehghani Moral Foundations Twitter Corpus: A Collection of 35k Tweets Annotated for Moral Sentiment. Social Psychological and Personality Science 11 (8), pp. 1057–1071. External Links: ISSN 1948-5506, 1948-5514, Document, Link Cited by: §2.1.

Hopp et al. (2021) F. R. Hopp, J. T. Fisher, D. Cornell, R. Huskey, and R. Weber The extended Moral Foundations Dictionary (eMFD): Development and applications of a crowd-sourced approach to extracting moral intuitions from text. Behavior Research Methods 53 (1), pp. 232–246. External Links: ISSN 1554-3528, Document, Link Cited by: §1.3, §2.1.

Landowska et al. (2023) A. Landowska, M. Robak, and M. Skorski What Twitter Data Tell Us about the Future?. arXiv. External Links: 2308.02035, Document, Link Cited by: Conclusion.

Nguyen et al. (2024) T. D. Nguyen, Z. Chen, N. G. Carroll, A. Tran, C. Klein, and L. Xie Measuring Moral Dimensions in Social Media with Mformer. Proceedings of the International AAAI Conference on Web and Social Media 18, pp. 1134–1147. External Links: ISSN 2334-0770, 2162-3449, Document, Link Cited by: §1.1, §1.3, §1.3, §1.3.

Nguyen et al. (2022) T. D. Nguyen, G. Lyall, A. Tran, M. Shin, N. G. Carroll, C. Klein, and L. Xie Mapping topics in 100,000 real-life moral dilemmas. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 16, pp. 699–710. Cited by: §1.1.

Pailler (2025) P. Pailler L’IA meilleure que l’humain sur les questions morales, selon une étude de l’Uni?. Virgule. External Links: Link Cited by: Acknowledgements.

Paun and Simpson (2021) S. Paun and E. Simpson Aggregating and Learning from Multiple Annotators. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, online, pp. 6–9. External Links: Document, Link Cited by: §2.2.

Preniqi et al. (2024) V. Preniqi, I. Ghinassi, J. Ive, C. Saitis, and K. Kalimeri MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussions. In Proceedings of the 2024 International Conference on Information Technology for Social Good, Bremen Germany, pp. 433–442. External Links: Document, Link, ISBN 9798400710940 Cited by: §1.1, §1.3, §1.3.

Roy and Goldwasser (2021) S. Roy and D. Goldwasser Analysis of Nuanced Stances and Sentiment Towards Entities of US Politicians through the Lens of Moral Foundation Theory. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, Online, pp. 1–13. External Links: Document, Link Cited by: §1.1.

Skórski et al. (2025) M. Skórski, A. Landowska, and K. Rajda Mapping technological futures: anticipatory discourse through text mining. Humanities and Social Sciences Communications 12 (1), pp. 899. External Links: ISSN 2662-9992, Document, Link Cited by: Conclusion.

Skorski and Landowska (2025) M. Skorski and A. Landowska The Moral Gap of Large Language Models. External Links: 2507.18523, Document, Link Cited by: Appendix A, Appendix D, §1.3, Prompt Sensitivity Limitations..

Skórski (2025) M. Skórski Bayesian Evaluation of LLMs’ Moral Values Understanding. Note: Supplementary materials External Links: Document, Link Cited by: Appendix B, §2.5.

Trager et al. (2022) J. Trager, A. S. Ziabari, A. M. Davani, P. Golazizian, F. Karimi-Malekabadi, A. Omrani, Z. Li, B. Kennedy, N. K. Reimer, M. Reyes, K. Cheng, M. Wei, C. Merrifield, A. Khosravi, E. Alvarez, and M. Dehghani The Moral Foundations Reddit Corpus. arXiv. External Links: Document, Link Cited by: §2.1.

Zangari et al. (2025a) L. Zangari, C. M. Greco, D. Picca, and A. Tagarelli ME2-BERT: Are events and emotions what you need for moral foundation prediction?. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, pp. 9516–9532. External Links: Link Cited by: §1.1, §1.3, §1.3.

Zangari et al. (2025b) L. Zangari, C. M. Greco, D. Picca, and A. Tagarelli A survey on moral foundation theory and pre-trained language models: current advances and challenges. AI & SOCIETY. External Links: ISSN 0951-5666, 1435-5655, Document, Link Cited by: §1.3.

Appendix A AI Prompt

For estimating moral competence under typical usage conditions, we deliberately avoided prompt engineering techniques that might bias models toward higher recall or precision (the work of Skorski and Landowska (2025) demonstrated that LLMs are somewhat sensitive in that regard). We tested prompt variants that explicitly request step-by-step reasoning, without noticing significant differences.

Appendix B TensorFlow Model Implementation

.Our Bayesian aggregation uses sparse tensor operations for efficient likelihood computation across annotator-item pairs. The implementation leverages TensorFlow’s sparse lookup operations to handle high-dimensional confusion matrices, enabling scalable inference on GPU hardware. The core algorithm computes marginal likelihoods over latent true labels by efficiently aggregating log-probabilities from annotator-specific confusion matrices, exploiting sparsity in the annotation pattern (most annotators do not label all items). The optimization uses gradient ascent on the log-posterior, jointly estimating class prevalences and per-annotator competence parameters. Details are available in our code repository Skórski (2025).

Listing 1: TensorFlow implementation

Appendix C Content Moderation Examples

Social media is full of expressive posts that occasionally may not be evaluated by AI providers who implement strict internal safety mechanisms, limiting research capabilities on real-world content. For this reason, we did not include the results of OpenAI. One rejected example is shown below.

Appendix D Moral Foundations Color Palette

We use the colorblind palette from Skorski and Landowska (2025), transitioning from individualistic (cooler) to collectivistic (warmer) foundations

Figure 6: Moral colors: individualism to collectivism