5,493,066 AI resources indexedWorking toward the world’s largest AI resource index.

Browse resources 13 sections
PDF

Effective Context Selection in LLM-based Leaderboard Generation: An Empirical Study

This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, o

By Kabongo, Salomon, D'Souza, Jennifer, Auer, SörenResearch paper

Resource information

Authors
Kabongo, Salomon, D'Souza, Jennifer, Auer, Sören
Research area
Artificial intelligence
Research topics
Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences
Publication date
2024
DOI
10.48550/arxiv.2407.02409
Upstream last modified
Jul 2, 2024
License
Creative Commons Attribution 4.0 International
Last record update
Nov 5, 2026

About this resource

This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.