Athena
Academic Research
Evaluating LLM-Driven Book Discovery: A Comparative Study of a RAG-Based Chat Interface vs. Traditional Search in Digital Libraries
This paper introduces Athena, a conversational, RAG-powered book-discovery system for Project Gutenberg, and compares it to traditional keyword search. Our user study shows that the LLM-based interface improves retrieval accuracy, user satisfaction, and exploratory search while highlighting challenges around trust and AI-generated content.
Authors
Nicole León, Pennsylvania State University
Matt Murtagh White, Trinity College Dublin
Yunkai Xu, Pennsylvania State University
Date
May 5th, 2025
Publication / Venue
Manuscript submitted to ACM; Presented at UIST 2025 in Busan, South Korea
Details
22-page manuscript
Empirical within-subjects study comparing a RAG-LLM conversational interface vs. the Project Gutenberg website
Key focus areas: exploratory search, trust in conversational agents, retrieval accuracy, visualization through knowledge graphs
From Queries to Conversations: How Interface Modality Shapes Exploratory Search Behavior
This study compares four search modalities: keyword, semantic, conversational, and conversational-graph to evaluate how interface design influences exploratory search, workload, confidence, and sense-making in large digital libraries. Conversational interfaces improved synthesis quality and reduced workload, but also increased confidence without increasing accuracy, revealing important risks in LLM-driven exploration.
Authors
Nicole León, Pennsylvania State University
Anonymous Author(s) (review-stage IUI submission)
Date
October 10th, 2025
Publication / Venue
Manuscript submitted to ACM (for IUI conference review)
Details
24-page mixed-methods study
Interfaces evaluated: keyword search (Project Gutenberg), semantic list search,
conversational LLM search, conversational-graph search
Measures: performance, workload (NASA-TLX), confidence calibration, synthesis quality
28-participant within-subjects lab study
Talk to Your Library: A Visual Conversational System for Book Discovery
Talk to Your Library is a research poster condensing the insights derived from the 'Evaluating LMM-Driven Book Discovery' research paper featured at the top of this page.
