Towards an Interactive ‘Evidence-RAG’ Peer-Review Workspace for the Journal of Digital History

AI through History, History through AI - Tuesday 16th June 2026

Mehrdad Almasi

Luxembourg Centre for Contemporary and Digital History (C²DH)

Frédéric Clavert

Luxembourg Centre for Contemporary and Digital History (C²DH)

Elisabeth Guerard

Luxembourg Centre for Contemporary and Digital History (C²DH)

Mirjam Pfeiffer

Luxembourg Centre for Contemporary and Digital History (C²DH)

Marion Salaün

Luxembourg Centre for Contemporary and Digital History (C²DH)

Plan

  1. Introduction
  2. Challenges about peer review
  3. Evidence RAG
    • Infrastructure
    • Pipeline
    • Application
    • Results
  4. What should be the future steps ?
  5. Advantages
  6. Proof of Concept

Introduction

Journal of Digital history is an international, academic, peer-reviewed and open-access journal, the Journal of Digital History (JDH) based on the principle of multi-layered articles.

As all academic journals, the JDH experiences a peer review crisis.


More than half of researchers now use AI for peer review — often against guidance see Naddaf (2025)

Challenges

  • Review processing is time-consuming.
  • Important evidence can be overlooked when papers are long and complex.
  • Limited transparency makes it difficult to justify LLM-supported scholarly assessments and decisions, see Martin-Boyle et al. (2026).
  • Manual review processes are difficult to scale as submission volumes increase, see Kim, Lee, and Lee (2025).

Evidence RAG

Infrastructure

Grid5000 is a large scale and flexible testbed for experiment-driven research in all areas of computer science.

  • Reliable research infrastructure.
  • 11 sites in France, Belgium and Luxembourg.
  • ~25,000 cores, 800 compute nodes, 550+ GPUs
  • Helped ensure more controlled and reproducible model testing.
  • Focus Areas: Parallel/distributed computing, cloud, HPC, Big Data, AI

Application

We created a JDH evidence audit application.

The final confidence score combines four components:

Confidence = 0.15 × Retrieval Strength + 0.25 × Model Support Strength + 0.45 × Evidence Specificity + 0.15 × Section Coverage

  • 🔎 Retrieval Strength: average retrieval score of the retrieved chunks.
  • 🤖 Model Support Strength: to what extent the model judges the reviewer comment as correct.
  • 🎯 Evidence Specificity: to what extent the retrieved items are related to the reviewer comment; this judges the quality of the retrieved evidence.
  • 🧩 Section Coverage: unique retrieved sections divided by total paper sections.

Results

  • paper 1 and paper 2: reviewers likely provided more accurate content
  • paper 3: reviewers seem to have more difficulties during the evaluation

What a human thinks

  • claude-qwen does a better job, because its ability to compare the article’s content and the review’s content is better

  • the system we set up is not good to evaluate reviewers’ second (or third) round of comments, because lacks the ability to compare several versions of the article

  • the model should have access to the full GitHub repository, because some reviewers look at the full repository

  • there might be a problem of sycophancy: sometimes, the model agrees with two divergent reviewers’ evaluations on one precise aspect of an article

Future Steps

What are our next steps ?

  1. Test the system with more papers and reviewer comments.
  2. Compare results across different models and retrieval settings.
  3. Improve the confidence score with editor feedback.
  4. Extend the system to support more journals and disciplines.

Advantages of our work

  • Supports editors by linking reviewer comments to paper evidence.
  • Makes peer review decisions more transparent and explainable.
  • Helps identify unsupported, partially supported, or unclear reviewer comments.
  • Keeps the human editor as the final decision-maker.

Proof of Concept (POC)

Get access to the video

Bibliography

Kim, Jaeho, Yunseok Lee, and Seulki Lee. 2025. “Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards.” arXiv Preprint arXiv:2505.04966.
Martin-Boyle, Anna, Cara A. C. Leckey, Martha C. Brown, and Harmanpreet Kaur. 2026. “PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-Based Scholarly q&a.” In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26). ACM.
Naddaf, Miryam. 2025. “More Than Half of Researchers Now Use AI for Peer Review — Often Against Guidance.” Nature 649 (8096): 273–74. https://doi.org/10.1038/d41586-025-04066-5.