RAGE Against the Machine: Retrieval-Augmented

LLM Explanations
Joel Rorseth Parke Godfrey Lukasz Golab Divesh Srivastava Jaroslaw Szlichta
University of Waterloo York University University of Waterloo AT&T Chief Data Office York University
[email protected] [email protected] [email protected] [email protected] [email protected]

Abstract—This paper demonstrates RAGE, an interactive tool RAG—instead demands high-level explanations of RAG in
for explaining Large Language Models (LLMs) augmented with
arXiv:2405.13000v1 [cs.CL] 11 May 2024

simple terms. While specific prompting strategies such as

retrieval capabilities; i.e., able to query external sources and pull chain-of-thought (CoT) prompting could serve as interpretable
relevant information into their input context. Our explanations
are counterfactual in the sense that they identify parts of the explanations, RAG has yet to receive dedicated focus in the
input context that, when removed, change the answer to the explainability literature. RAG is a leading prompting strategy
question posed to the LLM. RAGE includes pruning methods to for the use of modern LLMs in question answering (QA), as
navigate the vast space of possible explanations, allowing users CoT and other prompting techniques are less applicable and
to view the provenance of the produced answers. require specialized examples. Specific concerns about RAG,
such as the lack of provenance in LLM answers, or the “lost
I. I NTRODUCTION
in the middle” context position bias observed in recent LLMs
Motivation. Artificial Intelligence (AI) has seen remarkable [2], warrant dedicated study under an explainability lens.
growth in terms of both capability and popularity, exemplified Contributions. To fill this gap, we demonstrate RAGE,1
by recent large language models (LLMs) such as OpenAI’s an interactive tool designed to enable RAG Explainability for
ChatGPT, Microsoft’s Copilot, and Google’s Gemini. The LLMs.2 Our tool deduces provenance and salience for external
rapid progress in LLM capability is driven by scale, as AI re- knowledge sources used during RAG, exposing the in-context
searchers train increasingly complex models with increasingly learning behaviors of the LLM. Motivated by our prior work
large datasets using enormous computational resources. Within using counterfactual explanations for information retrieval [3],
a short span, state-of-the-art models have progressed from we derive provenance counterfactually by identifying minimal
training millions, to billions, and now to trillions of internal pa- context perturbations that change an LLM’s output. Our con-
rameters. However, this increased complexity further obscures tributions are summarized as follows.
the underlying decision-making process of LLMs, making it 1) Answer Origin Explainability. We introduce a novel
challenging to rationalize or troubleshoot their outputs. As framework to assess the origin of LLM answers, with
LLMs are adopted in critical sectors, it is imperative that respect to context knowledge sources, by evaluating coun-
verifiable explanations accompany their outputs, to build trust. terfactual source combinations and permutations.
The unique enhancements in capability that distinguish 2) Pruning Strategies. We present inference pruning strate-
LLMs from previous language models amplify their explain- gies to reduce the space of possible counterfactual expla-
ability concerns. Of particular relevance is retrieval-augmented nations, by prioritizing the evaluation of important context
generation (RAG), a popular prompt engineering strategy that perturbations.
leverages a powerful new LLM capability known as in-context 3) Interactive Demo. Participants will pose questions to
learning. With RAG, an LLM augments its trained knowledge an LLM augmented with knowledge sources from real
by learning from external knowledge sources, supplied directly datasets. RAGE will display explanations for RAG scenar-
via the LLM’s input context (prompt). RAG has been pivotal ios where answers are ambiguous, sourced from inconsis-
for LLMs in reducing their tendency to hallucinate plausible tent external knowledge, or traced through a chronological
yet incorrect outputs. This complex process, however, obfus- sequence. Participants will see how subjective questions,
cates the provenance of the produced answers. such as determining the greatest professional tennis player,
Background. Due to the recency of LLMs and their emer- can be answered differently by an LLM, depending on the
gent capabilities, few efforts have been made to explain their combination and order of context sources.
phenomena. Under the umbrella of mechanistic interpretabil- II. S YSTEM D ESCRIPTION
ity, low-level analyses have sought to understand the mecha-
A. Problem Description
nisms behind transformer-based language models, and capabil-
ities like in-context learning, by analyzing circuits that form Open-book question answering is a task where a system
amongst attention heads, or by assessing an LLM’s ability determines an answer to a given question using common
to override trained knowledge [1]. Our explainability focus— 1A video is available at https://vimeo.com/877281038.
which aims to trace the provenance of LLM answers during 2 The tool is available at http://lg-research-2.uwaterloo.ca:8092/rage.
 Users RAGE Knowledge
predicted answer. Combination-based counterfactual explana-
Retrieval Model tions, which can serve as citations, may be generated using
(Pyserini BM25) Lucene
Index a top-down or bottom-up search. A top-down counterfactual
Web App Answers Perturbation must remove a combination of sources (subset of Dq ) to flip
(Plotly Dash) Analysis Search LLM
the full-context answer to a target answer. On the other hand, a
Counterfactual Counterfactual
Llama 2 Chat 7B LLM
bottom-up counterfactual must retain sources to flip the empty-
Explanations Search
context answer to the target answer.
Fig. 1. The architecture of RAGE. In either case, the candidate solution search space is defined
as the set of all combinations of the given sources. We propose
knowledge about the topic and a provided set of sources. In an iterative algorithm that tests combinations in increasing
RAGE, we explain how an LLM performs this task, using its order of subset size. Specifically, we evaluate all combinations
own pre-trained knowledge and retrieved knowledge sources. containing k sources before moving on to those with k + 1
A user initiates the process by posing a search query q to sources. Since there may be multiple combinations of equal
a retrieval model M . Given an index of knowledge sources size, we iterate through these equal-size combinations in order
and a relevance threshold k, the retrieval model M scores and of their estimated relevance. This is calculated as the sum
ranks the k most relevant sources from the index with respect of the relative relevance scores of all P sources within the
to query q. The resulting ordering of sources, denoted as Dq , combination, which can be expressed as d∈Dq S(q, d, Dq ).
forms a sequence we refer to as the context. To estimate the relative relevance of a source d ∈ Dq ,
We combine Dq and q to form a natural language prompt the user can select from two scoring methods S. In the first
p for the LLM L. This prompt instructs L to answer question method, we aggregate the LLM’s attention values, summing
q using the information contained within the set of delimited them over all internal layers, attention heads, and tokens
sources from Dq . Although p serves as the final and sole input corresponding to a combination’s constituent sources. In the
to the LLM, we denote the answer a produced by the LLM for second method, we sum the relevance scores produced by the
a given query q and the sequence of knowledge sources Dq retrieval model for each source. Since we only compare scores
as a = L(q, Dq ). We also define S(q, d, Dq ) as the relative for combinations of equal size, there is no need to normalize
relevance score of a source d ∈ Dq with respect to the query combination scores by the number of sources.
q and other sources within Dq . To derive explanations, we To generate permutation-based counterfactual explanations,
assess the answers generated across various combinations or RAGE searches for the most similar source permutation (with
permutations of the sources in Dq . We refer in general to these respect to their given order) such that the LLM responds with
two methods as context perturbations. a different answer. These explanations quantify the stability
B. Architecture of the LLM’s answer with respect to the order of the con-
text sources, thus revealing any unexpected context position
RAGE is an interactive Python web application developed us-
bias. Our algorithm generates all length-k permutations for
ing the Plotly Dash web framework. We installed the 7B Llama
the k sources, then computes Kendall’s Tau rank correlation
2 Chat LLM [4] (meta-llama/Llama-2-7b-chat-hf ) through the
coefficient for each permutation (with respect to their given
Hugging Face Transformers library. Our software is, however,
order in Dq ). Once generated and measured, the permutations
fully compatible with any similar transformer-based LLM. All
are subsequently sorted and evaluated in decreasing order of
knowledge sources (documents) are retrieved from our locally-
similarity, based on decreasing Kendall’s Tau.
configured document indexes, using a BM25 retrieval model
For both combinations and permutations, our algorithm
from the Pyserini retrieval toolkit [5].
continues until it finds a perturbation that changes the answer,
We run our application on an Ubuntu 22.04 server, with
or until a maximum number of perturbations have been tested.
an Intel Core i9-7920x CPU, 128GB of DDR4 RAM, and
Before comparing against the original answer, we convert an-
an NVIDIA RTX 4090 GPU with 24GB of memory. We use
swers to lowercase, remove punctuation, and trim whitespace.
PyTorch’s CUDA library to run LLM operations on the GPU.
To supplement this counterfactual analysis, we analyze the
In RAGE, users can generate explanations in terms of two answers over a selected set of perturbed sources. To obtain a
complementary perturbations: source combinations or source set of combinations, RAGE considers all combinations of the
permutations. Combinations elucidate how the presence of retrieved sources Dq , or draws a fixed-size random sample of s
sources affects the LLM’s predicted answer, while permuta- combinations. Based on the user’s original question, a prompt
tions elucidate the effect of their order. Alongside counterfac- is created for each selected combination, which is then used to
tual explanations for each answer, RAGE presents a pie chart retrieve corresponding answers from the LLM. After analyzing
to visualize the distribution of answers, a list of perturbation- the answers, RAGE renders a table that groups combinations
answer rules, and a table associating different answers with by answer, along with a pie chart illustrating the proportion
the perturbations that produced them. of each answer across all combinations. A rule is determined
C. RAG Explanations for each answer, when applicable, identifying sources that
In generating counterfactuals, RAGE aims to identify minimal appeared in all combinations leading to this answer.
perturbations to the context that lead to a change in the LLM’s In a similar manner, the user can instruct RAGE to analyze
answers from a selected set of source permutations. The table
and pie chart illustrating associations between answers and
permutations resemble those of the combination case, with the
rule calculation adopting a unique definition. For each answer,
we determine a rule that identifies any context positions for
which all permutations leading to this answer shared the same
source. Users may again choose to analyze all permutations,
or a fixed-size random sample of s permutations.
For the latter, a naive solution might generate all k! permu-
tations of the k sources, then uniformly sample s permutations,
resulting in O(k!) time complexity. To improve the efficiency,
we propose an implementation using the Fisher–Yates shuffle
algorithm [6], which produces an unbiased random permu-
tation of any finite sequence in O(k). In our approach, we
invoke the Fisher-Yates algorithm s times to generate s random
permutations, resulting in an efficient O(ks) solution.
RAGE also allows the user to analyze the most optimum
permutations. As observed in recent works [2], LLMs often ex-
hibit a context position bias, paying more attention to sources
appearing at the beginning and end of the context than those in
the middle. As a result, sources that are important for obtaining
a given answer may not receive sufficient consideration by
the LLM. Given a distribution of the expected attention paid
to each position, this “lost in the middle” bias can be coun-
teracted by positioning important sources in high-attention
positions. By requesting “optimal permutations” from RAGE,
the user can analyze a set of permutations with optimum
placement of relevant sources in high-attention positions.
To estimate the relevance of a source, the user can choose to
use either the LLM’s attention scores or the retrieval model’s
assessed relevance score. If desired, the user can calibrate the
expected distribution of LLM context position attention by
selecting a predefined V-shaped distribution. Optimal permuta-
tions aim to maximize both the relevance and attention of their
constituent sources. A naive O(k!) solution might generate all
k! permutations, scoring each by summing the product of each
source’s relevance and attention, then sorting and selecting
the s highest-scoring permutations. Recognizing that optimal
permutations must maximize both the relevance and attention Fig. 2. Combination insights for the query about The Big Three.
of their constituent sources, we propose an efficient solution
by formulating this problem as an instance of the assignment A. Categorization of Use Cases
problem in combinatorics.
Numerous algorithms have been proposed to solve this The explanations generated by RAGE are applicable across
problem, which aim to find the most optimal assignment of all countless domains. Use cases can be categorized based on
k sources to all k context positions. Since RAGE allows the various factors, such as whether knowledge sources form a
user to request the top-s optimal permutations, our formulation timeline, or when questions are subjective, leading to am-
adopts a variant of the assignment problem that seeks the s biguous answers. In the former case, RAGE identifies salient
assignments with minimal cost. We use the algorithm proposed periods in time. In the latter case, it procures evidence to
by Chegireddy and Hamacher [7], which allows us to calculate support various answers. Knowledge sources may differ in
the s optimal permutations in O(sk 3 ). terms of their consistency. Our tool can identify consistent and
inconsistent sources. Sources may or may not share semantic
III. D EMONSTRATION P LAN dependencies, and may or may not share syntactic formats.
Conference participants will explore the provenance of RAGE will highlight source agreement and disagreement.
information included in retrieval-augmented LLM responses. In the following subsections, we introduce several use cases
They will then reinforce these findings by evaluating the that highlight the axes of this categorization. We begin by
importance of relative position among sources. exploring the possibility of an ambiguous answer, which
requires efficient evaluation over a large sample of knowledge asks RAGE to derive permutation insights. By reordering the
source combinations. Next, we present a scenario in which context documents in various configurations and analyzing the
sources are slightly inconsistent, testing RAGE’s ability to resulting answers, RAGE discovers that the LLM incorrectly
identify minor differences that can change the LLM’s answer. identifies the 2022 champion “Iga Swiatek” whenever the last
Last, we provide an example in which the sources form a document is moved towards the middle of the sequence. Using
timeline, requiring RAGE to strategically navigate alternate RAGE, the user has identified the up-to-date document that
timelines by minimally combining and permuting the sources. offers the correct answer, and has gleaned insights about out-
B. Use Case #1: Ambiguous Answers of-date documents and their ability to confuse the LLM.
The user asks an LLM to determine the best tennis player D. Use Case #3: Timelines
among “The Big Three” of Novak Djokovic, Roger Federer, The user consults an LLM to determine how many times
and Rafael Nadal. The user does not have any specific com- Novak Djokovic won the Tennis Player of the Year award
parison metric in mind, so they use the system to retrieve a set between 2010 and 2019. The user gathers relevant documents
of related documents, each containing a different ranking of from the system, each corresponding to one year’s winner.
The Big Three based on different metrics (e.g., total number Collectively, the documents form a timeline for the three win-
of match wins and number of weeks ranked first). The user ners: Rafael Nadal (2010, 2013, 2017, 2019), Novak Djokovic
expects that Novak Djokovic, who recently surpassed Rafael (2011, 2012, 2014, 2015, 2018), and Andy Murray (2016).
Nadal and Roger Federer in total Grand Slam wins, might be The user poses their question to RAGE, which reports that
the LLM’s choice. But when asked with the combination of all the LLM produces the expected answer of 5 when incorpo-
retrieved documents, the LLM’s answer is “Roger Federer.” rating the combination of all retrieved documents. To validate
Curious about why the LLM chose Federer, the user poses the LLM’s response, the user expects an explanation listing
the same query and documents to RAGE, requesting combi- each year Djokovic won the award, along with a citation to
nation insights. As illustrated in Figure 2, RAGE analyzes the a supporting document. To achieve this, the user reviews the
answers generated by the LLM using various combinations combination counterfactual generated by RAGE to determine
of the given documents, and discovers that the first document the minimal set of documents (and thus the exact years)
led the LLM to produce this answer. This document ranks required to infer the correct answer. RAGE cites five separate
various tennis players based on total match wins, with Federer documents from those provided, each documenting a different
ranking first at 369. RAGE’s answer rules formalize this year in which Djokovic won Player of the Year.
explicitly, asserting that this document was included in every Hoping to ensure that the LLM has not overlooked any
combination for which the LLM answered “Roger Federer.” time period covered by the documents, the user asks RAGE to
The user now comprehends why the LLM chose Federer derive permutation insights over the same inputs. By analyzing
but remains curious about the document’s relative signifi- a sample of permutations, the user is presented with a pie
cance. Reviewing the original ranking, they notice that this chart and answer table that indicate a consistent answer of 5.
document has prominent placement at the beginning of the The user observed that no rules were found, and concludes
context. To investigate the impact of this position, the user that the LLM consistently comprehends the entire timeline
requests permutation-based explanations for the same inputs. of the twenty-tens, regardless of the specific order of the
Surprisingly, RAGE reveals that moving the document to the timeline’s constituent documents. Through RAGE, the user has
second position altered the answer to “Novak Djokovic.” In successfully discovered which segments of the timeline were
short, these explanations have enabled the user to promptly crucial in determining the correct answer.
identify the document that influenced the LLM’s answer, and
R EFERENCES
to understand the impact of its relative position.
[1] J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu,
C. Use Case #2: Inconsistent Sources D. Huang, D. Zhou et al., “Larger language models do in-context learning
The user turns to an LLM for help in determining the most differently,” arXiv preprint arXiv:2303.03846, 2023.
[2] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and
recent winner of the US Open women’s tennis championship. P. Liang, “Lost in the middle: How language models use long contexts,”
A small set of documents is retrieved, each containing relevant 2023, arXiv:2307.03172.
statistics about US Open championships. The documents share [3] J. Rorseth, P. Godfrey, L. Golab, M. Kargar, D. Srivastava, and J. Szlichta,
“Credence: Counterfactual explanations for document ranking,” in ICDE,
similar format, but some may be more current than others. 2023, pp. 3631–3634.
Hoping that the LLM will pinpoint the most recent winner [4] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei,
across all documents, the user requests combination insights N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open
foundation and fine-tuned chat models,” arXiv:2307.09288, 2023.
in RAGE, and observes how the combination containing all [5] J. Lin, X. Ma, S.-C. Lin, J.-H. Yang, R. Pradeep, and R. Nogueira,
sources produces the response “Coco Gauff.” With no further “Pyserini: A Python toolkit for reproducible information retrieval research
explanation, the user aims to verify this result by identifying with sparse and dense representations,” in SIGIR, 2021, pp. 2356–2362.
[6] R. A. Fisher and F. Yates, Statistical tables for biological, agricultural
the source document behind the answer, and discovers that the aad medical research. Edinburgh: Oliver and Boyd, 1938.
last context document recognizes Gauff as the 2023 champion. [7] C. R. Chegireddy and H. W. Hamacher, “Algorithms for finding k-best
Curious whether other out-of-date documents could have perfect matchings,” Discrete Applied Mathematics, vol. 18, no. 2, pp.
155–165, 1987.
been mistakenly sourced for an incorrect answer, the user