Scribd
Upload
133 views

LLM With Knowledge Graphs

Uploaded by

Margaux Yap
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
133 views

LLM With Knowledge Graphs

Uploaded by

Margaux Yap
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 40

+ 50% Some Insights of mine

Interpreted by:
John Tan Chong Min
Knowledge
Graphs and
LLMs
Retrieval Augmented Generation (RAG)
• <Context 1>
• <Context 2>
• <Context 3>

• <Query>
Problems with LLMs
• Base LLM may not be able to recall all knowledge it is trained on

• May not be able to retrieve the right context using RAG


• Embedding vector may not be comprehensive enough for specific use case

• Knowledge Graphs may be useful to retrieve the right context by


searching the right keywords and retrieving the right subgraphs
ChatGPT may output differently with different
languages

Adapted from: Keynote, Knowledge Graph Conference 2023, Denny Vrandečić


The Future of Knowledge Graphs in a World of Large Language Models
https://www.youtube.com/watch?v=WqYBx2gB6vA
What is a Knowledge Graph?
• Triplets:
• {Source, Destination, Relation}

• Typically a Directed Graph


Hierarchy in Knowledge Graphs

Cyc Knowledge Base.


Computational Analysis on a Corpus of Political Satire Articles: A Theoretical and Experimental Study.
Stingo et al. 2016.
Cyc Knowledge Graph – Common-sense reasoning across nodes

Can we leverage the power of LLMs to


improve upon this?

https://www.youtube.com/watch?v=8NRvhGm9EOg
Knowledge Graphs and Causal Relations
• The relation between Source and Destination can be a causal link

Tired Awake
Drink Coffee
Can Knowledge Graph be viewed as a tool/memory?
• Perhaps a more efficient way to get data than querying large corpuses of text
• Extracting relevant parts of a Knowledge Graph can serve as a way to retrieve context

• May embody causal relations easily

• Knowledge Graphs can grow dynamically, much like memory

Get Information

Knowledge
LLM Agent Graph Database
Informs
Knowledge Graphs and LLMs: 3 Approaches
Approach 1
KG-Augmented LLMs
Approach 1: KG-augmented LLMs
• KG as Text: Pass in Knowledge Graph (Generative Agents: Interactive
Simulacra of Human Behavior, Park et al, 2023) as text

• KG as Object: Treat KG as an object and pass it into the model


• Typically uses Graph Neural Networks to embody KG information
KG to retrieve text for LLMs

Or non-gradient methods (aka Memory)


Recursive prompting using JSON structure to ground actions

[Agent’s Summary Description]


Eddy Lin is currently in The Lin family’s house: Eddy Lin’s bedroom:
desk
Related areas in The Lin family's house: Mei and John Lin’s
bedroom, Eddy Lin’s bedroom, common room, kitchen, bathroom,
and garden.
Eddy Lin knows of the following areas: The Lin family’s house,
Johnson Park, Harvey Oak Supply Store, The Willows Market and
Pharmacy, Hobbs Cafe, The Rose and Crown Pub.
* Prefer to stay in the current area if the activity can be done there.
Eddy Lin is planning to take a short walk around his workspace.
Which area should Eddy Lin go to?

• Recursively use prompting to prompt for lower and lower sub-


areas based on the JSON tree
• Ground the agent in feasible actions via prompting
• JSON structure to ground agent's actions is like grounding with a
Knowledge Graph

Generative Agents: Interactive Simulacra of Human Behavior. Park et al. 2023.


ERNIE – Hybrid Input/Output
• LM and KG are both jointly used for processing
QA-GNN – Two-way interaction between LM and KG
• LLM and KG influences each other
Approach 2
LLM-Augmented KGs
Approach 2: LLM-augmented KG
• Use LLM to generate KG via zero-shot/few-shot prompting
• e.g. LangChain (not very good)

• LLMs as text encoders (embedding space) to enrich KG


representations
Use LLMs to generate Knowledge Graphs
• LLMs can be more versatile and can be few-shot prompted to
generate relations!

• Better than SpaCy for Named-Entity Recognition (NER) for out-of-


distribution datasets (LLMs are superb learners if prompted well)
LLM zero-shot/few-shot prompting for KG generation
You are a networked intelligence helping a human track knowledge triples about all
relevant people, things, concepts, etc. and integrating them with your knowledge
stored within your weights as well as that stored in a knowledge graph.

Extract all of the knowledge triples from the text.

A knowledge triple is a clause that contains a subject, a predicate, and an object.

The subject is the entity being described, the predicate is the property of the
subject that is being described, and the object is the valuae of the property.
LLM zero-shot/few-shot prompting for KG generation
EXAMPLE
It's a state in the US. It's also the number 1 producer of gold in the US.
Output: (Nevada, is a, state), (Nevada, is in, US), (Nevada, is the number 1 producer of, gold)
END OF EXAMPLE

EXAMPLE
I'm going to the store.
Output: NONE
END OF EXAMPLE

EXAMPLE
Oh huh. I know Descartes likes to drive antique scooters and play the mandolin.
Output: (Descartes, likes to drive, antique scooters),(Descartes, plays, mandolin)
END OF EXAMPLE

EXAMPLE
{text}
Output:
LLMs as text encoders for Knowledge Graph Embeddings
• KG built based on embedding space
Approach 3
LLMs and KG two-way interaction
Approach 3: Synergy between Knowledge Graphs and LLMs
Knowledge Graphs for Fact-Checking!
• We can perhaps use knowledge graphs to perform fact-checking
• Here it is done as pre-training, but we can/should also use KGs during inference

FACTKB. Feng et. al. 2023.


LangChain Graph QA Example
LangChain Graph Creation and Utilisation Process
• Step 1: Generate triplets from context

• Step 2: Entity Extraction from query

• Step 3: Use entities to extract relevant triplets

• Step 4: Using triplets to answer question

https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/indexes/graph.py
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/chains/graph_qa/base.py
Step 1: Generate Triplets from Context
Generated Knowledge Graph
Steps 2-4: Extract and use relevant KG for query
Compare to just feeding in context directly
How I would do it
Strict JSON Framework
Step 1: Generate Knowledge Graph from context
Generated Graph
Compare with LangChain

LangChain Custom StrictJSON Framework


Step 2: Flexible Knowledge Graph parsing
Parsed Knowledge Graph
Step 3: Use Parsed Knowledge Graph to Answer Question
Questions to Ponder
• What are the failure modes of using KG for context representation for
LLMs?

• Should we utilise embedding space too for the Knowledge Graph, or just
for the LLM?

• How can LLMs help with a more flexible interpretation/construction of a


Knowledge Graph?

• How will we know what nodes are important to construct in a Knowledge


Graph? Is it entirely learnable, or are there fixed biases?

You might also like