2263 - Data Science and Artificial Intelligence

Data Science and Artificial

Intelligence – Lecture 6

Axel Polleres, Johann Mitlöhner

Institute for Data , Process and Knowledge Management

 Limits of second-Wave AI and Deep Learning:
• Data-Hungry:
• training neural networks needs a lot of correctly labelled training data.
• Models are not "explainable"
• Cannot "apply" abstract knowledge, e.g. to new, unseen examples, cannot “compute”
• Often not robust to changed contexts
• Cannot learn rules and constraints reliably (nor apply them)
• Transfer-Learning hits boundaries:
• fine-tuning still, often needs considerable amounts of labeled data
• not always possible/feasible in very specific domains
Limits of first-Wave AI and Logics:
• Lacking the capability to learn

• Focused (mostly )on discrete relationships

• Scalability/Combinatorial explosion (which sometimes can’t be

avoided)
What more did you learn on
(1) limits?
(2) approaches to address these limits?
 Limits of second-Wave AI and Deep Learning:
• Example –Cultural Heritage trying to apply pre-trained contemporary
image models to historic images:

Laptop 97%

Tabea Tietz, Jörg Waitelonis, Mehwish Alam and Harald Sack. Knowledge Graph based Analysis and Exploration of Historical
Theatre Photographs. Proceedings of the Conference on Digital Curation Technologies (Qurator 2020), Berlin, January 20-21,
2020. https://publikationen.bibliothek.kit.edu/1000105566
 Limits of second-Wave AI and Deep Learning:
Don't trust "end-to-end" Deep-
Learning solutions.
Hybrid AI models are the way
forward.

https://www.spiegel.de/plus/kuenstliche-intelligenz-in-der-krise-zu-dumm-zum-rechnen-a-00000000-0002-0001-0000-000166735218

https://www.nature.com/articles/d41586-019-03013-
5?utm_source=twt_nnc&utm_medium=social&utm_campaign=naturenews&sf221156507=1
 Truthfulness/Understanding:
• Not solvable without contextual+conceptual background knowledge!
 Trends: Transfer Learning/Fine-tuning:

• Idea: fine-tune existing,

pre-trained models (e.g.
large text corpora from
news, image recognition
models)
• to specific, related contexts
(e.g. classify trucks instead
of cars)

• Many Pre-trained models

available, for both images
and text:
• https://huggingface.co/
Trends: Zero-shot learning (wikipedia):
• “Unlike standard generalization in machine learning, where classifiers are
expected to correctly classify new samples to classes they have already
observed during training, in ZSL, no samples from the classes have been given
during training the classifier. It can therefore be viewed as an extreme case of
domain adaptation. “
• […]
• builds on the ability to "understand the labels"—represent the labels in the
same semantic space as that of the documents to be classified.
• Challenge: How to “detect”?“ new classes…
• Based on either attiditonal feature background knowledge:
• structured attribute combinations (e.g. “a Zebra is a Horse with stripes”)
• or (less common) in combination with unsupervised learning:
• E.g. https://arxiv.org/ftp/arxiv/papers/2001/2001.05624.pdf
“Third-Wave AI” or “Hybrid AI”
Setting priorities:
AAAI & Catalyst report 2019

https://cra.org/ccc/ai-roadmap-integrated-intelligence/
Typical applications nowadays need/use both:
Gameplaying:
Symbolic (1st wave): Subsymbolic (2nd wave):

12
Typical application need/use both:
Autonomous driving:
Symbolic: Subsymbolic:
Road network+ Routing algorithms:

Traffic Rules:

13
 Typical application need/use both:
Converstational agents/Chatbots:
Symbolic: Intent extraction:
Subsymbolic:
Response Policy Rules: Speech Recognition:

https://rasa.com/products/rasa-stack/

https://rasa.com/products/rasa-stack/
https://machinelearning.apple.com/2017/10/01/hey-siri.html

14
Recall: Good morning!
connecting to my previous lecture ;-)

https://twitter.com/nixcraft/status/1595472418084356096/
Also language models open up a whole lot of new
possibilities for hybrid combinations!
Not limited to natural language,
But also usable for
• Programming
• Database queries

For instance:

https://github.com/features/copilot

https://w.wiki/6Enj
Four pillars of today’s AI systems
(to be taken with a grain of salt):
(Old but still true: http://web.archive.org/web/20181018045806/https://www.reasoning.world/ai-is-about-machine-reasoning-or-when-
machine-learning-is-just-a-fancy-plugin/ )

• Learning: First, a system has to be taught. This can be done by single experts or a community is used where people teach the machine
bits of knowledge. This is what the machine uses to be able to learn on its own. You might think this way it doesn’t learn on its own,
but it does. Consider how a child learns. It learns by being taught by his parents, teacher, other children or anyone else teaching things
and it just copies and pastes everything with its “sensors” like ears and eyes. Thus, the AI learns best practices and reasoning from
experts. Knowledge is taught in atomic pieces of information that represent individual steps of a process.
• Semantic Graph: The taught knowledge has to be stored, which is done within a data store. The store is used to supply information for
the understanding of the world doing semantic reasoning. Like: I know that my mom is connected to dad. And I am connected to my
sister. And my sister is connected to her work colleagues. And she works in this city in that building. This is a semantic map of the
world that we know. That is part of our memory – a semantic graph. By creating a semantic data map, the AI understands the world in
which it operates.
• Process Engine: The engine is the central back-end service that puts everything together and thus delivers a solution to a certain
problem. The engine knows the map of the world where a system is acting in. In doing so, the engine takes everything it knows and
finds the correct solution to a specific problem on its own, step by step based on the knowledge it has.
• Problem Solving: Problem solving also known as machine reasoning (MR) is the ability to dynamically react to change and by doing
this, reusing existing knowledge for new and unknown problems. With machine reasoning, problems are solved in ambiguous and
changing environments. The AI dynamically reacts to the ever-changing context, selecting the best course of action. Thus, machine
reasoning is the basis for a general artificial intelligence (General AI).

Personal Opinion: I strongly consider the

last statement exaggerated for now!
17
Four pillars of today’s AI systems
(to be taken with a grain of salt):
https://www.reasoning.world/ai-is-about-machine-reasoning-or-when-machine-learning-is-just-a-fancy-plugin/

• Learning: First, a system has to be taught. This can be done by single experts or a community is used where people teach the machine bits of
knowledge. This is what the machine uses to be able to learn on its own. You might think this way it doesn’t learn on its own, but it does.
Consider how a child learns. It learns by being taught by his parents, teacher, other children or anyone else teaching things and it just copies
and pastes everything with its “sensors” like ears and eyes. Thus, the AI learns best practices and reasoning from experts. Knowledge is
taught in atomic pieces of information that represent individual steps of a process.
• Semantic Graph: The taught knowledge has to be stored, which is done within a data store. The store is used to supply information for the
understanding of the world doing semantic reasoning. Like: I know that my mom is connected to dad. And I am connected to my sister. And
my sister is connected to her work colleagues. And she works in this city in that building. This is a semantic map of the world that we know.
That is part of our memory – a semantic graph. By creating a semantic data map, the AI understands the world in which it operates.
• Process Engine: The engine is the central back-end service that puts everything together and thus delivers a solution to a certain problem.
The engine knows the map of the world where a system is acting in. In doing so, the engine takes everything it knows and finds the correct
solution to a specific problem on its own, step by step based on the knowledge it has.
We have not talked about this yet!
• Problem Solving: Problem solving also known as machine reasoning (MR) is the ability to dynamically react to change and by doing this,
reusing existing knowledge for new and unknown problems. Let’s
With get back
machine to those
reasoning, two
problems areguys!
solved in ambiguous and changing
environments. The AI dynamically reacts to the ever-changing context, selecting the best course of action. Thus, machine reasoning is the
basis for a general artificial intelligence (General AI).
What is a Knowledge Graph?

… good question!

Says more what a KG does

than what it is…
“interesting things and [understanding their]
relationships”
19
 Bing: Knowledge Graph

● over a billion entities

● 21 billion associated facts
● 18 billion links to key actions
● 5 billion relationships between entities

Bing Knowledge Graph Facts

20
More Knowledge Graphs
Other companies with knowledge graphs:
● Facebook Entity Graph
● LinkedIn
● How NASA Finds Critical Data through a Knowledge
Graph [Neo4j Blog]
● Amazon Product graph
● Yandex, Baidu
Some free open knowledge graphs:
●Dbpedia
●WikiData
●Linked Open Data project
21
Knowledge Graphs are Big Data:

https://twitter.com/rminHaller/status/1380309606618963969
What is a Knowledge Graph?

By properties:
McCusker, Chastain, Erickson, and McGuinness. What is a
Knowledge Graph? (unpublished, 2016).
identify some principles: p.2+3
“principled” aggregation of Linked Data? p.7

Rospocher, van Erp, Vossen, Fokkens, Aldabe, Rigau, Soroa,

Ploeger, Bogaard. Building event-centric knowledge graphs
from news. JWS (2016)

“knowledge-base of facts about entities typically

[Remark: often automatically] obtained from
structured repositories [such as Freebase]”

23
Applications of Knowledge graphs in AI:

Why are Knowledge Graphs important for hybrid (third wave) AI?
Improve AI (search/perceiving) by “knowing how things are connected”.
Enabling AI (reasoning) by “knowing how things are connected”.
Explain AI (learning/abstrations) by “knowing how things are connected”.

Goal: Knowledge Graphs could be the

“glue” to connect

24
AI(?) Applications of Knowledge Graphs:
Google – Search:
- Rich Snippets
- Personalised recommendations across services:

25
 AI(?) Applications of Knowledge Graphs:
IBM Watson - Reasoning :
• Used DBpedia as one of its fact bases! (predates the #LLM hype!)

https://youtu.be/P0Obm0DBvwI?t=951

26
AI(?) Technologies behind Knowledge Graphs:

• (Database-like) query languages…

• Cf. DMA lecture, we already learned SPARQL there as one such

language.

• … in gerenal, KGs are one of the “drivers” of the recent “revival” of

Graph Databases!

27
 KG enrichment by Rule-based Logical Reasoning:
RDFS (RDF Schema) and OWL: Two More standards
extending RDF to encode
• schema information
• taxonomic relationships
• and implicit triples

Basic logical Reasoning techniques:

• Materialisation (can be done by rules/queries) [2]
• Rewriting [1]
1. Stefan Bischof, Markus Krötzsch, Axel Polleres, and Sebastian Rudolph. Schema-agnostic query rewriting in SPARQL 1.1. In
Proceedings of the 13th International Semantic Web Conference (ISWC 2014), Lecture Notes in Computer Science (LNCS).
Springer, October 2014. [ .pdf ]
2. Axel Polleres, Aidan Hogan, Renaud Delbru, and Jürgen Umbrich. RDFS & OWL reasoning for linked data. In Reasoning Web
2013, volume 8067 of LNCS, pages 91--149. Springer, Mannheim, Germany, July 2013. [ .pdf ]
 Reasoning by Querying – Materialisation:
SELECT ?X WHERE
{
?X a dbo:Scientist ; dbo:birthPlace dbr:Bologna . No answer L
} … since implicit edges are missing!

instance data:
dbr:Marta_Grandi a dbo:Entomologist ;
dbo:birthPlace dbr:Bologna . Bologna
Marta_Grandi birthplace
dbr:Costanzo_Varolio a dbo:Medician;
dbo:birthPlace dbr:Bologna . a
Agent
a
⊑ ⊑
a
Ontology (schema data): a Person Organisation
dbo:Entomologist rdfs:subClassOf dbo:Scientist.
⊑
dbo:Medician rdfs:subClassOf dbo:Scientist.
dbo:Scientist rdfs:subClassOf dbo:Person. Scientist
dbo:Person rdfs:subClassOf dbo:Agent. ⊑ ⊑
dbo:Organisation rdfs:subClassOf dbo:Agent.
dbo:birthPlace rdfs:domain dbo:Person . Entomologist Medician
29
dbo:Organisation owl:disjointWith dbo:Place.
...
 RDFS deduction rules:
cf. https://www.w3.org/TR/rdf11-mt/

Could be read as Datalog deduction rules, e.g.:

triple(U,rdfs:subClassOf,S) :- triple(U,rdfs:subClassOf,V) , triple(V,rdfs:subClassOf,S) .

triple(V,rdfs:type,S) :- triple(U,rdfs:subClassOf,S) , triple(V,rdf:type,U) .
30
 RDFS deduction rules:
cf. https://www.w3.org/TR/rdf11-mt/

Could be read as Datalog deduction rules, e.g.:

triple(U,rdfs:subClassOf,S) :- triple(U,rdfs:subClassOf,V), triple(V,rdfs:subClassOf,S).

triple(V,rdfs:type,S) :- triple(U,rdfs:subClassOf,S), triple(V,rdf:type,U).
 RDFS deduction rules:
cf. https://www.w3.org/TR/rdf11-mt/

… and Datalog deduction rules could be written as SPARQL Construct statements:

CONSTRUCT {?U rdfs:subClassOf ?S} WHERE { ?U rdfs:subClassOf ?V. ?V rdfs:subClassOf ?S }

CONSTRUCT {?V rdfs:type ?S} WHERE { ?U rdfs:subClassOf ?S. ?V rdf:type ?U }
 Reasoning by Querying – Materialisation:
SELECT ?X WHERE Applying the rules of the previous
{ slides exhaustively (until a fixpoint),
?X a dbo:Scientist .
?X dbo:birthPlace dbr:Bologna . will yield additional implicit KG edges
} (i.e., RDF triples):

instance data:
dbr:Marta_Grandi a dbo:Entomologist ; dbr:Marta_Grandi a dbo:Scientist,
dbo:birthPlace dbr:Bologna . dbo:Person, dbo:Agent.

dbr:Costanzo_Varolio a dbo:Medician; dbr:Costanzo_Varolio a

dbo:birthPlace dbr:Bologna . dbo:Scientist, dbo:Person, dbo:Agent.

Ontology (schema data):

dbo:Entomologist rdfs:subClassOf dbo:Scientist. dbo:Entomologist rdfs:subClassOf
dbo:Medician rdfs:subClassOf dbo:Scientist. dbo:Person, dbo:Agent.
dbo:Scientist rdfs:subClassOf dbo:Person. dbo:Medician rdfs:subClassOf
dbo:Person rdfs:subClassOf dbo:Agent. dbo:Person, dbo:Agent.
dbo:Organisation rdfs:subClassOf dbo:Agent. dbo:Sienticst rdfs:subClassOf
dbo:birthPlace rdfs:domain dbo:Person . dbo:Agent.
33
dbo:Organisation
... owl:disjointWith dbo:Place.
 Reasoning by Querying – Query Rewriting:
SELECT ?X WHERE
{
{ {?X a dbo:Scientist } UNION {?X a dbo:Medician } UNION {?X a dbo:Entomologist } }
?X dbo:birthPlace dbr:Bologna .
}

instance data:
dbr:Marta_Grandi a dbo:Entomologist ; Alternatively, the rules can be
dbo:birthPlace dbr:Bologna . used “backwards” to rewrite the
original query to yield a more
dbr:Costanzo_Varolio a dbo:Medician; generic query!
dbo:birthPlace dbr:Bologna .

Ontology (schema data):

dbo:Entomologist rdfs:subClassOf dbo:Scientist.
dbo:Medician rdfs:subClassOf dbo:Scientist.
dbo:Scientist rdfs:subClassOf dbo:Person.
dbo:Person rdfs:subClassOf dbo:Agent.
dbo:Organisation rdfs:subClassOf dbo:Agent.
dbo:birthPlace rdfs:domain dbo:Person .
34
dbo:Organisation owl:disjointWith dbo:Place.
...
 Reasoning by Querying – Query Rewriting:
SELECT ?X WHERE
{
{ {?X a/subclassOf* dbo:Scientist}
?X dbo:birthPlace dbr:Bologna .
}

instance data:
dbr:Marta_Grandi a dbo:Entomologist ; Alternatively, the rules can be
dbo:birthPlace dbr:Bologna . used “backwards” to rewrite the
original query to yield a more
dbr:Costanzo_Varolio a dbo:Medician; generic query!
dbo:birthPlace dbr:Bologna .
You can also use SPARQL path
expressions in this query
Ontology (schema data):
rewriting! [1]
dbo:Entomologist rdfs:subClassOf dbo:Scientist.
dbo:Medician rdfs:subClassOf dbo:Scientist.
dbo:Scientist rdfs:subClassOf dbo:Person.
dbo:Person rdfs:subClassOf dbo:Agent.
1. Stefan Bischof, Markus Krötzsch, Axel Polleres, and Sebastian Rudolph. Schema-
dbo:Organisation rdfs:subClassOf dbo:Agent.
agnostic query rewriting in SPARQL 1.1. In Proceedings of the 13th International
dbo:birthPlace rdfs:domain dbo:Person .
35 Semantic Web Conference (ISWC 2014), Lecture Notes in Computer Science
(LNCS). Springer, October 2014. [ .pdf ]
dbo:Organisation owl:disjointWith dbo:Place.
...
 Existing KGs aren’t (logically) consistent L [1]
• E.g.
Dbpedia Ontology:

dbo:Agent owl:disjointWith dbo:Place.

You can think of this as a Constraint:

:- triple(X, "rdf:type, "dbo:Agent"),
triple(X, "rdf:type", "dbo:place").

dbo:Country rdfs:subClassOf dbo:Place.

dbo:Organisation rdfs:subClassOf dbo:Agent.

1. Stefan Bischof, Markus Krötzsch, Axel Polleres, and Sebastian Rudolph. Schema-agnostic query rewriting in
36 SPARQL 1.1. In Proceedings of the 13th International Semantic Web Conference (ISWC 2014), Lecture Notes in
Computer Science (LNCS). Springer, October 2014. [ .pdf ]
 Reasoning by Querying – Consistency checking:
SELECT
ASK {?X ?C1 ?C2}
WHERE { ?X a/subClassOf* ?C1;
a/subClassOf* ?C2.
?C1 owl:disjointWith ?C2. }

instance data:
dbr:European_Union a dbo:Country. Similarly, RDFS and OWL
inconsistency checking also can
dbr:European_Union a dbo:Organisation. be done by querying! [1]
(simplified)

Ontology (schema data):

dbo:Entomologist rdfs:subClassOf dbo:Scientist.

dbo:Medician rdfs:subClassOf dbo:Scientist.
dbo:Scientist rdfs:subClassOf dbo:Person.
dbo:Person rdfs:subClassOf dbo:Agent.
dbo:Organisation rdfs:subClassOf dbo:Agent.
dbo:birthPlace rdfs:domain dbo:Person .
dbo:Country rdfs:subClassOf dbo:Place.
37 dbo:Organisation owl:disjointWith dbo:Place.
 Often, you also need to deal with
contextualized information
• E.g. from
Doesn’t work
(since the data is
contextualized and
doesn't match the
pattern)

„Cities in the Italy with more than 1M population“:

Structured queries (SPARQL):

http://yasgui.org/short/UVOyhX8ft
https://en.wikipedia.org/wiki/Rome http://dbpedia.org/resource/Rome
PREFIX : <http://dbpedia.org/resource/>
PREFIX dbo: <http://dbpedia.org/ontology/>
Automatic PREFIX yago: <http://dbpedia.org/class/yago/>
Exctractors
SELECT DISTINCT ?city ?pop WHERE {
?city a yago:City108524735 .
?city dbo:country :Italy.
?city dbo:populationTotal ?pop

FILTER ( ?pop > 1000000 )

}

38
 Contextualised Information is better modeled
in another Open Knowledge Graph: Wikidata
• Wikidata can also be queried as RDF with SPARQL!

39
 Wikidata as RDF … can be queried by SPARQL
• “Simple” surface query:

SELECT DISTINCT ?city WHERE {

?city wdt:P31/wdt:P279* wd:Q515.
?city wdt:P1082 ?population .
?city wdt:P17 wd:Q38 .
FILTER (?population > 1000000) }

• What’s this?

40 SEITE 40
 Wikidata as RDF … can be queried by SPARQL
• However, Wikidata has more complex info:
(temporal context, provenance,…)
• Rome:
What do we learn?
• https://www.wikidata.org/wiki/Q220 • Data and meta-data
(context/ provenance)
… Can I query since when which Italian cities at the same level à
have more than 1M population with SPARQL? Yes! one RDF graph, mixing
reification and plain
data, cf. [Hernandez et
al. 2015]
• Quite some
Knowledge about the
ontology required!

SEITE 41
 Reification/Property Graphs:
• How to (best) describe contextual statements (aka meta-modeling) about triples in RDF is a bit open…

• One could use different Graph data models/Graph databases:

• Labeled Directed graphs (plain RDF) - supported by RDF triple stores:

Entity1 Entity1
predicate

• Property graphs – supported by other graph databases (e.g. Neo4J, BlazeGraph, etc.)
type: capitalOf
Name: Rome added: 2019-06-12 Name: Italy
Type: City author: @Axel Type: Country
added: 2019-06-11 added: 2019-06-11
42 author: @Sebastian author: @Sebastian
 Reification/Property Graphs:
• How to (best) describe contextual statements (aka meta-modeling) about triples in RDF is a bit open
• no real prevalent standard… the way Wikidata models it (see above) is just one option.
• various other options and proposals, inter-translatable but affect performance of querying:

• Example: "Rome is Italy's capital since 1861."

:Rome :capitalOf :Italy. [1861, [

Challenge:
RDF reification: [ a rdf:Statement;
How to
rdf:subject :Rome;
rdf:predicate :capitalOf; reformulate
rdf:object :Italy ] :yearBegins 1861 . the inference
rules or
“Named Graphs” :G1 {:Rome :capitalOf :Italy. } rewritings
:G1 :yearBegins 1861 . from slide 58?

“Singleton” properties: :Rome :p1 :Italy.

:p1 :subPropertyOf :capitalOf;
43 :yearBegins 1861 .
 Technology behind Knowledge Graphs:
• Storing & Querying Knowledge Graphs:
• Enriching/Verifying Knowledge Graphs via AI:
• Rule-based Reasoning
• Rule-Learning
• Link prediction via Graph Embeddings
• Advanced AI applications that leverage KGs

44
 Rule-Learning/Mining over Knowledge Graphs
• AMIE (Association Rule Mining under Incomplete Evidence)
• extracts supported and confident logical rules from a Knowledge Graph, such as:

?a rdf:type ?b :- ?e <http://xmlns.com/foaf/0.1/knows> ?a , ?e rdf:type ?b .

• uses sampling of "counterexamples", as acccording to a partial completeness assumption:

• If we have the triple (s p o) in the KG K and func(p) ≥ func(p-), then AMIE assumes that all (s p o') for o' !=
o do not hold in the real world
• This can be used to sample counter examples e.g.
• positive example:
<http://polleres.net#me> <http://xmlns.com/foaf/0.1/name> "Axel".
• generated negative example:
<http://polleres.net#me> <http://xmlns.com/foaf/0.1/name> "Hans".
• Both positive examples and sampled counter-examples are used for the training and rule mining.

https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/amie/
 Rule-Learning/Mining over Knowledge Graphs
• AMIE (Association Rule Mining under Incomplete Evidence)
• AMIE sample session:

?e <sameAs> ?a ?e <sameAs> ?b => ?a <sameAs> ?b

?e <knows> ?a ?e rdf:type ?b => ?a rdf:type ?b
?f <sameAs> ?b ?a <sameAs> ?f => ?a <sameAs> ?b
?a <sameAs> ?f ?f <knows> ?b => ?a <knows> ?b

https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/amie/
Code: https://github.com/lajus/amie/
 Technology behind Knowledge Graphs:
• Storing & Querying Knowledge Graphs:
• Enriching/Verifying Knowledge Graphs via AI:
• Rule-based Reasoning
• Rule-Learning
• Link prediction via Graph Embeddings
• Advanced AI applications that leverage KGs

47
 Embeddings, recall word embeddings:
• encoding words to vectors:
• frequency based (e.g. TF/IDF)
• context-based
• co-occurrence based
• prediction-based
• Encode/train NN to predict context-word(s) from word(s) n-grams
(e.g. training with sentences with missing words, or predicting next
words, essentially what LLMs do!)

• Applications:
• Document Classification
• Search (ranking)
• Machine translation
• Conversational AI
• Question answering
 Similar idea: Graph embeddings
• General idea:
• embed nodes and relations in a KG in a vectorspace…
• learn embeddings that should filfill certain properties using neural networks, e,g,
TransE [*] embedding:
Example:
Maria
married_to
Theresia
Franz Stephan von Lothringen
Elizabeth II
Other embeddings proposed Prince Philipp, Duke of Edinburgh

since: TransH (2014), TransR

(2015), RDF2Vev (2016)

• … where this vector embedding can be used again for other AI (particularly ML tasks.
• e.g. Link prediction, Node Classification (i.e. enriching a KG by missing links)
• item recommendation cf. https://www.slideshare.net/EnricoPalumbo2/an-empirical-comparison-
of-knowledge-graph-embeddings-for-item-recommendation

[*] Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, Oksana Yakhnenko:
Translating Embeddings for Modeling Multi-relational Data. NIPS 2013: 2787-2795
 Technology behind Knowledge Graphs:
• Storing & Querying Knowledge Graphs:
• Enriching/Verifying Knowledge Graphs via AI
• Advanced AI applications that leverage KGs
• Question Answering
• Semantic Search

50
 Some of our own research in this space:
● HDT - Lightweight/portable compressed graph/triple store.
● Extending SPARQL with “real” path queries (prototype)
● Connecting Knowledge Graphs to text and conversations
● Understanding and Searching Open Data by linking it to a Knowledge Graph

Svitlana Vakulenko, Maarten de Rijke, Michael Cochez, Vadim Savenkov, and Axel Polleres. Measuring semantic coherence of a conversation. In Proceedings of the 17th International
Semantic Web Conference (ISWC 2018), volume 11136 of Lecture Notes in Computer Science (LNCS), pages 634--651, Monterey, CA, October 2018. Springer. [ DOI | .pdf ]

Svitlana Vakulenko, Javier Fernández, Axel Polleres, Maarten de Rijke, and Michael Cochez. Message passing for complex question answering over knowledge graphs. In 28th ACM
International Conference on Information and Knowledge Management (CIKM2019, Beijing, China, November 2019. [ http ]

Sebastian Neumaier and Axel Polleres. Enabling spatio-temporal search in open data. Journal of Web Semantics (JWS), pre-print available at http://epub.wu.ac.at/6758/, 2019. to
appear (accepted for publication).

Javier D. Fernández, Miguel A. Martinez-Prieto, Claudio Gutiérrez, Axel Polleres, and Mario Arias. Binary RDF Representation for Publication and Exchange (HDT). Journal of Web
Semantics (JWS), 19(2), 2013. [ http ]

Vadim Savenkov, Qaiser Mehmood, Jürgen Umbrich, and Axel Polleres. Counting to k, or how SPARQL 1.1 could be efficiently enhanced with top k shortest path queries. In 13th
International Conference on Semantic Systems (SEMANTiCS), pages 97--103, Amsterdam, the Netherlands, September 2017. ACM. [ .pdf ]

51
Tackling hard NLP problems with Knowledge Graphs:

Svitlana Vakulenko, Maarten de Rijke, Michael Cochez, Vadim Savenkov, and Axel Polleres. Measuring semantic coherence
of a conversation. In Proceedings of the 17th International Semantic Web Conference (ISWC 2018), volume 11136 of
Lecture Notes in Computer Science (LNCS), pages 634--651, Monterey, CA, October 2018. Springer. [ DOI | .pdf ]

Idea: Use Graph

embeddings to a vector
space and/or shortest path
connections in a graph.
 Tackling hard NLP problems with Knowledge Graphs:

Idea: use unsupervised

message passing to
propagate confidence
scores obtained by
parsing an input
question and matching
terms in the
knowledge graph to a
set of possible
answers.
S.Vakulenko, J.Fernández, A. Polleres, M. de Rijke, and M.
Cochez. Message passing for complex question answering
over knowledge graphs. CIKM2019,
Admittedly, that one is easy for LLMs:
https://chat.openai.com/share/0a263a93-c978-4b1f-acd0-44a7bbaf3c51
Ongoing PhDs and Master theses:

PhD Nicolas Ferranti: Knowledg Graph Consolidation in Wikidata

cf. first results: https://www.semantic-web-journal.net/content/formalizing-and-

validating-wikidatas-property-constraints-using-shacl-and-sparql-0

MSc Gerhard Klager

Cf. first results published in a worksshop
https://ceur-ws.org/Vol-3447/Text2KG_Paper_11.pdf

Further MSc topics, cf. our list in Canvas ;-)

Time for our hands-on/hackathon exercise!

Challenge: Question Answering … how to combine LLMs and KGs?

• direct Question answering vs. leverageing a KG and SPARQL

• Possible substeps:
1. Question analysis? à a SPARQL pattern template
2. Named entity recognition à as set of “entities” and “relationships” mentioned in the question
3. Matching Entity labels à finding the identifiers matching the entities from step 2 in the KG
4. Query construction and execution à executable SPARQL Query
5. Answer formulation à Translating the answer back to Natural Language
Disclaimer:
1) you don’t need an OpenAI API key… Alternatives:
• use an Open Source Language model, e.g. via Colab:
• https://colab.research.google.com/github/oobabooga/text-generation-webui/blob/main/Colab-TextGen-GPU.ipynb
• Simply use the non-paid GPT interface

2) Be sure to NOT feed any sensitive (personal) or copyrighted data into OpenAI!
Recall:

Not limited to natural language,

But also usable for
• Programming
• Database queries

https://w.wiki/6Enj

• Interestingly, that didn’t work so well,

when I retried this morgning:

https://chat.openai.com/share/0a263a93-
c978-4b1f-acd0-44a7bbaf3c51
Appendix: More of our own (ongoing anf older) reseach using Knowledge Graphs
CiyData Pipeline:
Enriching and completing Open City Data with a combination of using Equational Knowledge and Machine Learning

PhD Stefan Bischof:

Thesis:
http://polleres.net/supervised_theses/Stefan_Bischof_Dissertation_2017.pdf
 Indicators

Problem: different
sources conatin
different indicators
(area, population,
popultiondensity,
economic factors, for
different years.

Idea: using both first-

wave and second wave
AI (ML&statistics)
methods
Step 1: Storing Data from different sources in
RDF:
• Data from some sources like eurostat come as multidimensional data -
Data Cube vocabulary (QB):
• Temporal (December)
• Unit of measurement (degrees Celsius)
• Aggregation (mean, min, max, …)
• Indicator (temperature, population density)

city Vienna
year 2016
indicator
source value
error population
1 852 997
eurostat 0.0

63
Use equational knowledge for missing
computed/computable values:
population area
indicator indicator
city Vienna city

year year
value 2016
error error
value
city
1 852 997 year 0.0
0.0
414.650
population
populationdensity =
area
indicator
derived From error value
population density
population 4 467
populationdensity Ü 0.0
area

64
equations: deal with
approximated values by using
error propagation
population area
indicator indicator
city Vienna city

year year source

value 2016 error

error
value KNN
city
1 852 997 year ϵ prediction
0.0
414.650
population
populationdensity =
area
indicator
derived From error value
population density
𝑒𝑞! population
populationdensity Ü
area
?
𝑝𝑟𝑜𝑝𝑎𝑔𝑎𝑡𝑒(0.0, 𝜖, 𝑒𝑞! ) 4 467

65
 More Details:
Stefan Bischof, Andreas Harth, Benedikt Kämpgen, Axel Polleres, and PatrikSchneider.Enriching integrated statistical open city data by
combining equational knowledge and missing value imputation. Journal of Web Semantics (JWS), October 2017.In press.

Main idea:
Combine
(1) ontological reasoning, (2)ML, (3)
equations “iteratively” with
QB equations
Evaluation Combination
PCA Regression + QB Equations
• Statistics one iteration (PCA regression + QB Equations)
• 991k observations from crawled data
• 522k new or better observations from PCA regression
• 230k better observations from QB Equations
• 232k new observations from QB Equations
• Same or better values (improved RMSE) for 80 of 82 indicators
• QB Equations are sensitive to correct error estimates

• More details: http://www.stefanbischof.at/slides/Rigorosum_Bischof.pdf

67
 City Data Pipeline
Prototype
citydata.wu.ac.at
• Search for indicators & cities
• obtain results incl. sources
• Integrated data served as Linked Open Data
• Predicted values AND estimated error rates for
missing data...

...it‘s not finished, but:

assumption: Predictions get better, the more Open
data we integrate...
CiyData Pipeline:
Enriching and completing Open City Data with a combination of using Equational Knowledge and Machine Learning

PhD Sebastian Neumaier:

Thesis:
http://polleres.net/supervised_theses/Sebastian_Neumaier_PhD_2019.pdf
 Tackling hard research problems with Knowledge Graphs:

Why is Search in Open Data a problem?

https://www.youtube.com/watch?v=kCAymmbYIvc

Structured Data in Web Search by Alon Halevy

vs.

Open Data Search is hard...

a) No natural language „cues“ like in Web tables...
b) Existing knowledge graphs don‘t cover the domain of "Open
Data“ well
c) Open Data is not properly geo-referenced

70
 Our research:
Knowledge Graphs for Natural Data Search &
Integration!

• 2 approaches how knowledge graphs could help to solve the Open Data search problem:
1. Hierarchical labelling of Labeling of numeric data
2. Hierarchical labelling of Spatio-Temporal entities

71
 Example Table

federal state district year sex population

Upper Austria Linz 2013 male 98157

Upper Austria Steyr 2013 male 18763

Upper Austria Wels 2013 male 29730

… … … … …

72
 Open Data CSVs look more like this

NUTS2 LAU2_NAME YEAR SEX P_TOTAL

AT31 Linz 2013 1 98157

AT31 Steyr 2013 1 18763

AT31 Wels 2013 1 29730

… … … …

73 Source: https://www.data.gv.at/katalog/dataset/e108dcc3-1304-4076-8619-f2185c37ef81
 Why not use the numeric values?
• Identifying the most likely semantic label for a bag of
numerical values
• Deliberately ignore surroundings

NUTS2 LAU2_NAME YEAR SEX P_TOTAL

AT31 Linz 2013 1 98157

AT31 Steyr 2013 1 18763

AT31 Wels 2013 1 29730

… … … …

74
 Why not use numeric values?
• Identifying the most likely semantic label for a bag of
numerical values
• Deliberately ignore surroundings

population (a district) (country Austria)

98157

18763

29730

75
 Background Knowledge Graph
• Cities
• Population
• Area
• Country
• Location (Coordinates)
• Economic indicators
• …
• Organisations:
• Revenues
• Board members
• …
• Persons (e.g. celebrities, sports)
• Name
• Profession
What’s in there? • Height
• Landmarks (e.g. famous buildings)
• Country
• Location
• Height
• Events
• Dates
• Location

76
 Background Knowledge Graph
• Find properties with
numerical range
• Hierarchical clustering approach

• Two hierarchical layers:

• Type hierarchy
(using OWL classes)

• Property-object hierarchy
(shared property-object pairs)

77
 Label based on Nearest Neighbors

2
4
3
6 5
1

78
 Example OD Labelling
populationTotal (a Settlement)
populationDensity (a City)

79

79 Source: http://data.wu.ac.at/iswc2016_numlabels/submission/col14.html
 Lessons learned
• We can assign fine-grained semantic labels
• If there is enough evidence in Background Knowledge Graph
• However: Missing domain knowledge for labelling OD

Future work:
• Complementary to existing approaches (column header labeling, entity linking and relation extraction)
• Combined approaches may improve results
• Focusing on core dimensions of specific domains e.g. city data, maye more promising than “general”
value labeling.

80
 What else can we do/use?
Focus on specific dimensions:
• Particularly temporal and geospatial queries require better support [2]

NUTS2 LAU2_NAME YEAR SEX AGE_TOTAL

AT31 Linz 2013 1 98157

AT31 Steyr 2013 1 18763

AT31 Wels 2013 1 29730

… … … …

[2] Emilia Kacprzak, et al.: A Query Log Analysis of Dataset Search. International Conference on Web Engineering (2017)

81
 Available Geospatial Knowledge Bases

82
82
 Geo-Knowledge Graph Construction
European Classification
Wikidata,
of Territorial Units
GeoNames

Wikidata links

Mapping OSM entities Wikidata links

to GeoNames regions

Extracting OSM
streets and places
83
Available Temporal Knowledge
 Temporal Knowledge Graph Construction

} • Named events and their

labels
• Links to parent
periods/events
• Links to the spatial
coverage

}
• Temporal extent, i.e. a
single beginning and end
date

85
85
 Dataset Labelling
Metadata descriptions
• Geo-entities in titles, descriptions,
organizations
• Restricted to „origin“ country of the
dataset (from portal)
• Temporal tagging using Heideltime
framework [3]

CSV cell value disambiguation

• Row context:
• Filter candidates by potential parents
(if available)
• Column context:
• Least common ancestor of the spatial
entities

[3] Strötgen, Gertz: Multilingual and Cross-domain Temporal Tagging. 86

86 Language Resources and Evaluation, 2013.
 Indexed Datasets

87
RDF Export 1/2:
Knowledge Graph

• Spatial and temporal base knowledge graph

• Annotated data points in metadata and CSV cells
• CSV metadata using CSVW vocabulary
• e.g., delimiter, encoding, header, …
 RDF Export 2/2: Annotate Datasets à
CSV on the Web Metadata [4]
• Note: no real cell level annotaitons, we
needed to add those!
• E.g.:
• csvwx:cell
• csvwx:hasTime
• csvw:refersToEntity
• …

Details: cf.:
http://data.wu.ac.at/ns/csvwx

[4] R. Pollock et al., Metadata Vocabulary for Tabular Data, W3C CSV on the Web (2015)
89
 Enable e.g. Search by GeoSPARQL Queries:
• Standard for representation and querying of geospatial linked data
• (Almost) no complete implementations of GeoSPARQL

90
 Search Interface
https://data.wu.ac.at/odgraphsearch/
Faceted query interface:
§ Timespan
§ Time pattern
§ Geo-entities
§ Full-text queries

Back end:
§ MongoDB for efficient key
look-ups
§ ElasticSearch for indexing
and full-text queries
§ Virtuoso as a triple store

91
91
 Lessons learned
• Geospatial and Temporal scope is the most useful search
feature for Open Data
• Respective Hierarchical Knowledge Graphs can be built
from existing Linked Data Sources
• Our algorithms annotate CSV tables and their metadata
descriptions

à KGs improve search (with some extra work)

92
92