T HESIS P REPARED FOR THE D EGREE OF

M ASTER OF S CIENCE (M. S C .)

Procedural Generation of Text

written by: Fedja Adam

Department: Computer Science
First Examiner: Prof. Dr. Wolf-Dieter Groch
Second Examiner: Prof. Dr. Elke Hergenrther

Issue date: March 16, 2015

Last updated: September 7, 2015
Submit date: September 16, 2015
Abstract

This work explores a simple approach on applying Procedural Generation to the

domain of authoring and varying natural language text and stories. A working
prototype system for the task is conceptualized, developed and tested, while
the most important parts of its implementation are discussed along with their
alternatives.
In the first part, Previous Works from the realms of text and story generation
are presented, as well as the topic of Procedural Generation in general. Each of
them is illustrated with a range of examples, which are assessed individually
in the context of this works goal, revealing both successes and drawbacks of
their respective approaches. Starting with 1977s plot generator Tale-Spin and
concluding with todays interactive fiction Almost Goodbye, it is shown that there
is no standard solution to the problem at hand.
With that in mind, the Development of a general-purpose text generation
framework is described. Based on a simple core approach of dynamically assem-
bling text from pre-written fragments, several improvements to the text genera-
tion process have been implemented, allowing a more versatile use of the system
than what the previously analysed systems offered.
Finally, a first Evaluation of the developed framework is performed in order
to determine whether it can productively assist human authors in the writing
process, as well as the degree to which this is the case. Over the course of several
test cases, it is used for introducing dynamic variation to existing short stories, as
well as generating short Wikipedia article summaries from a specifically prepared
film database. The results of these test cases demonstrate that the discussed text
generation approach can be used productively for low-frequency variation of
text, but requires further development when approaching the threshold where
variation becomes authoring.

i
Contents

1 Introduction 1
1.1 What is Procedural Generation? . . . . . . . . . . . . . . . . . . . 1
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

I Previous Works 4

2 Relevant Technologies 5
2.1 Natural Language Processing . . . . . . . . . . . . . . . . . . . . 5
2.2 Natural Language Generation . . . . . . . . . . . . . . . . . . . . 9

3 Related Software 13
3.1 Procedural Content in Games . . . . . . . . . . . . . . . . . . . . 13
3.2 Tale-Spin (and its Relatives) . . . . . . . . . . . . . . . . . . . . . 17
3.3 Almost Goodbye . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

II Development 31

4 Concept 32
4.1 Project Goal and Delimitation . . . . . . . . . . . . . . . . . . . . 32
4.2 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.3 Scope and Requirements . . . . . . . . . . . . . . . . . . . . . . . 35

5 Architecture 38
5.1 Preliminary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.2 Text Tree Generator . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3 Text Writer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

ii
CONTENTS CONTENTS

6 Main Components 43
6.1 Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.2 Ontology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.3 Dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.4 Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.5 Plugin API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

III Evaluation 65

7 Results 66
7.1 Same Story, Different Actors . . . . . . . . . . . . . . . . . . . . . 67
7.2 Changing the Mood . . . . . . . . . . . . . . . . . . . . . . . . . . 71
7.3 Generating Factual Texts . . . . . . . . . . . . . . . . . . . . . . . 75
7.4 Issues and Applications . . . . . . . . . . . . . . . . . . . . . . . . 84

8 Summary 86

9 Outlook 89

IV Appendix 101

A Text Generation Example 102

iii
1 | Introduction

1.1 What is Procedural Generation?

Procedural Generation is a way to create content dynamically using algorithms,
as opposed to simply delivering it statically in a pre-defined form. Rather than
displaying an image that was designed by a human, or a puzzle challenge that
was created by one, Procedural Generation is about teaching algorithms to be
designers and creators themselves.
This is not a small endeavour, and while it has managed to be a successful one
in certain domains with well-defined problems, it has yet to be proven practical
in others and under more general conditions. A lot of its applicability depends
on the degree to which a creative task can be modelled as a repeatable process.
Consider the task of creating a crossword puzzle for example: Given a database
of words, each with a short hint or description, the problem of creating a cross-
word puzzle can be reduced to finding a satisfying word layout while avoiding
repetitions and erroneous overlaps. An algorithm can do this, so rather than
delivering a digital crossword puzzle game with a large amount of hand-made
puzzles, they could be Procedurally Generated whenever the end user requests
one.

Give a man a puzzle and you entertain him for a day; teach his computer to
create puzzles and you entertain him for a lifetime.

Quote 1.1: Not the original English proverb, but it illustrates the point.

Other problems are more complicated to handle, especially when they re-
quire (what could be described as) common sense or creativity to solve. An
algorithm can create an image of a wooden texture, because it can be modelled
and understood to the point that allows to describe its creation process in an ab-
stract way. But can it paint a tree, or the moody landscape where it stands? Can

1
1.2. MOTIVATION INTRODUCTION

it do so in an expressive and meaningful way? Obviously, Procedural Generation

cannot hope to compete with a humans truly creative abilities anytime soon. In-
stead, one of its greatest merits is its ability to fabricate variation and diversity:
While a human artist can carefully craft a single piece of the desired content at
just about any level of quality and flexibility, a carefully adjusted algorithm can
create a thousand different pieces in a fraction of the time provided its task can
be successfully modelled on an abstract level.
The way I see it, Procedural Generation should not have the goal to replace
humans, but to collaborate with humans letting algorithms do what can be
automated, giving humans more time to do what can not. An algorithm might
fail thinking of suitable hints and words in a crossword puzzle, but a human will
need unnecessary minutes to arrange them, where an algorithm only requires
milliseconds. Algorithms alone can only do so much, but human creators are
limited in their own ways. With the help of Procedural Generation, it is possible
to achieve results that would have been near impossible without.

1.2 Motivation
Two things from the field of computer science have always fascinated me: Emer-
gent behaviour and artificial intelligence. I also enjoy playing video games once
in a while and even more so, developing the technology behind it. When I no-
ticed a small trend for Procedural Generation in modern games, I was intrigued:
Instead of game artists and designers preparing all the game content in advance,
clever algorithms took part in the process of creating, adapting and recombining
content; whenever needed, directly on the players computer. They started draw-
ing maps and assembling levels, randomizing items and creating player quests,
generating an endless stream of game content that would have been too tedious
and costly to be created all by hand.
But one thing was missing. Despite all the variation and different fields of
content generation, I did not encounter a single example of a procedurally gen-
erated story. Surely, applying Procedural Generation to something like the main
narrative of a story-driven game had to be tricky but could it be done? And if
it can be done, to what degree? Investigating this in its full depth would require
to develop not only a testbed game first, but also the complete infrastructure for
injecting a procedurally generated story and presenting it properly; a task that I
deemed too big for a few months work, and one that would also introduce a lot
of unnecessary side-tasks as well.

2
1.2. MOTIVATION INTRODUCTION

In the end, I decided to simply strip away all game-related topics as well as all
interactivity. Instead of game narrative, I decided to focus on written language as
a more common and generic medium. What remains is just a few core questions:
To what degree can Procedural Generation be applied to written stories, and text
in general? When setting the goal to receive practically usable results, what kind
of work can an algorithm be trusted to do in sufficient quality? Is it possible to
create a general-purpose framework for bringing human authors and algorithms
together as collaborators?
While dealing with these questions, the goal of this work is not to discover
something fundamentally new, but to investigate existing technologies and com-
bine them to a bigger whole creating the prototype of a productive system that
allows human authors to enrich their texts with computer-generated variation,
and dynamically adjust them to different contexts.

3
 Part I

Previous Works

4
2 | Relevant Technologies

This chapter deals with previous research and relevant techniques that are either
required or at least useful for understanding this works subject. Starting with a
general look on the field of Natural Language Processing, a broad overview
on the topic is provided before focusing on the discipline of Natural Language
Generation.

2.1 Natural Language Processing

Natural Language Processing is a computer science field that deals with various
forms of interaction between humans and machines through the use of written,
spoken or otherwise communicated1 language. Early research stretches back
as far as 1950, with the prospect of machine translation being a driving force
behind its development, but countless other applications have emerged since
and are pursued just as well. [39] As a discipline, Natural Language Processing is
far too broad to be covered in depth in a single chapter, and it is not the main
focus of this work. However, a general idea of NLP2 systems, as well as their
goals and applications, can help to put it into perspective.
There are many use cases and tasks in the field of NLP, some of which may
seem simple or trivial at first, but often reveal themselves to be difficult and
highly ambiguous problems upon closer inspection. Some company websites for
example offered Question Answering systems for customers, allowing them to
talk to a chatbot, which will try to identify what kind of information is re-
quested, and to provide a suitable answer. While the task of understanding a
users chat message can be solved in a naive approach using pattern matching al-
gorithms, these solutions are bound to a narrow context and merely guess the
users intention based on a set of pre-defined cues. For simple support queries
1
e.g. sign language, which requires spatial perception to be read.
2
A common abbreviation for Natural Language Processing.

5
2.1. NATURAL LANGUAGE PROCESSING RELEVANT TECHNOLOGIES

(which can usually be anticipated based on previous experience) this may be

sufficient, but for other tasks, a deeper understanding is required and process-
ing a text suddenly becomes far more complex. Overall, the task of interpreting
natural language can be modelled as a multi-step process: [23]

Tokenization: To an algorithm, a text is just a sequence of characters, so dis-

tinguishing individual sentences, words and tokens is a usual first step in
processing it. Splitting a text into sentences can be done using a pattern
matching approach, but simply looking out for individual characters like
dots, question and exclamation marks is insufficient, as they are also used
for decimal numbers or abbreviations (Fig. 2.1). In some languages, the
surrounding spacing and character casing can be used as hints towards one
or the other interpretation, but a full detection of sentence endings is hard
to do without more context.

Figure 2.1: A naive sentence split algorithm has plenty of opportunities to fail.
Correct results require more sophisticated algorithms.

Similarly, splitting a sentence into individual tokens isnt without problems

either: While in the English language, spacing is a solid way to tell apart
words, how should names like the Great Barrier Reef or Black Friday be
treated? Even though they do consist of multiple words, they should in
fact be regarded as a single token, because they would lose their specific
meaning when used individually (Fig. 2.2). Detecting named entities like
this, however, is a challenging task on its own.

Figure 2.2: A token can consist of multiple words. Failing to recognize them will
change their meaning.

6
2.1. NATURAL LANGUAGE PROCESSING RELEVANT TECHNOLOGIES

Part-of-Speech Tagging: After splitting a text into sentences and tokens, each
token can be tagged with dictionary information, such as the part of speech3
it represents. Again, ambiguity can make this task more difficult: The En-
glish word set could be either a noun, a verb or an adjective depending
on the context it is used in (Fig. 2.3).

Figure 2.3: For some words, there is no easy one-to-one mapping for parts of
speech, which can be solved trivially using a dictionary. Multiple interpretations
are possible, depending on the context.

Sentence Parsing: The process of transforming a tagged sequence of tokens

into a tree structure that highlights grammatical and semantic relations
between them is commonly called Parsing. This adds another layer of in-
formation: Not only is it now clear that words like take or put are verb
tokens, but it is also possible to derive their subjects and objects. Like in
most NLP tasks, the natural ambiguity of language is an issue that may
cause problems, since sentences can often be interpreted in multiple ways.
Telling apart likely interpretations from unlikely ones requires context and
world knowledge.

Figure 2.4: In the above parse trees, individual tokens are grouped recursively
into (noun) phrase (NP) nodes. Depending on how the original plain text phrase
is read during parsing, the age of the addressed gender changes. Source: [13]

Semantic Analysis: Given a completely parsed sentence, an algorithm still wont

have an idea what it means, because its just a careful arrangement of lin-
guistic building blocks with no equivalent that is meaningful to a piece of
3
In English, eight to nine parts of speech are classified: Noun, Pronoun Verb, Adjective, Ad-
verb, Preposition, Conjunction, Interjection and Article. [40]

7
2.1. NATURAL LANGUAGE PROCESSING RELEVANT TECHNOLOGIES

software. However, the parse tree can be analysed for cues that map purely
linguistic entities and concepts into the working domain at hand: From a
linguistic point of view, take or ball are just words but for a software
controlling a robotic arm, they are also actions and objects. Transform-
ing the linguistic data structure that a parsed sentence represents into a
stream of commands is the final step in NLP4 that can enable a robot to
follow orders and a chatbot to respond to complex user questions.

Figure 2.5: Once natural language input has been transformed into a form that
the software can work with directly, further processing can take place outside
the domain of NLP.

While the above list of steps can only provide a very rough overview on
the different tasks in a typical Natural Language Processing use case, it should
have become clear that many of them do not have obvious solutions, and their
straight-forward approaches often fall prey to the ambiguity of natural language,
or the lack of context and deep knowledge. In many cases, a sentence or even
a word could easily be interpreted in various different ways, and which of them
is the right one may not be immediately obvious. Until today, machines are
merely able to work with text, but the degree to which one could speak about
understanding text still remains arguable5 .
Still, it is important to note that Natural Language Processing is not only
about understanding language and that working with language, as opposed
to understanding it, is not solely caused by a lack of technical capabilities, but
characterising a different subset of goals as well. There are many applications
and tasks in NLP where truly understanding natural language is in fact not a
requirement to perform at a sufficient quality level.

Speech Recognition is an NLP discipline that deals with the task of transcribing
spoken language into text, which can be regarded as a largely independent
4
This is, of course, a large simplification. Several more layers of artificial intelligence tasks
may follow, but they are not necessarily bound to the domain of Natural Language Processing.
5
In fact, the task of truly understanding natural language is considered an AI-hard problem,
meaning that solving it would imply solving the problem of artificial intelligence in general.

8
2.2. NATURAL LANGUAGE GENERATION RELEVANT TECHNOLOGIES

preprocessing step before applying other NLP operations on the resulting

text. Among its commonly known use cases are early speech-to-text sys-
tems for text editing and writing, as well as personal assistant systems, like
Apples Siri.

Machine Translation aims at finding the closes representation of a text in a dif-

ferent language. While naive word-by-word translations fail due to differ-
ing sentence structures, idiomatic phrases and subtly different word mean-
ings, more complex systems use knowledge about a languages grammar
and large dictionaries with meta information to provide usable results. Re-
cent approaches often use statistical data of already translated text corpora
instead. Although prominent examples like Google Translate sometimes
fail to reach a satisfying text quality, they have the advantage of not re-
lying on large databases and rule sets that have to be manually built and
maintained.

Sentiment Analysis describes the task of determining whether a certain term is

mentioned in a positive or negative way in any given text. Viable use cases
for this are approximating numeric ratings from plain text user reviews, or
researching public opinion on a certain topic.

Automated Summarization of texts such as news- or encyclopaedic articles is

typically used by search engines and similar applications that aggregate
information to provide a quick overview to human users. Even without
understanding the actual content of a text, a rough summary can still be
created by using key sentences from the original text, which are selected
using a rating algorithm for their relevance. [31]

Of course, the above list is largely incomplete, but it illustrates that most of
the common NLP tasks deal with reading and processing natural language - but
not producing it.

2.2 Natural Language Generation

The task of generating human-readable text based on a non-language input is
far less prominent than the various other applications of the NLP domain, but
nonetheless relevant. In their work Building Applied Natural Language Generation
Systems, researchers Robert Dale and Ehud Reiter list several viable use cases for
Natural Language Generation systems: [14]

9
2.2. NATURAL LANGUAGE GENERATION RELEVANT TECHNOLOGIES

Generating weather forecast texts based on graphical weather maps.

Summarizing both data and statistical meta information from spreadsheets

and databases in a textual form.

Describing the reasoning chain of an expert system.

Wording the answer texts to user questions in Question Answering systems.

They also describe a second category of NLG6 systems called Authoring Aids,
which help people create routine documents. An example in their own words:

A doctor, for example, may spend a significant part of her day writing
referral letters, discharge summaries, and other routine documents;
while a computer programmer may spend as much time writing text
(code documentation, program logic descriptions, code walkthrough
reviews, progress reports, and so on) as writing code. Tools which
help such people quickly produce good documents may considerably
enhance both productivity and morale.

Authoring Aid software doesnt necessarily exhibit intelligent behaviour

and, in a sense, every usable text editing program with a sophisticated way
to define, insert and adjust pre-written text fragments could already be con-
sidered to be part of that category. However, the more complex and dynamic
these fragments get, the more intelligent the Authoring Aid needs to become in
order to be actually useful. An ideal Authoring Aid would, once set up, be able
to autonomously write text that matches a certain author intent. Hypothesizing
such an autonomous system, the problems it would need to solve may range
from relatively simple text processing issues7 to complex planning-based tasks
for structuring and composing text on a larger scale.
From a more general point of view, Dale and Reiter identify six different
stages in Natural Language Generation:

Content determination: Given an available or potential set of information, the

system should do a selection which parts of it to include in the text, and
which to omit. Factors influencing this decision are the purpose and target
audience of the text, as well as the desired level of detail and how trivial
or unexpected a certain information is.
6
Natural Language Generation
7
Example: Adapting a pre-written sentences grammatical person to match a subject name.

10
2.2. NATURAL LANGUAGE GENERATION RELEVANT TECHNOLOGIES

Document structuring: Given a set of information entities or data points, the

order in which they appear needs to be determined. Some of these or-
derings may be preferable over others, depending on the nature of the
conveyed information, as well as author intent. On a higher level, para-
graphs and topics can be ordered purposefully in order to present them in
logical order, following a narrative golden thread, or even to bias their
importance as perceived by human readers.

Aggregation: Given a set of sentences or concepts, an algorithm needs to decide

which of them can be merged and which is the best way to do so - all in
order to reduce redundancy, shorten the resulting text and make better use
of the limited human attention span when presenting it. Simple merging
operations can happen on a syntactic level and involve the omission of
repeated elements as well as usage of conjunctions and similar linguistic
devices. In a simple example, an Aggregation step could merge the two
sentences The day was warm. The day was sunny. into a less redundant
The day was warm and sunny..

Lexical choice: Given a conceptual phrase to express in concrete text, the prob-
lem of lexical choice deals with selecting the right words for it. When
writing about a downward price change on the stock market, whether its
value is described to decrease, fall, or even plummet depends not only on
the magnitude of the change, but contextual knowledge as well. Deciding
which of N synonymous words to use for any given case is at the core of
this stage, which attempts to create a suitable mapping from semantics to
wording.

Referring expression generation: In order to identify an entity8 , a text needs

to establish a reference to it - either using a unique name, or in more natu-
ral ways using pronouns, aliases and paraphrases. Creating the appearance
of natural speech requires not only selecting appropriate ways for referring
to an entity, but also making use of a certain degree of variation when
doing so multiple times.

Realisation: Given a complete syntactical representation of a text, this last stage

applies rules of grammar, punctuation and formatting in order to create the
final text for human consumption.
8
such as an object, place, time, concept or other.

11
2.2. NATURAL LANGUAGE GENERATION RELEVANT TECHNOLOGIES

Each of these stages represents a complex task to be solved, not all of which
are well-understood and practically explored at a general level. Thus, it is im-
portant to note that, even though Dale and Reiter described the above stages
in detail and proposed potential approaches for dealing with them, their article
is focused primarily at providing a basic overview and summary of generating
natural language texts, as well as presenting typical problems that arise in the
process. Overall, the field of Natural Language Generation appears to be far from
reaching a conclusive state on a large scale, even though promising results may
have been achieved in various sub-tasks and simplified or specialized problem
setups.

12
3 | Related Software

The concept of Procedural Generation is nothing new and in fact, there have
been a lot of works dealing with it in one way or another. In the context of this
thesis, three cases are especially interesting and will be presented in this chapter:
Procedural Content in Games will be used as a starting point, because it
exhibits a great variety of generation techniques that can contribute to giving a
basic overview. Getting more specific, the section about Tale-Spin will present
some of the story generating experiments that have been conducted until today
and demonstrate why their focus does not match with the goals of this work. Fi-
nally, the story experiment Almost Goodbye is chosen as an example to illustrate
a more suitable approach.

3.1 Procedural Content in Games

Procedural Generation has long been used as a technique to dynamically cre-
ate video game content, thus making games a colorful example of an applied
real-world use case. This section will give a basic overview on several of its ap-
proaches, which will later serve as a role model for different ways to generate
content in general.
Originally being utilized as a way to save the once very limited resources
of both disk space and memory, 1984s space simulation game Elite was one of
the first prominent examples to use procedurally generated content: Instead of
delivering a manually authored or pre-generated game world as data package
alongside the game binaries, the required portions of it were generated on-the-
fly on they players machine. [38] That way, Elite was technically able to feature
248 galaxies with 256 solar systems each, despite the hardware limitations at the
time1 . Using a fixed seed value for generating the random numbers that made up
its universe, Elite always generated the same world for all players and sessions
1
Although the content was later cut down to eight galaxies in order to avoid overwhelming
the player or rendering the world unbelievable.

13
3.1. PROCEDURAL CONTENT IN GAMES RELATED SOFTWARE

the goal not being to dynamically create content, but to serve a huge amount of
static content in an extremely efficient way. [15]
However, as storage devices and system memory showed a drastic increase of
capacity in the following years, Procedural Generation was no longer a necessity
to deliver large or detailed game worlds all of the required content could be de-
livered directly with the game and simply accessed when needed. Additionally,
video games became gradually more sophisticated in matters of audiovisual pre-
sentation, and quality requirements on game content consequently scaled with
them up to the point where the expectations of players and developers left
creative algorithms unable to compete with human artists. [1]
In recent years, the Procedural Generation of game content is experienc-
ing a tentative renaissance, with increasing attention from developers and a
more frequent usage in commercial games. [1] Its common purpose, however,
has changed: Rather than being a vessel for delivering game assets in the most
memory-efficient way possible, its focus has shifted towards achieving a seem-
ingly infinite variety of game content that would be too time-consuming or im-
practical for human artists to create manually. Among the typical usage examples
for Procedural Generation in modern games, three closely related forms can be
identified:

Creation of entirely new game content that can be described by an abstract

model, using an algorithm which is able to apply it. Creating such models
for a certain type of game content can be quite tricky, as most of them dont
fit easily into a schematic; but there are certain exceptions where even a
mathematical function can already yield sufficient results. For illustrating
this with an example, consider a very simple, two-dimensional landscape:

Figure 3.1: A simple example for a procedurally generated 2D landscape.

14
3.1. PROCEDURAL CONTENT IN GAMES RELATED SOFTWARE

When assuming that there are no holes or caves crossing it, a landscape can
be described as a sequence of height values along a horizontal axis. These
height values can be calculated using a mathematical function depend-
ing on their horizontal position and any number of additional parameters.
When choosing a simple term like a sine function, the landscape may look
artificial or alien to the human observer but depending on the context the
game provides, more complex terms can yield results that are sufficiently
convincing.
Because the Procedural Content of a game is usually generated as needed
on the players computer, there is no practical way of reviewing each re-
sulting instance or manually removing undesirable outcomes, so content
generating algorithms are required to directly produce results of the de-
sired quality. This restriction leaves only a narrow window for feasible
use cases, especially when operating largely without human input. One of
the more common ones is generating a believable terrain, like the world
generation algorithm in Minecraft, to name one prominent example. In
a completely autonomous setup, it is able to create an arbitrary amount
of (practically) infinite maps (Fig. 3.2) that are consistent with the game
world. [32]

Figure 3.2: A procedurally generated world in Minecraft. Source: [47]

Adaption of human-authored content depending on the in-game situation, or

other parameters. Modifying the walking animation of a character model
to make sure the contact points of its feet match varying ground levels is
a common example. Techniques like this allow to create a more believable

15
3.1. PROCEDURAL CONTENT IN GAMES RELATED SOFTWARE

impression than using completely pre-defined content that doesnt react

to its environment or completely dynamic content that lacks the artistic
quality of human-created work.
Applied to a different kind of content, roleplaying games like The Elder
Scrolls: Skyrim let players select a basic character model from a fixed pool,
but then allow to customize it using a set of parameters that modify its
appearance dynamically like exaggerating or scaling certain facial fea-
tures. Going a step further, the game No Mans Sky procedurally generates
all animals and plants (Fig. 3.3) in the games universe by randomizing the
parameters of generic Templates which are not only used to modify their
appearance, but also to derive AI behaviour. [24]

Figure 3.3: Four different variations of the same animal template in No Mans
Sky. Source: [24]

Recombination of pre-defined content "atoms" while satisfying certain constr-

aints. In this case, game entities are not derived from mathematical func-
tions or similar, but built from existing elements that are combined in a
meaningful way. Although nothing inherently new is created, difficulty can
arise from complex requirements regarding which of the existing atoms
can be combined in what way, as purely random combinations would often

16
3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

produce nonsensical results like using plants as decorative elements of

a moon surface landscape. Generator algorithms from this realm can in-
corporate concepts from query languages (to find matching elements) and
problem solvers (to deal with constraints).
As an example, The Binding of Isaac uses a recombination approach to gen-
erate all of its levels using a set of pre-defined rooms that are selected,
joined and configured procedurally. Constraints in the generation algo-
rithms assure both a consistent game world and an appropriate difficulty
level in order to provide a satisfying gameplay experience.

Of course, the above characterization is not to be regarded as a discrete list of

categories, but as a continuous spectrum heavily relying on pre-defined human
input on one side and being a mostly autonomous setup on the other (Fig. 3.4).
Surely, there are many applications of Procedural Generation that do not strictly
exhibit only one of the above forms.

Figure 3.4: Different forms of Procedural Generation.

Since it was first introduced, Procedural Generation has become more and
more capable, but is in most cases still far away from generating results on a
quality level that is comparable to human-authored assets. However, modern al-
gorithms slowly aspire to be valuable collaborators in the content creation work-
flow of games, enabling developers to provide more varied, situation-dependent
and customized experiences.

3.2 Tale-Spin (and its Relatives)

Besides being a somewhat popular term in game development, Procedural Gen-
eration has also been explored in a more academic environment, and specifically

17
3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

focused to the field of narrative and text generation as well.

One of the first works on procedurally generating narrative was Tale-Spin by
James Meehan, published in 1977. [2] Its goal was to write stories based on a
knowledgebase of characters, their relationships, physical locations, story struc-
ture and the English language. In an approach that was often mirrored by similar
projects, Tale-Spin interpreted the plot of a story as a problem solving process
and essentially simulated plausible character behaviour in a fictional world.
While all abstract world knowledge and behavioural rules, as well as the
initial world state are user-defined, actions and consequences in the story output
are inferred by the algorithm and arise naturally from the initial configuration.
Given a set of character goals, an integrated problem solver will use them to
determine sub-goals and actions, which are added to the systems world state
memory. An inference mechanism then derives the tree of consequences each
action invokes some of which being new character goals that can be fed to the
next simulation step, completing the algorithms main loop (Fig. 3.5). In the end,
a text generation module would transform a trace of relevant goals, actions and
state changes into an English text a simple story, if you will.

Figure 3.5: An abstract schematic of Tale-Spins main loop.

In his publication, Meehan presents some of the generated text outputs, in-
cluding both the expected results (Lst. 3.1) and some interesting failure cases,
which he comments to shed some light on their technical cause and possible so-
lutions. Most of them are attributed to a lack of knowledge, either by failing
to properly simulate the perception that characters have of the world, or by not

18
 3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

providing certain details of abstract knowledge in the first place.

 
ONCE UPON A TIME GEORGE ANT LIVED NEAR A PATCH OF GROUND. THERE WAS A NEST
IN AN ASH TREE. WILMA BIRD LIVED IN THE NEST. THERE WAS SOME WATER IN A RIVER.
WILMA KNEW THAT THE WATER WAS IN THE RIVER. ONE DAY WILMA WAS VERY
THIRSTY. WILMA WANTED TO GET NEAR SOME WATER. WILMA FLEW FROM HER NEST
ACROSS A MEADOW THROUGH A VALLEY TO THE RIVER. WILMA DRANK THE WATER.
WILMA WASN'T THIRSTY ANYMORE.

GEORGE WAS VERY THIRSTY. GEORGE WANTED TO GET NEAR SOME WATER. GEORGE
WALKED FROM HIS PATCH OF GROUND ACROSS THE MEADOW THROUGH THE VALLEY
TO A RIVER BANK. GEORGE FELL INTO THE WATER. GEORGE WANTED TO GET NEAR THE
MEADOW. GEORGE COULDN'T GET NEAR THE MEADOW. WILMA WANTED GEORGE TO
GET NEAR THE MEADOW. WILMA WANTED TO GET NEAR GEORGE. WILMA GRABBED
GEORGE WITH HER CLAW. WILMA TOOK GEORGE FROM THE RIVER THROUGH THE
VALLEY TO THE MEADOW. GEORGE WAS DEVOTED TO WILMA. WILMA LET GO OF
GEORGE. GEORGE FELL TO THE MEADOW. THE END.

Listing 3.1: The usual output that Tale-Spin produced.

One of the given examples modeled a character falling down as Gravity

moving him, which led to a story where one character fell into a river and
soon after, Gravity drowned: The system inferred that, when Gravity moves
someone into the river, it must itself be located inside the river afterwards but
unable to escape, because it doesnt have legs, wings or friends to call for help.
Meehan observes that Tale-Spin didnt know enough about gravity, suggesting
that this specific case was later fixed by adding enough abstract world knowledge
for the system to not handle concepts like gravity in the same way it handles
actual characters.
In a different example, Meehan attempted to re-create the fable of The Fox
and the Crow, where a fox uses flattery to trick a crow into losing a piece of food
to him2 . After putting all the relevant parts into place to match the fables initial
setup, and providing the required knowledge about them, the algorithm should
be able to derive the actual fable by itself. However, the problem solver had
different plans (Lst. 3.2) and deduced that it would make much more sense if
the crow just ate the piece of food right away. And why wouldnt it? Of course
this defies the purpose of the tale, so Meehan fixed this by defining that the crow
had eaten recently and thus was not hungry.
A general focus of his analysis appears to be the topic of knowledge, and
2
Sitting on a tree branch well out of reach, a crow holds a bit of cheese in its mouth, when
a fox approaches it. In order to get the cheese for himself, he praises the crows appearance and
requests it to sing a song, suggesting that it would be equally beautiful. As the crow opens its
beak, the cheese falls to the ground and is picked up by the fox.

19
 3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

 
Once upon a time there was a dishonest fox and a vain crow. One day the crow was sitting in
his tree, holding a piece of cheese in his mouth. He noticed that he was holding the piece of
cheese. He became hungry, and swallowed the cheese. The fox walked over to the crow. The
end.

Listing 3.2: Transcript of Tale-Spins rather "pragmatic" The Fox and the Crow
interpretation.

a lot of failure cases are commented in a way that suggests the existence, lack
or transfer of knowledge to be the main culprit. While this assumption sounds
reasonable with regard to the potentially very limited testing database and the
prototype nature of the presented work, it certainly does not cover the full scale
of the problem. The above example of The Fox and the Crow is picked by Meehan
to demonstrate a case where adding more knowledge can produce unexpected
results [2] because as soon as the crow knew about the food in its mouth,
eating it seemed like the most reasonable course of action.
Albeit being a correct observation, this does not deal with the actual root of
the problem: Tale-Spin simply didnt understand the purpose of the story it was
supposed to tell, and it had no concept of a writers intention either. It did not
write a fable in order to convey a message or illustrate a morale, but merely
documenting the progress of an internal character simulation3 . This is not the
way a typical human writer would work, and one can imagine that an approach
like this will hardly be able to deliver plotlines of hand-authored quality, as a
meaning can only emerge through human interpretation of coincident acts.
Going forward, this is only one of the problems which has since been ad-
dressed in Tale-Spins successors, and there have been many similar works on
the Procedural Generation of narrative until today. Dealing with all of them in
full detail is beyond the scope of this work, but fast-forwarding to more recent
times the following selection of prominent examples should provide a rough
overview:

Author (1981) was a planning-based4 story generator algorithm like Tale-Spin

but progressing upon the observation that the lack of purpose led to
trivial story setups, a different approach for creating the main plot was
3
Meehan was well aware of this and I believe that anything more was simply not part of the
projects scope at the time. As he states himself: [Tale-Spins] output may be regarded as a trace
through problem-solving procedures..
4
Using information about the current world state A, and the desired world state B, a planning
system finds a sequence of actions, which transforms A into B. The set of possible actions or
action templates is pre-defined.

20
3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

chosen. [3] Instead of the World Simulation as implemented in Tale-Spin,

Author focused on simulating the goals and intentions of an author, as
he composes the plot of his story. These may include invoking certain
feelings in the reader, conveying a pre-conceived message or morale, or
maintaining an engaging story arc all of which can be broken down into
sub-goals that influence directed actions. Here, the creation of a plot is not
understood as an emergent property of tracing through each characters
individual problem solving process, but as the main problem itself. This
meta-problem is solved by a virtual author, who is constrained by a complex
web of his own goals and requirements.

Universe (1983) built up on both Tale-Spin and Author, but aimed for some-
thing slightly different: Not a self-contained, complete story was the de-
sired output, but a never-ending series of episodes, featuring complex char-
acter interaction and development. [4] This particular story generator algo-
rithm resembled a TV series director more than a novel author and was
focused heavily on the interaction of a multitude of characters. As such,
both individual character traits and author goals were tracked, in order
to achieve a certain level of consistency and at the same time add an el-
ement of drama to keep things going. In addition to a planning system
like previous algorithms used, Universe also made use of pre-defined plot
fragments with stereotypical roles that could be filled dynamically when
their requirements were met with the characters at hand. Resorting to this
mixed approach of human-authored content and dynamic planning, Uni-
verse was able to generate more complex and complete fictional worlds
than similar prototypes at the time. [18]

Minstrel (1993) could be considered the conclusion of an over a decade long

research effort on the topic on procedural narrative generation, inspired
by Tale-Spin and its various close relatives. [19] Like Author, Minstrel uses a
planning-based simulation approach with a strong focus on the intentions
of a virtual author and like Universe, it also relied on a database of typical
plot fragments which would be used for reference. But going a step further,
Minstrel was not only able to apply these fragments to new stories, but
could also adapt them to fit new situations. [20] This allowed it to create
new and unexpected variations, but also resulted in a very unstable output
quality; in an evaluation of Minstrel, [6] Rafael Prez y Prez and Mike
Sharples write:

21
 3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

The reader can imagine a Knight who is sewing his socks and
pricked himself by accident; in this case, because the action of
sewing produced an injury to the Knight, Minstrel would treat
sewing as a method to kill someone.

Brutus (1999) was a story generator focused on the topic of betrayal and the
first program that actually attempted to produce high-quality text in addi-
tion to a meaningful plotline. Based on a mathematical theorem prover,
it uses logical pattern matches to assemble a story from pre-defined ele-
ments and was largely dependent on human input in order to generate a
story [21] which enabled it to create texts on a comparatively high qual-
ity level (Lst. 3.3). They actually looked human-written, because (for the
most part) they were. [7] Its authors consequently stated that Brutus is not
creative at all, but a reverse engineering effort to see if a particular story
can be re-assembled by an algorithm, given the appropriate input. [17]
 
Dave Striver loved [...] the fact that the university is free of the stark unforgiving trials of the
business world only this isn't a fact: academia has its own tests, and some are as merciless as
any in the marketplace. A prime example is the dissertation defense: to earn the Ph.D., to
become a doctor, one must pass an oral examination on one's dissertation. [...]

Dave [...] needed the signatures of three people on the first page of his dissertation, the
priceless inscriptions which, together, would certify that he had passed his defense. One of the
signatures had to come from Professor Hart, and Hart had often said [...] that he was honored
to help Dave secure his wellearned dream. At the defense, Dave thought that he eloquently
summarized Chapter 3 of his dissertation. [...] There were no further objections.

Professor Rodman signed. He slid the tome to Teer; she too signed, and then slid it in front of
Hart. Hart didn't move. "Ed?" Rodman said. Hart still sat motionless. Dave felt slightly dizzy.
"Edward, are you going to sign?"

Listing 3.3: A short excerpt from a text that was created by Brutus, using pre-
written text fragments. [7]

Fabulist (2010) was progressing on works like Tale-Spin, Author and Universe,
and had a strong focus on balancing character and plot consistency. Al-
though being an author-centric planner, i.e. simulating the goals of a vir-
tual author, the system was designed to make sure that character behaviour
within the authored plot was consistent and believable. While Universe at-
tempted the same by enforcing character traits to match with their roles
in plot fragments, Fabulist expanded on this by evaluating each characters
individual motives and goals as well. Fabulists author Mark Riedl explains
the distinction with an example:

22
 3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

[...]if a character has a niceness value of -10, it is consistent for

that character to force someone to marry his son, but it does not
mean that the character has a reason to force someone to marry
his son.

When a plot sequence requires characters to act in ways that are not deemed
believable, Fabulist would attempt to repair the story plan by finding rea-
sons for said characters to act like they are required to. This form of back-
tracking allows it to achieve a higher level of character and plot consistency
than Universe. [8] Like most of its predecessors, Fabulist created rather tech-
nical English transcripts of the plot it had modelled without focusing too
much on literary value (Lst. 3.4).
 
[...] The genie is in the magic lamp. Jafar rubs the magic lamp and summons the genie out of
it. The genie is not confined within the magic lamp. Jafar controls the genie with the magic
lamp. Jafar uses the magic lamp to command the genie to make Jasmine love him. The genie
wants Jasmine to be in love with Jafar. The genie casts a spell on Jasmine making her fall in
love with Jafar. Jasmine is madly in love with Jafar. Jasmine wants to marry Jafar. [...]

Listing 3.4: A short excerpt from a text that was created by Fabulist. [16]

Moral Storytelling System (MOSS) (2013) is a software prototype that was

designed to write stories that carry morales. Given a set of rules on how
certain morales can be expressed in terms of a sequence of character emo-
tions, the algorithm then uses a planning approach to generate sequences
of actions that lead to the required emotional configurations. [9] In the final
step, an English text is generated that represents the plot (Lst. 3.5).
 
Once upon a time there lived a unicorn, a knight and a fairy. The unicorn loved the knight.
One summer's morning the fairy stole the sword from the knight. As a result, the knight didn't
have the sword. The knight felt distress that he didn't have the sword anymore.
The knight felt anger towards the fairy about stealing the sword because he didn't have the
sword anymore. The unicorn and the knight started to hate the fairy.
The next day the unicorn kidnapped the fairy. As a result, the fairy was not free. The fairy felt
distress that she was not free.

[36]
Listing 3.5: A text that was created by MOSS.

Although approaching the topic from different angles and sometimes with
different goals, a certain characteristic is quite common among all of the above:
The assumption that at least part of a story generation process can be interpreted
as a planning problem in some form. Most of the mentioned systems focus on

23
3.2. TALE-SPIN (AND ITS RELATIVES) RELATED SOFTWARE

modelling said planning problem, and solving it in a way that leads to satisfac-
tory results with regard to the hypothesis that motivated their creation (Fig. 3.6).
However, none of them were actually intended to be used productively they
were vessels for exploring abstract ideas and concepts on the domain of narra-
tive generation and computational creativity.

Figure 3.6: An overview of Tale-Spin and its relatives, as presented in this chap-
ter.

It feels like a natural conclusion of this kind of exploration, that issues from
the area of concrete applicability have largely been left untouched. From an
academic point of view, a lot has been achieved in the last forty years of occa-
sional research on the topic yet, when asking a layman about the quality of the
generated texts, he would hardly notice a similar improvement (See Lst. 3.1 and
Lst. 3.5). One prominently absent topic appears to be the overall writing style
of generated texts: Looking at the output of Tale-Spin and similar systems, they
usually follow a fixed structure and read more like technical transcripts than
actual stories5 .
Of course, this observation misses the point of these works, but at the same
time demonstrates a dilemma that authors of story generating systems are fac-
ing: If one text generator is able to creatively build new stories while another one
merely assembles pre-written fragments, the first one obviously appears prefer-
able; but what if the first one only returns texts of poor quality, while the second
5
The only exception here is Brutus, which resorted to using human-authored text fragments
for constructing its final results.

24
3.3. ALMOST GOODBYE RELATED SOFTWARE

one produces output that is practically usable?

3.3 Almost Goodbye

Almost Goodbye is a short piece of interactive fiction that uses Procedural Gener-
ation to increase variety for the reader, as well as react to the decisions made by
the player. It can be played in the web browser and will highlight the procedu-
rally generated parts of its text when requested. [28] Unlike Tale-Spin and similar
experiments, Almost Goodbye did not try to reflect human creativity, simulate
character behaviour or a virtual author. Its goal was not to create an abstract
model of narrative and using that to explore a certain concept but to create a
good piece of interactive fiction; and use Procedural Generation in the process
wherever practically viable. At the same time, the resulting text should not suffer
in terms of quality, but be still as good as [...] hand-authored. [10]
This premise marks an interesting contrast to the works that were briefly pre-
sented in the previous chapter; where quality was assessed mostly in terms of
an algorithms capability to produce an abstract arrangement of plot segments,
rather than whether or not someone would generally read [the text] for plea-
sure. By choosing this path, the projects author Aaron Reed is moving a little
more in the realms of engineering than science, and also a significant step closer
to productive usage in various applications.
For his playable software prototype, Reed set a small and manageable goal:
Writing satellite sentences in an otherwise hand-authored text, in order to set
the mood and context of a scene dynamically. The story puts the reader into the
role of an interstellar colonist, who is about to leave earth forever; and it is up
to his choice, which of five persons are visited for saying goodbye, where they
are visited and in what order. These choices define the context of a scene: The
place to meet each person is selected directly using a multiple choice question,
but the overall mood is a state that can only be affected indirectly and depends
on the order of visits and the selected dialogue options.
The core of each encounter is pre-written and remains exactly the same each
time it is played out: These kernel sentences are directly narrating the events
of a story [10] and remain static. On the other hand, so-called satellite sentences
do not contribute to the main plot and are scattered sparsely across the text
(Table 3.1). They fill in the outline [...] by maintaining, retarding, or prolonging
the kernel events [11] and are only loosely coupled to the rest of the text. This
makes them good candidates for Procedural Generation, because maintaining

25
3.3. ALMOST GOODBYE RELATED SOFTWARE

Anchoring Sentences Pacing Sentences

In the hot sunlit pastures yel- There was a silence.
low flowers bloomed. He smiled again.
It was getting late. She let that sink in.
The day was waning, and the He took a deep breath.
muttering of the city contin- The tension was unbearable.
ued outside.

Table 3.1: Examples for two kinds of satellite sentences, as given by Reed. [10]

consistency becomes less of an issue. Consequently, in Almost Goodbye, satellite

sentences were chosen as the main interface between a human writer and an
algorithm: The author specifies a set of formal grammars6 that allow the system
to assemble satellite sentences from short text fragments. He also guides the
system where to insert them using markers in the main text. An algorithm will
then be able to fill in the blanks dynamically (Fig. 3.7).

Figure 3.7: A sample text from Almost Goodbye, before and after filling in satel-
lite sentences.

In order to determine a suitable text replacement for a blank marker, the

algorithm interprets certain cues from the markers surroundings for example
whether it is located at the start, end or in the middle of an actors dialogue lines
and uses them to select a starting symbol, which will then be expanded into
readable text. The rules that will be applied in the expansion process are defined
in the satellite grammar, which is composed of multiple smaller grammars that
are added and removed depending on the context (Fig. 3.8):
6
A formal grammar consists of a set of replacement rules. In our context, they take a form
like symbol X can expand to statements A, B or C, as in FRUIT can expand to apple, pear,
strawberry. Each statement is a text fragment, which can also include references to the starting
symbols of other rules, so a simple replacement graph can be defined.

26
3.3. ALMOST GOODBYE RELATED SOFTWARE

Figure 3.8: When a TENSION symbol is encountered, the way it is expanded

depends on the set of currently active grammars.

For example: When the main character is afraid, a rule is active that allows to
replace a TENSION symbol with Maybe its too late for this and similar sen-
tences (Lst. 3.6). This is defined in a specific afraid grammar that will only be
active as long as the current context features a main character with a matching
emotional state. On the other hand, one that feels remembered would result
in deactivating the afraid grammar and activating one that contains rules ex-
panding to a more positive inner monologue like a simple Its going to be
okay. Similarly, descriptions of the scenes environment will be replaced differ-
ently, depending on the location that was chosen or the time of day.
 
TENSION : *NOGO * ANXIOUS

ANXIOUS : * IFEEL *ANXIOUSWORD but *ANXIOUSREASON * IFEEL *ANXIOUSWORD I can ' t -

breathe I can ' t breathe , the air * AIRQUALITY but starved somehow for oxygen -
Maybe this doesn ' t matter Maybe it ' s too late for this

IFEEL : I feel I can ' t help feel

ANXIOUSWORD: anxious vaguely anxious nervous tense
ANXIOUSREASON : *MAYBE it ' s just the imminent departure *MAYBE it ' s just pretrip -
jitters that ' s to be expected , *MAYBE who wouldn ' t , I suppose

Listing 3.6: An excerpt from the context-bound afraid grammar.

While each grammar provides a certain degree of variation by allowing dif-

ferent and sometimes nested text expansions for each given symbol, in Almost
Goodbye, they are bound to a specific context. In order to account for variation
in the context itself, a set of grammars is required, where at least one accounts
for each supported state: One for each emotion, one for each time of day and
location and one for each character to meet, plus a few generic ones, amounting
to a total of 23 small grammars. [10]

27
3.3. ALMOST GOODBYE RELATED SOFTWARE

In order to generate a specific, yet consistent sentence, these grammars also

have to interact with each other: When a sentence refers to the character being
talked to in a scene, it will depend on the gender of the addressed person. For
a fragment from an emotion grammar, whether it is I listen to him or I lis-
ten to her is unknown at the time of writing because the player might visit
each character with any emotional state. Reed solves this problem by defining
those sentences generically as I listen to HIMHER and supplying a matching
expansion for the HIMHER symbol in each characters grammar.
Reed concludes that the system served its purpose of assembling high-quality
literary text about 80% of the time, but still considers it embryonic while
addressing the problem of creating trustworthy [Procedural Generation] col-
laborators only to some extent. One topic where he sees room for improvement
is providing more context to the text generator, either explicitly by the author di-
rectly, or implicitly using digital dictionaries like WordNet7 . Looking into Reeds
work from the perspective of this thesis, it seems that Almost Goodbye man-
aged to do a lot of things right, but fell short on certain aspects that are deemed
important for this work:

3 High Output Quality: As mentioned earlier, Reed focused on output qual-

ity instead of algorithmic quality. His work doesnt do anything fun-
damentally new, but succeeds because he is able to put well-established
things to good use. Almost Goodbye merely combines existing text frag-
ments rather than creatively writing something entirely new8 , but then
again is able to generate texts that can be used practically and without
further human observation.

3 Simple: Similar to the Procedural Generation approach in Almost Good-

bye, all the human-authored content is kept as simple as possible. There
are only two major Components satellite grammars and core texts and
both of them are easy to read, understand and write.

7 Highly Specific: First and foremost, Almost Goodbye is a specific piece

of interactive fiction that is based almost completely on dialogue - and
naturally, the software framework behind it is tailored exactly to the kind
7
WordNet is a large and well-known digital dictionary that is designed for usage in automated
NLP applications, in effect being geared towards algorithms instead of human readers. It is
publicly available and maintained by the Princeton University. [27]
8
This is similar to the Recombination kind of Procedural Generation, as presented in the
Games chapter.

28
3.3. ALMOST GOODBYE RELATED SOFTWARE

of text it deals with: Satellite sentences can only react to dialogue, and the
dialogue itself is restricted to two characters, one of which being the first
person narrator of the story. All the story-specific parameters like emotions,
progression of time and locations are part of the system as well, so there
would be no way to change them without rewriting parts of it.

7 Doesnt Scale Well: Because all text variation depends on a set of small,
interdependent grammars, consistent sentence structure and narration can
only be achieved through careful construction. Many grammar symbols
represent globally used sentence fragments like AIRQUALITY or LIGHTVERB
that can only be used when knowing what exactly they might expand to,
and defined when knowing what exactly they are expected to represent.
This knowledge, however, cannot be acquired easily: Since all context sen-
sitivity in the narration is achieved by activating and deactivating small
sub-grammars, the definition and usage of certain symbols is scattered
across the whole grammar database, and each newly introduced gram-
mar could both use and define any symbol. There is no such thing as a
scope, everything is defined globally and naming collisions between gram-
mar symbols are likely to occur and potentially hard to track. All of this is
perfectly manageable in a small story like Almost Goodbye, but becomes
hard to maintain once the number of grammars grow and it becomes hard
to imagine setting up a general-purpose text generation library with this
concept.

7 Very Limited: The purely grammar-based text generation system in Almost

Goodbye does work well in a limited scope, but falls short when it comes to
implementing more complex logic: For example, if a certain text passage
should depend on whether or not character A is older than character B,
there is no elegant way to achieve this, as there is neither a data layer
to access this information, nor a way to evaluate logic expressions like
this. Most satellite sentences can work without this kind of intelligence,
but it would seem like a worthwhile goal to go beyond and also generate
passages of the main text, or introduce some more meaningful variation to
them.
Furthermore, except for adding more satellite grammars and kernel texts
to a story, there is no way to efficiently extend the text generation system
behind Almost Goodbye. Adding custom logic can only be done by directly

29
3.3. ALMOST GOODBYE RELATED SOFTWARE

modifying the source code, which can get complicated quickly in decen-
tralized environments like internet communities9 .

Summarizing this, Almost Goodbye is, within its scope, a great example for
practically applied Procedural Generation but not a general-purpose solution.

9
When two authors develop extensions independently from each other and a third one wants
to use both, he will manually have to merge them into one consistent patch.

30
 Part II

Development

31
4 | Concept

In the course of this thesis, the software prototype of a custom text generator
has been implemented. This chapter describes the core ideas behind its creation
and illustrates the mindset that governed the process. As a reference to previous
works and hint towards its practically oriented nature, the name Scribe has been
chosen as a working title.

4.1 Project Goal and Delimitation

The goal of this work is to create a general-purpose software framework for ap-
plying Procedural Generation to text, as a collaboration device between human
authors and text generator algorithms.
Turning away from creativity and more intelligent aspects of text generation
algorithms is a pragmatic choice: Systems like Tale-Spin incorporate different
kinds of planning and simulation in order to mimic the reasoning of an author
and come up with a consistent plot but humans are pretty good at that kind
of reasoning themselves, and the capability gap that an algorithm would need
to bridge seems to be quite far. Creative writing is not yet a domain where
Procedural Generation can be applied in a useful and productive way. On the
other hand, focusing on the less creative aspects of writing offers a far better
chance for becoming a reliable collaborator to human authors.
Regarding previous works, a distinction should be made to highlight the most
important differences to the prototype presented here:

No Story Focus: There is no specific focus on stories in particular, although they

can serve as a useful example for meaningful text variation in general.

Not a Plot Generator: Even in the context of storytelling, this work does not
deal with plot generation or variation, as previous works like Tale-Spin
did; but rather with textual representations of stories. The difference in

32
4.1. PROJECT GOAL AND DELIMITATION CONCEPT

terminology here is subtle, but important: While a plot can be considered

an abstract representation of the hard facts, the course of action, a story
is much more than just a sequence of events. It is the concrete presentation
of a plot, or in its written form: A text. It requires writing skills to trans-
form an existing plot into a story, not planning skills. Thus, the prototype
presented here should not be considered a spiritual successor or competitor
of Tale-Spin and its relatives, but as an extension on a different domain1 .

Not Specialized: This work aims for a general-purpose text generation system
that could be used in various ways. It is not tied to a specific use case
like for example Almost Goodbye. It is also not designed for interac-
tive fiction, although it could certainly be used as a key component of an
interactive fiction system.

Not Knowledge-Based: While the generation of factual texts and reports is of

course among the potential uses for a system as conceptualized here, this
is certainly not its only purpose. As a result of this, a mainly knowledge-
based approach as seen in other systems is not deemed favourable in the
context of this work. Even though having access to a distinct data layer is
later shown to be beneficial (Section 6.2), using it as the primary device
for structuring and generating all text content would be out of place for
more creative text genres.

Productive Requirement: The ultimate goal of this effort is to produce directly

usable text, i.e. text that does not require human review and corrections
in order to be used the way it was intended to. Though Almost Goodbye
worked under the same premise, most of the other presented systems did
not.

As its name suggests, Scribe aims to be primarily One who writes focused
on the practical aspect of writing and not being a truly creative author in its own
right, but a diligent helper. Humans may be far better text generators than any
current software system could hope to be, but their throughput is limited; they
cant write thousands of specialized text variants in a reasonable time. This is
where Scribe has its place as a tool for generating variation, expressing data
and assembling human-authored text dynamically.
1
A plot generator like Tale-Spin and a text generator like Scribe would actually make a good
team: Their tasks are mostly distinct and each could contribute some puzzle pieces that the
other can not.

33
4.2. USE CASES CONCEPT

4.2 Use Cases

Leaving concrete system specifications and requirements aside for now, there is a
range of use cases and applications where a non-creative text generation system
like Scribe might prove helpful. The following list is not necessarily represen-
tative of the actual prototype feature set, but can be seen as an example for
potential applications:

Story Variations and Flavours: Authors could write stories with varying text
passages, settings or moods. Important themes and morales can remain
constant, but could be represented differently. Characters could be al-
tered, scenes relocated and props switched out. In an experimental setup,
an artist could even release a printed story, where each books content is
slightly different from the next. In a more grounded use case, fables or
short stories could within limits adapt to the reference frame of differ-
ent cultures, or to different target audiences in general.

Structured Reports: Certain repetitive and well structured types of news arti-
cles could be generated using data about the event other systems have
demonstrated this, for example, by writing experimental articles about lo-
cal soccer matches. By spending less time on writing boilerplate text
that uses a similar structure and phrases most of the time, news authors
could afford to cover smaller events that might otherwise have fallen out
of scope. [30] Likewise, any type of report that is primarily dependent on the
data it seeks to communicate could be automatically generated as well.

Personalized Advertising: When possible, internet ads are often selected with
regard to the specific user that is visiting a site. Text-based adverts could
go a step further and adjust depend on the interests and traits of its current
viewer or depending on the website that currently incorporates them, in
order to blend in and have a higher chance of being perceived as appropri-
ate or useful.

Wikipedia Entries: Online encyclopaedia Wikipedia is facing quite a big task:

Providing all of humanitys knowledge, in all languages. A text generation
system could be a useful collaborator for writing standard paragraphs that
do not require special attention, or even generating whole articles from
abstract data. These would likely follow a very rigid structure and use

34
4.3. SCOPE AND REQUIREMENTS CONCEPT

common or generic phrases, but could still provide a more valuable starting
point for human authors and readers than a non-existent page. [37]

Game Development: Like Almost Goodbye, games have already proven to hold
a viable place for Procedural Generation. They are an interactive medium,
and interactivity can be a tricky thing to handle when all content is pre-
defined and all situations have to be anticipated. Procedural Generation
allows to create content on-the-fly in dynamic situations whenever they
arise. Scribe could be part of that as well either as a software system for
purely generating game texts when needed, or as a partial reference model
for implementing certain types of Procedural Generation systems.

Of course, there is no reason to regard the above list as closed and complete,
but a general tendency already becomes apparent: Most of the above would
leverage a text generation system in order to reduce repetitive writing and
increase the re-use of common text fragments and phrases, adding variation to
otherwise static text, or adapting it to fit various different scenarios.

4.3 Scope and Requirements

Starting from the project goal definition, a set of requirements has been deter-
mined for a first prototype system:

Data Driven: Being designed as a general-purpose system, there should be no

specific reference to any kind of textual content within its source code.
All data that can be worked with needs to be loaded dynamically from an
external storage such as files or databases.

Language Agnostic: If possible, there should be no core dependency on any

specific natural language. As a result, the system cant incorporate fixed
grammar or text parsing rules and needs to rely entirely on the data that
is provided. This would allow the same base framework to be used across
multiple languages.

Simple: Authors of literature and authors of program code are not commonly
known to do each others job very well, so requiring too much programming
skills is not desirable for providing text content. Within reasonable limits,
the data that powers the system should be as easy to read, understand and
write as possible.

35
4.3. SCOPE AND REQUIREMENTS CONCEPT

Predictable: Being a reliable collaborator to human authors requires delivering

predictable results. Each text output should originate from the input data
set in a comprehensible way, so unsatisfying results can be attributed to
a cause, which can then be adjusted accordingly. A system that exhibits
chaotic behaviour would be interesting, but likely of no practical use.

Deterministic: In addition to predictability, even random text variation should

follow deterministic rules, meaning that each output should be reproducible
only depending on input data and a seed2 value. This is a vital requirement
in terms of debugging and analysing specific text outputs if it wasnt pos-
sible to reproduce an output, there would be no way to examine it later.

Modular: For both data storage and overall framework organization, a modu-
lar approach should be favoured over a monolithic one. This should ease
collaboration between multiple authors, as each author can work only on
a small, isolated part of the whole system. It could also allow the creation
of self-contained text libraries that deal with a specific topic and can be
added as a module to existing text projects.

Extensible: Programmers and technically skilled authors should be able to ex-

tend the systems core capabilities without diving into its source code. These
extensions should be modular as well, so they can be shared easily.

All of the above points have been considered in the design of the presented
prototype. As a general course of action, a top-down approach was chosen:
Beginning with a completely pre-written sample text, parts of it have been re-
placed with their own abstract representation, which can dynamically expand
to the original text, or a semantically equal variation. This was repeated in an
iterative process until further abstraction seemed no longer reasonable.
With regard to the overall goal of maintaining a high quality of output text,
this approach had an obvious advantage over attempting to generate new texts
from scratch: Due to the nature of a bottom-up approach, a rough, yet func-
tional sketch of the whole text generation framework would have been required
in order to receive any kind of result and evaluate that result for quality. Since
neither framework architecture nor the degree to which Procedural Generation
2
Random Number Generators (or RNGs) cant really create random numbers, because com-
puters are inherently deterministic. To overcome this limitation, most RNGs maintain an internal
state that is transformed in a complex way to produce a series of pseudo-random numbers. The
initial state of an RNG determines the complete sequence of numbers following it, and is conse-
quently called the seed value.

36
4.3. SCOPE AND REQUIREMENTS CONCEPT

can be applied were known at the start of this project, this seemed very imprac-
tical. On the contrary, the iterative top-down process that was chosen produced
a readable output at all times and could easily be monitored and evaluated after
each change.

37
5 | Architecture

The text generation process as implemented in Scribe requires various compo-

nents to work together and communicate. Giving both information flow and
system layout a clean and understandable structure is a requirement that arises
naturally from goals like maintainable extensibility and modularity. This chapter
aims to give a broad overview on Scribes architecture and some of the decisions
involved in designing it.

5.1 Preliminary
In its essence, the task that needs to be performed by Scribe can be regarded as
the process of transforming abstract information into human-readable text.
This process is usually not a sequential one, neither for human authors, nor
algorithms: Texts are much more than just a sequence of words, and one can
imagine that it is impossible even for the most talented authors to write a bigger
text one word at a time without planning ahead. Human-readable text is not a
continuous stream, but a very structured kind of transporting information al-
beit not in the same sense that computer scientists tend to think about structured
data.
On one of the lower levels, there is sentence structure, as expressed through
their syntax. Paragraphs are not randomly grouped sentences either, but instead
show some resemblance to a thought process, each with an introduction, a core
narrative and a closing part wrapping it up. Multiple paragraphs form sections
or chapters, which together can form even bigger structural elements, and so
on. Although the actual usage and meaning of each of those elements may vary
strongly, they are neither random nor without purpose and likely implement a
certain concept that helps readers to follow through. As a result of the structured
nature of text, an abstract representation will exhibit similar structured elements
and so will a Procedural Generation system for text that has to deal with them.

38
5.2. TEXT TREE GENERATOR ARCHITECTURE

However different or technically adjusted this internal structure may turn

out to be, it carries useful diagnostic information that is lost when flattening
all text output to a mere stream of words in the end. Since Scribe is to become
a tool for text authoring, it should be designed in a way that allows to not only
generate texts, but also to access their internal structure; both for debugging the
generation process and getting a better grasp on its inner workings. In order to
achieve this, the Scribe text generation process is split in two distinct steps: Cre-
ating structured text content with a Text Tree Generator and then proceeding
to transform it into the desired form of plain text using a Text Writer. Both will
be described in the following sections.

5.2 Text Tree Generator

In the first step, a text tree data structure is generated. It contains the entire
output text that was assembled by the system, but retains the structured form
that was used internally during the generation process.
Picking Almost Goodbye as an example, each layer within the tree structure
would resemble the result of an expansion step of formal grammar symbols,
allowing user to see exactly how the final text came to be. Scribe is not based
solely on formal grammar like Almost Goodbye, but, as a spiritual successor to its
approach, does not require a radically different data structure either: The text
generation process starts with a single root node, expands it into a set of child
nodes and then subsequently iterates over each child to do the same. (Fig. 5.1)

Figure 5.1: Formal grammars and similar nested structures can be visualized
as a tree that is recursively expanded. In Scribe, a text tree is built during the
generation process.

The algorithm at the core of the text tree generator is rather simple in con-
cept. For each tree node that is expanded, beginning with the root node, the
following basic procedure is run (Fig. 5.2):

39
5.2. TEXT TREE GENERATOR ARCHITECTURE

1. Query knowledge from the source data, and perform basic reasoning when
requested. The resulting information will be used in the text generation
process.

2. Evaluate dynamic expressions in the current node. This may include in-
corporating information into the text, as well as deciding how to expand
certain symbolic parts of that text.

3. Resolve the symbols that were determined in the previous step, to get a
hold of the text segments to insert in their place.

4. Create child nodes from the previously resolved text segments.

5. Iterate recursively over each child node.

By recursively expanding nodes, the algorithm itself operates with a top-

down approach as chosen during the concept phase. In a first, trivial imple-
mentation, the root node can simply contain the complete, pre-written text with
no processing necessary. Proceeding from there, certain parts of that text can
be replaced with abstract symbols that can be dynamically expanded into child
nodes; in the next steps more and more concrete text sections can be replaced
with dynamically expanded ones, until the text has reached a satisfying level of
variety and flexibility.

Figure 5.2: An illustration of the five steps of the text tree generation algorithm.
The terms Ontology and Templates will be discussed in detail in chapter Main
Components. A concrete step-by-step example of the generation algorithm is
available in Appendix A.

Like it was stated in the concept chapter before, a top-down approach is use-
ful to make sure that the desired high quality of output texts can be reached:

40
5.2. TEXT TREE GENERATOR ARCHITECTURE

As more and more parts of a formerly pre-written text are substituted with dy-
namically generated ones, a sudden loss in quality will be easier to spot during
development than a failure to gain it when trying to create a completely new
text from scratch.
Building up on the concept of recursive expansion, the data structure of a tree
emerges naturally. An advantage of also using it for representing the generated
text itself is the fact that each tree node can again be regarded as the root of
a sub-tree, each of which individually represents a smaller, but more concrete
portion of the output text. This not only allows the system to easily re-generate
or modify only certain segments of a previously created text, but also helps users
to understand its results by providing them in a hierarchically structured way.
(Fig. 5.3)

Figure 5.3: Each text tree node represents a text portion that includes the ones
represented by all of its child nodes.

When looking at the tree of text elements, it also resembles the way a human
author might approach writing that text: Its root node represents the text that
has yet to exist and the seed from which it will grow. Going one layer deeper,
there is a very broad structure of the text, like a separation into introductory
chapters, conclusions and the main narrative all of it still being abstract, but
not quite as much as their parent node. Each subsequent child layer adds detail
and fleshes out the content a bit more, until a leaf node is reached that really
just contains a concrete text fragment. Similarly, a human might start with an
idea, then broadly draft the narrative structure and refine it until he is ready to
concretize it by writing. (Fig. 5.4)
Although the text tree generator does most of the work in Scribe, the task
of text generation is not yet done when it has finished working. After all, it
didnt produce an output text, but an abstract representation of it which can be
subject to further processing and modification, if the user so wishes. A text tree
may be a useful data structure, but it cannot be read by a human; transforming
it into a more accessible format is a step that still needs to be done.

41
5.3. TEXT WRITER ARCHITECTURE

Figure 5.4: A text tree is also the result of recursive concretization of an abstract
origin.

5.3 Text Writer

In the last step of text generation, a visitor algorithm creates a text transcript
of the previously generated tree. Since it already contains all the required text
fragments, this task is solved trivially by iterating over all tree nodes in the cor-
rect order and performing a string concatenation in each step. A text writer
implementation doing this is already included in Scribe.
The systems design, however, allows to handle the transformation into a fi-
nal output in different ways by implementing alternate text writers. Rather
than returning a raw text, the system could also emit an HTML document that
highlights its internal structure, or slice off expanded child nodes at a certain
layer to produce an interactive cross section of it. In a more productive usage, a
custom text writer implementation could generate LaTeX output, E-Mails or just
about any other representation of text that might be required, while using the
internal text tree structure as a hint for text formatting.

42
6 | Main Components

The Scribe prototype can be split into five different areas of responsibility. Each
of them will be explained in this chapter to shed some light on the thought and
design process behind them and illustrated with select implementation details.
First, a generic Template system is described that expands on the approaches
of Almost Goodbye to define pre-written text fragments in a structured way. In
a consequent extension of this, the implementation of an Ontology layer is de-
tailed, which acts as a knowledge backend of the system; for more language-
specific data, a simple Dictionary is introduced next. Tying all of these systems
together, an Expression parser allows to evaluate small logical statements within
Template texts in order to replace them with a suitable text or invoke system
functionality. Finally, a Plugin API opens the framework to external additions in
a modular way, without requiring source code access.

6.1 Templates
At its core, Scribe is built upon a data structure that is called a Text Template, or
simply Template. In the very first iteration of the prototyping process, a Template
consisted of a fixed text block that was forwarded to the output mechanism
which was of course not very Procedural. As a first measure for introducing
text variation, a feature was added to allow Templates to leave blanks within
their text body, which could refer to a different Template that would be filled in
later (Fig. 6.1). In its simple form, this was equivalent to the formal grammars
that were used in Almost Goodbye; each Template being a replacement rule
mapping its name symbol to a text statement.
Still, the question remains why Scribe would use pre-written text in the first
place: In a different approach, a text generation system could try to synthesize
natural language text by itself based on structured data and knowledge about
how to form sentences which would enable it to deviate from pre-defined frag-

43
6.1. TEMPLATES MAIN COMPONENTS

ments and express any kind of meaning1 . However, this would also deprive
users of any direct influence on the text output beyond modifying abstract data
models. Writing quality would be limited by the capabilities of a text synthe-
sis module, which is not desirable with regard to the previously stated project
goals. A big advantage of choosing a grammar-like backend is not only its high
text quality potential, but also the added flexibility and predictability: Since all
output text is derived directly from pre-written and recombined text fragments,
an author has full control over the generated results and can apply his writ-
ing skills in a useful way. Stylistic devices, corporate speech2 or just peculiar
preferences of a single artist can be incorporated into generated texts without
problems.

Figure 6.1: A trivial Template design similar to the formal grammars from Almost
Goodbye.

Each Template is referred to using a name. It doesnt have to be unique and

whenever multiple Templates of the same name are available at once, one of
them is selected randomly (Fig. 6.2). This allows authors to easily increase the
variety of certain text passages by introducing new Templates, without having
to touch any of the existing data, making collaboration between multiple users
much easier.
Looking up Templates by name is done using the Template Database which
is simply an interface for querying and managing available Template definitions.
While it could be replaced by other means of data storage as required, the cur-
rent prototype implementation is an in-memory collection of Templates, which
is loaded from the file system. Following the same modular mindset as the Tem-
plate naming, there is no pre-defined database file, but instead a folder that
will be searched for matching data files, and all of them will be loaded additively.
In order to add a new Template or set of Templates, all that needs to be done
1
In a simple example, an abstract input like Subject Alex, Predicate Eat, Object Apple,
unspecific could be transformed into Alex eats an apple.
2
e.g. preferring certain terminology or wordings over others.

44
6.1. TEMPLATES MAIN COMPONENTS

is adding the file defining them to the appropriate folder; again, this kind of
workflow reinforces the modular nature of the system and aims towards easing
collaboration. (Fig. 6.3)

Figure 6.2: When multiple Templates share the same name, they form a pool
from which one instance is selected randomly.

As mentioned earlier, the Template design presented so far is nothing more

than an elaborately wrapped formal grammar. It suffers from the same limi-
tations as the one discussed in the Almost Goodbye chapter. Maybe the most
important one to address was its poor scalability due to its high demand for
manual organization by the user. Managing the complexity that arises from a
high amount of intertwined text replacement rules is not a trivial task, especially
when each rule can depend on every other rule. (Fig. 6.4)

Figure 6.3: Multiple Templates can form a module, which can be added to and
removed from the Template Database individually.

To reduce the need for these dependencies, Template Parameters were in-
troduced as a feature: When referring to a Template, a special function call
syntax can be used to pass any number of arguments to it. They can be of any
type, including various forms of raw data like numbers and text strings, but most

45
6.1. TEMPLATES MAIN COMPONENTS

importantly Template symbols as well; which can then be expanded within the
text, or passed on with subsequent calls. This allows different Template modules
to communicate with each other without having to know or access their inter-
nals they can instead provide a public API for others to invoke, and exchange
information through Parameters. (Fig. 6.5)

Figure 6.4: With a pure formal grammar approach to Templates, interacting

modules need to access each others internals to exchange information.

Relying entirely on Parameters for information exchange is a great fit for

small, self-contained modules and Template groups. It can get a little more diffi-
cult when dealing with bigger texts with a lot of variable aspects; even the short
story Almost Goodbye would require a two-digit amount of Parameters for stor-
ing all of its context information. Modelling all of this as Template Parameters
and requiring the user to carefully pass them on would introduce a lot of poten-
tial for mistakes and likely be perceived as an unnecessary burden so a second
device for information storage was introduced.
The so-called Template Features take the role of an implicit data context. Each
Template can define Features similarly to Parameters and provide expressions or
queries to determine their value. They can be used in the same way and also
take on a written form within the expanded text fragment, but rather than being
limited to the Template that defined them, their scope is extended to all invoked
child Templates as well: That way, a storys root Template can provide a lot of
contextual information without having to pass it on explicitly, or inadvertently
making it accessible to other modules.
Unfortunately, with Parameters and Features around, text variation became
a tedious task on the user side: Being achievable only through defining multiple
Templates with the same name, all variants had to include a copy of the same
Parameter and Feature definitions, each of them had to be kept in sync after

46
6.1. TEMPLATES MAIN COMPONENTS

Figure 6.5: Template Parameters can help reduce complexity by rectifying access
patterns to flow in one direction only. Here, Module B is entirely self-contained
it doesnt depend on any outside information and can be used in various setups.

changing something. Not only was this diminishing usability, it also required
careful maintenance and would otherwise be an opportunity for mistakes. This
issue was addressed by allowing each Template to define more than one text
body, one of which would be selected randomly when being invoked while
sharing the same context.
To allow a more specialized text generation regarding Feature and Parameter
values, optional Conditions were introduced for both Templates and individual
text bodies: Whether they are considered available in a random selection event
is determined by the result of a boolean expression which can depend on all
Parameters and Features that are present in their scope. This allows users to sup-
ply very specific text fragments, which are used only when their equally specific
conditions are met resulting in a more fine-grained and less generic context
awareness.
In their final design (Fig. 6.6), Templates have become a versatile device
for assembling text from pre-written fragments. They can be defined in a self-
contained, modular way, provide the means for configuration through parame-
ters and scoped data access, as well as variation and conditional specialization.
Unlike the formal grammars from Almost Goodbye, they do not double as vari-
ables or context carriers, but are clearly aimed towards generating text. Using
Parameters, any kind of data can be passed to a Template for evaluation and text
generation, which can then be made implicitly available to subsequently invoked
Templates through Features.
One thing that wasnt detailed so far is that Features also have a second
purpose: They allow Templates to submit queries for retrieving data themselves.

47
6.2. ONTOLOGY MAIN COMPONENTS

Figure 6.6: The final Template design, in a visualized form.

Without a device for data retrieval, Templates would be largely dependent on

a programmer to pass data from the outside via Parameters an unnecessary
detour through program code, from a data-driven perspective. The key to these
user-defined queries is a unified data model and a query language to match.
Both are handled by Scribes Ontology component.

6.2 Ontology
Ontology deals with describing domain-specific knowledge in a structured form.
While Templates only deal with parameterized written text and its variation, an
Ontology layer can provide the added value of both world and domain-based
knowledge.
In the context of Scribe, this solves the problem of information availability:
Consider a Template that provides a text passage describing the early Monday
morning of a character. Following modular principles, the characters name is
passed as a Parameter, so it can be incorporated in the text but that name
alone doesnt tell us much about him. Does he prefer coffee, tea or something
else? Is he a morning person? Does he like his job? The Template which has the
job of describing that persons morning has no idea about all of this, and thus
can hardly describe anything in a meaningful way. Passing all of this information
explicitly via Parameters or Features seems impractical and short-sighted, as it
would limit future text fragments to a fixed set of variables that strictly define
what can be known. The calling site might have all the information, but the
invoked Template has no way of asking for it and would be bound to whatever
kind of useful information was anticipated when first defining it. (Fig. 6.7)
With an added Ontology layer, this is no longer a problem. Rather than
passing a characters name as a Parameter, an Ontology symbol that refers to that
character can be passed; allowing the Template to not only access his name for

48
6.2. ONTOLOGY MAIN COMPONENTS

Figure 6.7: In a pure Template-based approach, world knowledge is intertwined

with structural information and thus, limited by scope and API decisions.

a textual representation, but also any kind of information that is available in the
data layer. This includes relations to other entities and abstract knowledge as
well: Does he have a pet? Whats the name of that pet? Is it a dog? Suddenly,
all of this becomes available for text generation and specialization, where a lack
of information previously would have forced authors to resort to writing rather
generic texts (Fig. 6.8). It also allows a limited degree of reasoning: If the
character has a pet, which is a dog, and dogs tend to wake their owners, it can
be assumed that it would be reasonable for the character to be woken by his
pet3 .
Another benefit of introducing a distinct Ontology layer is the clear separation
of responsibility into writing and knowledge representation. Templates are no
longer required to work on both domains (using facilities that arent up to the
task), but can instead focus on assembling text bodies, while querying data from
a component that is specialized in providing it. In a more metaphorical sense,
when regarding Templates as subset of a virtual authors writing abilities, the
Ontology layer would assume the role of his semantic memory.
For representing knowledge in the Scribe prototype, an implementation of the
Resource Description Framework (RDF) was chosen. Using this specification, all
knowledge is modelled as a set of relations. Each relation consists of a subject,
a predicate and an object, and is typically called a data triple. Consequently,
an RDF dataset consists of any number of triples and can be visualized as a
relational graph of semantic entities. Unlike more traditional, object-oriented
3
Example: If the main character has a OwnsPet relation to a Toby entity, which has an IsA
relation to Dog, and Dog has a WakesUpOwner property, an appropriate Ontology query can reveal
that the main character will be woken by Toby.

49
6.2. ONTOLOGY MAIN COMPONENTS

Figure 6.8: With an added Ontology layer, all knowledge is available to all Tem-
plates. Information about specific entities can be queried using a symbol.

approaches, there is no definition of an objects structure or properties: Instead

of declaring that a Person has a Name, an Occupation and a FavouriteColor and
requiring each instance of a Person to define those values, the triple structure
of RDF leads to a more decentralized approach, where each instance can be
considered an aggregation of individual statements. (Fig. 6.9)

Figure 6.9: The object-oriented approach on the left results in a single, atomic
block of information that can be neither extended, nor split up. Contrarily on
the right, the RDF approach allows both.

This approach is reinforced by the core concept of RDF, where anyone can
say anything about anything [12] a very useful trait with regard to the modular
structure that Scribe is aiming for. When designing the Template Database in the
previous chapter, it was a main goal to keep it open for additions: Each module
could be designed in a self-contained way, but still be extended by any other
module, and introducing new modules can be done simply by dropping them
into a file system folder. For the same usability and maintainability reasons,
the systems knowledge representation layer should aim for a similar degree of

50
6.2. ONTOLOGY MAIN COMPONENTS

modularity and extensibility; and this is where the triple structure of RDF really
pays off: Because each object is an aggregate of independent triples, data can
be organized freely across multiple distinct datasets. For a single object, differ-
ent Ontology modules can contribute different kinds of knowledge in the same
way that different books can deal with different aspects of a single topic. Mirror-
ing the Template components approach, the Ontology Database additively loads
every RDF dataset that is located within a certain file system directory it acts
like a metaphorical bookshelf where individual books can be added and removed
as needed.
For accessing knowledge, an implementation of the SPARQL query language
was chosen. The main application for a query language in Scribe is to obtain a
list of semantic entities that match certain conditions and SPARQL is a text-
based way of expressing those conditions, so the underlying query engine can
process it. Being designed specifically for RDF, a SPARQL query defines a set of
relations that need to be matched; either as simple as checking for a property
value (Lst. 6.1), or involving more complex relations (Lst. 6.2).
 
SELECT ? name
WHERE
{
@ {Param : SomeEntity } P r o p e r t y : Name ? name .
}

Listing 6.1: The core of a simple SPARQL query in Scribe - it retrieves the name
of an entity that is passed via parameter.

In Scribe, queries are typically defined by Templates, which make use of them
for setting default values of Parameters and Features. When a query yields mul-
tiple result values, each Feature or Parameter can be configured individually to
either accept the first result, a random result, or all of them at once. When
no results are available at all, a sequence of fallback queries and values can be
specified, so the Template can still produce a generic text when the required spe-
cific data is not available. To further configure queries, they have access to all
Features and Parameters in their defining Templates scope.
With RDF and SPARQL, Scribes Ontology component is able to provide arbi-
trary amounts of world and domain knowledge, which can be used in the text
generation process. This not only allows to manage information in a much more
structured and maintainable way than with a pure formal grammar approach as
used by previous systems, but also provides limited reasoning capabilities on top
of that.

51
6.2. ONTOLOGY MAIN COMPONENTS

 
SELECT ? entity
WHERE
{
? entity P r o p e r t y : IsA * ? actor .
? actor P r o p e r t y : HasAttribute F l a g : Actor .

? entity P r o p e r t y : IsA * ? prey .

@ {Param : MainActor } P r o p e r t y : PredatorOf ? prey .
}

Listing 6.2: A more complex query. It retrieves a semantic entity that is both
flagged as an actor, and preyed upon by the main actor, which is passed as a
parameter.

An Ontology layer can also serve as a solution to deal with certain gram-
matical aspects of writing: When an actor is provided via Parameter, or chosen
dynamically using a query, text fragments need to address the topic of grammat-
ical gender accordingly. Is it he, she, or it? Is it his arm, her arm or its arm? What
is the plural of wolf ? A correct sentence cannot be formed without knowing this,
and clearly, providing pre-written sentences for each grammatical configuration
is not a viable solution.
Instead, each semantic entity in the Ontology database could specify the
proper wording and Templates could access it dynamically via Ontology query.
The entity John Smith would specify he and his, while Jane Smith would
return she and her. An entity Wolf could specify Wolves as a plural form.
By abstracting away grammatical information, Templates can use the same base
fragments and still produce correct texts independently from an entities gender.
Unfortunately, this approach has several problems: For one, Templates would
get cluttered with a lot of queries, because each single word that depends on
the grammar of an entity would require one. An author would not be able to
form a single sentence without having to think about queries and data access
which is a no-go from a usability perspective. More importantly though, storing
grammatical information is not what the Ontology layer was designed for in
the first place; it should store world and domain knowledge, but not language-
specific lookup tables. Lastly, creating a strong tie between Ontology and specific
languages only for the purpose of looking up grammatical cases seems like an
unnecessary way to introduce synchronization issues in multi-language setups.
It appeared obvious that a different solution should be considered.

52
6.3. DICTIONARY MAIN COMPONENTS

6.3 Dictionary
Dictionaries are a way to store language-dependent data of words and phrases;
and they can provide a solution to the problem of creating grammatically correct
sentences without knowing all their elements up front.
In the domain of Natural Language Processing, a Dictionary usually describes
a database that holds information about words how to use them in a grammati-
cally correct way, and sometimes additional data like synonymous groups, trans-
lations or pronunciations. Using this database, text synthesis and translation
programs are able to form grammatically correct sentences, find translations or
provide text-to-speech services. Manually applying grammatical rules requires
deep knowledge of the language at hand, and also introduces strong ties be-
tween the grammar algorithm and the language it is dealing with neither of
this is considered to be a necessary addition to this work, so a much simpler,
language-independent and entirely data-driven approach was chosen.

Singular Plural
Noun:Wolf wolf wolves
Noun:Sheep sheep sheep
Noun:... ... ...

Table 6.1: An excerpt from a simple noun Dictionary. Each symbol can map to
multiple words, depending on the required form.

In Scribe, a Dictionary maps abstract symbols to a set of concrete word forms.

It can be visualized as a table, where each row represents a word and each
column represents one of its forms. A Dictionary for nouns might have one
column for singular forms and one for plural forms, where a Wolf symbol might
address a row that provides Wolf as singular form and Wolves as a plural
(Table 6.1). Following the same pattern, grammatical gender can be expressed
as a row each, with columns referring to the correct pronoun4 forms (Table 6.2).
When writing Template text fragments, Dictionary entries can be referred to
using their row symbols in the same way that Parameters, Features or Ontology
entities can. Speaking in programmer terms, each form is accessible as if it was
a property of the object that is referenced by the symbol. Of course, directly
using Dictionary symbols in text fragments can only be of limited use when
exactly knowing which word and form to use beforehand, why not write the
4
See the he / she / it example above.

53
6.3. DICTIONARY MAIN COMPONENTS

Nominative Oblique Reflexive Poss. Det. Poss. Pro.

Gender:Male he him himself his his
Gender:Female she her herself her hers
Gender:Neutral it it itself its its

Table 6.2: When referring to a dynamically chosen entity using a pronoun, the
correct form can be retrieved from a special Dictionary that holds gender-specific
word forms.

word itself directly instead? The real advantage of Dictionaries emerges when
allowing to link Ontology entities to Dictionary symbols: They do no longer have
a direct string representation that is inserted into the text, but instead refer to
a Dictionary row; which enables fragment writers to refer to the entities plural
without knowing which entity it will be (Lst. 6.3).
 
Of all the { Var : SomeAnimal . Plural } , one was especially clever .

Listing 6.3: Linking Ontology and Dictionaries allows accessing the correct form
of an entity that is not known when writing the text.

Ontology entities can also be linked to additional Dictionary symbols for

retrieving the correct pronouns and other grammatically relevant word forms.
However, this has the disadvantage of not having a central aggregate of all gram-
mar knowledge for a certain word: As long as the only connection between dif-
ferent types of Dictionary data on the same term is an Ontology entity that is
associated with each of them, there is no convenient way to pass around words
without either losing information or wrapping them inside an entity. Relying on
the Ontology as an aggregation for grammatical data also compromises the pre-
viously introduced clean separation between domain and linguistic knowledge
why should an Ontology entity care about which kinds of different Dictionary
symbols need to be linked to it?
To provide a cleaner solution, Dictionary rows were implemented in a way
that allows them to be linked together: Each of them can specify any number of
symbols referring to different rows, which are then treated as if they were part
of the current one. Instead of having a Wolf Ontology entity link separately
to Noun:Wolf, Gender:Neutral and potentially other Dictionary symbols, all
of them can be grouped in a single Dictionary row (Table 6.3). That way, a one
Dictionary symbol can be used to access all grammatical data of a certain term
(Fig. 6.10).

54
 6.3. DICTIONARY MAIN COMPONENTS

Singular Plural <Links>

Noun:Wolf wolf wolves (Gender:Neutral), ...
Noun:Sheep sheep sheep (Gender:Neutral), ...
Noun:... ... ... ...

Table 6.3: By directly linking the Noun:Wolf row to the grammatical one of
the appropriate gender, it inherits all gender-specific data and can access the
appropriate word forms as if they were defined directly.

Overall, this rather simple Dictionary approach to handling grammar may

seem primitive compared to other NLP systems, but this simplicity is also its
main advantage: As nothing more but a two-dimensional mapping mechanism,
it isnt tied to any specific language, or even grammar at all. In alignment with
the overall data-driven goal of the system, it depends entirely on user-defined
content, and neither rows nor columns are fixed in any way, providing a high
level of flexibility. Writers can define Dictionaries for any type of word and stay
in complete control over the exact shape of all generated sentences; there is
no sophisticated grammar algorithm and thus, one system less to worry about
failing and becoming a roadblock in that one special case. All grammar in
Scribe is entirely human-made it just provides the means to use it in a dynamic
situation.
 
{ Var : SomeActor } opened { Var : SomeActor . PossDet } eyes to a sharp ringing noise that -
tore through the comforting silence of { Var : SomeActor . PossDet } room . { Var : -
SomeActor . Nominative } reached out for the alarm clock .

 
John opened his eyes to a sharp ringing noise that tore through the comforting silence of his
room. He reached out for the alarm clock.

 
Mary opened her eyes to a sharp ringing noise that tore through the comforting silence of her
room. She reached out for the alarm clock.

Figure 6.10: Using linked Dictionary rows, all grammar information is accessible
with only one main symbol that can be accessed as if it had defined all forms
itself. This allows to conveniently write texts without previously knowing the
gender of certain entities.

In a later iteration, Dictionaries were expanded to also support the definition

of synonymous word groups: When accessing the group symbol, the Dictionary
would choose one of its synonymous terms at random. This was not at all a vital
feature, but an easy way to add a useful mechanic for writers, so they can let the

55
6.4. EXPRESSIONS MAIN COMPONENTS

system decide which word to choose for a certain entity while at the same time
maintaining grammatical correctness as described above.
Aware of the projects modularity goal, the Dictionary component also follows
in the footsteps of Ontology and Templates: Users can define any number of
Dictionaries, which are additively loaded from a file system folder. This approach
was successfully used in previously described main components for creating a
modular interface to user-generated content, and there was simply no reason to
deviate from that path, as Dictionaries are inherently modular anyway.

6.4 Expressions
Expressions are instructions for processing a set of values to produce a new value.
In Scribe, they play a vital role in connecting human-written text to the rest of
the system.
In the very first Template implementation, a text fragment was nothing but a
completely static block of text: It would simply be passed to the system output
as is, and there was no way to insert variables or other Templates at specific
locations. While conceptualizing these first steps and adding more and more dy-
namic text generation tools to the system, it quickly became obvious that simple,
generic % markers like Almost Goodbye used wouldnt be enough: They solved
the issue of knowing where to insert generated text, but not what kind of text
to generate which was not a problem in the specific and limited scope of a
short piece of interactive fiction, but it certainly would be in a general-purpose
environment.
 
His favourite cat was { Var : FavouriteCat . Age } years old .

Listing 6.4: Similar to accessing word forms from a Dictionary in Fig. 6.10, it
should be possible to access Ontology entity relations to retrieve data as well.

To the very least, each marker in the text should be able to specify a symbol
that refers to the Template or variable value to insert. This first setup already
allowed users to access most of a Templates feature set, simply by allowing them
to specify which kind of content to insert, and where to insert it in the text. It
was not able, however, to account for accessing different word forms using the
Dictionary component, or a more in-depth usage of the Ontology layer: Both of
them require the user to specify not only the name of a target entity, but also
the name of the member to access (Lst. 6.4). Furthermore, Templates that

56
6.4. EXPRESSIONS MAIN COMPONENTS

expose parameters require some way to specify their value when referring to
them, similar to a typical function call in a programming language. Lastly,
there needs to be some way to not only access Ontology data, but also process it
like checking whether one character is older than another in order to get the
most out of it.

Use Case Example Case Syntax Requirement

Randomized Content %. Insertion Markers
Variable Values Var:FavouriteCat Identifiers
Child Templates Template:PlotTwist Identifiers
Template Parameters Template:PlotTwist(A, B, ...) Invocation
Dictionary Lookup Var:SomeAnimal.Plural Member Access
Ontology Access Var:FavouriteCat.Age Member Access
Conditional Text Var:GuyA.Age > Var:GuyB.Age Logical Operations
Inferring Data Var:GuyA.Age Var:GuyB.Age Arithmetic Operations
Var:GuyA.Age + 10 Literals

Table 6.4: An open list of different use cases for expressions in text fragments.

With arithmetic and logical operations, function calls and member access
already on the growing requirement list, the stretch from sophisticated text in-
sertion markers to full expression evaluations became smaller and smaller. So,
rather than extending the initial support for symbol markers, they were re-
placed entirely with a custom expression language that is able to provide both
the required basic features of text insertion, as well as more complex opera-
tions. Its syntax and semantics were specified directly in source code, using a
library called Sprache [34] , which does the low-level work of parsing and trans-
forming any provided input text accordingly. Depending on the specification that
is passed to the library, its parsing output can be any kind of structured data
like the abstract representation of an XML document. In Scribes implementa-
tion, the resulting data structure is an expression tree [35] that can be compiled
into a method at runtime, which can then be executed in order to evaluate the
expression. (Fig. 6.11)
Both of these processing steps parsing and evaluation require a certain de-
gree of context information to work. When parsing an expression, identifier sym-
bols can mean different things depending on where that expression is located:
An identifier like Var:Actor can refer to one variable when used in Template A,

57
6.4. EXPRESSIONS MAIN COMPONENTS

Figure 6.11: The Expression Parser transforms text expressions to executable

code. When the text generation process encounters an inline expression, it can
invoke its compiled form to evaluate it.

but a different one when used in Template B, depending on the scope of each.
From an implementation perspective, this can be solved by passing the required
context data on to the expression parser via parameter; but since it would re-
quire either anticipating what kind of context data is needed, or blindly offering
all of it, this could later turn out to be a poorly maintainable choice. Not only
would the exact specification of each context parameter affect the way in which
both the parser and the environment invoking it need to be implemented, it
would also make adding, removing or changing these parameters more difficult.
(Fig. 6.12)

Figure 6.12: Passing all context information via parameter ties parser and envi-
ronment strongly together. This can make maintenance more difficult, especially
when having multiple different environments to support.

To prevent this, an interface for querying specific context data was defined.
By passing only the interface on to the expression parser, it remains entirely
decoupled from its environment while at the same time being able to access
context information as needed. A setup like this also makes it easier to use the
expression parser in different environments - like inside of Ontology queries, or
as conditional statements for Templates and individual text bodies. (Fig. 6.13)

58
6.4. EXPRESSIONS MAIN COMPONENTS

Figure 6.13: Introducing an interface that allows the parser to retrieve context
data from its environment can help providing a clear separation.

When evaluating an expression by invoking the compiled method, there is a

similar need for context information: Resolving variables and object members to
their values, as well as performing invocation operations is strongly tied to Tem-
plates, Ontology and Dictionaries, as it potentially needs to query each of them
yet, it is not the responsibility of the expression parser to handle any of these
and it certainly shouldnt be for maintenance reasons. But without knowing or
having access to these domains, how should it be able to create and compile an
expression that does?

Figure 6.14: Similar to the IParserContext interface from Fig. 6.13, the
IEvalContext interface provides a clean separation between expression eval-
uation and the context they are evaluated in.

Again, defining an interface for providing the required functionality turned

out to be a viable solution: Even without having access to an instance of said
interface, just knowing how it is defined allows the parser to use it in the ex-
pressions it emits. Consequently, the compiled methods that result from this
will require a concrete interface reference, which can then be provided by the
environment which is evaluating the expression.
Overall, the expression language of Scribe resembles programming languages
like C++ and C# , but supports only a small subset of their syntax and functional-
ity. It wouldnt add much value to deal with both of them in full detail, but there
are some points that might be worth looking at:

59
6.4. EXPRESSIONS MAIN COMPONENTS

Expressions Only: Contrary to real programming language, Scribe expres-

sions are really just that: Expressions. A set of instructions to process a
set of input values to produce one output value. There are no blocks, no
loops, functions or any other means of introducing a more complex pro-
gram flow. As a rule of thumb, expressions can do what the right-hand
side of an assignment operation in C# or C++ can do but nothing more.
They are not designed to allow users to write complex programs, but to
provide a way for quickly evaluating small bits of logic, embedded directly
within text, queries or other environments. And, of course: To provide a
language for inserting Templates or variables into specific locations within
text fragments.

Conditionals: Although there is no complex program flow in Scribe expres-

sions, there is a ternary conditional operator, as known from C# and C++ :
condition ? whenTrue : whenFalse. Using this form allows to insert
conditional passages and values without the requirement for creating a set
of wrapper Templates to achieve this goal.

Return Values: When an expression is evaluated, a return value is produced.

By default, this value is simply converted into a string representation to
be incorporated into the final text result. However, if the returned value
happens to be a reference to a Template, it will be invoked. That way,
simple insertion markers like Template:PlotTwist from Table 6.4 can be
expressed naturally, and also be incorporated into logical operations and
conditionals.

Identifiers: All identifiers in the Scribe expression language are treated as liter-
als, which are resolved at runtime. That way, they can be passed around
or stored in variables, similar to pointers in C or C++ . Since every object
is referred to via identifier, this works not only with Templates, but also
Dictionary entries, Ontology entities and even with variables themselves.
Being able to handle all kinds of objects that way makes it a lot easier for
Templates of different modules to communicate with each other because
they have the means to exchange the building blocks on which the whole
text generation system is based upon.

Short Access Patterns: As detailed in the Dictionary section, Ontology entities

are relationally linked to Dictionary rows, each with a set of different word

60
6.4. EXPRESSIONS MAIN COMPONENTS

forms to be accessed. Simply inserting the name of an entity that is stored


in a variable would require the following expression: 
Var : SomeAnimal . P r o p e r t y : Text . Singular


First, the variable is accessed using its identifier Var:SomeAnimal. It con-
tains an Ontology entity, so accessing any of its member-like relations re-
quires another identifier. In this case, the Dictionary link relation is ac-
cessed, which is identified using Property:Text. But addressing a Dictio-
nary row alone is not enough: A column name needs to be specified to
complete the lookup operation, in this case Singular. Evaluating this ex-
pression, the proper singular form of the stored Ontology entitys name will
be inserted into the text but the code required to achieve this is obviously
far too long. To counter this, some (optional) shortcuts were implemented:
When omitting the Ontology relations prefix, all relations of the entity
will be searched for one that matches the accessed member name. This
is a more expensive operation than directly accessing the relation with a
fully known identifier, but Scribe is not a very time-critical application and
losing time in the scale of less than a millisecond seems like a reasonable

tradeoff. 
Var : SomeAnimal . Text . Singular


In the very specific case of simply trying to find a suitable text represen-
tation of an Ontology entity, the explicit mention of the Text relation can
be omitted as well. Whenever the expression evaluation encounters an
identifier operation that doesnt work like trying to access a non-existing
Singular Ontology relation it is assumed that instead of the Ontology

entity itself, the user is actually referring to its linked Dictionary row. 
Var : SomeAnimal . Singular


Additionally, Dictionaries always have a default column, i.e. word form
that will be accessed when no specific other form is specified. For nouns,
it makes sense to specify their singular form as a default, which allows an

even shorter access pattern. 
Var : SomeAnimal


For the plural form, there still needs to be the explicit Plural access. To
improve the reading flow of the text fragments that incorporate these ex-
pressions, column aliases were introduced to Dictionaries, allowing them

61
6.5. PLUGIN API MAIN COMPONENTS

to specify an intuitive short name for each word form. The plural of a noun
can be shortened simply with an s, resulting in a very minimal to request
the proper plural form of a variables Ontology entity:
 
Var : SomeAnimal . s

Overall, Scribe expressions are not intended to be an actual programming

interface and shouldnt be judged by the same measures as one. They are de-
signed as an elaborate way of tying together the systems individual components:
Serving as anchors to insert Templates and variables at the right spot within the
text, as well as allowing to access knowledge and react to different inputs. There
are situations, where users might require the help of an actual algorithm to per-
form a complex task that wasnt anticipated during the design of Scribe but
expressions only take a minor role in this.

6.5 Plugin API

One of the main goals in designing Scribe was to achieve a certain degree of
extensibility, contributing to making its general-purpose usage feasible. Since
there is no practical way of anticipating all requirements, needs and problems
for all potential use cases, providing a modular, non-intrusive way to extend the
software is vital to its usability - not only because of the previously discussed
problem of merging different custom source code versions, but also because re-
quiring its users to understand and maintain Scribes original source code doesnt
make a good learning curve in the first place.
There is more than one way to address this issue, and different setups require
different solutions. The text generation library Rant [29] approaches this topic by
providing a complete custom programming language that can be used within
text fragments, allowing users to implement arbitrary algorithms directly as part
of their texts - which is a powerful concept, but not without its drawbacks:

Mixing Algorithms and Content makes it harder to re-use previously devel-

oped logic with a new set of content and, when applied poorly, can obstruct
the view on both due to a lack of proper separation. This is not an easy
problem to avoid, but providing the means for full in-line programming
could encourage extensive usage with the potential to be heavily inter-
twined with concrete text passages and content.

62
6.5. PLUGIN API MAIN COMPONENTS

Specialization using a domain-specific language can improve efficiency by strip-

ping away unnecessary requirements and adding shortcuts to common
functionality, but it also complicates knowledge transfer: Getting accus-
tomed to an entirely new programming language takes time, and even
more so if that language only exists in its own niche. Relying on existing
and widely known programming languages instead allows users to simply
apply their previous knowledge to the new context.

Maintenance efforts are putting a high workload on the task of creating a new
programming language and also introduce a second point of failure: Does
my script produce the wrong output because there is an error in my script,
or because there is a problem with the language it is written in? Especially
in the context of this thesis, implementing and maintaining a custom pro-
gramming language would carry a high potential to be a significant time
sink.

As an alternative concept, rather than allowing user to write program code in

a domain-specific language, a comparatively easy way to implement an extension
mechanism is by providing a plugin API. The core idea behind this is to allow
users to write a regular C# library, which references the Scribe framework and
exposes classes that implement a certain interface from it. Scribe will then load
the compiled Assembly of these libraries at runtime and use said interface for
communicating with them (Fig. 6.15).

Figure 6.15: By referencing Scribe at compile time, a plugin has knowledge

about the interface it needs to implement - which Scribe can use to load and
execute unknown code at runtime.

The way a plugin interface is specified will also define the contact points
between core and plugin; where exactly it is possible to add custom functionality
and how it is done. In the current Scribe implementation there are two of them:

63
6.5. PLUGIN API MAIN COMPONENTS

Custom Methods can be defined by writing regular C# methods within a spe-

cialized host class and flagging them with a ScriptMethod attribute that
defines an identifier to be used for calling that method from within Scribe
expressions, as seen in text fragments, conditions and ontology queries.
As a typical example, the Scribe framework itself defines all of its mathe-
matical functions in a plugin - Min, Max, Abs, Round and many others are
made available through the custom method definition mechanic of Scribes
plugin interface. Similarly, users can define their own functions and add or
remove them as needed.

Postprocessing of a fully generated text tree is an optional step that can be

defined by implementing a specific class interface. Any amount of post-
processing steps can be introduced by plugins and are executed in a well-
defined order using simple priority sorting. Scribe itself defines various
postprocessing steps in different plugin modules that take care of simple
tasks like removing duplicate whitespace characters from the text, enforc-
ing capital casing at the beginning of sentences, or replacing the English
article a with an where followed by a vowel.

While there surely would be a lot of opportunities to expand on this, even

with only these two very basic contact points, it has been possible to successfully
overcome all of the (sometimes unplanned for) challenges that arose through
the evaluation phase of Scribe.

64
 Part III

Evaluation

65
7 | Results

In the concept phase of this work, a number of potential use cases for Scribe
was defined, most of them falling into at least one of three categories: Using
text generation as a way to automate writing repetitive text segments, using it
to introduce variations to otherwise static text, and finally leveraging it as a tool
to deal with the complex task of adapting a text according to multiple variables.
With these categories in mind, a set of three test cases has been determined in
order to evaluate the prototype that was created in the course of this thesis:

1. Telling the same story with different actors can illustrate Scribes ability
to complete a text fragment with variables, including both finding suitable
solutions for variable constraints and properly reacting to them within the
text for example, adapting to different genders of an actor.

2. Changing the mood of a story should determine whether Scribe can be

used to introduce consistent variation to a story aspect that spans the entire
text.

3. Generating short factual texts from data serves the purpose of testing
Scribes usefulness on a different domain than varying fictional stories,
where text is written and consumed primarily to deliver information, rather
than for its entertainment value.

For each of these test cases, a concrete example has been chosen in order to
add an element of practicability. While in some instances, further consideration
was necessary to find an optimal angle on how to use Scribes facilities, no more
system changes were implemented at this point. All following discussion deals
purely with different ways to use the previously described text generation system
and should not be regarded as a continuation of development, but rather as an
exemplary display of potential workflow elements.

66
 7.1. SAME STORY, DIFFERENT ACTORS RESULTS

7.1 Same Story, Different Actors

The Wolf in Sheeps Clothing is a widely known fable in which a wolf disguises
itself as a sheep in order to prey at the flock without being noticed conveying a
morale of not always taking things for face value. Aside from cultural preferences
and associations, there is no fundamental reason why this fable would need to be
based exactly on a wolf and a sheep, but rather two kinds of actors, one of which
is preying upon the second and using deceit to get past their defences. This
abstraction can be used to select the stories pro- and antagonists dynamically
from a set of known actors, while satisfying a set of constraints that bind them
together.
 
A Wolf found great difficulty in getting at the sheep owing to the vigilance of the shepherd and
his dogs. But one day it found the skin of a sheep that had been flayed and thrown aside, so it
put it on over its own pelt and strolled down among the sheep.

The Lamb that belonged to the sheep whose skin the Wolf was wearing began to follow the
Wolf in the Sheep's clothing. So, leading the Lamb a little apart, he soon made a meal off her
and for some time he succeeded in deceiving the sheep, and enjoying hearty meals.

Appearances are deceptive.

Listing 7.1: The version of The Wolf in Sheeps Clothing that was chosen as a basis
for this test case. Source: [44]

When inspecting the original text for fragments to be determined dynami-

cally, it becomes obvious that there are more than only the two main actors to
be replaced here:
The Wolf is considered the main actor, which can be chosen freely using a query
to the Ontology layer. The only constraints that apply here are merely
filtering out completely invalid choices, like an entity that isnt even a living
thing but an abstract concept, or one that does not qualify as an actor at
all. These kinds of constraints apply to all actors equally.

The Sheep is considered the secondary actor, which, in addition to the basic
actor constraints, is required to be an entity that the main actor is preying
upon.

The Lamb is a special version of The Sheep, which is led astray and devoured by
the main actor. It is not vital to the story and can be described simply as a
child or young of the secondary actor, in case there is no special entity in
the database for filling that role.

67
 7.1. SAME STORY, DIFFERENT ACTORS RESULTS

The Shepherd is, on an abstract level, the protector of the secondary actor, who
watches over them and keeps the main actor at bay. This role is required
for the original version of the fable, but, in cases where the Ontology layer
does not provide such an entity, can be skipped by assuming that the group
of secondary actors simply protect or watch over themselves.

The Dog is an optional actor that serves the role of the protectors helper. Sim-
ilar to The Lamb, its appearance is non-vital and more of a decoration to
the overall story.

The first three roles are determined in the stories root template, as they will
be used throughout the story independently from any variation that might be in-
troduced. While the young version of the secondary actor remains optional due
to its paraphrase fallback, main and secondary actor themselves are required for
the story to exist in the first place. Stories where either of those roles remain un-
filled will be discarded by the system at an early stage and trigger a backtracking
step to find a valid solution. On the contrary, the protector roles are optional
for a different reason: When no suitable protector can be found, text fragments
that require one simply become unavailable, leaving only the ones where it is
assumed for the secondary actor to not have a protector. Rather than requiring
certain entities to exist, the text is shaped based on their availability.
 
Once upon a time, there was a hyena who liked to eat antelopes, yet had a hard time getting at
them, because they were very careful and kept their eyes open for intruders. But the hyena
was patient and followed the antelopes in some distance, day in and day out, collecting old
parts of their fur that they had left behind, and making a cloak out of it.

After a while, an unsuspecting antelope ended up separated from the rest, and alone with the
hyena, who soon swallowed it.

Don't judge with your eyes alone.

Listing 7.2: A generated version of the fable with different actors. Note that,
since there is no protector defined for antelopes, the text has been altered to not
require one.

In addition to choosing the five main actors dynamically using Ontology

queries, a second layer of variation was introduced by defining multiple pos-
sible text bodies and templates for each structural element of the story. During
text tree generation, one of each will be chosen at random, leading to various
different combinations of text fragments and thus, a text with a level of variation
that is only limited by the number of defined text fragment options.

68
 7.1. SAME STORY, DIFFERENT ACTORS RESULTS

Difficulty arises from the fact that, while the fables morale only requires a
very abstract setup, its concrete form includes various details depending on its
choice of actors. The secondary actor having a protector or not is an example
of the more significant ones, but it also shows in details like the construction
of a disguise: Finding the pelt of a flayed secondary actor only makes sense if
that actor both has a pelt in the first place and is a domesticated animal that is
flayed or sheared. At the same time, other kinds of secondary actors might offer
different ways to obtain a disguise that might be described instead.
 
A fox found great difficulty in getting at the chickens, as they always keep their eyes open for a
threat. But the fox was patient and followed the chickens in some distance, day in and day out,
collecting old parts of their feathers that they had left behind, and making a cloak out of it.

Without being noticed, one of the chickens' children started to follow the fox in chicken's
clothing and was swallowed soon after.

Don't trust strangers.

Listing 7.3: Another generated version of the fable. In this case, chickens are
preyed upon who do not have a pelt like the original actor. Instead, the fox
ends up collecting their feathers.

The process of concretizing an abstract idea is a creative one that feels natu-
ral to humans, but is yet very difficult (if not impossible) to solve for a machine.
Scribe itself has no way of coming up with new ideas, or even determining that
picking up the flayed pelt of a chicken doesnt make any sense. It doesnt under-
stand the text it is writing, but simply evaluates and executes the patters that are
defined by previous human input. Consequently, its problematic inability to be
truly creative is solved by not trying to be creative at all and instead leveraging
human creativity wherever possible. When describing the act of obtaining a dis-
guise in Scribe, a viable practice is to provide various conditional text fragments
with specific descriptions1 , and a generic text template2 as a fallback. While the
generic variant ensures that there is a valid text description in all cases, specific
versions can paint a more lively picture of the concerned plot point whenever
their conditions are met.
Although this was only the first test case for Scribe, it quickly became clear
that its Expressions and Dictionary components were able to serve their purpose
1
For example a specific text fragment for Animals that are flayed, or one for Animals with
some form of fur or coat.
2
For example simply not detailing how exactly the disguise is obtained and just stating that
this is the case.

69
7.1. SAME STORY, DIFFERENT ACTORS RESULTS

as intended and provide an easy way to integrate dynamic variables into the text
across different grammatical situations. As detailed previously, grammar was
not applied by the system in any way, but the human author writing the text
templates, and using member and dictionary lookups on variables in order to
invoke different forms of an entities text representation.
However, a rather fundamental issue became evident as well: Re-using a
text fragment in different contexts and thus reducing human workload is only
possible if that text fragment is generic enough to be equally applicable in all
of them; but especially in creative texts, this genericness can easily be perceived
by the reader as lifelessness, lacking the carefully arranged detail one would
expect from a human-written text. Purely factual texts, like business reports or
corporate E-Mail dont have this problem to the same degree, because a certain
sobriety is expected from them. As far as more creative text for recreational
reading is concerned, this problem can be countered by preferring a high number
of specific text fragments over a few generically varied ones. (Fig. 7.1)

Figure 7.1: Text generation based on pre-written and adapted text fragments
puts the author in a position of conflicting goals writing generic text to re-
duce workload and writing specific texts to improve quality. The ideal balance
between the two depends on the nature of the written text.

A not as technically inclined but interesting observation was made when it

came to varying the fables morale in the end: In the original text, the final sen-
tence concluded that Appearances are deceptive, which can be varied using
alternate wordings of the same or similar statement: Dont take things for face
value, Dont judge with your eyes alone and others. However, it turned out
that it was easily possible to replace the morale with an entirely different one
while still retaining consistency with the story that was told before. Dont trust
strangers stays mostly in line with the initial statement, but focuses on a differ-

70
 7.2. CHANGING THE MOOD RESULTS

ent aspect, while One who has a goal will find a way completely switches from
the sheeps cautious perspective to appraising the value of the wolfs cleverness.
The likely reason this drastic change still works so well is because the human
reader does all the work by reinterpreting the text according to the morale it
tells him to convey although nothing changed in the text itself, it appears that
the readers view of it can be shifted easily using only a few anchoring cues. This
could have implications not only in the domain of procedural story generation,
but also when it comes to creating variations of factual texts and reports that
present the same content with different sentiments and shifted views. However
interesting, expanding on this would exceed the scope of this work and will be
left to the investigation of future works.

7.2 Changing the Mood

Scarecrow is a short story by Luke Thompson, written from the perspective of a
fictional person who hired someone to scare away birds from a plot of land. The
story isnt really conveying any specific message, but delivers a mostly uncom-
mented impression of an occurrence.
 
So we hired this scarecrow. Yincent. My sister saw the ad in the Cambrian News and said 'Is he
for real?' so I called him up to see. We met in the afternoon and in the evening I called him
again to say the job was his. This was Sunday.

I showed him the plot and we watched at the kitchen window so he could see the birds he was
meant to scare. He took it all in, but I could see his hands shake. I said 'Are you nervous?' He
said he had Essential Tremors and he probably drank too much, but the shaking helped his
work. 'Makes you look real,' he said.

Before dawn every morning I let the cat out, and I see Yincent setting up. I think it's nice he's
there and I wave, and he waves.

Listing 7.4: The original text of the short story Scarecrow. Source: [45]

Without apparent hidden meanings or layers, and a slightly positive mood,

Scarecrow was chosen as a minimalistic proving ground for two things:

1. Creating a story template with a fixed plot that can be procedurally varied
to be delivered with either a positive or a negative sentiment. Where the
first test case could theoretically afford to operate mostly on an easy to do
insert actor here replacement basis, this is not possible here. Rather than
being a part of the text itself, the overall sentiment is a meta information

71
 7.2. CHANGING THE MOOD RESULTS

that emerges from the sum of all content and wording, as well as the in-
teraction between the readers expectation and writers intention. Changing
the mood does not require to replace core facts, but to alter the way in
which they are presented.

2. Extending an existing story template with a new one in a modular way. This
point is about testing whether a viable modularity can be achieved when
designing procedural story templates and data structures. In theory, would
an author be able to write a distinct set of templates and data that adds
a different spin to the story, without modifying any parts of the existing
setup?

The first one was achieved by defining an environment variable that stores
the overall mood value positive or negative and allows all of the stories
text fragments and templates to react to it by describing things either in a good
or in a bad light. First results dont indicate fundamental problems with this
approach and it should be possible to scale it accordingly when using it on bigger
text bodies. Of course, reducing a stories mood to a simple, constant, binary
representation of positive or negative is hardly an adequate practice for more
complex texts, but beyond the scope of a simple proof-of-concept test case, there
is no reason not to use a more fine-grained adjustment when needed.
 
So we hired this scarecrow. Frederick. One of my friends had this cousin who offered to do the
job and I thought it sounded like a weird job to do at all; so I called him up to see if he was
serious. He came around before I drove to work to meet up and I just gave him the job right
away.

I walked him around the plot and we watched from a distance so he could see the birds he was
meant to scare. He glared at them with a strange look in his face, and I could see his hands
shake. I said 'Are you nervous?'. He didn't answer at first, and with his empty eyes stared into
oblivion; then snapped out of it and said something about 'Essential Tremors' and drinking too
much.

When the cat wants out every morning, I see Frederick setting up. Our cat doesn't really want
him here, and I'm having second thoughts about this as well.

Listing 7.5: One of the negative versions of the story Scarecrow.

One way to extend an environmental mood setting would be to provide mul-

tiple continuous variables instead of a single boolean one. Each of them could
define a scale on a single emotional axis, such as aggressiveness, apprehen-
sion or acceptance, or domain-specific settings such as product-sentiment or
favouring-solar-energy. Depending on their value, some text fragments could

72
7.2. CHANGING THE MOOD RESULTS

be statistically favoured over others, or certain words could be replaced with

synonyms that match the specified scale more closely in the given context. The
result would be a text that creates a different perception of its main topic, based
on outside settings. However, the degree to which this process can be automated
in the long run would be crucial to its viability and aside from flavouring works
of fiction, a practice of biasing a factual text depending on the person who reads
it would seem rather questionable.
Another issue with sentiment flavouring of texts is the fact that emotional
hints are usually bound to a specific context or subject: Intuitively speaking,
there is only very rarely such a thing as a mood setting that spans an entire
text; except if that text is specifically dealing with a single subject, to which
the sentiment is tied. The previously discussed introduction of a set of domain-
specific scale variables can help with keeping sentiments in context, but would
likely result in a badly maintainable jungle of variable definitions when the
generated text grows and extends its scope beyond a few core topics. A simple
way to provide a cleaner structure for this kind of data would be to move it
into the Ontology layer and store it associated to certain entities, which can be
accessed when needed, but remain invisible otherwise (Fig. 7.2). One could even
go so far as to create a procedural work of fiction, where the main characters
traits are part of the Ontology layer as well and have an impact on how parts
of the plot are described, or even on how small parts of the plot develop.

Figure 7.2: Providing detailed scales like domain-specific sentiments or fictional

character traits can lead to an abundance of global environment variables. Stor-
ing them in the Ontology layer instead is a more structured way to handle this
kind of data.

Tinkering with ideas from the realm of plot variation, it is important to keep
in mind that unlike TaleSpin and its relatives, Scribe has no way of performing an
actual character or world simulation. All dynamic variation would be the result

73
 7.2. CHANGING THE MOOD RESULTS

of an authors careful planning, and the combinatoric complexity that arises from
assembling a lot of dynamically adjusted small parts to a single whole. Whether
or not a system like Scribe could be sufficiently used to create a procedural text
from the scale of a book is yet to be determined and while a reasonable scepti-
cism is indeed advisable on that matter, the test cases that have been performed
so far have not yielded a fundamental reason to reject this possibility.
Shifting focus to the domain of extensibility, the second test case was per-
formed in order to find out whether it was possible to extend existing procedural
text template in a non-intrusive and modular way. Aside from the previously
existing positive / negative mood setup, would it be possible to introduce a dif-
ferent theme to the story, without the need for modifying existing structures? As
an example, an alternative version should be introduced where the scarecrow is
perceived as a ghost or supernatural entity, in order to give the story a different
spin with a touch of mystery. The example case was consciously chosen to in-
troduce a new element to the text, without the comfort of being able to simply
re-use existing structures with additional data.
 
We hired a scarecrow. Gudrun. One of my friends had this cousin who offered to do the job
and I thought it sounded like a weird job to do at all; so I called her up to see if she was
serious. She came around noon to meet up and I just gave her the job right away.

She seemed like a nice girl, but I somehow couldn't really focus on what she said. Her words
just slipped my mind. I walked her around the plot and we watched from a distance so she
could see the birds she was meant to scare. None would show up, which was a little strange.
'They know when I want them to leave', she said, 'and I don't want them to bother you'.

Gudrun is setting up every morning now, but most people don't notice her. Only our cat does.
Scout was always a bit anxious around the wide open field, but since we have Gudrun, that
somehow seems to be fine. I think it's nice she's there.

Listing 7.6: The basic story Scarecrow in a positive mood, but using ghost
story templates that were introduced using a second module.

This is where the early considerations on regarding the systems modularity

in both API and data structure can demonstrate their value: Since text templates
are loaded from whichever files are located in the template directory and all text
fragments are considered equally, it is easy to introduce new variants to existing
texts. Consistency, however, remains a concern: When a ghost story fragment
is as likely to occur as a regular section, there is no guarantee that segments
of the main part are properly introduced by matching themes in the beginning.
Text templates need to be designed in a way that allows the algorithm to make
an initial decision on certain aspects of the story, and then stick to it. One

74
7.3. GENERATING FACTUAL TEXTS RESULTS

way to achieve this using only existing capabilities of Scribe is to define an

environment variable when a certain theme is present, which can be checked for
in each templates precondition. By simultaneously setting the statistical weight
of a conditional template to a very high value, the system can be forced by the
author to use a certain template version when the matching theme flag is set.
(Fig. 7.3)

Figure 7.3: In order to give an existing story a new spin, its root Template can
be wrapped in a dummy which sets an environment variable. Conditional child
Templates can then check that variable in order to override existing text frag-
ments with different ones.

A problem with this approach is that it works by replacing templates from

the base text entirely - which, depending on the granularity of the templates, is
not as non-intrusive as would be desirable with regard to the initial goal of mod-
ular extensibility. If one of the replaced templates is later changed in its base
version, none of these changes will have an effect as long as the replacement
template is active; and writing any of those templates also requires deep knowl-
edge of their respective base versions, creating a strong connection that is prone
to change-induced failure. With a full override as the only way of introducing
structural changes to existing base texts, the gained advantage of separating dif-
ferent themes of a text into different modules remains limited.

7.3 Generating Factual Texts

This last test case had the goal of generating text with the sole purpose of de-
livering information in a human-friendly way. The text in this case acts as a
presentation layer to an underlying data set, which needs to be communicated
appropriately. Examples of these texts include corporate E-Mails, reports, ency-

75
7.3. GENERATING FACTUAL TEXTS RESULTS

clopaedia articles or (factual) news like the previously mentioned experimen-

tal soccer match articles. For this test case, the goal was set to generate short
Wikipedia article stubs3 from an ontology dataset.
Generating Wikipedia article stubs automatically is certainly not a new idea.
The Swedish linguist Prof. Lars Sverker Johannson developed an automated
article writing program named Lsjbot in 2009 [37] , which, by the time of 2014,
had created around 2.7 million Wikipedia articles in Swedish, Cebuano4 and
Waray-Waray5 .

Figure 7.4: A typical Lsjbot-generated article contains a few introductory sen-

tences, a human-readable data table and a list of sources. Source: [50]

While human authors are certainly much more capable of writing encyclo-
paedic articles than an algorithm that has no way of understanding the topic it
is dealing with, the main advantage of automatically created text on Wikipedia
is language coverage: English and German are among the most represented lan-
guages between one and five million articles each, but around 160 languages
are well below the 10.000 mark, and a lot of them dont even reach a thou-
sand articles. [41] Even the least represented languages Kanuri and Herero have
around five million speakers but not a single article beside their front page.
Supplying the less maintained language-specific Wikipedia sites with content is
a huge effort with only very few active contributors. Since a lot of articles can
be grouped into categories with a core set of information that can be described
3
In Wikipedia terms, a stub article is one that contains only a few sentences of very basic
information, without any in-depth exploration of the topic. Articles of this kind usually act as
placeholders, until someone invests the time and research effort to expand them.
4
Cebuano, also known as Visayan, is a language spoken by about 20 million people in the
Philippines.
5
Waray-Waray is spoken by around 2.6 million people and, like Cebuano, originates from the
Philippines as well.

76
7.3. GENERATING FACTUAL TEXTS RESULTS

generically6 , it is possible to implement a transformation from the available core

facts to human-readable text. Even though these generated articles may con-
vey only a very limited amount of information, they can still provide a rough
categorization, and a starting point for human authors to expand upon.
Johannsons program Lsjbot was focused mainly on living creatures; as for this
test case, the domain of films was chosen in order to limit its scope to a single
knowledge domain as well. An ontology dataset with around 500 films was ob-
tained from DBpedia [33] , which is an open, community-driven project that aims
to provide a structured data layer for existing Wikipedia pages which means
that the data that Scribe will use to generate Wikipedia articles has been pre-
viously extracted from the (human-written) original article. By comparing the
procedurally generated result to the actual article, a relatively easy assessment
of the achieved quality level is possible. It also highlights the fact that all arti-
cles generated here are to be regarded as a test exercise, since there would be
no point in replacing an existing human-written article with an algorithmically
generated one.
As to be expected from a generic and open dataset of that scale, the ontology
from DBpedia was not directly usable for the purpose of article stub generation at
sufficient quality, and a significant amount of preprocessing steps was necessary
in order to transform it accordingly:

Removal of Irrelevant Data: The source dataset contained a lot of information

that was considered irrelevant for the task at hand, yet accounted for a high
fraction of the data volume and, as a result, loading times when parsing it.
To streamline the testing process and increase the general signal-to-noise
ratio, the affected ontology properties were identified and removed using
a simple blacklist during processing.

Merging Redundant Data: Some information like the name or the run length
of a film was expressed redundantly through multiple data properties, but
neither of them was guaranteed to be used at all. Since iterating through
a set of potentially existing properties at runtime in order to search for a
bit of information that may or may not exist seemed like an unnecessary
point of failure, all redundant data was mapped to a single, well-defined
property each. As a result, the name or run length of a film can be looked
6
Plants and animals have a scientific classification, works of art have an artist and a time of
creation, etc.

77
7.3. GENERATING FACTUAL TEXTS RESULTS

up using a single identifier and if that doesnt yield any results, it can be
safely assumed that the requested information is simply not available.

Normalizing Irregular Multi-Row Data: As a result of the data mining tech-

niques that were used by DBpedia contributors to gather ontology data,
some entries containing list data are represented incorrectly as a single
RDF data triple with multi-line string value, rather than a set of RDF data
triples with regular values. While the multi-line string version is a human-
readable representation of a list, its individual items remain indistinguish-
able to an algorithm that operates on RDF triples alone. A specialized pre-
processing step can detect and transform these multi-line lists into multi-
triple datasets.

Extracting Implicit Data: Evaluating explicit property data like the name or
runtime of a film is trivial, but covers only a part of the overall available
information. In the ontology data that was obtained from DBpedia, there
is a notable amount of information encoded within the categories a certain
entity is associated with. If a film is categorized under 1940s American
Western, its (rough) release date, nationality and genre can be derived
from it most of which would have been unknown otherwise. Implicit in-
formation like this can then be transformed into explicit property values to
allow easy and well-defined access patterns.

Retrieving Dictionary Information: Due to the nature of film data, a short

summary may contain a lot of names starring actors, directors, producers
and other people that were involved in the making of it. In order to include
these names in a text in a grammatically correct way, they require a Dic-
tionary entry that associates their name with a gender. Unfortunately, the
gender of a person is not part of the DBpedia dataset, and only a fraction
of referenced persons are referencing an ontology entry at all; for most of
them, the reference just consists of a string containing their name.
In order to provide gender information on person references, a simple al-
gorithm is run for each newly encountered one: Given an ontology entity
identifier of the person, the full name is retrieved using an ontology query
to the database. The name is then evaluated with a list of known male and
female first names [46] if a match is found, the corresponding gender will
be used. If the name is not part of any of the provided lists, but the persons
ontology entity provides an encyclopaedic text description, it can be anal-

78
7.3. GENERATING FACTUAL TEXTS RESULTS

ysed for cues on its subjects gender. An easy way to do this is counting the
number of occurrences of each genders personal pronouns7 and assuming
that the gender with the highest count is likely the one of the person that
is described.

As it turned out, preparing the film ontology dataset for its usage in Scribe
was a non-trivial task that took its time to complete, mainly due to prevalent
problems with the source data. The transformation process into a more usable
form is still far from perfect, but was deemed good enough for the task at hand,
especially since the main focus of this exercise are mostly Scribes text generation
capabilities, rather than practical data mining applications. The final ontology
dataset that was used by Scribe consists of a guaranteed Name property and a
range of additional properties (See Table 7.1), which may or may not be defined
for any given film entity.
Once the film ontology dataset and Dictionary was generated and ready to be
used, writing the actual text Templates in Scribe was a comparatively small task,
consisting mostly of coming up with suitable ways to describe each fact, struc-
turing the text accordingly and reducing repetitive wording in the final result.
A recurring problem was the fact that nearly all data properties are optional,
resulting in a vast amount of conditional text fragments that are used, replaced
with a fallback or ignored, depending on the existence of their required proper-
ties. Simply remarking a films writer and director in the same phrase requires
four different variants, depending on which of the properties are available and
which arent:
 
written by { Var : Film . Writer }

 
directed by { Var : Film . Director }

 
written and directed by { Var : Film . Director }

 
written by { Var : Film . Writer } and directed by { Var : Film . Director }

Other text fragments form full sentences based on property values and are
either included in the final text, or simply ignored when no appropriate data is
available; which is easier to handle on the author side, but can lead to a list of
disjoint sentences with a bad reading flow. Building at least parts of the final text
on a sub-sentence level allowed to manually join sentences and create transitions
7
he, him, himself, his / she, her, ...

79
7.3. GENERATING FACTUAL TEXTS RESULTS

Use Case Count Description

Name 1 A Dictionary entry that represents the films
name.
Release Year 1? The release year of the film, as an integer value.
Runtime 1? The total run length of the film, in minutes.
Genre n? A set of Dictionary entries that each represent a
genre this film belongs to.
Nationality n? Where the film was produced.
Director n? A set of Dictionary entries that each represent a
director who was involved in the making of this
film.
Producer n? Similar to Director.
Writer n? Similar to Director.
Starring Actors n? Similar to Director.
Real Events 1? A boolean value that indicates whether this film
was based on real events.
Set in Year n? A list of years this film was set in.
Set in Place n? Similar to Set in Year.
Director Debut 1? Indicates whether this film was the first public
work of its director(s).
Visual Technique 1? Describes whether the film is in color, black-and-
white or animated.
Gross Revenue 1? The total box office revenue of the film in dol-
lars.

Table 7.1: A list of relevant film data entries from the processed result ontology.

instead at the cost of a more complex Template structure and higher effort for
the text author. This again highlights the issue of specialization versus reusability,
as pictured earlier in Fig. 7.1: When trying to achieve a continuous reading flow
and narrative, specialized text is required; whereas the easiest thing to do with
a dataset is generating a list of disjoint sentences, each stating a simple fact.
The more specialized a text gets, the less reusable and dynamic it becomes, so
pursuing the goal of text quality appears to be in direct conflict with attempting
to reduce the overall workload by sticking to generic patterns.
Evaluating the quality of Scribes generated text can be a bit tricky to do in an
objective way, because the quality of a text cant be easily defined when moving

80
 7.3. GENERATING FACTUAL TEXTS RESULTS

beyond describing stylistic issues. [22] Instead, each generated film summary was
compared to the corresponding excerpt from the human-written Wikipedia text
in order to put it into perspective. When asking test candidates to decide which
version of a text was taken from Wikipedia, and which one was generated, they
were usually able to tell the difference but not always easily so. Although
without an actual study and a lot more participants, this finding has little more
than anecdotic value, it was still notable that some candidates even scored as
low as 50 percent in classifying the origin of a given text; meaning that, in these
specific cases, the generated texts remained essentially indistinguishable from
actual Wikipedia excerpts.
When assuming that these first results are representative of Scribes poten-
tial, it can be expected to be a viable helper for populating some of the lesser-
maintained languages on Wikipedia, one specific domain at a time.
However, it also became apparent that generating more than just a quick
summary would pose a serious problem to the system, as writing a whole article
about a topic has more creative elements to it than wrapping a predefined set
of data points into sentences; in fact, even this task alone turned out to be a
less than ideal fit to Scribes capabilities: Being primarily a tool for varying and
adjusting pre-defined text Templates, it is easy to generate individual sentences
from facts but there is no way to make sure these sentences actually form a
continuous text, a single narrated entity. (See Fig. 7.5)
 
It was written by {Var:Film.Writer}. It was directed by {Var:Film.Director}. It yielded a gross
revenue of {Var:Film.Revenue}.

 
Written by {Var:Film.Writer} and directed by {Var:Film.Director}, it yielded a gross revenue of
{Var:Film.Revenue}.

Figure 7.5: The first text is easy to generate, because each sentence is optional
and completely unrelated to the others which also makes it hard to read. The
second text is much easier to read, but requires a lot more effort in its imple-
mentation, because it is unclear which of the required facts are available and the
more fluent version creates a strong dependency between them.

Since all writing is done by an author, there is no easy way to insert a new fact
into the existing text, because all sentences have been carefully crafted to work
in the absence or presence of various fragments and yield a nicely readable text
result. Adding a new sentence inbetween requires complete knowledge of the
surrounding text generation behaviour which complicates the writing process

81
7.3. GENERATING FACTUAL TEXTS RESULTS

significantly. While Scribe works reasonably well with a high text-to-data ratio
like in the previous two test cases, it starts to be a lot less useful when it comes to
transforming raw data into a tightly packed text summary. It is a system that is
most helpful when the main issue is organizing text and adjusting it to different
variables; not for writing the text itself.

82
7.3. GENERATING FACTUAL TEXTS RESULTS

Generated Text Wikipedia Text

Undercurrent is an American drama Undercurrent (1946) is a film noir
with Film Noir aspects that was released drama directed by Vincente Minnelli.
in 1946 as one of the later black-and- The screenplay was written by Edward
white films. It was written by Ed- Chodorov, based on the novel You Were
ward Chodorov and directed by Vin- There by Thelma Strabel. The mo-
cente Minnelli. Notable on screen ap- tion picture features Katharine Hepburn,
pearances include Katharine Hepburn, Robert Taylor, Robert Mitchum, and oth-
Robert Mitchum and Robert Taylor. ers.
Magnificent Obsession is an American Magnificent Obsession is a 1954 Univer-
drama that was released in 1954. It was sal Technicolor romantic feature film di-
written by Robert Blees and directed by rected by Douglas Sirk; starring Jane
Douglas Sirk. Notable appearances on Wyman and Rock Hudson. The screen-
screen include Agnes Moorehead, Bar- play was written by Robert Blees and
bara Rush and Jane Wyman. Wells Root, after the 1929 book Magnifi-
cent Obsession by Lloyd C. Douglas. The
film was produced by Ross Hunter.
Wild Animals is a South Korean indepen- Wild Animals is Korean director Kim Ki-
dent movie that was released in 1996. duks second film, released in 1996. It is
It was written and directed by Ki-duk a crime-drama film set in Paris, and stars
Kim. It was starred by Cho Jae-hyun, Cho Jae-hyun, Dong-jik Jang and Ryun
Jang Dong-jik and Jang Ryun. Jang.
Iron Man is an American drama that Iron Man is a 1931 American sport
was released in 1931 given its time, drama film directed by Tod Browning
a natural representative of black-and- and starring Lew Ayres, Robert Arm-
white movies. Written by Francis Ed- strong and Jean Harlow.
ward Faragoh and directed by Charles
Albert Browning, it was among others
played by Jean Harlow, Lew Ayres and
Robert Armstrong.

Table 7.2: A direct comparison between a select set of film summaries, generated
from data on one side and manually written by humans on the other.

83
7.4. ISSUES AND APPLICATIONS RESULTS

7.4 Issues and Applications

The previously discussed test cases have revealed both strengths and weaknesses
in the prototype system that was developed over the course of this work:

3 Structure: The main benefit of using Scribe over just writing a lot of text
variants is the framework it provides for not only structuring data and
text, but also integrating both to produce a result. Templates, Dictionary,
Ontology and Expressions work together as anticipated and allow authors
to separate data from text, access it using an abstract layer and present
it using a pre-defined scheme of text fragments, which are dynamically
assembled to a complete text. Especially in cases with a high number of
variables and variations, this can provide a helpful structure to keep track
of it all.

3 Predictability: Since all text fragments are written by a human author, as

well as all rules on how these fragments are to be combined, the systems
output is entirely predictable. There are no unknown variables and, for
any given input, the text result is completely stable.

3 Throughput: Once set up, the system requires no further human input
besides the data it should process during text generation. Given a uniform
dataset and a set of Templates on how to transform this data into a text,
the time consumption of generating a new output becomes negligible. This
can help with tasks like generating Wikipedia article stubs, but also reduce
maintenance efforts for introducing changes to a uniform text template
compared to maintaining text variants manually: Independently from the
amount of generated texts, as long as they are generated procedurally, all
it takes to modify them is to adjust their generation Template.

Complex Setup: Whether to use a system like Scribe needs to be carefully

evaluated on a case-by-case basis, mostly because it takes a lot of time to
initially set up the text Templates for generation, and prepare the data for
processing. Additionally, designing these Templates is far more complex
than just writing text, and most texts are likely better off written manually
in the first place. The Procedural Generation of text is not a one size fits
all solution and will, when applied to the wrong use cases, do more harm
than good.

84
7.4. ISSUES AND APPLICATIONS RESULTS

Improvable Usability: The framework that was described in this work

is still in a rough and experimental state and there is a lot of ground to
cover on the usability domain, before any actually usable software can
come out of this. Requiring users to edit data files by hand, without proper
error checking or reporting, and no preview or user interface at all is not
a reasonable way to interact with authors. Perhaps, an integration with
commonly know text editing software8 could yield desirable results or a
standalone UI application that is specialized in text generation.

7 Illiterate: The system doesnt understand any part of the text on a deeper
layer and, as a result, is unable to work with it in a more meaningful way
than basic character string operations. Since it cannot perform tasks that
require a certain degree of semantic knowledge, all of them have to be
performed by the user. The previous issue of manually joining sentences
and writing transitions due to the systems inability to help on that matter
is one prominent example of this.

Wrapping this up, it can be expected that a high potential for real-world use
cases of Scribe and similar systems is situated in domains where a recurring
body of text fragments and phrases is used and re-used in order to convey a set
of basic data points to a reader. Generating Wikipedia stub articles is a successful
example of this, but one can easily imagine to use a similar system for generating
corporate E-Mail and text reports as well. The originally anticipated idea of
procedurally generating or varying stories can be successfully implemented as
well, even though it is important to note that the system isnt actually a creative
entity by itself, but designed to help a creative author organize and maintain a
large body of variations.

8
Such as Open Office or Microsoft Word.

85
8 | Summary

In this work, the concept of Procedural Generation was applied to the domain of
authoring natural language text and, in doing so, the capabilities and limitations
of a specific approach were explored practically. As a vital requirement of this,
the prototype system Scribe was developed and tested with three distinct use
cases, each focusing on different aspects of the generation and writing process.
Although the scope of this project later shifted towards a more general direc-
tion, the original idea was to determine to which degree fictional stories can be
procedurally generated without severely impacting their quality.

Figure 8.1: The Scribe testing interface that was used for displaying generated
text results.

Over the course of the initial research effort, various former story generation
systems were described and analysed, including early plot generators like Tale-
Spin, Minstrel and Brutus, as well as the more productively oriented procedural
writing experiment Almost Goodbye. While the discussed plot generators rely on

86
 SUMMARY

a world simulation approach with only little reliability and limited text quality,
Almost Goodbye manages to satisfy all production quality requirements using a
formal grammar based approach albeit a strongly specialized one. In an effort
to build up on this, the presented prototype system was conceptualized.
Instead of recursive text replacement using formal grammars, a more com-
plex design was crafted, involving distinct system components for text fragments
(Templates), grammatical word forms (Dictionary), domain knowledge / data
(Ontology) and generation logic (Expressions, Plugins). By relying on a special-
ized solution for each problem of the chosen approach, Scribe is able to expand
the core idea of fragment-based procedural text generation as seen in Almost
Goodbye to a more general-purpose scope. A proof of concept for this was con-
ducted as part of the evaluation and testing process. (See Fig. 8.1 and Fig. 8.2)

Figure 8.2: A debug view of generated text that allows to inspect its internal
structure.

A basic functionality test of all five system components was done by adapt-
ing a pre-written fable to different actors that were dynamically chosen from
a database while solving for a set of constraints. The systems ability to intro-
duce not random, but thoroughly consistent variation was tested by changing
the mood of a pre-written short story, both in a basic version and an extended
setup where entirely new aspects were introduced into the plot for a subset of
generated stories. Finally, Scribe was successfully used to generate Wikipedia ar-
ticle stubs from a database, in order to prove its viability in a completely different
domain with different requirements.

87
 SUMMARY

Initially focusing on generating and varying stories as a specific kind of text,

it soon became apparent that an equally viable application of the presented text
generation approach is creating instances of more predictable text genres, like
domain-specific reports or generic news- and encyclopaedia articles.
Despite the adequately positive results, certain drawbacks have been de-
scribed as well some of which are originating from the roughness of a prototype
system, others being inherent in the text generation approach itself. Regardless
of how sophisticated its implementation is, the main issue of a fragment-based
text generator remains its inability to work with text and language on a deeper
level than character string operations.
Upon closer inspection, most human-written text is highly adjusted to domain
and context and most sentences are strongly intertwined with their surround-
ings. To humans, written language is a multi-layered, subjective, emotional and
highly diverse means of communication, and text generation systems have yet
to find a way to interact with text on a similar level. While fragment-based ap-
proaches acknowledge this by instead leveraging human creativity for the writ-
ing process, they could still benefit from more intrusive NLG techniques when
it comes to hiding the seams between the pre-written building blocks they are
based upon.
As the introduction may have alluded to, another viable use case for this pro-
totype is in game development, where a players interaction with the game rein-
forces a dynamic environment with a lot of variables which need to be adapted
to. In works of interactive fiction, a fragment-based text generator can assume
the core role of organizing and assembling all the text that will later be pre-
sented to the player. Additionally, a lot of the concepts that were described in
the context of Scribes development would be equally applicable to other Proce-
dural Generation fields from the Recombination realm, as seen in Fig. 3.4 in
which case certain aspects of this work could serve as a potential role model.

88
9 | Outlook

Does fragment-based text generation have a future? It is a tricky question with

more than one aspect and viewing angle. To get to the bottom of this, it is
important to first answer a different question: What is it exactly that a user
would expect from a perfect text generator? A diligent helper that produces
completely predictable results according to the input it is supplied with or a
truly creative entity that is able to fill in all the gaps by itself when prompted
with an open-minded writing task?
It doesnt seem like a far stretch to assume that, given an infinite technolog-
ical potential, there would still be a viable use for both of them, but each one
solving a different set of problems; and as a result, systems like Scribe could
actually have their place in information technology. However, this doesnt nec-
essarily mean that Scribe itself does as well. Even though it was developed with
the goal of productive application in mind, above all it was (and still is) a largely
experimental prototype.
The most obvious improvement that would need to be made with an ac-
tual use in mind comes from the usability realm. As stated before, Scribe in its
current state is not very usable. Setting up a new instance of the text gener-
ation system, including Ontology database, Dictionary and Templates requires
proficiency in three different file formats, a lot of manual editing and learning
a custom expression language for tying it all together; in addition to getting a
hold of concepts like nested Templates and text trees. Managing these tasks and
succeeding in them requires a programmer; yet, writing creative text requires
an author, and as stated previously, these two are not commonly known to do
each others job very well. Getting rid of the usability issue will be of utmost
importance, should the prototyped system get any chance of user adoption.
A straightforward way of improving usability would be to provide a conve-
nient editing system for the complete stack of data that is involved in the gener-
ation process:

89
 OUTLOOK

Templates: The difference between a Template and a text-based formal gram-

mar rule is not only the fact that Template text is interleaved with complex
Expressions, but also the added structure and data interface that consists
of Template Parameters and Features, as well as their optional Ontology
queries, conditions and multiple alternate text fragments. While all of
them certainly make the concept of a text Template more powerful, they
also require a higher level of understanding than human-readable formal
grammars and plain text. Reading a Template on the source side and pic-
turing its actual result text is not always intuitive.
A user-interface based editor for Templates could provide live previews of
potential results on one side of the screen, while editing the contained
text fragments on the other (Fig. 9.1). This preview could be scoped to
the currently edited fragment, or include a fixed level of parent and / or
child Templates to provide some context, if the user so wishes. In order
to establish a stronger focus, the current position of the text editing cursor
could be simultaneously highlighted on both the editor and preview side.
Result text regions that were provided solely for context could furthermore
be displayed greyed out while not active.

Figure 9.1: In a Live Editing approach, the user is able to see and interact with
both the source Template as well as the resulting generated text at the same
time.

Template Features and Parameters could be defined using a fixed user in-
terface as opposed to typing text representations of them, and when writ-
ing embedded Expressions or Ontology queries, their respective specialized
editors could be used directly to guide users through the process.
In order to provide some additional debugging tools, the preview could
also include an advanced mode, which would allow to inspect the values of

90
 OUTLOOK

local variables, individual query and expression results, as well as potential

Parameter and Feature values, as derived from possible parent Templates
of the current one.
To get a better overview on a texts complete set of Templates, an interactive
visual graph could be generated, where each Template is a single node,
and each dependency forms a single, directional connection. The resulting
visual representation of the edited Template body would likely help users
to gain an intuitive understanding of it, simply by relying on well-known
and easily understandable visual cues and metaphors. (Fig. 9.2)

Figure 9.2: A node graph visualization of a set of Templates. In this example,

Template StoryA is highlighted, including its dependencies.

Dictionary: Unlike Templates, the Dictionary is build upon a comparatively triv-

ial concept that is easy to understand and doesnt require a lot of special
attention: Each word, as represented by an abstract symbol, can define an
arbitrary amount of concrete forms, each of them accessible through its
(grammatical) role, such as Singular and Plural. Still, a significant usabil-
ity improvement could be made providing a specialized user interface for
organizing, creating, editing and searching Dictionaries. Even something
as simple as an editable table view1 (Fig. 9.3) could be much more con-
venient, efficient and intuitive than requiring users to edit text files with a
custom data format, as the current prototype does.

Ontology: Scribes Ontology layer is built upon RDF, a system that relies on data
Triples to express both object properties and relations between objects.
Providing an editor for this kind of data can be done in multiple ways with
different weights on overview and editing convenience.
1
For reference, see applications like Microsoft Excel, Google Sheets or LibreOffice Calc.

91
 OUTLOOK

To provide a quick overview, the edited dataset can be interpreted as a

set of distinct objects, described by a set of Triples each and displayed
in a visual node graph or a flat list. While this approach can provide a
first overview on the available data, it doesnt play well with advanced
editing operations, such as simultaneously changing all values of a certain
property, or adding a new property to all objects of a certain type.
On the other hand, a specialized editable table view that allows to inter-
act directly with the set of available Triples could enable advanced users
to quickly perform both trivial and complex operations on the Ontology
dataset. (Fig. 9.3)

Figure 9.3: An example of an editable table view for Ontology data Triples,
visualized using Google Sheets.

Expressions: Scribe Expressions are the glue that ties together all other sys-
tems, so it is even more important to provide a convenient way of editing
and testing them. Because they represent a limited subset of a program-
ming language, experience from providing usable source code editing soft-
ware can be applied here as well: When editing an Expression, syntax
highlighting could help users parse the code using visual cues and auto-
complete features could help writing it. After the user stops typing, Expres-
sions could be verified for correctness and show error details if necessary
and if there are no errors, the system could automatically derive sample
input from the Expressions context in order to display a selection of likely
outputs.

92
 OUTLOOK

To introduce less technically inclined users to the concept of Expressions, a

visual editing system could be provided as well. When active, users could
avoid writing code alltogether and instead assemble Expressions from pre-
defined code blocks that could be arranged using drag-and-drop func-
tionality.

Usability can often seem like an open-ended topic and indeed, most of the
above ideas and concepts would require a tremendous effort to be researched,
implemented and evaluated properly. Even though resorting to existing solutions
for the tasks at hand could greatly reduce the workload: Designing, developing
and iterating upon what essentially amounts to being a full Integrated Develop-
ment Environment is not a small endeavour. Yet, it might be a worthwhile one
when it comes to turning the developed prototype into a usable product.

Figure 9.4: An example of a text editor integration for Scribe, visualized using
Google Docs.

As it was mentioned before, another way of improving user experience would

be an integration into existing text editing software, such as Microsoft Word, Li-
breOffice Writer or Google Docs (Fig. 9.4). Most likely, users are already familiar
with either the software itself, or at least the concepts and metaphors behind
it, which could ease the transition from writing text to writing text Templates.
When mapping the concept of a Document to the one of a text Template and
providing some additional user interface options for specifying Template Fea-
tures and Parameters, users might be eased into the new workflow with less of
a culture shock and potentially higher user acceptance and productivity. How-
ever, as writing itself and authoring Procedural Generation Templates are applied

93
 OUTLOOK

to different use cases and require a different set of skills, integrating both in a
single application would need to be carefully evaluated for viability.
Going deeper into the core of the implemented prototype system, there are
quite some opportunities for improvement as well. As described earlier, a re-
curring problem in the examined test cases was the fact that Scribe isnt able to
handle the actual content of a Template and, as a result, can only use them as
fixed building blocks. There is no way for a Template to react to its surround-
ings, which can lead to several stylistic problems in the generated output text.
Especially problematic are the inadvertent occurrence of word repetitions and
sequences of simple or similar sentences, both of which contribute to an imme-
diate loss in perceived writing quality to the human reader yet, the localized
nature of Scribes text Templates makes it impossible to counteract on the issue
efficiently. Without knowing the final text tree structure, there is no way to de-
tect these stylistic problems and even with knowing it, a Template has no easy
way to react accordingly. Addressing these issues requires a global view of the
generated output, because they occur only after assembling the text. With the ex-
isting system in mind, there are two techniques that could significantly improve
its output quality:

Stylistic Reprocessing: As soon as a text nodes sub-tree has been fully gener-
ated, the final text portion of that node is available and can be analysed
with regard to its stylistic quality and rated among a uniform scale between
zero and one. [22] With the presented system, it would be easily possible to
apply a threshold to the stylistic rating and re-generate the sub-tree, should
its value be too low: Since most Templates implement a certain degree
of variation in structure and wording, it is likely that some instances will
be rated better than others. Contrary to Scribes current implementation,
where the first valid instance will be the final one, this would force the
system to recursively re-write parts of the text until a satisfying style rating
is achieved or a maximum number of attempts is exceeded.2

Stylistic Postprocessing: Even though re-generating parts of the text tree has a
good chance to get the best out of the source material, it will still fail when
a stylistic improvement requires the alteration of text that remains static
across different text versions, or a change in structure that wasnt antici-
pated by Template authors. When two subsequent Templates each define a
2
Interestingly enough, this behaviour also seems to mirror the process of iteratively reviewing
and polishing a text as found in human authors.

94
 OUTLOOK

trivial sentence, varying each sentence individually will not solve the prob-
lem of a bumpy reading flow. Instead, an Aggregation of both sentences
would be required, as described in Dale and Reiters Natural Language Gen-
eration stages (Section 2.2). Similarly, a Lexical Choice algorithm could
attempt to switch out overused words with an equivalent synonym.

Scribes text Templates may be a useful device of structuring and recombining

human-authored content, but when it comes to primarily text-based operations,
they do not offer additional value and might actually get in the way of efficient
processing by limiting access patterns. Hence, any Stylistic Postprocessing oper-
ations are likely best to be performed on either the raw text output directly, or a
transparent mapping between the generated node tree and the text it encodes.
Another limitation of the system arises from its rather simplistic Dictionary
implementation: Being designed as a trivial mapping between a word and its
pre-defined grammatical forms, it provides an easy to use and versatile way of
writing grammatically correct text, even though its dependent parts are deter-
mined dynamically. However, it is easy to imagine that this approach will be
insufficient when encountering languages with more extensive inflection3 and
agglutination4 traits than, for example, English and German. Instead of rely-
ing entirely on human input, an improved system could make use of Natural
Language Generation techniques in order to derive correct grammatical forms
automatically. Going as far as introducing a completely automated Realization
stage as proposed by Dale and Reiter may or may not be beneficial in the context
of this work, and would be subject to further investigation.
Considering the Different forms of Procedural Generation (Fig. 3.4) again, it
becomes apparent that Scribe operates almost entirely in the domain of Recom-
bination, with only a few aspects of Adaption: Both Dictionaries and Templates
are merely devices for assembling a body of canned text in a sophisticated
way, but in no way will the system alter the text itself except for a few text
tree (post-)processors, which automatically adjust whitespaces, character casing
and English articles5 in the generated result. Even though this author-centric ap-
proach does have the previously discussed advantages of delivering predictable
and directly usable results, there is still a lot of potential in learning to work
3
Inflection: Adjusting a word in order to express different grammatical categories such as
tense, mood, voice, aspect, person, number, gender and case. [42]
4
Agglutination: Forming complex words by stringing together morphemes without changing
them in spelling or phonetics. [43]
5
A simple automatic distinction between a and an.

95
 OUTLOOK

with the processed text content on a deeper level. To expand on this, shifting
the projects focus more towards Creation and Adaption (Fig. 3.4) of text should
certainly be among the goals of future prototypes.

96
Bibliography

[1] T OGELIUS , J ULIAN ; S HAKER , N OOR ; N ELSON , M ARK J.: Procedural Content Generation in
Games: A Textbook and an Overview of Current Research
Published by Springer, 2015

[2] M EEHAN , JAMES: TALE-SPIN, An Interactive Program that Writes Stories

1977 in "Proceedings of the 5th International Joint Conference on Artificial Intelligence"

[3] D EHN , N ATALIE: Story Generation after Tale-Spin

1981 in "Proceedings of the 7th International Joint Conference on Artificial Intelligence"

[4] L EBOWITZ , M ICHAEL: Creating a Story-Telling Universe

1983 in "Proceedings of the 8th International Joint Conference on Artificial Intelligence"

[5] T URNER , S COTT: Minstrel - A Computer Model of Creativity and Storytelling

Published by University of California, 1992

[6] P REZ Y P REZ , R AFAEL ; S HARPLES , M IKE: Three Computer-Based Models of Storytelling:
BRUTUS, MINSTREL and MEXICA
2004 in "Knowledge-Based Systems, Volume 17, Issue 1"

[7] B RINGSJORD , S ELMER ; F ERRUCCI , D AVID: Artificial Intelligence and Literary Creativity: In-
side the Mind of BRUTUS, a Storytelling Machine
1999 in "Computational Linguistics, Volume 26, Number 4"

[8] R IEDL , M ARK ; Y OUNG, M ICHAEL: Narrative Planning: Balancing Plot and Character
2004 in "Journal of Artificial Intelligence Research 39"

[9] S ARLEJ , M ARGARET; RYAN , M ALCOM: Moral Storytelling System (MOSS)

2013 in "Interactive Storytelling: 6th International Conference, ICIDS 2013, Istanbul"

[10] R EED , A ARON: Sharing Authoring with Algorithms

2012 in "Proceedings of the third workshop on Procedural Content Generation in Games"

[11] C HATMAN , S EYMOUR: Story and discourse. Narrative structure in fiction and film
Published by Cornell University Press, 1980

[12] K LYNE , G RAHAM ; C ARROLL , J EREMY; M C B RIDE , B RIAN: Resource Description Framework
(RDF): Concepts and Abstract Data Model
2002-08-29,
http://www.w3.org/TR/2002/WD-rdf-concepts-20020829, Last accessed 2015-04-27

97
BIBLIOGRAPHY BIBLIOGRAPHY

[13] B IRD , S TEVEN ; K LEIN , E WAN ; L OPER , E DWARD: Natural Language Processing
Published by Distributed with the Natural Language Toolkit, 2008

[14] D ALE , R OBERT; R EITER , E HUD: Building Applied Natural Language Generation Systems
2000 in "Cambridge University Press"

[15] S PUFFORD , F RANCIS: Masters of their universe, edited extract from Backroom Boys: The
Secret Return Of The British Boffin
2004-07-31 in "The Guardian"

[16] R IEDL , M ARK: Fabulist

https://research.cc.gatech.edu/inc/fabulist, Last accessed 2015-04-11

[17] G ERVS , PABLO: Story Generator Algorithms

2013-09-12,
http://www.lhn.uni-hamburg.de/article/story-generator-algorithms, Last accessed 2015-04-07

[18] WARDRIP-F RUIN , N OAH: Lebowitzs Universe, part 2

2006-03-26,
http://grandtextauto.org/2006/03/06/lebowitzs-universe-part-2/, Last accessed 2015-04-07

[19] WARDRIP-F RUIN , N OAH: Turners Minstrel, part 1

2006-03-21,
http://grandtextauto.org/2006/03/01/turners-minstrel-part-1/, Last accessed 2015-04-07

[20] WARDRIP-F RUIN , N OAH: Turners Minstrel, part 2

2006-03-22,
http://grandtextauto.org/2006/03/02/turners-minstrel-part-2/, Last accessed 2015-04-07

[21] WARDRIP-F RUIN , N OAH: Brutus

2008-03-26,
http://grandtextauto.org/2008/03/06/ep-73-brutus/, Last accessed 2015-04-07

[22] A DAM , F EDJA: Checking for Stylistic Errors

2014 in a h_da course work on "Natural Language Processing"

[23] H ARRIEHAUSEN -M HLBAUER , B ETTINA: The seven levels of language understanding

2013 in "Natural Language Processing"

[24] G AME I NFORMER: A Behind-The-Scenes Tour Of No Mans Skys Technology

2014-12-05,
https://www.youtube.com/watch?v=h-kifCYToAU, Last accessed 2015-03-31

[25] B ENSE , H ERMANN: Ontologien als Schlsseltechnologie fr die automatische Erzeugung

natrlichsprachlicher Texte
2014-07-11,
http://ontology4.us/download/presentations/140711_Ontology4_TextGenerierung_Ernst-
Schroeder-Zentrum_Darmstadt.pdf, Last accessed 2015-07-26

[26] B ENSE , H ERMANN: Schreib-Maschine Beispiel-Textgenerierung

http://ontology4.us/manual/Text-Composer/Beispiel-Textgenerierung/index.html, Last accessed
2015-07-26

[27] P RINCETON U NIVERSITY: What is WordNet?

2015-03-17,
https://wordnet.princeton.edu/wordnet/, Last accessed 2015-07-25

98
BIBLIOGRAPHY BIBLIOGRAPHY

[28] R EED , A ARON: Almost Goodbye

http://almostgoodbye.textories.com/, Last accessed 2015-04-12

[29] F LECK , N ICHOLAS: Rant

http://rantlang.github.io/, Last accessed 2015-06-06

[30] B ENSE , H ERMANN: Automatische Erzeugung natrlichsprachlicher Texte

2014,
http://www.schreib-maschine.info/, Last accessed 2015-04-20

[31] X U, X IAO: PyTeaser

https://github.com/xiaoxu193/PyTeaser, Last accessed 2015-07-02

[32] P ERSSON , M ARKUS: Terrain Generation, Part I

2011-03-23,
http://notch.tumblr.com/post/3746989361/terrain-generation-part-1, Last accessed 2015-04-02

[33] DB PEDIA: DBpedia - About

http://wiki.dbpedia.org/, Last accessed 2015-06-12

[34] N ICHOLAS B LUMHARDT: Sprache - Tiny C# Monadic Parser Framework

https://github.com/sprache/Sprache, Last accessed 2015-05-04

[35] M ICROSOFT: Expression Trees (C# and Visual Basic)

https://msdn.microsoft.com/en-us/library/bb397951.aspx, Last accessed 2015-05-04

[36] R EILLY, C LAIRE: Absolutely fabulist: The computer program that writes fables
2014-08-03,
http://www.cnet.com/news/absolutely-fabulist-computer-program-writes-fables/, Last accessed
2015-04-11

[37] W IKIPEDIA: Lsjbot

http://en.wikipedia.org/wiki/Lsjbot, Last accessed 2015-04-20

[38] W IKIPEDIA: Elite (Video Game)

http://en.wikipedia.org/wiki/Elite_(video_game), Last accessed 2015-03-31

[39] W IKIPEDIA: Natural Language Processing

https://en.wikipedia.org/wiki/Natural_language_processing, Last accessed 2015-06-27

[40] W IKIPEDIA: Part of speech

https://en.wikipedia.org/wiki/Part_of_speech, Last accessed 2015-06-28

[41] W IKIPEDIA: List of Wikipedias

http://meta.wikimedia.org/wiki/List_of_Wikipedias, Last accessed 2015-06-12

[42] W IKIPEDIA: Inflection

https://en.wikipedia.org/wiki/Inflection, Last accessed 2015-07-27

[43] W IKIPEDIA: Agglutination

https://en.wikipedia.org/wiki/Agglutination, Last accessed 2015-07-27

99
BIBLIOGRAPHY BIBLIOGRAPHY

Text Sources

[44] A ESOP: The Wolf in Sheeps Clothing

http://www.eastoftheweb.com/short-stories/UBooks/WolShe.shtml, Last accessed 2015-06-07

[45] T HOMPSON , L UKE: Scarecrow

http://www.eastoftheweb.com/short-stories/UBooks/Scar880.shtml, Last accessed 2015-06-08

[46] V OGEL , A DAM ; J URAFSKY, D AN: Gender in the ACL Anthology

2012,
http://nlp.stanford.edu/projects/gender.shtml, Last accessed 2015-06-16

Image Sources

[47] M INECRAFT W ORLD S EEDS: Triple Crater Caverns Seed

2014-10-24,
http://topminecraftworldseeds.com/triple-crater-caverns, Last accessed 2015-03-31

[48] W IKIMEDIA C OMMONS: Orange Folder Icon

2009-06-07,
http://commons.wikimedia.org/wiki/File:Folder-orange.svg, Last accessed 2015-04-24

[49] W IKIMEDIA C OMMONS: Cog Wheel Icon

2013-01-26,
http://commons.wikimedia.org/wiki/File:Cog_font_awesome.svg, Last accessed 2015-05-04

[50] W IKIPEDIA: Spilochroa guttata

https://sv.wikipedia.org/wiki/Spilochroa_guttata, Last accessed 2015-06-16

[51] V ECTOR A RTS: Free Sale Labels vector graphics

http://www.vectorarts.net/objects/free-sale-labels-vector-graphics/, Last accessed 2015-06-28

[52] F REEPIK: Robotic arm, free icon

http://www.flaticon.com/free-icon/robotic-arm_17969, Last accessed 2015-06-29

100
 Part IV

Appendix

101
A | Text Generation Example

The goal of this appendix is to demonstrate a full text generation cycle (Fig. 5.2)
with a concrete example. For the sake of simplicity and in order to focus on
providing a basic overview, certain implementation details may be skipped.

Source Data
Scribes text generation process depends heavily on the available source data.
Excerpts from external files that are relevant to it are displayed as follows:
 
<Template name=" Template:EasySample " r o o t=" t r u e ">
<F e a t u r e name=" V a r : P r e d a t o r ">
<Query mode=" Random ">
SELECT ? obj
WHERE { ? obj System:IsA Knowledge:Animal . }
</ Query>
</ F e a t u r e>
<F e a t u r e name=" Var:HuntedAnimal ">
<Query mode=" Random ">
SELECT ? obj
WHERE {
? obj System:IsA Knowledge:Animal .
@ { Var:Predator } Knowledge:PredatorOf ? obj .
}
</ Query>
</ F e a t u r e>
<Text>
{ Template:Title }

There was a { Var:Predator } who liked to prey on { Var:HuntedAnimal . s } .

</ Text>
</ Template>

<Template name=" T e m p l a t e : T i t l e ">

<Text>A Simple Example</ Text>
</ Template>

Listing A.1: Templates\EasySample.xml

102
 TEXT GENERATION EXAMPLE

 
Knowledge : Animal
System : Text Noun : Animal .

Knowledge : Sheep
System : IsA Knowledge : Animal ;
System : Text Noun : Sheep .

Knowledge : Wolf
System : IsA Knowledge : Animal ;
System : Text Noun : Wolf ;
Knowledge : PredatorOf Knowledge : Sheep .

Listing A.2: Ontology\EasySample.ttl

 
Column : EasySample : Nouns , singular , null , primary
Column : EasySample : Nouns , plural / s , null

Noun : Animal > animal , /s , ( Gender : Neutral )

Noun : Sheep > sheep , /, ( Gender : Neutral )
Noun : Wolf > wolf , ves , ( Gender : Neutral )

Listing A.3: Dictionary\EasySample.dct

 
Column : Grammar : Genders , I / he , he , primary
Column : Grammar : Genders , me / him , him
Column : Grammar : Genders , myself / himself , himself
Column : Grammar : Genders , my / his , his
Column : Grammar : Genders , mine / hiss , his

Gender : Male > he , him , himself , his , his

Gender : Female > she , her , herself , her , hers
Gender : Neutral > it , it , itself , its , its

Listing A.4: Dictionary\Gender.dct

103
 TEXT GENERATION EXAMPLE

Generation Process
The following is a simplified step-by-step description on the process of generating
a text from the available source data.

Preparation
1. The system is initialized by generating the root node of a text tree.

2. The root Template "Template:EasySample" it is picked as a basis for the

text tree and assigned to the root node.

3. The root node is set as the currently active one.

Query
Since Template:EasySample defines two Feature variables, their values need to
be determined. Both Var:Predator and Var:HuntedAnimal define a query in
order to do so.

1. The query from Var:Predator is processed, yielding all known animals

Knowledge:Sheep and Knowledge:Wolf as results. By default, when a
query yields multiple result values, one of them is selected randomly. In
this case, let us assume that Knowledge:Sheep was picked.

2. The query from Var:HuntedAnimal is processed, yielding all animals which

are preyed upon by Var:Predator. However, there is no known animal
which Var:Predator (being a sheep) preys upon. The query yields no
results and is considered a failure, leaving variable Var:HuntedAnimal un-
defined.

3. Because the failed query of Var:HuntedAnimal is not flagged as optional,

the system attempts to find a different solution and performs a backtrack-
ing operation to step one, while marking the current solution as invalid.

4. The query from Var:Predator is processed again, yielding Knowledge:Sheep

and Knowledge:Wolf as results. Since Knowledge:Sheep is now flagged as
being an invalid result for this specific query and context, the only remain-
ing result is Knowledge:Wolf, which is picked as value for Var:Predator.

104
 TEXT GENERATION EXAMPLE

5. The query from Var:HuntedAnimal is processed again, yielding Knowledge:Sheep

as a result, because Var:Predator (now being a wolf) preys upon it.

6. All variables of the current node are now defined, no further queries are
required.

Evaluate
The current nodes text body has three inline expressions: Template:Title,
Var:Predator and Var:HuntedAnimal.s. Each of them needs to be evaluated
in order to determine a suitable text representation.

1. Template:Title and Var:Predator are reference symbols without any op-

eration to be performed. They will be returned as is, no further evaluation
required.

2. Var:HuntedAnimal.s performs a member access of s on the object refer-

enced by Var:HuntedAnimal. In order to do so, the symbol Var:HuntedAnimal
will be resolved to its contained value Knowledge:Sheep, which will then
become subject to the member access operation. For user convenience,
the same member access syntax is used for accessing both Ontology and
Dictionary data, making it ambiguous expression.
The system will first assume the object to be a Dictionary entry and attempt
to lookup the specified member s as one of its word forms. However, be-
cause Knowledge:Sheep is not a Dictionary entry, but an Ontology entity,
this will fail and initiate the second step, where Knowledge:Sheep is as-
sumed to be an Ontology entity.
Again, due to user convenience, Ontology member acccess is an ambigu-
ous operation, as the specified member could either refer to an Ontology
relation, or a word form of the Ontology entitys corresponding Dictionary
entry. The second case is checked first by looking up the Knowledge:Sheeps
Ontology member Knowledge:Text, which succeeds and returns Noun:Sheep.
The specified member s is now used to perform a column lookup in the
Noun:Sheep entry, eventually resolving the alias s to the plural column
and returning word form sheep.

105
 TEXT GENERATION EXAMPLE

Resolve
The current nodes expressions have been evaluated, but except for Var:HuntedAnimal.s,
none of them can be inserted into the text directly. The remaining values need
to be resolved.

1. Template:Title is a reference symbol. To resolve it, a lookup is performed

in various sources, until a match is found. First, the plugin API is queried in
order to find out, whether a plugin method is registered using that symbol.
As this is not the case, the Template database is queried next, which yields
the text Template Template:Title as a result. The node is considered
resolved.

2. Var:Predator is a reference symbol as well. The same lookup operations

as with Template:Title are performed, but none of them yield success-
ful results. Finally, the current nodes variable scope is queried, which
returns a match with the local variable Var:Predator. Since it contains
Ontology entity Knowledge:Wolf, a text representation can be retrieved by
performing an Ontology lookup for its Knowledge:Text member, returning
its corresponding Dictionary entry Noun:Wolf. A word form lookup with its
primary column singular results in the text wolf. The node is considered
resolved.

3. Var:HuntedAnimal.s already contains the text string value sheep. The

node is considered resolved.

Create
Each inline expression can potentially spawn a sub-node of the current one, de-
pending on the resolved value.

1. Expression Template:Title was previously resolved to a text Template,

which is now instantiated as a child node of the currently active one. The
referenced Template is assigned to the newly created child node.

2. Expressions Var:Predator and Var:HuntedAnimal.s have been resolved to

text strings and do not spawn any child nodes.

106
 TEXT GENERATION EXAMPLE

Iterate
All previously described steps will now be performed recursively on the current
nodes child nodes. In this case, there is only one child node that was spawned
by Template:Title. Since it only contains a static text, its generation process is
considered trivial and will not be described in detail

Results
Due to the limited Template and Ontology databases, there is only a single pos-
sible result for text tree generation, which consists of a root node with a single
child node and some expressions that have been resolved to string representa-
tions as described above. Transforming the tree into a simple text string results
in the following:
 
A Simple Example

There was a wolf who liked to prey on sheep .

Listing A.5: The resulting sample text.

107