www.linkalab.

it

LLMs for Information Management

Modulo Prompt Engineering (MP)

1
 www.linkalab.it
Prompt Engineering Introduction
● Prompt Engineering, is a technique to improve the
performance of language models.
● It is based on the concept that the more specific a prompt
is, the higher the probability of obtaining a desired
output, from text generation to linguistic translation to
creative writing.
● A prompt is a short string of text that must be clear and
concise to provide the language model with the
information it needs to generate accurate and relevant
output.
● Since generative NLP models output words deemed most
probable based on a certain context, they ensure the
likelihood of the result but not its adherence to the data
used in the training phase.
● Therefore, when only the former but not the latter exists,
the phenomenon of so-called hallucinations occurs.
2
 www.linkalab.it
Prompt Engineering Introduction
● Six strategies for getting better results

3
 www.linkalab.it
Prompt Engineering Introduction
● OpenAI API Playground (https://platform.openai.com/playground)

4
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ These models can’t read your mind.
■ If outputs are too long, ask for brief replies.
■ If outputs are too simple, ask for expert-level writing.
■ If you dislike the format, demonstrate the format you’d like to see.
○ The less the model has to guess at what you want, the more likely you’ll get it.
○ Tactics:
■ Include details in your query to get more
relevant answers
■ Ask the model to adopt a persona
■ Use delimiters to clearly indicate distinct
parts of the input
■ Specify the steps required to complete a
task
■ Provide examples
■ Specify the desired length of the output
5 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Include details in your query to get more relevant answers

6 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Ask the model to adopt a persona (OPEN IN PLAYGROUND)

7 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Use delimiters to clearly indicate distinct parts of the input
(OPEN IN PLAYGROUND)

8 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Use delimiters to clearly indicate distinct parts of the input
(OPEN IN PLAYGROUND)

9 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Use delimiters to clearly indicate distinct parts of the input
(OPEN IN PLAYGROUND)

10 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Specify the steps required to complete a task (OPEN IN PLAYGROUND)

11 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Provide examples (OPEN IN PLAYGROUND)

12 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Provide examples (OPEN IN PLAYGROUND)

13 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Provide examples (OPEN IN PLAYGROUND)

14 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Specify the desired length of the output (OPEN IN PLAYGROUND)

15 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Specify the desired length of the output (OPEN IN PLAYGROUND)

16 www.linkalab.it
 Six strategies for getting better results
1. Strategy: Write clear instructions
○ Tactic: Specify the desired length of the output (OPEN IN PLAYGROUND)

17 www.linkalab.it
 Six strategies for getting better results
2. Strategy: Provide reference text
○ Language models can confidently invent fake answers, especially when asked
about esoteric topics or for citations and URLs.
○ In the same way that a sheet of notes can help a student do better on a test,
providing reference text to these models can help in answering with fewer
fabrications.
○ Tactics:
■ Instruct the model to
answer using a
reference text
■ Instruct the model to
answer with citations
from a reference text

18 www.linkalab.it
 Six strategies for getting better results
2. Strategy: Provide reference text
○ Tactic: Instruct the model to answer using a reference text
(OPEN IN PLAYGROUND)

19 www.linkalab.it
 Six strategies for getting better results
2. Strategy: Provide reference text
○ Tactic: Instruct the model to answer with citations from a reference text
(OPEN IN PLAYGROUND)

20 www.linkalab.it
 Six strategies for getting better results
3. Strategy: Split complex tasks into simpler subtasks
○ Just as it is good practice in software engineering to decompose a complex
system into a set of modular components, the same is true of tasks submitted
to a language model.
○ Complex tasks
■ tend to have higher error rates than simpler tasks.
■ can often be re-defined as a workflow of simpler tasks in which the outputs
of earlier tasks are used to construct the inputs to later tasks.
○ Tactics:
■ Use intent classification to identify the
most relevant instructions for a user query
■ For dialogue applications that require very
long conversations, summarize or filter
previous dialogue
■ Summarize long documents piecewise
and construct a full summary recursively
21 www.linkalab.it
 Six strategies for getting better results
3. Strategy: Split complex tasks into simpler subtasks
○ Tactic: Use intent classification to identify the most relevant instructions for a
user query
■ For tasks in which lots of independent sets of instructions are needed to
handle different cases, it can be beneficial to first classify the type of query
and to use that classification to determine which instructions are needed.
■ This can be achieved by defining fixed categories and hardcoding
instructions that are relevant for handling tasks in a given category.
■ This process can also be applied recursively to decompose a task into a
sequence of stages.
■ The advantage of this approach is that each query will contain only those
instructions that are required to perform the next stage of a task which can
result in lower error rates compared to using a single query to perform the
whole task.

22 www.linkalab.it
 Six strategies for getting better results
3. Strategy: Split complex tasks into simpler subtasks
○ Tactic: Use intent classification to identify the most relevant instructions for a
user query (OPEN IN PLAYGROUND)

23 www.linkalab.it
 Six strategies for getting better results
3. Strategy: Split complex tasks into simpler subtasks
○ Tactic: Use intent classification to identify the most relevant instructions for a
user query (OPEN IN PLAYGROUND)

24 www.linkalab.it
 Six strategies for getting better results
3. Strategy: Split complex tasks into simpler subtasks
○ Tactic: For dialogue applications that require very long conversations,
summarize or filter previous dialogue
■ Since models have a fixed context length, dialogue between a user and an
assistant in which the entire conversation is included in the context
window cannot continue indefinitely.
■ There are various workarounds to this problem, one of which is to
summarize previous turns in the conversation.
■ Once the size of the input reaches a predetermined threshold length, this
could trigger a query that summarizes part of the conversation and the
summary of the prior conversation could be included as part of the system
message.
■ Alternatively, prior conversation could be summarized asynchronously in
the background throughout the entire conversation.
■ An alternative solution is to dynamically select previous parts of the
conversation that are most relevant to the current query.
25 www.linkalab.it
 Six strategies for getting better results
3. Strategy: Split complex tasks into simpler subtasks
○ Tactic: Summarize long documents piecewise and construct a full summary
recursively
■ Since models have a fixed context length, they cannot be used to
summarize a text longer than the context length minus the length of the
generated summary in a single query.
■ To summarize a very long document such as a book we can use a
sequence of queries to summarize each section of the document.
■ Section summaries can be concatenated and summarized producing
summaries of summaries.
■ This process can proceed recursively until an entire document is
summarized. If it’s necessary to use information about earlier sections in
order to make sense of later sections, then a further trick that can be useful
is to include a running summary of the text that precedes any given point
in the book while summarizing content at that point.

26 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ If asked to multiply 17 by 28, you might not know it instantly, but can still work it
out with time.
○ Similarly, models make more reasoning errors when trying to answer right away,
rather than taking time to work out an answer.
○ Asking for a "chain of thought" before an answer can help the model reason its
way toward correct answers more reliably.
○ Tactics:
■ Instruct the model to work out its own
solution before rushing to a
conclusion
■ Use inner monologue or a sequence of
queries to hide the model's reasoning
process
■ Ask the model if it missed anything on
previous passes
27 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Instruct the model to work out its own solution before rushing to a
conclusion (OPEN IN PLAYGROUND)

28 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Instruct the model to work out its own solution before rushing to a
conclusion (OPEN IN PLAYGROUND)

29 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Use inner monologue or a sequence of queries to hide the model's
reasoning process
■ The previous tactic demonstrates that it is sometimes important for the
model to reason in detail about a problem before answering a specific
question. For some applications, the reasoning process that a model uses to
arrive at a final answer would be inappropriate to share with the user.
■ For example, in tutoring applications we may want to encourage students
to work out their own answers, but a model’s reasoning process about the
student’s solution could reveal the answer to the student.
■ Inner monologue is a tactic that can be used to mitigate this.
■ The idea of inner monologue is to instruct the model to put parts of the
output that are meant to be hidden from the user into a structured format
that makes parsing them easy.
■ Then before presenting the output to the user, the output is parsed and
only part of the output is made visible.
30 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Use inner monologue or a sequence of queries to hide the model's
reasoning process (OPEN IN PLAYGROUND)

31 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Use inner monologue or a sequence of queries to hide the model's
reasoning process (OPEN IN PLAYGROUND)
■ Alternatively, this can be achieved with a sequence of queries in which all
except the last have their output hidden from the end user.
■ First, we can ask the model to solve the problem on its own. Since this initial
query doesn't require the student’s solution, it can be omitted.
■ This provides the additional advantage that there is no chance that the
model’s solution will be biased by the student’s attempted solution.

32 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Use inner monologue or a sequence of queries to hide the model's
reasoning process (OPEN IN PLAYGROUND)
■ Next, we can have the model use all available information to assess the
correctness of the student’s solution.

33 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Use inner monologue or a sequence of queries to hide the model's
reasoning process (OPEN IN PLAYGROUND)
■ Finally, we can let the model use its own analysis to construct a reply in the
persona of a helpful tutor.

34 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Ask the model if it missed anything on previous passes
■ Suppose that we are using a model to list excerpts from a source which
are relevant to a particular question.
■ After listing each excerpt the model needs to determine if it should start
writing another or if it should stop.
■ If the source document is large, it is common for a model to stop too early
and fail to list all relevant excerpts.
■ In that case, better performance can often be obtained by prompting the
model with follow up queries to find any excerpts it missed on previous
passes.

35 www.linkalab.it
 Six strategies for getting better results
4. Strategy: Give the model time to "think"
○ Tactic: Ask the model if it missed anything on previous passes
(OPEN IN PLAYGROUND)

36 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Compensate the weaknesses of the LLM by feeding it the outputs of other tools.
○ For example, a text retrieval system (RAG or retrieval augmented generation)
can tell the model
augmented aboutcan
generation) relevant documents.
tell the model about relevant documents.
○ A code execution engine, for example, can
help the model do math and run code.
○ If a task can be done more reliably or
efficiently by a tool rather than by a language
model, offload it to get the best of both.
○ Tactics:
■ Use embeddings-based search to
implement efficient knowledge retrieval
■ Use code execution to perform more
accurate calculations or call external APIs
■ Give the model access to specific
functions
37 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Use embeddings-based search to
implement efficient knowledge retrieval
■ Models can use external informations if
provided as part of its input.
■ This can help models to generate more
informed and up-to-date responses.
■ For example, if a user asks a question
about a specific movie, it may be
useful to add high quality information
about the movie to the model’s input.
■ Embeddings can be used to
implement efficient knowledge
retrieval, so that relevant information
can be added to the model input
dynamically at run-time.
38 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Use embeddings-based search to
implement efficient knowledge retrieval
■ The semantic features of embeddings,
and the availability of fast vector
search algorithms, imply that they can
be used to implement efficient
knowledge retrieval.
■ A text corpus can be split up into
chunks, and each chunk can be
embedded and stored.
■ A given query can be embedded and
vector search can be performed to find
the embedded chunks of text from the
corpus that are most related (closest in
the embedding space) to the query.
39 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Use code execution to perform more accurate calculations or call external
APIs (OPEN IN PLAYGROUND)
■ LLMs cannot be relied upon to perform arithmetic accurately on their own.
■ Where this is needed, a model can be instructed to write and run code
instead of making its own calculations.
■ In particular, a model can be instructed to put code that is meant to be run
into a format such as triple backtick.
■ When the output is ready, the code can be extracted.
■ Finally, if necessary, the output from the code execution engine can be
provided as input to the model for the next query.

40 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Use code execution to perform more accurate calculations or call external
APIs (OPEN IN PLAYGROUND)
■ Another good use case for code execution is calling external APIs.
■ If a model is instructed in the proper use of an API, it can write code that
makes use of it.
■ A model can be instructed in how to use an API by providing it with
documentation and/or code samples showing how to use the API.

41 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific
functions
■ The Chat Completions API allows passing
a list of function descriptions in requests.
■ This enables models to generate function
arguments according to the provided
schemas.
■ Generated function arguments are
returned by the API in JSON format and
can be used to execute function calls.
■ Output provided by function calls can
then be fed back into a model in the
following request to close the loop.
■ This is the recommended way of using
OpenAI models to call external functions.
42 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

43 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

44 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

45 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

46 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

47 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.
● Function specification for the function we'd like the API to generate
arguments for:

48 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.
● Function specification for the function we'd like the API to generate
arguments for:
Notice that we
are inserting
the database
schema into
the function
specification.
This will be
important for
the model to
know about.
49 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

50 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

51 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

52 www.linkalab.it
 Six strategies for getting better results
5. Strategy: Use external tools
○ Tactic: Give the model access to specific functions
■ In our next example, we'll demonstrate how to execute functions whose
inputs are model-generated, and use this to implement an agent that can
answer questions for us about a database.

[system: Answer user questions by generating SQL queries against the Chinook Music Database

[user: Hi, who are the top 5 artists by number of tracks?

[assistant: Function(arguments='{\n "query": "SELECT Artist.Name, COUNT(Track.TrackId) AS

TrackCount FROM Artist JOIN Album ON Artist.ArtistId = Album.ArtistId JOIN Track ON
Album.AlbumId = Track.AlbumId GROUP BY Artist.ArtistId ORDER BY TrackCount DESC LIMIT
5;"\n}', name='ask_database')

[function (ask_database): [('Iron Maiden', 213), ('U2', 135), ('Led Zeppelin', 114),
('Metallica', 112), ('Lost', 92)]

53 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Sometimes it can be hard to tell whether a
change makes your system better or worse.
○ Looking at a few examples may hint at which is
better, but with small sample sizes it can be hard
to distinguish between a true improvement or
random luck.
○ Maybe the change helps performance on some
inputs, but hurts performance on others.
○ Evaluation procedures (or "evals") are useful for
optimizing system designs. Good evals are:
■ Representative of real-world usage (or at
least diverse)
■ Contain many test cases for greater statistical
power (see table below for guidelines)
■ Easy to automate or repeat
54 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Evaluation of outputs can be done by computers, humans, or a mix; computers
can automate evals with objective criteria (e.g., questions with single correct
answers) as well as some subjective or fuzzy criteria, in which model outputs are
evaluated by other model queries.
○ Model-based evals can be useful
when there exists a range of possible
outputs that would be considered
equally high in quality (e.g. for
questions with long answers).
○ The boundary between what can be
realistically evaluated with a
model-based eval and what requires
a human to evaluate is fuzzy and is
constantly shifting as models
become more capable.
55 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers

56 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers
(OPEN IN PLAYGROUND)
■ Here's an example input where both points are satisfied:

57 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers
(OPEN IN PLAYGROUND)
■ Here's an example input where only one point is satisfied:

58 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers
(OPEN IN PLAYGROUND)
■ Here's an example input where none are satisfied:

59 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers
■ The following variation tracks the kind of overlap between the candidate
answer and the gold-standard answer, and also tracks whether the
candidate answer contradicts any part of the gold-standard answer.

60 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers
(OPEN IN PLAYGROUND)
■ Here's an example input with a substandard answer which nonetheless
does not contradict the expert answer:

61 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers
(OPEN IN PLAYGROUND)
■ Here's an example input with answer that directly contradicts the expert
answer:

62 www.linkalab.it
 Six strategies for getting better results
6. Strategy: Test changes systematically
○ Tactic: Evaluate model outputs with reference to gold-standard answers
(OPEN IN PLAYGROUND)
■ Here's an example input with a correct answer that also provides a bit more
detail than is necessary:

63 www.linkalab.it
 www.linkalab.it
Examples
● https://platform.openai.com/examples

64