Long prompts != better prompts When helping companies with their prompts, one of the first things I try to do is slim them down. More often than not, prompts are overly verbose, leaving more opportunities to lead to misinterpretations. Some findings from a recent paper (linked below) on code generation tasks give some more quantitative proof on this: - Shorter, more concise prompts lead to higher success rates. - All six LLMs studied failed to solve tasks with prompts longer than 180 words. - Among tasks with prompts longer than 150 words, 64.0% resulted in "garbage code," and 37.5% produced meaningless outputs.
I routinely use prompts that are several pages long with GPT4 and assistants for coding. Writing a precise and effective prompt takes time and thought which is why I describe it as coding. I don't like the methodology. "Therefore, we use a 0 50 100 150 200 250 Prompt Length (# Words) (a) Density plot of the number of words in HumanEval’s prompts CodeGen-16B InCoder-1.3B GPT-3.5 GPT-4 SantaCoder StarCoder 0 50 100 150 200 Prompt Length (# Tokens) Pass Fail (b) Distribution of passed/failed tasks’ prompt length Figure 6: Distribution of prompt length prompt’s length (i.e., the number of words) as a proxy metric for task complexity. It is a pragmatic choice but we acknowledge its limitation" What they are saying is interesting, but their 100% failure rate at that size is purely a result of the methodology and a lack of prompt quality.
Good share, Dan; I struggle with this. I feel my prompts are thorough but probably too long. Also, if you are doing a a few shots prompting the examples, do make them long. I will be reading the paper, though it seems to focus on coding, so I wonder about writing.
This is great, but I don't think it is so strange. Even humans have a short working memory! It is also a question of costs and limits: there are too many tokens to memorize. I use short prompts and examples and recap to refresh "ideas." :) Too many think about LLMs as search engines; they are not!
Great point. What I noticed, is that the longer the prompt, the more contradictions it tends to have, which messes with LLM's reasoning and undermines the result that it brings
As with so many situations in life, more words often does not make things better.
Totally agree Dan Cleary ! Sometimes I find a piece-wise/step-wise approach more successful 🙌a bit like building Lego with pieces of building blocks 😀
Size matters. The shorter the better (my personal view).
Co-founder of PromptHub | Helping companies and individuals write, iterate, and collaborate on prompts
8moLink to paper: https://arxiv.org/pdf/2406.08731