Derek Thomas’ Post

Excellent guide on how to deploy and benchmark LLMs.

Excited to have presented our paper on Bayesian Prompt Ensembles at ACL 2024. In this paper, we explore the use of multiple prompt re-formulations to capture model uncertainty in LLMs. We derive the connection between prompt re-formulation and model uncertainty/Bayesian NNs and propose a method to learn the relative importance of different prompts for best performance. As part of this work, we open-sourced a code package and tutorial with which you can a) easily use LLMs to perform classification tasks and b) apply our method, BayesPE, to improve uncertainty estimation through the use of multiple prompts. Hope it is useful to folks out there! Paper: https://lnkd.in/etpRRwJC Code: https://lnkd.in/eDHdQ6k4

What is your biggest challenge in evaluating LLMs today? Here's a must-read article by Rama Ramakrishnan on gaining value from LLMs with a practical LLM cost equation. What sort of data curation will you need? How will you fix errors? Can you estimate ongoing run costs? #GenAI#ArtificialIntelligence#BusinessTransformation https://lnkd.in/e4tyKWzi

I just published a blog on how to fine tune LLMs and create custom datasets. In this blog, I cover: - What fine-tuning is and its benefits - When to fine-tune a model - Methods for fine-tuning, including full fine-tuning and parameter-efficient fine-tuning - How to prepare your custom dataset Check it out here: https://lnkd.in/gDcrrBGP

https://goo.gle/3AVM3yc Another scenario where RAG is a better choice to ground responses... "Trade-offs of the RAG approach Advantages to using this approach are that RAG automatically benefits from ongoing model evolution, particularly improvements in the LLM generating the final response. As this LLM advances, it can better utilize the context retrieved by RAG, leading to more accurate and insightful outputs even with the same retrieved data generated by the query LLM. A disadvantage is that modifying the user's prompt can sometimes lead to a less intuitive user experience. In addition, the effectiveness of grounding depends on the quality of the generated queries to Data Commons."

🥳 New small article published!!  📄 Key metrics for evaluating a RAG system 📄 RAG systems are becoming increasingly popular, but questions remain about evaluation methods to ensure they are working well. In this article you will find some innovative approaches to evaluate your RAG system and see how good it is without much effort. It has been written to be intuitive and easy to follow. You can see it displayed on Medium. Thank you very much for reading it, I hope you like it!! Medium publication: https://lnkd.in/dqcyb6_s #rag #llm #genai #machinelearning

**WorkBench - New LLM Evaluation Dataset** LLM agents can access external tools, such as calendars and search engines, via API calls. They’re able to overcome some common shortcomings of LLMs like i) inability to interact with the real world, and ii) hallucinations. Businesses are excited about agents. They take LLMs from "cool they can summarize an email for me" to "OMG they can send emails, schedule meetings and chase leads too!" Even better, this can all be done via a chat interface like Slack. But are these agents reliable in practice? WorkBench shows that, for agents out-the-box, the answer is No. WorkBench is a sandbox environment with databases and tools representing a realistic workplace environment. We tested 5 models and found the best-performing approach (ReAct with GPT-4) was **only 43% accurate**. In many cases, errors led to severe consequences such as sending private emails to the wrong people! I want to thank all the people who helped shape WorkBench: Olly, my co-lead author, for making this a lot of fun to work on. Bertie, Patricio, Tanaya for contributions throughout, this paper hugely benefited from frequent iteration. Tom, Duncan, Tishtrya, Daniel, Tony, Vlad for great feedback at the end, making this accessible to a non-academic audience. Paper: https://lnkd.in/eU44UC_J Github: https://lnkd.in/eRVHSzJG

Well written with speed comparison tables

Anthropic context retrieval where u can simply add context to your embeddings. for better understanding

I just read an interesting post about the possible expiration of RAG (Retrieval Augmented Generation) based frameworks. Recall that these frameworks extend the capabilities of a large language model in documentation or Q&A (Questions and Answers) systems by chunking the documentation, vectorizing it with an embedding engine, retrieving the most relevant chunks for the answer by measuring the semantic distance to the input and using the aggregate of these segments to generate the answer. A Twitter user named @shaun.agi has shared a brief analysis claiming that the future context window of new LLMs, with Google's Gemini at the forefront, renders the advantages of RAG systems moot. Here is his analysis: https://lnkd.in/d2DVXcq4 In my view, this conclusion is not accurate, especially in the short term. Firstly, while 10M tokens might seem substantial, and indeed it is for a single issue, it falls short for specialized domain data products or documentation amassed over years by significant organizations. At Red Team Shield, S.L., for instance, we manage documentation several orders of magnitude larger. Secondly, the use of proprietary LLMs outside an organization's premises is not always viable for handling the most sensitive information. It might be suitable for Open Source information, but not for data we do not want to leave the company's premises. For these reasons, I believe the author's claim that a 10M context window can solve 90% of use cases is an overstatement. On the contrary, I anticipate that sensitive information will continue to be managed in the short to medium term using modular versions of RAG systems, complete with all necessary redundancies and guardrails, and achieving the best balance with the investment of human hours in information management. Remember, humans may be out of the loop in operations, but never in responsibilities. Short term == summer 2024 Medium term == summer 2025 Who knows == september 2025