Henry Peter’s Post

“What We Learned from a Year of Building with LLMs“ A wonderful article on building LLMs in production. Thank you Eugene Yan ,Bryan Bischof, Charles Frye, Hamel H., Jason Liu and Shreya Shankar! My highlights below 👇 ---- Part One focuses on tactical insights, including prompting tips, evaluation strategies, Retrieval-Augmented Generation (RAG), and fine-tuning considerations. Parts two and three cover operational aspects and strategy. [Links in comments] ---- Prompting: 1 - Start every LLM application with prompting. 2 - Build on various prompting techniques: in-context learning w/ n-shot prompts, chain of thought (CoT), relevant resources (RAG, etc.). 3 - For n, aim for 5 examples per prompt; don’t hesitate to go up to 12. 4 - Lean towards smaller, focused prompts over large, multi-purpose ones. ---- RAG (Retrieval-Augmented Generation): 1 - Don’t rely solely on vector embeddings for search. 2 - Combine keyword search with embeddings for a hybrid approach. 3 - Start with RAG before fine-tuning; it's cost-efficient and effective. 4 - RAG remains relevant even with longer context models. ---- Fine-Tuning: 1 - Consider fine-tuning only when prompting and RAG are insufficient. 2 - Fine-tuning adds complexity and costs: requires annotated data, model evaluation, and hosting. ---- Evaluation: 1 - Use LLMs as judges - decent correlation to human judges 2 - Evaluation metrics: Likert scales, binary classifications, pairwise comparisons. Binary easiest cognitive load. 3 - Safety and PII defects are managed well, but hallucinations still around ~5-10% and tough to get under 2%. ---- Overall, a dense (but readable) article offering valuable tactical insights into LLM development. Highly recommended for a deep dive into current best practices and tips for effective LLM implementation. #genai #enterprise #llms

What We’ve Learned From A Year of Building with LLMs https://lnkd.in/gETctFFs #genai #LLMs

💡 Your RAG pipelines may NOT always require context retrieval; at times, they can simply provide answers based on pre-trained knowledge. An often overlooked aspect of RAG pipelines in the wild is that there are instances where retrieved knowledge isn't necessary to answer common questions, or where pre-trained knowledge outperforms what's retrieved. 🤔 How do you train your LLM in the RAG pipeline to decide? ⛳ A recent paper "When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively" proposes a tailored training approach for LLMs, called ADAPT-LLM, where they are trained to generate a special token (⟨RET⟩) when they don't know the answer to a question. 📖 Some more insights ⛳ Two approaches are discussed: Closed-book question answering, relying solely on parametric memory, and open-book question answering, coupling LLMs with IR systems. ⛳ They authors then propose a hybrid approach, where LLMs use parametric memory for popular questions and employ IR systems for less popular ones ⛳ An evaluation conducted on an open-domain question answering dataset demonstrates that ADAPT-LLM effectively discerns when to retrieve information and when to rely on its parametric memory. ⛳ ADAPT-LLM consistently outperforms traditional LLMs under various configurations, showing improvements in question answering accuracy. ⛳ The primary bottleneck for ADAPT-LLM's performance lies in the IR system, suggesting potential areas for improvement in future research. I really like the idea of optimizing LLMs for RAG applications and this is a promising direction. I remember discussing ADAPT-RAG not long ago, which had a similar approach, but this one looks much neater as it doesn't require any additional components. 🚨 I post #genai content daily, follow along for the latest updates! #genai #llms #rag

Ever wondered how to manage massive documents without losing crucial context? Unlock the secrets of document loaders, text splitting, and LangChain creation, using tools like Beautiful Soup and recursive text splitters, to handle even the largest files with ease. Master the art of maintaining content integrity and efficiently summarizing vast datasets with advanced techniques. #openai #llms #generativeai

Ever wondered how to manage massive documents without losing crucial context? Unlock the secrets of document loaders, text splitting, and LangChain creation, using tools like Beautiful Soup and recursive text splitters, to handle even the largest files with ease. Master the art of maintaining content integrity and efficiently summarizing vast datasets with advanced techniques. #openai #llms #generativeai

When optimizing RAG, teams often obsess over the shiny thing— embedding, rerankers, or other complex techniques. Yet, the🥇 is somewhere else.. In our recent work, we saw that the most valuable uplift can come from the simplest part — chunking. Here's something that really surprised me: After testing different chunk sizes, simply switching to page-sized chunks (2048 tokens) alone improved our hit rate by 27.7 points. That's a massive jump—and it happened without changing anything in our embedding models or metadata. Just chunk size. But, as with most things, there's a catch. Larger chunks bring both improvements and challenges: - Improved context understanding - Better retrieval accuracy - Higher computational costs - Increased processing time So how do you find the optimal chunk size for your system? Well, it depends. We started with a simple line search, checking different variations of the param. We also went full “Bayesian Optimization” mode with Optuna, but this is typically overkill. The use case and complexity of your data are important. We worked with financial PDFs so it made a lot of sense to go for —- page-sized chunks. I know…Is it even considered “chunking” if you take the whole page? Why not simply use Long Context? Honestly, I didn’t care as long as the results were dramatically better. Long-context has shown in the past it’s power in other experiments (see first comment). I still believe that the future is in RAG. Key learning: Chunking is a powerful tool for improving retrieval accuracy, but it's not just about making everything bigger. You need to test and find the sweet spot that works for your specific needs. What chunk sizes have worked best for your use case?

Ever wondered how to manage massive documents without losing crucial context? Unlock the secrets of document loaders, text splitting, and LangChain creation, using tools like Beautiful Soup and recursive text splitters, to handle even the largest files with ease. Master the art of maintaining content integrity and efficiently summarizing vast datasets with advanced techniques. #openai #llms #generativeai

Ever wondered how to manage massive documents without losing crucial context? Unlock the secrets of document loaders, text splitting, and LangChain creation, using tools like Beautiful Soup and recursive text splitters, to handle even the largest files with ease. Master the art of maintaining content integrity and efficiently summarizing vast datasets with advanced techniques. #openai #llms #generativeai

Will emergent models with new skills and better benchmarks soon make auto-prompt optimization obsolete? The question is better phrased as “What does it take to build product-grade LLM apps now and in the future?” Currently, we need to combine three types of optimization: - Model fine-tuning - Auto-prompt optimization - Hyperparameter optimization, such as tuning the number of chunks and chunk size in RAG GPT4o1 with built-in reasoning and Claude 3.5 Sonne with computer use, are examples of how LLMs are growing both vertically and horizontally in their capabilities. The future is both exciting and uncertain. 1️⃣ When will we not need model fine-tuning? AGI: 100% accuracy on any task for any pre-trained LLM with perfect latency and the lowest cost. If fine-tuning can save end users' costs and improve latency, fine-tuning will continue to exist. 2️⃣ When will we not need auto-prompt optimization? If both fine-tuned and pre-trained models can achieve 100% accuracy on your task If both LLM prompt rephrasing and few-shot learning become ineffective in improving accuracy 3️⃣ When will we not need hyperparameter optimization in our LLM application? If each stage reaches 100% accuracy, including not only the LLM but also embedding models. And how far are we from reaching 100% accuracy on all existing skills and all new, upcoming skills? We’re as far from it as we are from GPT4 to GPT100. What do you think? #adalflow #artificialintelligence #machinelearning #llms #autoprompting

Ever wondered how to manage massive documents without losing crucial context? Unlock the secrets of document loaders, text splitting, and LangChain creation, using tools like Beautiful Soup and recursive text splitters, to handle even the largest files with ease. Master the art of maintaining content integrity and efficiently summarizing vast datasets with advanced techniques. #openai #llms #generativeai