Mehul Marvania

Check out this highly recommended course for those interested in optimizing their models: - Introduction to Quantization - Symmetric and Asymmetric Quantization Modes - Hands-on Implementation from Scratch using PyTorch - covering Per Tensor, Per Channel (Axis), and Per Group techniques - Weights Packing: Exploring 2 or 4 Bits Quantization Methods https://lnkd.in/gMAuWXHm

43

Many of ML practitioners might have heard the term "Extending context length" for LLMs. In case if you want to dive deeper into this topic along with its code implementation, give this blog a read. I have described about Position Interpolation, NTK aware and Dynamic NTK method, which are widely used for extending context length.

69

Resharing it: This is a great summary shared by the LinkedIn engineering team about their #llmops and LLM deployment work and their key learnings. #llms #chatgpt4 #gemini #aiml #mlops #llmops https://lnkd.in/d2bH5GRR

2

What if your queries required access to multiple sub-related data from different documents, for an LLM to parse and make sense of before returning a coherent and accurate output? We often ask complicated questions, right? 🔹Well to solve this problem, these researchers released Multi-Head RAG (MRAG) which leverages activations of Transformer’s multi-head attention layer, instead of the decoder layer, as keys for fetching multi-aspect documents. They are taking advantage of the fact that multi-head attention can learn to capture different data aspects. 🔹The interesting part is that MRAG is both cost-effective and energy-efficient. In other words, it does not require additional LLM queries, multiple model instances, increased storage, or multiple inference passes over the embedding model. 🔹Moreover, MRAG can be seamlessly integrated with existing RAG frameworks and benchmarking tools like RAGAS as well as different classes of data stores. 🔹The researchers provide a comprehensive evaluation methodology, including specific metrics, synthetic datasets, and real world use cases to demonstrate MRAG’s effectiveness. 🔸Do check it out! #genai #technology #artificialintelligence Click #linkedangle and follow for more content Set the 🔔 notification on my page, don’t miss a post!

181

10 Comments

Perhaps the most critical thing in a Retrieval Augmented Generation (RAG) system is the retrieval or searching strategy. Poor retrieval will lead to irrelevant documents going to the LLM leading to irrelevant generated answers or worse hallucinations. Is cosine similarity everything? Do check out below illustration which talks about four of the most common strategies for playing around with retrieval strategies in RAG systems: 1. Similarity-based retrieval: This is based on embedding similarity, you can play around with different metrics like cosine, maximal margin relevance etc. 2. Similarity with Threshold retrieval: Useful where we cap the similarity threshold and can help in removing irrelevant document but can drop recall 3. Multi-query retrieval: Make the LLM generate variations of your query and get documents based on these multi-versions of your initial query, often useful to get more recall or coverage of documents 4. Reranker retrieval: Use a model specialised in ranking documents based on search relevance Often you can combine some of these techniques and even use ensemble retrievers. Do explore them and go beyond just cosine similarity!

17

When it comes to "Retrieval" phase of RAG, choosing the right strategy makes all the difference. Sparse retrieval (like BM25/TF-IDF) is great for finding exact keyword matches, while Dense retrieval (like BERT) understands the meaning behind words for more accurate results. Hybrid retrieval combines both, offering the best of both worlds. Each method has its strengths. Understanding when to use them is key to better search results. 

10

4 Comments

Going from PoC to Production for LLMs? 🤔 In the age of GPTs, chat interfaces are ubiquitous. But what if we add a voice/conversational layer on top? Imagine a C-suite member checking sales on the go or a new employee understanding leave policies through a natural voice interface. Intriguing, right? This got me thinking: how do we move from a PoC running on a local machine to a full-fledged production application? Here's what I found out: 1️⃣ Data Management and Knowledge Distillation: Handle and version your knowledge base (PDFs, Images, Excel, ER Diagrams). Keep data up-to-date and manage new information efficiently. How do you ensure information in the data is not outdated? What’s the best way to encode and chunk this data? The questions arise! 2️⃣ Access control and Citations: How do you ensure people cannot access data they are not supposed to have access to? How do you retrieve specific documents to ensure better answers (LLM models work the best when provided with specific information) and lower bills (Each LLM service charges by the number of tokens)? How do you remove any Personally Identifiable Information (PII)? What kind of caching strategies to use? (Pareto principle, 80% of user queries would be for 20% of the questions that get repeated) 3️⃣ Security and Reproducibility: Protect against bad actors and prompt injection. Create secure networks. How do you safeguard your application? It’s a growing field with evolving challenges. 4️⃣ Heart of the Operations (Sort of, I still think your knowledge base is the heart): Now that you have designed and taken care of the above aspects, there is still more. What size of the model to use? What environment to host it in? how to scale and replicate? (If you have a fine-tuned model). 5️⃣ Generating responses: How do you ensure the results generated are accurate and correct? While it is an active area of research, the only effective solution at this point is humans in the loop. Productionizing LLMs is complex, with more questions than answers. I’d love to hear your experiences and insights. What challenges have you faced, and what solutions have you found? If you can think of important things that I may have missed out, feel free to comment below! Ps. I will be writing and building more on this idea in the coming days, stay tuned for more :) #LLM #AI #MachineLearning #DataScience #VoiceInterface

12

1 Comment

It’s been a while since I have shared my experiences of working on AI projects using LLM’s that I have worked on in last 2 years. Hope these experiences will help folks. 😊 so here is the first one   Language Classification with GPT – We were working on a problem scenario where 2 languages to be classified in separate classes were very much similar in terms of words available in them and up to a good extent semantically also, but the languages were actually different. As the languages were domain specific, it was expected that we have to empower GPT. We used different models starting with GPT 3.5, GPT4 Turbo 0125, and then GPT 4 Omni and at last GPT-4 Turbo 2024-04-09. Of all the models we got the best accuracy with the last one. The last 2 are the latest ones available and as per Microsoft recommendation the go-to GPT models. However with GPT-4 Turbo 2024-04-09 also we didn’t got the desired result by the same pipeline we were using (words filter + vector search + RAG). While searching for the method to resolve the issue, I came across a research paper around ‘Feedback loop’ and how it can improve the results. (Sorry didn’t remember the paper name now , will try to find and post that).  We applied the same with some experimentations and it worked. So the focus here is – if you have a use cases where GPT/LLM’s is getting confused in language classification after application of latest techniques like vector search and RAG, feedback loop can be introduced after the final output as a last step. It can enhance the accuracy in very confusing but important cases. Hope this will help community. 😊 #NLP #LLM #GPT #RAG #DataScience #AI #OpenAI #Chatgpt OpenAI

42

2 Comments

- SwiftKV is a novel model transformation and distillation procedure (by folks at Snowflake) designed to optimize large language model (LLM) inference. Very cool findings in the paper (as presented in the vLLM meetup on 11/13). - SingleInputKV: Prefills later layers' KV cache using earlier layer outputs, reducing computation on prompt tokens. The key point is that the first few layers of attention in LLMs are similar (nice find), allowing SwiftKV's SingleInputKV mechanism to skip computations in later layers during inference. By using a single earlier layer's output to compute the KV cache for subsequent layers, it significantly reduces both computation (by 50%) and memory usage (by 62.5%) without sacrificing model performance. - AcrossKV: Merges KV caches of neighboring layers to minimize memory usage and increase batch size for higher throughput. - For models like Llama-3.1-8B and 70B, SwiftKV reduces prefill compute by 50%, memory usage by 62.5%, and achieves up to 2× higher throughput and 60% lower time per token. https://lnkd.in/guhEBW5G

62

Gen AI in Identity Verification I'm proud to highlight the continued success of MetaMap‘s Generative AI solution for identity verification, launched in Q1 2024. After months of production use, it has demonstrated remarkable stability and efficiency. Key contributors: 1. Vikram Kalabi : Your robust ML pipeline has proven its resilience, consistently delivering accurate results at scale. 2. Vladislav I. : The human-computer interface you developed has been pivotal, allowing seamless human-in-the-loop feedback and continuous improvement. 3. Alan Garcia : Your operational leadership has ensured smooth integration of human oversight, maintaining high quality standards. This project exemplifies how cutting-edge AI, when thoughtfully implemented, can bring long-term value and reliability to critical processes. Kudos to Vikram, Vladislav, Alan, and the entire team for their ongoing commitment to excellence. #GenerativeAI #IdentityVerification #SustainedInnovation

76

5 Comments

Watching Gradient drop a Fortune 100's Databricks jobs costs by >50% in a few days is crazy. Everyone was shocked. 🔥 🔥 Best part was - it was all automatic. 🔥 🔥 While everyone was in meetings, building other pipelines, or just grabbing lunch - Gradient was at work dropping costs with no overhead at all. 💰 Gradient literally prints money for our customers ⚡ What's even crazier is Gradient isn't even done yet, it's still early on its journey in optimization The power of AI/ML in the cloud infrastructure world is, in my opinion, the killer app. And it's amazing to see this new technology in action helping real people. But don't take my word for it. Try it out for yourself. Book a demo here if you want help automating and managing your Databricks jobs Sync Computing

45

4 Comments

🌿 SandalQuest: Digitizing Knowledge of Sandalwood Cultivation with AI 🌿 I’m excited to share our project SandalQuest, created as part of the ML-Fiesta: AI/ML Hackathon organized by International Institute of Information Technology Bangalore! Our team developed an AI-powered pipeline to preserve and provide easy access to the indigenous knowledge surrounding sandalwood cultivation in Karnataka. 🔑 What We Built: - An Automatic Speech Recognition (ASR) model tailored for colloquial Kannada, capturing informal language and dialect variations. - A speech-based question-answering system that allows users to ask questions and retrieve relevant answers from a transcribed audio database. - A searchable audio database that helps users access traditional knowledge through speech. 💻 Tech Stack: - Python, PyTorch for AI & Machine Learning - Whisper for Speech Recognition - Hugging Face Transformers for NLP - MongoDB for database management 🔍 Impact: We’re helping preserve valuable knowledge on sandalwood cultivation and making it easily accessible for farmers, researchers, and enthusiasts. 🔗 Check out the project: https://lnkd.in/dMxRP76s 🚀 Team Name: Code Wizards Team Members: - Shani Sinojiya (AI/ML & Backend Developer) - Mohammad Anas Africawala (AI/ML Engineer) - Tisha Patel (Full Stack Developer) It’s been an amazing experience working on this project, and I’m proud of what we’ve achieved. Looking forward to applying these insights to real-world applications! #AI #MachineLearning #SpeechRecognition #Kannada #ColloquialSpeech #Sandalwood #DataScience #NaturalLanguageProcessing #AIForGood #Hackathon #TechForGood #GoogleCloud #AIResearch #DeepLearning

10

2 Comments

We've just brought out a new update of my book, Generative AI with LangChain. It covers a lot of new information and code examples with new models: https://amzn.to/3XlSYbA. If you've bought the book already, you should be able to access the updated pdf online. If not, here's your chance to get a book about applying #genai with #Python. #llms #openai #gemini

60

1 Comment

I came across an excellent GitHub repository packed with hands-on Jupyter notebooks for anyone eager to explore LLMs. It covers: 🔹 Introduction to LLMs – Understanding the basics. 🔹 Tokens & Embeddings – The foundation of language modeling. 🔹 Model Internals – Peeking under the hood of LLMs. 🔹 Text Applications – Classification, clustering, and topic modeling. 🔹 Prompt Engineering – Crafting prompts for better performance. 🔹 Advanced Text Generation – Tools and techniques. 🔹 Semantic Search – Intelligent information retrieval. 🔹 Multimodal Models – Combining text and image understanding. 🔹 Customization – Building embeddings, fine-tuning BERT, and generative models. Big thanks to the repo creator for sharing such a valuable resource. Here is the repo: https://lnkd.in/gUKg7Jp7

77

2 Comments

📊 Optimizing RAG Applications with TruLens: A Quick Guide Recently explored TruLens for evaluating and fine-tuning Large Language Models (LLMs) in Retrieval Augmented Generation (RAG) applications. Here's what I learned: Key Evaluation Metrics: ✅ Groundedness: Ensures responses are factually supported by context ✅ Answer Relevance: Measures if responses directly address queries ✅ Context Relevance: Evaluates quality of retrieved information The Process: Set up automated evaluation pipelines using TruLens Track performance across different configurations Use metrics to optimize: Chunking strategies Retrieval parameters Context window sizes Embedding models Benefits: Data-driven optimization Automated quality assessment Systematic performance tracking Clear metrics for improvement Result: More accurate, relevant, and reliable RAG applications with measurable improvements in response quality. #AI #MachineLearning #LLM #RAG #DataScience #TruLens https://lnkd.in/gHQnjREi

3

As promised last week, here is my blog on how to evaluate Machine learning models for Sparse classification problems such as opioid overdose, where only 2% of the dataset will be positive and the remaining dataset will be negative. Take a look and share your thoughts! https://lnkd.in/gGqCvab9 #machinelearning #classification #sparsedata #modelmetrics #precision #recall #deeplearning #artificialintelligence

5

The hype has shifted from LLMs to Agents. The first ideas emerged right after the recent "AI-resurrection" (read: ChatGPT release 2 years back) and now it has become a buzzword on all levels. Don't get me wrong, I am an "Agent-optimist". Both consumers and businesses want to move on from "request-based assistance" ("summarize me this article") to autonomous minions helping us with everyday tasks without explicit orders given. We have already seen these attempts of introducing Agents to our email services and allowing them to take over the computer control. The explosive burst of such tools will happen in 2025, though – just scan the billboards of AI conferences' participants and ask your favorite LLM to extract the most commonly used term. What is less obvious is that Agents will eventually follow the LLM's path – huge models are great for consumers and everyday support. Still, enterprise businesses require distilled dedicated fine-tuned versions tailored to specific tasks. Think of Agents as not of a single almighty deity but more like a team of skilled professionals working on assigned projects (and later starting their own ones). Each of these "distilled Agents" will have their core competency, communicate with each other, and enrich the world picture for the whole group. The skills will not be fully isolated, though each has an area of strength. In the same way as it is now a reality in effective software development teams: an engineer or PM can draw a mock-up and it is warmly welcomed, then the final prototype work is better left to the designer. There are opportunities and challenges. Companies (or single project owners) will be able to "hire" a specific Agent from the marketplace to "join the existing A-Team". The biggest technical riddle is about having an efficient and universal protocol for Agents of various origins to effectively communicate with each other. And not drink too much coffee in the kitchen between tasks. #ai #ml #future #futureofai #aiagents

8

4 Comments