Hardik Jain

Most (maybe all?) LLM performance benchmarks like Artificial Analysis go in depth on *online* inference. *Batch* inference seems simpler since almost all companies run some form of embarrassingly parallel workloads. But batch inference is different from other map-reduce style workloads. The biggest differences are the use of GPUs and often mixed CPU + GPU compute. Today, tons of energy is going into improving systems for batch inference, and batch inference is just as important as online inference (companies have accumulated tons of data over the years, and if you want to use AI to extract insights or make decisions based on that data, you will need to process that data in a batch fashion). A bunch of companies are speaking at #RaySummit on their best practices for batch inference: Apple: "petabyte scale embedding generation" Roblox: "efficient batch inference at Roblox" Uber: "large scale generative AI batch prediction with Ray and vLLM" eBay: "pioneering next-gen AI platform" Handshake: "job description parsing with batch LLM inference" https://lnkd.in/deWkzHEC

260

4 Comments

🚀 Bengaluru-based Conversational AI Startup Gnani AI Raises $3M 💰 Gnani AI, India's first voice-first small language model (SLM) for vernacular languages, has secured $3M in funding from tech holding company Info Edge. The investment will fuel the startup's expansion plans. Key Highlights: • Gnani AI is building a series of voice-based SLMs, trained on millions of hours of proprietary audio datasets and billions of conversations in 12 Indian languages • The startup has over 100 customers in banking, financial services, insurance, telecom, automotive, and healthcare industries • Gnani AI's models run on the edge, reducing costs and enabling quick fine-tuning based on enterprise needs • The startup specializes in creating customized models for individual companies, adding real value to their data CEO Ganesh Gopalan emphasizes the importance of multimodal models, focusing on voice and text fusion. By taking their models to enterprises and helping them build on top of it, Gnani AI is revolutionizing the conversational AI landscape in India. #ConversationalAI #VoiceFirst #IndianLanguages #Funding #Expansion

12

1 Comment

Interesting to read actual evidence of what one could expect as an inevitability of scale!

21

3 Comments

I was shocked to see the below graph 🤯. I saw this Google Search trend graph on Chat GPT VS Perplexity VS Claude. I now feel like I am in a bubble surrounded by tech folks who use claude etc and to me it feels normal. But as you can see in the graph, the actual use of Claude & Perplexity is very very minute compared to Chat GPT. Especially Perplexity feels like its non-existent. And now a month back Chat GPT Search is also launched which will definitely take a share of this audience. Looking at this graph also makes me optimistic about building, It clearly shows that there is a huge gap in the market, and similar products can be built. What do you guys think ? #ai #startups #chatgpt #claude #perplexity

21

4 Comments

While Gen AI engineers in India are rightly concerned about #hallucination in LLMs, yet most would rarely appreciate that genius folks in our own country evolved extraordinary measures to arrest hallucination thousands of years ago, which included sophisticated #guardrails which were to be 'assimilated' in the brain itself, impacting the way it perceived reality and 'generated' content (Isn't the brain one of the most hallucinating natural 'LLMs' after all ?). This was not just to ensure that facts were rendered correctly. The attempt was to prevent disasters of another kind and of a much more serious nature. After all, language in India was considered an important vehicle for connecting to the divine, and not merely for communicating with one another. And we often fail to realize this #opensource framework, so to say, was built upon by different intellectuals in different parts of the country, and India ended up having several wonderful languages in different regions, languages which look different, but have a lot in common. Did you not realize that no matter where in India we are from, names of most of us can be traced to this one language? https://lnkd.in/g4Njx4mG #llms #nlp #generativeai

16

3 Comments

Exploring Hybrid Retrieval-Augmented Generation (RAG) for Enhanced Information Retrieval In my recent research in the  realm of information retrieval, I came across Hybrid RAG which balances between semantic (vector similarity) and syntactic (keyword) search approaches for achieving accurate and comprehensive results. A cutting-edge solution that combines the strengths of both methods to revolutionize how we interact with data. Key Features: - Semantic Search (Vector Similarity): Ideal for handling queries with typos or nuanced intents, leveraging powerful embeddings like HuggingFace to capture semantic relationships. - Syntactic Search (Keyword Matching): Perfect for precise matches on keywords, abbreviations, and names, employing techniques such as BM25Encoder for effective keyword search. - Hybrid Approach: By blending vector and keyword search scores using advanced fusion techniques like Reciprocal Rank Fusion (RRF), Hybrid RAG ensures robust and context-aware retrieval. Why Hybrid RAG Matters: Semantic search (using dense embeddings) excels in capturing semantic similarity and context, making it effective for understanding the meaning behind queries and documents. On the other hand, syntactic search (using techniques like BM25) is better at capturing exact matches and structural similarities. Combining both ensures a more comprehensive coverage of relevant information. Implementing Hybrid RAG: -- Set up with Pinecone for indexing and retrieval. -- Utilize HuggingFaceEmbeddings for semantic search. -- Employ BM25Encoder for efficient keyword matching. -- Integrate PineconeHybridSearchRetriever for seamless fusion and retrieval. Unlock the power of Hybrid RAG to enhance your information retrieval capabilities. Explore more about how this innovative approach can elevate your data analytics and decision support systems. #HybridRAG #InformationRetrieval #SemanticSearch #SyntacticSearch #AI #MachineLearning #DataScience #TechnologyInnovation #PineconeAI #GenAI #MLOps 

7

Controversial take 🚀 Large Language Models (LLMs) are akin to monoliths similar to OS packaging in early days where core OS and applications were all bundled together. A lot of such application packages were never used. Just like unused OS applications and packages, a lot of knowledge in todays model may not be useful for many specialized use-cases particularly in B2B, which is also evident in need for fine-tuning or ICL. In the applications world, as containerization took center stage, it became flexible to add or remove packages needed for a system. Docker emerged as the go-to for this purpose, allowing engineers to build enterprise systems by excluding unnecessary packages besides using a relevant starting point, further adding their own packages. For example, users can begin with a base OS container and customize it by adding or removing packages. Over the last year, with LoRA Adapters(and similar variants) have been goto methods for bringing in your knowledge bits to create customized models. Other techniques such as modelmerge help merge multiple models into a single one. These are interesting directionally, however it needs to become even simpler similar to layers in containers, where knowledge bits can be added/excluded seemlessly. Looking ahead, the future of LLM seems like would mirror containers. Starting with a pre-trained base model tailored to your use-case, you can then add layers your knowledge bits on top. "Language Model as a container" offers the simplicity of packaging your custom knowledge bits. #LLM #huggingface #SLM

16

6 Comments

Week 10 NeuroNuggets 🧠 : Prompt Engineering Using LLMs - Part 3   In this week's NeuroNugget, we will go over a second set of prompting techniques to improve an LLM's performance:   1) Chain of Thought ⛓: Chain-of-Thought (CoT) prompting, contrasts with the standard few-shot 💉 prompt approach by encouraging the language model (LLM) to reveal its reasoning process. Instead of just providing examples and expecting direct answers, CoT prompts guide the LLM to unfold its thought process. This method often leads to answers that are not only more accurate but also easier to understand and interpret. It's like having a detective 🕵‍♀️ mentor who not only shows you evidence but also teaches you how to think like a detective to crack the case on your own.   2) ReAct: Reason & Action 🏹(ReAct) prompting is a method that helps language models do two things at once: explain their thinking and decide what to do next. This approach lets them think through problems and make plans that they can change and improve as they go. It's like having a smart friend 👧 who not only gives you advice but also helps you understand the reasons behind it, making your plans more organized and effective.   3) Directional ↗ Stimulus: Directional Stimulus Prompting (DSP) introduces a small set of tokens called "directional stimulus" into the prompt. These tokens act like hints or clues, guiding the language models (LLMs) towards the right direction. For instance, in a summarization task, the directional stimulus could be specific keywords that should be included in the summary.   4) Meta Prompting: A Meta Prompt serves as a generic, structured template 🎶 that outlines how to approach a category of tasks without specific examples. It guides LLMs in reasoning through tasks by providing a framework that can be adapted with specific details. This method enhances efficiency and precision in utilizing LLMs, focusing on the methodology of problem-solving rather than the specific content. For eg, a meta prompt to create a product description would outline sections like key features, specifications, use cases, and conclusion, emphasizing structured content generation over specific product details.   This would be my last post in this series ;) Hope these posts have been helpful! Thank you for learning with me 💌  

3

Perhaps the most critical thing in a Retrieval Augmented Generation (RAG) system is the retrieval or searching strategy. Poor retrieval will lead to irrelevant documents going to the LLM leading to irrelevant generated answers or worse hallucinations. Is cosine similarity everything? Do check out below illustration which talks about four of the most common strategies for playing around with retrieval strategies in RAG systems: 1. Similarity-based retrieval: This is based on embedding similarity, you can play around with different metrics like cosine, maximal margin relevance etc. 2. Similarity with Threshold retrieval: Useful where we cap the similarity threshold and can help in removing irrelevant document but can drop recall 3. Multi-query retrieval: Make the LLM generate variations of your query and get documents based on these multi-versions of your initial query, often useful to get more recall or coverage of documents 4. Reranker retrieval: Use a model specialised in ranking documents based on search relevance Often you can combine some of these techniques and even use ensemble retrievers. Do explore them and go beyond just cosine similarity!

17

Training LLMs using reinforcement learning using human feedback. I was slacking on this topic for 6 months and finally was able to go through it, and wow I am amazed at what technologies are being worked on in production-ready systems these days. Generally in RL of LLMs, we have a reward model (let's say a good/bad text classifier) that gives a reward (prediction 1 for good, -1 for bad) and an RL algorithm (based on multiple loss functions) updates the weights of the LLM model to achieve better results. This is an iterative process between the reward model, the LLM, and the RL algorithm. Overall, my take is the most important aspect of training LLM using RLHM is obtaining that human feedback and training a reward model.

28

Brain Teaser #10: Suppose we are fine-tuning a large language model using Low-Rank Adaptation (LoRA). We possess two low-rank update matrices, δWA​ and δWB​​, both associated with the rank "r". Considering our objective to preserve the model's current knowledge while incorporating new data from diverse features, what "mathematical rationale" should guide the selection of one of the update matrices (δWA​ or δWB) for low-rank adaptation? Hint: Consider the specific characteristics or properties of the matrices that can help in developing mathematical reasoning. Here is my response: The norm ratio, calculated as the ratio of the nuclear norm to the Frobenius norm of the update matrix, is an essential metric for assessing how updates impact a model. It provides insights into the model's stability and adaptability. Here’s a brief overview: https://lnkd.in/ekB_q9XX #MachineLearning #DeepLearning #NaturalLanguageProcessing #LoRA #FineTuning #AI #DataScience #ModelOptimization #NeuralNetworks #ArtificialIntelligence #TechInnovation #DataDriven #KnowledgeIntegration #Research #LLMs

30

4 Comments

Thanks to Siddhi Shroff I am debugging complex pytorch systems much more efficiently in VScode. A simple config parameter that boost productivity "debug.inlineValues":"on" For those working with pytorch tensors of 3+ dims - this makes the shapes permutation so much clearer. https://lnkd.in/g7bkE6-7.

32

1 Comment

I came across an excellent GitHub repository packed with hands-on Jupyter notebooks for anyone eager to explore LLMs. It covers: 🔹 Introduction to LLMs – Understanding the basics. 🔹 Tokens & Embeddings – The foundation of language modeling. 🔹 Model Internals – Peeking under the hood of LLMs. 🔹 Text Applications – Classification, clustering, and topic modeling. 🔹 Prompt Engineering – Crafting prompts for better performance. 🔹 Advanced Text Generation – Tools and techniques. 🔹 Semantic Search – Intelligent information retrieval. 🔹 Multimodal Models – Combining text and image understanding. 🔹 Customization – Building embeddings, fine-tuning BERT, and generative models. Big thanks to the repo creator for sharing such a valuable resource. Here is the repo: https://lnkd.in/gUKg7Jp7

77

2 Comments

There is a revolution happening in AI IDEs. IDE have been idle for quite a few years now, the last good thing that happened in that arena was VS Code by Microsoft which was Open-Source. Thanks to this One project todays folks are building completely new IDE experience on Steroids. - Cursor.sh is shipping Coding Agents and the IDE is super cool. - Zed.dev is bring human & AI together. - Replit just updated their UI and started shipping their own Agents too. - Windsurf by Codeium is another Agentic IDE. The flip side of this is a different species of prompt based development tool : - Vo.dev by Vercel - Llamacoder by Hassan El Mghari - Bolt.new - Claude Artifacts Developers, founder, Investors who are missing out on these will feel the pinch later. All of them are super cool and very soon will become the norm. Microsoft is already betting big on AI for the github Co-Pilot. The next wave will be about "image-to-code" system that we will start seeing. Super excited for this sector. Is anyone building anything like this in India then kindly comment or dm me. #AIAgents #startups #ai

22

10 Comments

🚀 Exciting Update: Fine-Tuning Sarvam 1 for Bhojpuri NLP! 🌟 Two months ago, I fine-tuned Sarvam 0.5 (a 2B LLM) on an instruction-set Bhojpuri dataset for just 1 hour. While the results were promising, the model often mixed Hindi with Bhojpuri in its responses. Fast forward to now, I’ve been working with Sarvam 1, the latest version of the model. After training it for just 1 epoch (about 5 hours) on a much better GPU, I’m thrilled with the results! The fine-tuned model produces accurate and fluent Bhojpuri responses without mixing in Hindi. I’m proud to share that this fine-tuned Bhojpuri model is now available on Hugging Face for everyone to explore and use. 🎯 This marks a significant step towards improving regional language processing, and I’m excited to see how this model will contribute to the Bhojpuri NLP ecosystem. Check it out, and let me know your thoughts! 👇 https://lnkd.in/g28wfEKr Note: Original Sarvam 1 is not pre-trained on Bhojpuri language data and it is a text completion model, not an instruction tuned. #NLP #Bhojpuri #AI #LLM #FineTuning #Sarvam

21

3 Comments

𝐌𝐚𝐡𝐚𝐫𝐚𝐬𝐡𝐭𝐫𝐚 & 𝐔.𝐒. 𝐄𝐱𝐢𝐭 𝐏𝐨𝐥𝐥𝐬: 𝐂𝐚𝐧 𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐯𝐞 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 𝐊𝐞𝐞𝐩 𝐔𝐩? 🗳️📊 The Maharashtra exit polls landed yesterday, and last week, the U.S. polls were trending everywhere. But here’s the real question: 𝐖𝐡𝐲 𝐝𝐨𝐞𝐬 𝐩𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐯𝐞 𝐦𝐨𝐝𝐞𝐥𝐥𝐢𝐧𝐠, 𝐞𝐯𝐞𝐧 𝐰𝐢𝐭𝐡 𝐜𝐮𝐭𝐭𝐢𝐧𝐠-𝐞𝐝𝐠𝐞 𝐀𝐈, 𝐬𝐭𝐢𝐥𝐥 𝐬𝐭𝐫𝐮𝐠𝐠𝐥𝐞 𝐰𝐢𝐭𝐡 𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧𝐬? 𝐖𝐡𝐲 𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐯𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 𝐅𝐚𝐥𝐥 𝐒𝐡𝐨𝐫𝐭: ❓ How can models adapt when up to 10% of voters decide on election day? ❓ What happens when exit polls sample only 5-10% of voters, often skewed toward urban and affluent groups? ❓ Can we avoid overfitting to outdated coalition patterns when history suddenly changes? ❓ How do we manage voter data with hundreds of variables while avoiding overwhelming complexity? 𝐇𝐨𝐰 𝐖𝐞 𝐂𝐚𝐧 𝐃𝐨 𝐁𝐞𝐭𝐭𝐞𝐫: ✨ 𝐇𝐲𝐛𝐫𝐢𝐝 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡𝐞𝐬:: Bayesian models paired with ML to better manage uncertainty. ✨ 𝐃𝐲𝐧𝐚𝐦𝐢𝐜 𝐀𝐥𝐠𝐨𝐫𝐢𝐭𝐡𝐦𝐬:Real-time adjustments with reinforcement learning. ✨ 𝐌𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠: Combine exit polls, social media sentiment, and even economic indicators. ✨ 𝐄𝐱𝐩𝐥𝐚𝐢𝐧𝐚𝐛𝐥𝐞 𝐀𝐈: Build trust using tools like SHAP and LIME to interpret predictions. 𝐓𝐡𝐞 𝐑𝐨𝐚𝐝 𝐀𝐡𝐞𝐚𝐝 🚀: Elections are unpredictable, but that’s what makes refining predictive modelling so exciting. By addressing biases, embracing real-time adaptability, and leveraging advanced methods, we can better understand voter dynamics. Also, read my recent article experimenting with U.S. election predictions on Towards AI - "https://lnkd.in/gZ7h-2cD " 𝐖𝐡𝐚𝐭 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞𝐬 𝐨𝐫 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬 𝐝𝐨 𝐲𝐨𝐮 𝐭𝐡𝐢𝐧𝐤 𝐜𝐨𝐮𝐥𝐝 𝐫𝐞𝐝𝐞𝐟𝐢𝐧𝐞 𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧 𝐟𝐨𝐫𝐞𝐜𝐚𝐬𝐭𝐢𝐧𝐠? 𝐃𝐫𝐨𝐩 𝐲𝐨𝐮𝐫 𝐢𝐝𝐞𝐚𝐬 𝐢𝐧 𝐭𝐡𝐞 𝐜𝐨𝐦𝐦𝐞𝐧𝐭𝐬 𝐛𝐞𝐥𝐨𝐰! 💬 🔔 Love Machine Learning insights? Follow Sanjay Nandakumar for more thought-provoking posts, cutting-edge ideas, and deep dives into ML applications! 🚀 💬 React ❤️ | Comment 💭 | Repost 🔄 #ElectionForecasting #PredictiveModeling #MachineLearning #DataScience #AI #Statistics #DeepLearning #ArtificialIntelligence #ExitPolls #Maharashtra #MaharashtraEletion2024 #USA #USElections2024 #Trump

6

4 Comments

Want to build AI apps and get paid for it? I am looking for a self-organized AI Engineering Intern to work at Unwind AI. Here's what you'll work on: ↳ LLMs ↳ AI Agents and Multi-agent apps ↳ RAG Systems ↳ New and upcoming opensource frameworks ❌ No resumes needed. ❌ No formal interviews. Instead: ✅ Check out the opensource tutorials repo ✅ Create a new AI tutorial with an AI framework of your choice ✅ Open a PR with your tutorial A complete tutorial would typically have: ↳ Code Files ↳ Requirements ↳ Detailed README The most unique and complete tutorial wins the internship at Unwind AI. Look at already existing examples for reference. Want in? Wear your opensource hat and get started now: https://lnkd.in/dW6b_dEn Please like 👍 and share ♻️ this post with your network. #AI #Internship #OpenSource #Python

283

29 Comments

🚨Paper Alert 🚨 ➡️Paper Title: SGP-Bench: Can Large Language Models Understand Symbolic Graphics Programs? 🌟Few pointers from the paper 🎯Assessing the capabilities of large language models (LLMs) is often challenging, in part, because it is hard to find tasks to which they have not been exposed during training. Authors take one step to address this challenge by turning to a new task: focusing on symbolic graphics programs, which are a popular representation for graphics content that procedurally generates visual data. 🎯LLMs have shown exciting promise towards program synthesis, but do they understand symbolic graphics programs? Unlike conventional programs, symbolic graphics programs can be translated to graphics content. Here, Authors characterize an LLM's understanding of symbolic programs in terms of their ability to answer questions related to the graphics content. 🎯 This task is challenging as the questions are difficult to answer from the symbolic programs alone -- yet, they would be easy to answer from the corresponding graphics content as they verified through a human experiment. To understand symbolic programs, LLMs may need to possess the ability to imagine how the corresponding graphics content would look without directly accessing the rendered visual content. 🎯They used this task to evaluate LLMs by creating a large benchmark for the semantic understanding of symbolic graphics programs. This benchmark is built via program-graphics correspondence, hence requiring minimal human efforts. They evaluated current LLMs on their benchmark to elucidate a preliminary assessment of their ability to reason about visual scenes from programs. 🎯They found that this task distinguishes existing LLMs and models considered good at reasoning perform better. 🎯Lastly, they have introduced “Symbolic Instruction Tuning (SIT)” to improve this ability. Specifically, they queried GPT4-o with questions and images generated by symbolic programs. Such data are then used to finetune an LLM. They also found that SIT data can improve the general instruction following ability of LLMs. 🏢Organization: Max Planck Institute for Intelligent Systems, University of Cambridge, Mila - Quebec Artificial Intelligence Institute, Université de Montréal, University of Tübingen, Massachusetts Institute of Technology 🧙Paper Authors: Zeju Qiu, Weiyang Liu, Haven (Haiwen) Feng, Zhen Liu , Tim Z. Xiao, Katherine M. Collins, Joshua Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf 1️⃣Read the Full Paper here: https://lnkd.in/gu7E4VWX 2️⃣Project Page: https://lnkd.in/gud2vcGD 3️⃣Code: https://lnkd.in/geF-tvYR 🎥 Be sure to watch the attached Video -Sound on 🔊🔊 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.

8