Prem Sagar Gali

Interesting to read actual evidence of what one could expect as an inevitability of scale!

21

3 Comments

🚀 Bengaluru-based Conversational AI Startup Gnani AI Raises $3M 💰 Gnani AI, India's first voice-first small language model (SLM) for vernacular languages, has secured $3M in funding from tech holding company Info Edge. The investment will fuel the startup's expansion plans. Key Highlights: • Gnani AI is building a series of voice-based SLMs, trained on millions of hours of proprietary audio datasets and billions of conversations in 12 Indian languages • The startup has over 100 customers in banking, financial services, insurance, telecom, automotive, and healthcare industries • Gnani AI's models run on the edge, reducing costs and enabling quick fine-tuning based on enterprise needs • The startup specializes in creating customized models for individual companies, adding real value to their data CEO Ganesh Gopalan emphasizes the importance of multimodal models, focusing on voice and text fusion. By taking their models to enterprises and helping them build on top of it, Gnani AI is revolutionizing the conversational AI landscape in India. #ConversationalAI #VoiceFirst #IndianLanguages #Funding #Expansion

12

1 Comment

Responsible engineers should avoid copying code directly from online sources, including LLMs or platforms like Stack Overflow, without fully reviewing, understanding, and adapting it for safe integration into their codebase. Thoughtful assessment is essential to ensure code behaves as expected and meets project requirements. That said, I believe LLMs can be a fantastic resource to accelerate learning and deepen understanding across a range of topics—as long as they’re used with careful consideration. #LLM #engineering #ChatGPT

10

1 Comment

While Gen AI engineers in India are rightly concerned about #hallucination in LLMs, yet most would rarely appreciate that genius folks in our own country evolved extraordinary measures to arrest hallucination thousands of years ago, which included sophisticated #guardrails which were to be 'assimilated' in the brain itself, impacting the way it perceived reality and 'generated' content (Isn't the brain one of the most hallucinating natural 'LLMs' after all ?). This was not just to ensure that facts were rendered correctly. The attempt was to prevent disasters of another kind and of a much more serious nature. After all, language in India was considered an important vehicle for connecting to the divine, and not merely for communicating with one another. And we often fail to realize this #opensource framework, so to say, was built upon by different intellectuals in different parts of the country, and India ended up having several wonderful languages in different regions, languages which look different, but have a lot in common. Did you not realize that no matter where in India we are from, names of most of us can be traced to this one language? https://lnkd.in/g4Njx4mG #llms #nlp #generativeai

16

3 Comments

This is detailed guide to calculate GPU Memory 💾 requirement for serving LLMs. Ps - it doesn't take context length into consideration Website 📖- https://lnkd.in/gf7YzqJt

116

3 Comments

Most (maybe all?) LLM performance benchmarks like Artificial Analysis go in depth on *online* inference. *Batch* inference seems simpler since almost all companies run some form of embarrassingly parallel workloads. But batch inference is different from other map-reduce style workloads. The biggest differences are the use of GPUs and often mixed CPU + GPU compute. Today, tons of energy is going into improving systems for batch inference, and batch inference is just as important as online inference (companies have accumulated tons of data over the years, and if you want to use AI to extract insights or make decisions based on that data, you will need to process that data in a batch fashion). A bunch of companies are speaking at #RaySummit on their best practices for batch inference: Apple: "petabyte scale embedding generation" Roblox: "efficient batch inference at Roblox" Uber: "large scale generative AI batch prediction with Ray and vLLM" eBay: "pioneering next-gen AI platform" Handshake: "job description parsing with batch LLM inference" https://lnkd.in/deWkzHEC

260

4 Comments

📢 Starting a thread on Fine-tuning Large Language Models (LLMs) 📢 1/n ● Large Language Models (LLMs) have revolutionised computational systems' ability to understand and generate human language. Building upon traditional language models, LLMs address limitations in handling rare words, overfitting, and capturing intricate linguistic patterns. ● Fine-tuning is a crucial process that adapts pre-trained LLMs to specific tasks or domains. This involves further training the model on a smaller, targeted dataset, refining its pre-existing knowledge and enhancing performance on specific tasks with reduced data and computational requirements. ● The fine-tuning process typically involves a seven-stage pipeline: dataset preparation, model initialisation, training environment setup, fine-tuning, evaluation and validation, deployment, and monitoring and maintenance. Each stage is essential for tailoring the LLM to meet specific requirements and ensuring optimal performance. ● There are different types of fine-tuning: unsupervised fine-tuning, supervised fine-tuning (SFT), and instruction fine-tuning via prompt engineering. The choice of method depends on the availability of labeled data and the specific task requirements. ● Parameter-Efficient Fine-Tuning (PEFT) techniques have emerged to address the computational challenges associated with fine-tuning large models. PEFT methods, such as LoRA, QLoRA, and adapters, allow for efficient fine-tuning with fewer parameters, reducing computational costs and mitigating overfitting. ● Evaluation and validation are critical for assessing the performance of fine-tuned LLMs. Metrics such as cross-entropy, perplexity, and accuracy are used to measure the model's ability to generalize and perform the target task effectively. ● Deployment of fine-tuned LLMs involves making them operational and accessible for specific applications. This includes setting up the necessary infrastructure, developing APIs, and optimizing model performance for inference. ● Monitoring and maintenance are essential to ensure the ongoing performance and reliability of deployed models. Continuous monitoring of metrics, retraining on updated data, and addressing potential biases or security vulnerabilities are crucial for maintaining model effectiveness and ethical considerations. ● Various platforms and frameworks facilitate LLM fine-tuning, such as Hugging Face, OpenAI, Google, and NVIDIA. These platforms provide tools for data preparation, model selection, hyperparameter tuning, and deployment, streamlining the fine-tuning process. ● Fine-tuning LLMs raises ethical considerations regarding bias, fairness, accountability, and transparency. Addressing these concerns is vital for responsible AI development, ensuring that models are deployed ethically and their outputs are trustworthy. Stay tuned for upcoming posts 📬 where I go through these in detail 📤 .

7

My take is that the training process performs a kind of negative sampling to filter out samples that have low information, hence improving training efficiency.

4

3 Comments

How are Graph Convolutional Neural Networks (GCNNs) a generalisation of Convolutional Neural Networks (CNNs)? My next blog in the Graph series on "Deep Learning with Graphs" is on the way (well, it's already written on paper) and it's going to be a blockbuster. The beauty of Graph Machine Learning is that the more you read about it, the more you feel that your neurons (real ones!) are learning to think better! And you're already in love with it! Coming to the point: One might recall the convolution operations in a CNN. You have a grid of pixels representing an image. You have a 3 x 3 filter. You slide the filter over the image and carry out aggregation to obtain a new pixel value. You proceed in the same fashion over the whole grid, sliding towards the right , bottom, carrying out the convolution operation and finally getting a new transformed, reduced image through the series of aggregations. You repeat this through the different convolutional layers which together form a Convolutional Neural Network Now think of a Graph. It is not simple grid like an image. The topology of a Graph is arbitrary. There is no notion of left, bottom, right in a Graph like in an image which is represented through a grid of pixels. You cannot slide over a Graph as simplistically you did and performed the convolutional operations in an image. How do you approach the problem? How do you carry out convolutional operations in a Graph? Now take a node in the Graph. Take all the nodes which are the immediate neighbours of this node. The convolutional operation here is: You aggregate the information from all the neighbours and pass it to the node. The information is nothing but the emedfings of the neighbour nodes. You add together the emedding of the node under consideration. You transform the aggregated embedding by multiplying with a weight Matrix and transform the node's own embedding by multiplying with a different weight matrix. This is the typical linear transformation operation in a NN. You apply non linearity - the activation function like ReLu. Now think about a CNN taking the same grid and considering a central pixel. Consider the rest of the pixels around the central pixel. You are essentially doing the same thing: taking information from the neighbour pixels along with the central pixel, aggregating, as I described above for the node in the Graph. Hence a Graph Convolutional Network is a generalisation of CNN The story goes, you do the same for every node on the Graph. Each node in the Graph represented through a different neural network architecture! This is amazing 🔥🔥🔥💓💓💓 #graphs #machinelearning #deeplearning My Technical Blogs Library: 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on: Data Science, Data Engineering, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics

22

The possible downgrading risks for NVIDIA (or any similar high-growth stock) typically stem from a mix of internal and external factors. Here are the key risks: 1. Valuation Concerns High P/E Ratio: NVIDIA's P/E ratio is significantly above industry averages, raising concerns that the stock is overvalued. DCF Valuation Gap: If actual growth doesn’t meet expectations, analysts might downgrade their price targets. 2. Macroeconomic Factors Interest Rate Hikes: Rising interest rates can hurt growth stocks as future cash flows are discounted more heavily. Recession Risks: Economic slowdowns could reduce demand for NVIDIA’s products in gaming, AI, and data centers. 3. Industry-Specific Risks Semiconductor Cycles: The semiconductor industry is highly cyclical, and any downturn could lead to reduced revenue. Competition: Intense competition from AMD, Intel, and emerging players in AI chip markets could erode market share. Supply Chain Challenges: Global supply chain issues, including chip shortages, could hurt production and profitability. 4. Regulatory Risks Geopolitical Tensions: U.S.-China relations could impact NVIDIA’s revenue from China, a significant market for its products. Government Regulations: Stricter export controls or antitrust regulations might restrict growth opportunities. 5. Technological Risks Innovation Pressure: Failure to stay ahead in AI, GPUs, or other high-tech sectors could lead to downgrades. Customer Concentration: Heavy reliance on a few large customers (e.g., cloud service providers) exposes NVIDIA to risk if these relationships weaken. 6. Execution Risks Overestimated AI Growth: While AI is a strong growth driver, over-optimistic forecasts could lead to disappointment. Mergers and Acquisitions: Failed or costly acquisitions (e.g., ARM deal termination) can impact investor sentiment. 7. Market Sentiment Profit-Taking: If NVIDIA’s stock has experienced a significant rally, any bad news could trigger profit-taking. Short-Term Earnings Miss: Missing quarterly earnings expectations can lead to immediate downgrades. Mitigation for Investors Diversify portfolios to balance sector risks. Stay updated on earnings reports and industry trends. Monitor macroeconomic indicators and geopolitical developments. Considering these risks, NVIDIA's long-term potential remains strong, but investors should prepare for possible downgrades during periods of market volatility or sector corrections. NVIDIA

2

1 Comment

100 𝐌𝐨𝐬𝐭 𝐈𝐧𝐟𝐥𝐮𝐞𝐧𝐭𝐢𝐚𝐥 𝐏𝐞𝐨𝐩𝐥𝐞 𝐢𝐧 𝐀𝐈 (𝐓𝐢𝐦𝐞 𝐌𝐚𝐠𝐚𝐳𝐢𝐧𝐞) Surprises in this list [1] Elon Musk missing in the list [2] Anil Kapoor in the list. #timemagazine #ai #generativeai #nlproc #llms #top100 #influential #people

32

3 Comments

Recently, I have heard about NVIDIA NIM (NVIDIA Inference Microservices), which is claimed to be a groundbreaking platform transforming AI model deployment. Some features that I got were: 1. NIM optimizes inference engines for various models and hardware, ensuring top-notch performance and cost efficiency. 2. As part of NVIDIA AI Enterprise, it delivers enterprise-grade support with rigorous validation and regular security updates. 3. NIM supports diverse AI models, from large language models to medical imaging, making it versatile across multiple domains. 4. Its microservices seamlessly integrate with existing infrastructures and popular AI frameworks like Hugging Face and LangChain, simplifying deployment and enhancing scalability. 5. Backed by major cloud providers and infrastructure partners, NVIDIA NIM is the go-to solution for efficient, scalable AI deployment. Learn more about how NVIDIA NIM can revolutionize your AI projects! Currently, looking into it that how can it be a great option for developers and Engineers. #AI #MachineLearning #NVIDIA #TechInnovation Sources: NVIDIA Technical Blog: https://lnkd.in/dy2GeTAh NVIDIA Newsroom: https://lnkd.in/dbrnhkpw

15

optimizing Large Language Models (LLMs) from a software engineering perspective can significantly impact latency and costs. with my recent time invested in building production-grade RAG solutions, here are a few key pointers I felt can be taken into consideration: - Long context prefill in the prompt passed to the LLM can increase latency and costs. opt for prompt compression or decomposition to mitigate this issue. - avoid using an agentic framework with multi-turn behavior, like self-iterating loops, or techniques like self-consistency prompting, to prevent higher latency and LLM costs. agentic frameworks like crewAI or autogen by ms can be overkill with higher inference time - if using vector embeddings, opt for a vector db with lower retrieval latency, such as sqlite-vss or redis. - implement semantic caching to reduce LLM API calls and lower response time. reducing output length can also aid in processing or decoding phases. you can also explore using YAML instead of JSON for structured output as YAML is less verbose and hence, fewer tokens, however, I still believe this is task-dependent. - tasks performed by LLMs that are not strictly sequential can be split into parallel calls. sequentially required steps can be merged into a single LLM call to avoid round-trip latency. - ensure proximity between LLMs and tools infrastructure-wise. Allowing third-party plugins to deploy on the infrastructure where large models run can further reduce latency. - using async programming paradigms ( for eg. concurrent API calls ) or parallel programming ( for eg. using multi-threading for parallel API requests ) can also help reduce the inference time and increase the throughput. do check LiteLLM (YC W23) though. - using query routing strategies like semantic router by Aurelio AI or Martian or Unify or RouteLLM by Anyscale and LMsys can also help boost cost efficiency and speed. I am yet to test these out though, but hopefully soon. some notes: - there is a tradeoff between throughput and time per output token: if we process let’s say multiple user queries concurrently, we'll have higher throughput compared to running the queries sequentially, but we'll take longer to generate output tokens for each user. - large language Models (LLMs) generate text in a two-step process: "prefill", where the tokens in the input prompt are processed in parallel, and "decoding", where text is generated one 'token' at a time in an autoregressive manner. Each generated token is appended to the input and fed back into the model to generate the next token. Generation stops when the LLM outputs a special stop token or when a user-defined condition is met (e.g., some maximum number of tokens has been generated). what are the other potential strategies for building production-grade rag solutions? #SoftwareEngineering #LLMOptimization #DataProcessing #GenAI #llms

42

10 Comments

LLM Visualization LLMs are advanced deep-learning models based on transformer decoders. It is essential to understanding various components of LLMs like - Embeddings - LayerNorm - Self Attention - Projection - MLP - Softmax - Output #llms #visualization #nlproc #deeplearning #embedding

1,420

41 Comments

LLM Visualization LLMs are advanced deep-learning models based on transformer decoders. It is essential to understanding various components of LLMs like - Embeddings - LayerNorm - Self Attention - Projection - MLP - Softmax - Output #llms #visualization #nlproc #deeplearning #embedding

1,685

65 Comments

𝗟𝗹𝗮𝗺𝗮 𝟯 𝘂𝗻𝘃𝗲𝗶𝗹 𝗮 𝗳𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 𝘁𝗿𝗮𝗱𝗲𝗼𝗳𝗳 𝘁𝗵𝗮𝘁 𝗳𝗲𝘄 𝗮𝗿𝗲 𝘀𝗲𝗲𝗶𝗻𝗴⬇️ 🦙Llama 3 just came out and to the surprise of many people, the L3-8B version can match the L2-70B one on several benchmarks. Several people also argued that the chinchilla paper (https://lnkd.in/dXaRqyea) was wrong because the llama 3 scaling is very far from chinchilla-optimal. 🤔 In my opinion, this is due to a misunderstanding of what the chinchilla paper is about and more generally of the effects of scale in deep learning. The bitter lesson (https://lnkd.in/d3sVYYgp) has become common knowledge in the field thanks to the GPT family that popularized it. We hear everywhere that "bigger models are better": Yes it's true, but in what sense? Many people have a mental model along the lines of "bigger models are more capable" meaning they would be able to achieve levels of performance that smaller models can't. ❌ This is not true with LLMs and is mainly because we train only once on tokens (most of them at least) and therefore overfitting is not possible. The 𝘳𝘦𝘢𝘭 advantage of bigger models over smaller ones is that they converge *faster*. In other words, you can reach a given level of NLL (modeling performance) with fewer training iterations. ❔ Then a natural question arises: Should I train a smaller model with more or a bigger model with fewer iterations? That's where the chinchilla paper brings a response. They say that the question they are answering is: "Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?" (implying "to achieve the best possible NLL"). 🧐 Coming back to Llama 3, it's obvious that the Llama team is aware of this work and there is a reason why they didn't follow this recipe. My guess is that they are taking into account the inference cost in the balance (which is completely out of the scope of the chinchilla paper). When one "overtrains" an 8B model, one ends up with a worse "NLL to training compute" ratio, but also with a better "NLL to inference compute" ratio. ✅ Now if we think about it from a business perspective, it's a very powerful tradeoff because you can choose to buy performance either at a fixed cost (training) or at a variable one (inference). Hope this post helped you! #Llama3 #Meta #LLM #AI

9

3 Comments

Thanks to Siddhi Shroff I am debugging complex pytorch systems much more efficiently in VScode. A simple config parameter that boost productivity "debug.inlineValues":"on" For those working with pytorch tensors of 3+ dims - this makes the shapes permutation so much clearer. https://lnkd.in/g7bkE6-7.

32

1 Comment

𝗟𝗶𝗴𝗵𝘁 𝗻𝗼𝘁𝗲𝘀 𝗳𝗼𝗿 𝗮 𝗠𝗼𝗻𝗱𝗮𝘆 𝗲𝘃𝗲𝗻𝗶𝗻𝗴: I had been reading through Advanced RAG techniques and GraphRAGs late evening yesterday in perparation of my next blogs on LinkedIn. Some points that caught my attention: In order to improve the retrieval accuracy in the Retrieval-Augmented Generation (RAG) approach, we may use: ⭕Re-Ranking of Context: we use LLMs like GPT-4 or specialised models like ColBERT or Cohere to re-rank the context and then pass only the top most relevant contetxts to the LLM to get the final answer. 𝗧𝗵𝘂𝘀, 𝘄𝗲 𝘂𝘀𝗲 𝗮𝗻 𝗟𝗟𝗠 𝘁𝗼 𝗿𝗲-𝗿𝗮𝗻𝗸. 𝗧𝗵𝗲 𝗶𝗱𝗲𝗮 𝗯𝗲𝗶𝗻𝗴 𝘁𝗼 𝗶𝗺𝗽𝗿𝗼𝘃𝗲 𝘁𝗵𝗲 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗽𝗮𝘀𝘀𝗲𝗱 𝘁𝗼 𝘁𝗵𝗲 𝗟𝗟𝗠 ⭕GraphRAGS: Going from Community Summaries -> Community Answers ->Global Answers 📌 Fundamentally in a GraphRAG , we get chunks out of the source documents as in the standard RAG process 📌 We then carry out Entity Extraction and Relationship Capture from these chunks. This is essentially building up of Nodes and Edges of the Knowledge Graph. 📌 We carry out Element Summmaries: This is the summarization of all the nodes and relationships extracted. 📌 The community summaries are randomnly shuffled and the reason for shuffling being that they shouldn't be concentrated in a single context window. 📌 We make multiple calls to the LLM to answer the query against each of the chunk of the community summaries. 𝗔𝗻𝗱 𝗳𝗼𝗿 𝗲𝗮𝗰𝗵 𝗰𝗵𝘂𝗻𝗸 𝘄𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲 𝗶𝗻𝘁𝗲𝗿𝗺𝗲𝗱𝗶𝗮𝘁𝗲 𝗮𝗻𝘀𝘄𝗲𝗿𝘀 𝗮𝗻𝗱 𝗲𝗮𝗰𝗵 𝗮𝗻𝘀𝘄𝗲𝗿 𝗶𝘀 𝘀𝗰𝗼𝗿𝗲𝗱 𝗮𝗻𝗱 𝗿𝗮𝗻𝗸𝗲𝗱. 📌𝗔𝗴𝗮𝗶𝗻 𝗮𝗻 𝗟𝗟𝗠 𝗶𝘀 𝘂𝘀𝗲𝗱 𝘁𝗼 𝘀𝗰𝗼𝗿𝗲 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿 𝗯𝗲𝘁𝘄𝗲𝗲𝗻 𝟬 𝗮𝗻𝗱 𝟭𝟬𝟬 𝗮𝗻𝗱 𝗳𝗶𝗻𝗮𝗹𝗹𝘆 𝘁𝗵𝗲 𝘁𝗼𝗽 𝗿𝗮𝗻𝗸𝗲𝗱 𝗮𝗻𝘀𝘄𝗲𝗿𝘀 𝗮𝗿𝗲 𝘁𝗮𝗸𝗲𝗻 𝗮𝗻𝗱 𝘂𝘀𝗲𝗱 𝘁𝗼 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲 𝗮 𝗳𝗶𝗻𝗮𝗹 𝗴𝗹𝗼𝗯𝗮𝗹 𝗮𝗻𝘀𝘄𝗲𝗿. The point I'm trying to make is: The above remined me of ChatGPT training process [https://lnkd.in/dWpuyCpV] . Here, we had human labellers rank the ChatGPT responses based on “Likert Scale” – the ranking being based on the quality of the responses. This was part of the Step 2 of ChatGPT training process. 𝗔𝗻𝗱 𝘁𝗵𝗲 𝘀𝘁𝗼𝗿𝘆 𝗮𝗳𝘁𝗲𝗿 < 𝟮 𝘆𝗲𝗮𝗿𝘀: the human ranking is already replaced by LLM ranking! Coming Soon: ⭕RAG Beyond Basics: Part 15 of my Graph Series of Blogs ⭕The Marriage of Retrieval Augmented Generation with Knowledge Graphs: Part 16 of my Graph Series of Blogs Watch this space! #graphs #deeplearning #rags #llms My Tech Blogs Library 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on Data Engg, Data Science, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics Image credits: Claudia Sonnet 3.5!

12