Gaurav Yengera

Interesting to read actual evidence of what one could expect as an inevitability of scale!

21

3 commentaires

At the AWS Summit Berlin this year, I was asking around everyone one question- "Do you have any LLM application running in production?" The most common response involved hesitation followed by a "No- we have experimented a bunch though". Almost 1.5 years after the chatgpt revolution began, this is the reality as far as industrial adoption of LLMs is concerned. In fact, these are the usage trends: - Most companies have done some form of experimentation and maybe have some prototypes they're struggling to trust- so, nothing in production. - Some companies are actively using some LLM-based applications to help improve their internal processes and workflows (as opposed to exposing LLM results to their end customers). - Still fewer companies have an LLM-based app that their customers interacts with in some way (either directly via a chatbot or so OR by consuming the output of an LLM in some form). - An extremely tiny portion of companies have succeeded in using opensource LLMs in production in addition to the enterprise ones. Despite all the cool developments in the area, the adoption rate as far as productionization of these models go, remains low. I'll ask around the same question at next year's AWS summit and hopefully, will get a lot of yeses ;) #awssummit #llm #ai #opensource #chatgpt

32

3 commentaires

Love this take (and this question) from Ravid Shwartz Ziv . My personal view is that current frontier AI is so far from the threshold of "Intern-AGI" that it would be difficult to produce a benchmark for it where results were greater than 0. But there are some subcomponents that such an agent would need, that we could probably benchmark on their own. * Complex long-chain instruction following with wrong and missing steps and adversarial perturbations * Formal self validation for task completion, rejection or redirection defining a policy with 99.9 trajectory match with specific groups of humans. IE, a SWE Intern should be able to guess, with 99.9 percent accuracy or better, whether it has completed a required subtask for its meta task, or needs more help * A zero shot understanding of how to navigate complex space layouts and estimate the time cost of actions with very very high precision

7

⚡ Meta AI’s LayerSkip Speeds Up LLMs ⚡ Meta’s FAIR team has introduced LayerSkip, a game-changing technique that accelerates large language models (LLMs) while cutting down on memory and power usage! 💡 By using methods like layer dropout, early exit predictions, and self-speculative decoding, LayerSkip achieves 2x speedups without compromising accuracy. 🚀 Now open-source, developers can experiment and enhance their own models for tasks like text summarization and code generation. 🧑‍💻✨ https://lnkd.in/gYemn5ck #MetaAI #LLMs #AIInnovation #LayerSkip #OpenSource #MachineLearning #AI #TechBreakthrough

1

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

386

12 commentaires

Brain Teaser #11: What mathematical arguments can explain how **Layer Normalization** enhances training stability in large language models (LLMs) and Transformer architectures, particularly in mitigating issues like vanishing and exploding gradients? Answer: Training destabilization in neural networks, particularly due to gradient explosion or vanishing, typically occurs because of the high condition numbers of weight matrices during backpropagation. These large condition numbers lead to unstable gradient scaling, either amplifying gradients excessively or diminishing them, hindering effective learning. How Layer Normalization overcomes and brings stability in training is briefly discussed here: https://lnkd.in/gmWm42tQ #MachineLearning #DeepLearning #Transformers #LargeLanguageModels #LayerNormalization #GradientStability #AIResearch #DataScience #NeuralNetworks #AIInsights

16

5 commentaires

Google "Quantum error correction below the surface code" unedited version available in Nature: https://lnkd.in/dqMWEYUB Older version, with more supplemental material (and open access), in arxiv:

13

1 commentaire

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

47

2 commentaires

Highly reliable electronics can be designed at high cost using techniques like triplication and voting. To address reliability challenge at extreme scale, we need energy efficient *and* high-performance reliable systems. PULP Platform is moving in this direction: https://lnkd.in/dhFwwVme

165

5 commentaires

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 commentaire

The amount of human effort that went into RLHF methods at OpenAI for alignment on top of Pretraining + SFT , are surely reducing after gpt3.5 . Meaning you have now LLMs doing most of the RLHF methods such as ranking of responses, given they are already aligned. Of course this process would also be human controlled at top level but the overall human effort surely decreases with every release. Its almost like already aligned AIs now aligning the next generation AI. Just like how humans have created Governments , and Governments once established, can align individual at each level as they pass on the hierarchy of policies from top to down.

1

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

Great to see Pytorch roadmap showing strong investment in composability as strategy for handling scaling. When it comes to scaling hardware customized high performance algorithms on specialized ML accelerators, composability still holds the key but is a challenge. At d-Matrix, we are building our compiler roadmap to address this challenge.

23

🌟 Excited to share my latest contribution to the AWS Machine Learning Blog! 🌟 In this post, we delve into how Amazon trains Sequential Ensemble Models at scale using Amazon SageMaker Pipelines. This approach has enabled us to: ✅ Streamline model orchestration, ✅ Scale training efficiently, and ✅ Enhance predictive accuracy for complex business problems. This work showcases the power of Amazon SageMaker in tackling large-scale machine learning workflows while ensuring scalability and efficiency. 🚀 Check out the blog to learn more: #MachineLearning #SageMaker #AWS #AI #DataScience

9

Exploring the Pros and Cons of Multi-token Prediction in Language Models : A recent study, "Better & Faster Large Language Models via Multi-token Prediction," by researchers from FAIR offers a fascinating look at how predicting multiple tokens simultaneously can significantly enhance the efficiency of large language models like GPT and Llama. The researchers of the paper claim that the models developed via the technique discussed in the paper solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models Key Highlights: Enhanced Efficiency: The multi-token prediction method boosts sample efficiency without additional training time or memory overhead, presenting a compelling case for its adoption in larger models. Improved Performance: This technique has shown to dramatically improve performance, particularly in complex coding tasks and natural language processing, consistently outperforming traditional models. Pros of Multi-Token Prediction: Scalability and Versatility: The method scales impressively with larger models and demonstrates benefits across various data types, including both code and natural language. Inference Speed: Models trained with this method can be up to 3× faster at inference, making them not only more intelligent but also quicker. Potential Challenges: Complexity in Training Dynamics: The method involves multiple output heads which could complicate model training dynamics and require careful tuning. Optimal for Large-Scale Models: While the benefits are clear for large models, smaller models may not see as significant improvements, possibly limiting its applicability in resource-constrained projects. Generalization across Tasks: The effectiveness across all types of tasks is not fully explored, which may limit its applicability in certain scenarios. Future Directions: The paper discusses promising future research paths, including automating the selection of the number of tokens to predict and optimizing vocabulary sizes for multi-token prediction. This approach represents an exciting advancement in AI training techniques. However, it also poses new challenges that necessitate further exploration to fully understand and mitigate potential drawbacks. Paper link : https://lnkd.in/d69Je8wU #AI #MachineLearning #LanguageModels #Innovation #Research

14

Now we have a scientific paper showing that LLMs are bad at reasoning - by 6 engineers from Apple - https://lnkd.in/gFJE9AWU I had previously talked about LLMs being just "Large Language Models" and not really being good for reasoning. This you can deduce from the architecture of LLMs - they just predict the next best probable word in a sequence of words (tokens to be specific), mimicing the patterns in the data it was used to train on. Now, you have experiments proving the same, published in the above paper. This reiterates the fact that one needs to be very careful in designing solutions using Generative AI (LLMs) - you really can't use it for decisions concerning human beings, without human experts in the loop. Having said that, there are innumerable applications of LLMs that where it can be used as an assistive tool for humans - improving their productivity and efficiency being one example. The trick is in carefully designing solutions using LLMs, in ways it can be used, and identifying at the start itself of your AI project, whether the business case you are trying to solve is indeed one of them. Here’s one of my posts from 10 months ago talking about reasoning ability of LLMs: https://lnkd.in/gDq3x9PW

20

1 commentaire

Happy to share that our latest paper is out! In this study, we delve into the world of Slow Feature Analysis, a powerful technique to transform measured data into uncorrelated signals ranging from slow to fast. We've introduced a novel approach that goes beyond traditional methods, addressing the challenge of nonstationary and oscillating features. Our semi-supervised encoder-decoder architecture incorporates a statistical preference for these characteristics, paving the way for more accurate modeling. Curious about the results? We put our approach to the test on both simulated and real industrial processes, with promising outcomes. Thanks to my supervisor, Prof. Biao Huang. Read more about our findings in the full paper! https://lnkd.in/gK6naG5U

61

2 commentaires