Sarath Shekkizhar

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

47

2 Comments

It’s been an honor working with Gil Weinberg! Thanks so much! This demo is from one of the chapters in my thesis on #AI driven raga agnostic automatic accompaniment for alapana (non-rhythmic improvisations) in South Indian Classical Music. Since the algorithm is raga agnostic, the system can accompany for any raga performed! More demos on a violin playing #robot coming soon! #MusicTechnology #MachineLearning #HumanRobotInteraction #HumanComputerInteraction #AutomaticAccompaniment

217

32 Comments

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

386

12 Comments

❓ Struggling to keep up with generative AI’s rapid evolution—or wondering how to turn research into real-world impact? Join me and Ravin Kumar, Senior Research Data Scientist at Google Labs, for a live podcast recording where we’ll tackle these questions and more. 🎙 What we’ll cover: 📚 Generative AI at Google: How tools like Notebook LM bridge cutting-edge research with practical applications for technical and non-technical users alike. 🌍 Democratizing AI: The shift from training models to API-driven solutions, and the trade-offs between open-source and hosted tools. 🧩 Cultural & Practical Challenges: How to adapt generative AI tools for diverse global audiences and improve their user experience. 🛠 AI Roles & Personas: The evolving landscape of AI—from researchers fine-tuning models to developers building with APIs—and how to get started. 🏁 The Competitive Landscape: How companies like Google are pushing the boundaries of generative AI with new tools and features. This is your chance to hear from someone working on the frontlines of AI research and products—don’t miss it! Sign up here: https://lnkd.in/gwmzHunh

11

Drawing from his transformative experience, Yann LeCun emphasizes the urgent need for India to foster domestic players capable of creating pre-commercial R&D environments that bridge the gap between academia and industry, enabling talented researchers to thrive. These inspiring insights were shared during his recent visit to India, where he encouraged the younger generation to embrace research as a meaningful and impactful pursuit. Speaking on Nikhil Kamath YLog as well (https://lnkd.in/eh-swBuD), he reflected on parallels from a few decades ago in France and current India, where, despite humble beginnings and attending a regular school, he built a remarkable career in research through perseverance, continuous publishing, and exploration. Today, he stands as one of the pioneers of AI—often regarded as a "godfather" of the field. His message resonates deeply: while opportunities may not always be abundant, dedication and passion can carve out a fulfilling and impactful career path. As a society, we need to recognize that research is not limited to academia but is a viable and rewarding profession. Spending years in a PhD or working in academic/industrial research lab should not be seen as a failure, but rather as the relentless effort of a dedicated researcher contributing to groundbreaking ideas. Through rigorous work—publishing at top conferences with both theoretical insights and empirical evidence—they illuminate directions that may seem unfamiliar, unexplored, or even daunting at first but hold immense potential for innovation. Researchers excel in the 'Art of Persuasion' through a foundation of scientific rigor. I am particularly inspired by individuals with a similar vision for India, such as Samir Saran, Shamika Ravi, Gaurav Aggarwal, Manish Gupta, Mitesh Khapra, Sunayana Sitaram and their colleagues & connections, who are driving research and entrepreneurial innovations on an Indian scale.

13

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

🚨 **Addressing the Reproducibility Crisis in ML-Based Science** 🚨 Another fascinating post from “AI Snake Oil“, where Princeton based Arvind Narayanan and Sayash Kapoor argue that ML-based science is facing a reproducibility crisis. Here's a summary of their findings and proposed solutions: 🔍Key Issues Identified: *Data leakage is a major cause of reproducibility failures. *Complex ML code and lack of standardization complicate performance evaluation. *Overoptimism* and biases in publication contribute to the crisis. 📊 Evidence of the Crisis: *41papers from 30 fields show widespread errors affecting 648 studies. *Common pitfalls include no train-test split, non-independence, and improper feature selection. 🔧 Proposed Solution: *Develop a taxonomy of data leakage to understand and mitigate errors. *Introduce model info sheets to connect ML model performance with scientific claims, enhancing transparency and reproducibility. 💡 Authors demonstrate their observations over an interesting case study: *Civil war prediction studies claiming superior ML model performance were found to be flawed due to data leakage. *According to authors, correcting these errors showed no substantial improvement over traditional models! In summary, Narayanan and Kapoor urge the adoption of best practices and ongoing scrutiny to ensure the reliability of ML-based scientific research. I highly recommend following these two and #AISNAKEOIL #MachineLearning #Reproducibility #DataScience #ResearchIntegrity #DataLeakege https://lnkd.in/dhrCthMm

2

2 Comments

"Are large language models superhuman chemists?" So what these researchers did was make a test -- a benchmark. They made a test of 7,059 chemistry questions, spanning the gamut of chemistry: computational chemistry, physical chemistry, materials science, macromolecular chemistry, electrochemistry, organic chemistry, general chemistry, analytical chemistry, chemical safety, and toxicology. They recruited 41 chemistry experts to carefully validate their test. They devised the test such that it could be evaluated in a completely automated manner. This meant relying on multiple-choice questions rather than open-ended questions more than they wanted to. The test has 6,202 multiple-choice questions and 857 open-ended questions (88% multiple-choice). The open-ended questions had to have parsers written to find numerical answers in the output in order to test them in an automated manner. In addition, they ask the models to say how confident they are in their answers. Before I tell you the ranking, the researchers write: "On the one hand, our findings underline the impressive capabilities of LLMs in the chemical sciences: Leading models outperform domain experts in specific chemistry questions on many topics. On the other hand, there are still striking limitations. For very relevant topics the answers models provide are wrong. On top of that, many models are not able to reliably estimate their own limitations. Yet, the success of the models in our evaluations perhaps also reveals more about the limitations of the exams we use to evaluate models -- and chemistry -- than about the models themselves. For instance, while models perform well on many textbook questions, they struggle with questions that require some more reasoning. Given that the models outperformed the average human in our study, we need to rethink how we teach and examine chemistry. Critical reasoning is increasingly essential, and rote solving of problems or memorization of facts is a domain in which LLMs will continue to outperform humans." "Our findings also highlight the nuanced trade-off between breadth and depth of evaluation frameworks. The analysis of model performance on different topics shows that models' performance varies widely across the subfields they are tested on. However, even within a topic, the performance of models can vary widely depending on the type of question and the reasoning required to answer it." And with that, I'll tell you the rankings. You can log in to their website at ChemBench.org and see the leaderboard any time for the latest rankings. At this moment I am seeing: gpt-4: 0.48 claude2: 0.29 GPT-3.5-Turbo: 0.26 gemini-pro: 0.25 mistral_8x7b: 0.24 text-davinci-003: 0.18 Perplexity 7B Chat: 0.18 galactica_120b: 0.15 Perplexity 7B online: 0.1 fb-llama-70b-chat: 0.05

3

🔥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 - 𝐓𝐨𝐩 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 🔥 🤖 I've conducted a mock interview with my student, Bheesetti Anand, focused on Generative AI and LLM (Large Language Models) interview questions. This session is designed to benefit other students preparing for similar interviews, helping them secure their dream roles in the field of Generative AI. 📔 Here’s why watching the video can help you make a 𝐬𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 𝐜𝐚𝐫𝐞𝐞𝐫 𝐬𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈: ✅ 𝐂𝐨𝐦𝐩𝐫𝐞𝐡𝐞𝐧𝐬𝐢𝐯𝐞 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞: Detailed exploration of top interview questions related to Generative AI and Large Language Models. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Real-world examples and practical applications to understand complex concepts better. ✅ 𝐄𝐱𝐩𝐞𝐫𝐭 𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞: Tips and strategies to approach and answer interview questions effectively. ✅ 𝐂𝐚𝐫𝐞𝐞𝐫 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧: Essential knowledge and skills to prepare for interviews and stand out to potential employers. 📽 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐥𝐢𝐧𝐤: https://lnkd.in/g_nkP7fs #generativeai #llm #interview #jobpreparation #mockinterview

14

2 Comments

🔍 Quantum AI Series: Modeling Sequences with Quantum States Picking up where I left off in my #QuantumAI series, let’s talk about Modeling Sequences with Quantum States, with insights from a fascinating paper on the topic. Classical probability distributions can be modeled using quantum states—particularly entangled states that retain information from subsystems. This approach allows quantum models to efficiently represent sequences without losing essential details. Tensor networks play a key role in optimizing the process, providing a compact, low-rank factorization of data. As I've shared previously, Singular Value Decomposition (SVD) is foundational to these models, as it helps represent the complex correlations present in sequential data. Matrix Product States (MPS) capture these rich interactions by using reduced densities to efficiently compress sequence representations while preserving their structure. This is the essence of keeping representation while building efficient lightweight models. In summary, quantum probability models allow us to go beyond classical limits in sequential tasks, contributing to information retention. Sharing the paper for a deeper dive if you'd like so: https://lnkd.in/d5aqXyyS #QuantumAI #TensorNetworks #SVD #QuantumComputing #AI

5

Confounding? Detect it! A brand new paper on detecting and measuring confounding using causal mechanism shifts (presented this week at NeurIPS) In their new NeurIPS 2024 paper, Abbavaram Gowtham Reddy and Vineeth N. Balasubramanian (IIT Hyderabad) propose a framework for detecting and measuring confounding in a number of scenarios, based on shifts in causal mechanisms. The authors propose three complementary measures of confounding (with conditional and multivariate variants), designed to assess relative strengths of confounding in a variety of settings. The proposed measures complement one another depending on the available contextual information and offer a unified way to study confounding across diverse scenarios where data from different contexts is available. The authors generously shared Python implementation of their solution on GitHub. 💡 Paper: https://lnkd.in/dGXbnqAy 💡 Code: https://lnkd.in/dpczZvcU

142

7 Comments

Researchers at CMU Introduce TriForce: A Hierarchical Speculative Decoding AI System that is Scalable to Long Sequence Generation Quick read: https://lnkd.in/gr_he4kw Researchers from Carnegie Mellon University and Meta AI (FAIR) Introduce TriForce, a hierarchical speculative decoding system designed for scalable long sequence generation. TriForce utilizes the original model weights and dynamic sparse KV cache via retrieval as a draft model, serving as an intermediate layer in the hierarchy. Maintaining the full cache allows for superior KV cache selection using retrieval-based drafting, characterized as lossless compared to eviction-based methods like StreamingLLM and H2O. The hierarchical system addresses dual memory bottlenecks, pairing a lightweight model with a StreamingLLM cache for initial speculations to reduce drafting latency and accelerate end-to-end inference. TriForce introduces a hierarchical speculative decoding system with retrieval-based KV cache selection. The hierarchical system addresses dual bottlenecks, enhancing speed-up. Retrieval-based drafting segments the KV cache, highlighting relevant information. Lightweight models with StreamingLLM cache accelerate initial speculations, reducing drafting latency. TriForce utilizes model weights and KV cache to enhance LLM inference speed for long sequences. The implementation utilizes Transformers, FlashAttention, and PyTorch CUDA graphs, maintaining full layer sparsity while minimizing kernel launching overhead. Paper: https://lnkd.in/ga45hspw Machine Learning Department at CMU

26

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

GPT-4o says "9.11 is bigger than 9.9". It is extremely fascinating how AI models which can understand quantum mechanics, have read books from Tolstoy to Principia, and are now achieving super-human scores on knowledge benchmarks can fail at something as trivial as interpreting the decimal point system. It would almost be like if a top scientist made groundbreaking discoveries in physics and then said 1x1=2 *. These issues are at the heart of why AI models are considered unreliable. To build a world where AI models are ubiquitous, we need to figure this out ASAP. I think the most important step here is understanding WHY these mistakes happen. To this end, the recent interpretability work done by Anthropic will be crucial. https://lnkd.in/gvMKkGXX Further thoughts in comments. * Any resemblances to actual people are purely coincidental 😜 #AI #Anthropic #LLMs

58

28 Comments

🔎 Demystifying LLMs: A New Video Series! 🔎 Struggling to wrap your head around Large Language Models (LLMs)? I feel you! That's why I'm launching a new video series, LLM Explained, in collaboration with Uthiralakshmi S. from Worcester Polytechnic Institute. Each episode will be a bite-sized 2-minute explainer, tackling a specific LLM concept. This week, we're diving into the foundations: tokenization and embedding! Whether you're a seasoned data scientist or just starting your AI journey, this series is designed to make LLMs accessible and engaging.✨ Stay tuned for the next episode! In the meantime, hit that follow button and join the conversation! If you learned something valuable, share it with your network! 👨‍💻 👥 #GenAI #LLM #LargeLanguaeModels #DeepLearning #NeuralNetwork #AI #MachineLearning #Tokens #Tokenization #Embedding #Education #AIEducation #Data #DataScience #NEU #WPI

18

2 Comments

🚀 Excited to present our work on "AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations" at the #NeurIPS2024 Workshop on Adaptive Foundation Models! 📅 December 14, 2024, 10:45 AM 📍 West Exhibition Hall A 🔴 Problem: Multimodal web agents struggle to adapt to unseen tasks/websites/domains, relying on large-scale pre-training and fine-tuning, which limits efficiency. 🛠️ Methods: We introduce AdaptAgent, enabling agents to adapt with few-shot human demonstrations via in-context learning for proprietary models and meta-learning for open-weight models. 📈 Results & Importance: AdaptAgent boosts task success by a relative increase of 21.03% to 65.75% with just 1-2 multimodal demonstrations, offering a more efficient, adaptable solution for web agents. 📄 For more details, check out our arxiv paper: https://lnkd.in/g46s2Pys. This work is led by our intern Gaurav Verma during his time at J.P. Morgan AI Research, along with fellow collaborators Nishan S., Zhen Zeng, Tucker Richard Balch, and Manuela Veloso ❤️🤝 PS: We’re also looking for an intern in multimodal agents! 🤖 If you're passionate about multimodal systems, reach out (here or at NeurIPS 2024) or apply: https://lnkd.in/gtN6P5w9. #MultimodalAgents #FewShotLearning #JobOpening

7