Revanth Gangi Reddy

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

Check out this highly recommended course for those interested in optimizing their models: - Introduction to Quantization - Symmetric and Asymmetric Quantization Modes - Hands-on Implementation from Scratch using PyTorch - covering Per Tensor, Per Channel (Axis), and Per Group techniques - Weights Packing: Exploring 2 or 4 Bits Quantization Methods https://lnkd.in/gMAuWXHm

43

Brain Teaser #10: Suppose we are fine-tuning a large language model using Low-Rank Adaptation (LoRA). We possess two low-rank update matrices, δWA​ and δWB​​, both associated with the rank "r". Considering our objective to preserve the model's current knowledge while incorporating new data from diverse features, what "mathematical rationale" should guide the selection of one of the update matrices (δWA​ or δWB) for low-rank adaptation? Hint: Consider the specific characteristics or properties of the matrices that can help in developing mathematical reasoning. Here is my response: The norm ratio, calculated as the ratio of the nuclear norm to the Frobenius norm of the update matrix, is an essential metric for assessing how updates impact a model. It provides insights into the model's stability and adaptability. Here’s a brief overview: https://lnkd.in/ekB_q9XX #MachineLearning #DeepLearning #NaturalLanguageProcessing #LoRA #FineTuning #AI #DataScience #ModelOptimization #NeuralNetworks #ArtificialIntelligence #TechInnovation #DataDriven #KnowledgeIntegration #Research #LLMs

30

4 Comments

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

385

12 Comments

Jianbo Shi and Jitendra Malik's paper on "Normalized Cuts and Image Segmentation" (https://lnkd.in/dcyU5Skr) published in 2000, is a seminal work that introduces an interesting method for image segmentation by using concepts from graph theory and linear algebra. It is one of the most highly cited papers in Computer Vision with over 20k+ citations. By modeling the image as a graph, where pixels represent nodes and the edges between them encode the similarity, the segmentation problem transforms into one of graph partitioning. The segments then can be thought of as subgraphs. The key idea is to partition the graph to minimize the normalized cut (a criterion that evaluates the cut cost: the total weight of the edges removed to split the graph) relative to the total edge connections within each group. This problem is then elegantly converted into a linear algebra problem: finding the partition corresponds to solving for the second smallest eigenvector of the Laplacian matrix of the graph. Here's a short report on the entire paper, prepared as part of our project component for the Linear Algebra and its Applications course last semester. Please feel free to comment or address any issues if there are any! #ImageSegmentation #MachineLearning #ComputerVision #DataScience #ScientificResearch #LinearAlgebra #GraphTheory #Report #project

125

5 Comments

Computer science professor at University of California, Berkeley : Stop pursuing CS = You are welcome to study CS for Complexity slaying of Computational challenges but not for jobs. Here is what Prof. James O'Brien said: I've been a professor of computer science at UC Berkeley since 2000. Recently, I've noticed how difficult it is for tech graduates to enter the job market right now. At least in technology, it's a very different landscape than it has been in the past. It's not just my students; it's a hard time for everyone right now. Over the years, students I've worked with, taught, or known from other schools have typically had little trouble finding a job after graduating with a tech degree. There were lots of opportunities. Quite often, recent graduates would receive multiple offers and be left to try to decide between two really great choices. But now, things have changed. There's still demand for people, and some people are getting jobs, but not across the entire industry. Our view from Berkeley is also a little distorted because we're one of the top schools, so most of our students are still getting offers. But rather than getting five great offers, they might only get one great offer or an offer from their second choice. This is certainly not what we've seen in the past. There are also people coming from other schools that don't have quite the strong reputation that Berkeley does, and they're not getting job offers at all. Some programming roles are disappearing. It's not just the technology jobs that are being affected. Any "hands-on keyboard job" is at risk of being contracted out offshore or outsourced to AI. Another problem is that many companies are also trying to consolidate a bit. These things are all coming together like a perfect storm. In tech, programming positions that aren't that demanding are disappearing. They are still entry-level jobs that involve basically babysitting the AI — but these systems are getting better very quickly. = This matches No Work Theory. What? No one should ever work. WTF?? Work is the source of nearly all the misery in the world. Almost any evil you’d care to name comes from working or from living in a world designed for work. In order to stop suffering, we have to stop working. That doesn’t mean we have to stop doing things. It does mean creating a new way of life based on play; in other words, a ludic revolution. By “play” I mean also festivity, creativity, conviviality, commensality, and maybe even art. There is more to play than child’s play, as worthy as that is. I call for a collective adventure in generalized joy and freely interdependent exuberance. Play isn’t passive. Doubtless we all need a lot more time for sheer sloth and slack than we ever enjoy now, regardless of income or occupation, but once recovered from employment-induced exhaustion nearly all of us want to act. Pursue 007 life based on Universal Basic Income instead of 996 corporate slavery. That is Utopia.

45

37 Comments

Excited to share our latest paper titled - 𝐇𝐨𝐰 𝐭𝐨 𝐂𝐡𝐨𝐨𝐬𝐞 𝐚 𝐓𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝 𝐟𝐨𝐫 𝐚𝐧 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐌𝐞𝐭𝐫𝐢𝐜 𝐟𝐨𝐫 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬. This work addresses the critical gap in research on setting robust thresholds for LLM evaluation metrics, which are essential for the reliable deployment of LLMs. Drawing inspiration from model risk management practices in regulated industries like finance, we propose a systematic methodology to define these thresholds. The approach involves identifying risks associated with the specific application, understanding stakeholders' risk tolerance, and applying statistically rigorous procedures using ground-truth data.

58

4 Comments

Happy to share that our paper, "UniMTS: Unified Pre-training for Motion Time Series", has been accepted to the Neural Information Processing Systems (NeurIPS) 2024. This is the first pre-trained model specifically designed for human motion time series, excelling in zero-shot activity recognition across a wide range of wearable devices, regardless of body placement or orientation. Our approach follows a CLIP-inspired pre-training framework, where we leverage contrastive learning to align motion data with LLM-enhanced activity descriptions. https://lnkd.in/gjJcVSiR #NeurIPS2024 #neurips #neurips2024 #neurips24 #MachineLearning

92

9 Comments

Ran a quick test on the new #OpenAI #o1mini to solve IIT-JEE Advanced 2024 maths questions. It accurately solved 13 out of 17 questions with an impressive accuracy of ~76%. Quick thoughts: 1. The hidden chain of thought is quite detailed and extensive. Breaks the problem, solves and verifies and iterates multiple times. This might increase the output token cost by a big margin. 2. Seems there are tools integrated for calculations. Much needed as #gpt4o does a bad job at it. 3. Latency is quite high. Some questions took more than 50 seconds to get answered. This needs to come down for production user-facing use cases. 4. Hallucination still exists. Check the 1st question in the link, it had the correct step-by-step reasoning but hallucinated at generating the final answer. 5. Overdoing self-thought. Saw a couple of instances where it did generate correct answers in the process but eventually made wrong answers. Would give a clearer picture once the rate limits are increased and benchmarking is done on a bigger sample. Let me know any interesting findings and thoughts 😀 https://lnkd.in/giSCezyG #GenAI #OpenAI #PhysicsWallah

139

5 Comments

It’s been an honor working with Gil Weinberg! Thanks so much! This demo is from one of the chapters in my thesis on #AI driven raga agnostic automatic accompaniment for alapana (non-rhythmic improvisations) in South Indian Classical Music. Since the algorithm is raga agnostic, the system can accompany for any raga performed! More demos on a violin playing #robot coming soon! #MusicTechnology #MachineLearning #HumanRobotInteraction #HumanComputerInteraction #AutomaticAccompaniment

217

31 Comments

I am happy to share that our years of effort have finally culminated in a VLDB paper titled "Datalog with First-Class Facts." We introduce SLOG (a variant of Datalog), where facts are first-class entities, with each fact uniquely identified by an identifier. This constrained approach proves particularly efficient for reasoning about tree-structured data using Datalog, with successful applications across database systems, AI, and programming language domains. We have developed a data-parallel MPI-based backend that can scale to thousands of processes. Check out the scaling result on a supercomputer for performing Context-Sensitive Points-to-Analysis (CSPA) in our language. The collaborative work was done by Thomas Gilray, Kristopher Micinski Yihao Sun, Sowmith K, and myself. Thanks to Argonne Leadership Computing Facility for supporting our research. #database #hpc #MPI #datalog

93

1 Comment

Shashank Sonkar and colleagues are back with another great paper and matching GitHub repo, "Pedagogical Alignment of Large Language Models". You can read the paper on arXiv here - https://lnkd.in/enZfCJgi, and browse the code here - https://lnkd.in/gyW5hSzp. They demonstrate an interesting approach to generating synthetic data to train open weights models to behave more pedagogically, as well as developing some new evaluation metrics. I think there's lots here for those of us interested in this kind of work to learn from and build on. (If synthetic data generation is new do you, check out this post from answer.ai earlier this week - https://lnkd.in/ecsG7R4b.)

38

4 Comments

Fine tuning gpt2 on indic-gpt dataset I ran one epoch of fine tuning on an arbitrary dataset of Indic-gpt and managed to train it for language detection with Colab.Have a look: https://lnkd.in/dZJUY-UV #indic-gpt #fine tuning #language detection

11

🔥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 - 𝐓𝐨𝐩 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 🔥 🤖 I've conducted a mock interview with my student, Bheesetti Anand, focused on Generative AI and LLM (Large Language Models) interview questions. This session is designed to benefit other students preparing for similar interviews, helping them secure their dream roles in the field of Generative AI. 📔 Here’s why watching the video can help you make a 𝐬𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 𝐜𝐚𝐫𝐞𝐞𝐫 𝐬𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈: ✅ 𝐂𝐨𝐦𝐩𝐫𝐞𝐡𝐞𝐧𝐬𝐢𝐯𝐞 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞: Detailed exploration of top interview questions related to Generative AI and Large Language Models. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Real-world examples and practical applications to understand complex concepts better. ✅ 𝐄𝐱𝐩𝐞𝐫𝐭 𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞: Tips and strategies to approach and answer interview questions effectively. ✅ 𝐂𝐚𝐫𝐞𝐞𝐫 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧: Essential knowledge and skills to prepare for interviews and stand out to potential employers. 📽 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐥𝐢𝐧𝐤: https://lnkd.in/g_nkP7fs #generativeai #llm #interview #jobpreparation #mockinterview

14

2 Comments

To(ring) or not to ring. That's the question we tried to answer. Last quarter, I along with Pranava Singhal and Harshvardhan Agarwal worked on a project to answer a simple question in Graph ML: Can adding virtual nodes to represent molecular rings improve performance on graph-level tasks? 🌟 Why do rings matter? Rings in Molecules play a role in determining their properties, like hydrophobicity, which influence solubility, and bioavailability. Inspired by this, we proposed a novel RingSAGE algorithm to explicitly tackle these structures during the message-passing stage in GNNs. 💡 Why This Matters Traditional GNNs treat molecular graphs uniformly, missing the unique properties of rings. Our approach introduces virtual nodes to capture ring-specific information. However, this leads to an increase in the number of learned parameters, so for a fair comparison, we compare our method against a variant of GIN with more parameters than our model. 🔍 What We Achieved Our model outperformed standard GNN models on the ZINC dataset, a widely used molecular drug solubility prediction benchmark. We provide a notebook to replicate these experiments, a codebase to use RingSage for your own usecase and a blog elaborating our method and results.

72

6 Comments

Last month, a paper from Sakana AI called “AI Scientist” introduced an intriguing idea. Now, don’t get me wrong, I love AI as an assistant. But after getting an LLM-generated review for my NeurIPS paper last month (which was total crap and useless), I’m a little skeptical about AI discovering true novelty. Here’s my take on the whole AI Scientist concept: The paper talks about a paradigm shift in research, where AI can ideate by identifying challenges and opportunities, build hypotheses, design and run experiments by generating code and simulations, analyze results, refine hypotheses, and even write research papers and reviews. The authors even generated three full research papers using prompt templates to prove the concept! Now, I agree that this could be a game-changer for interdisciplinary research, or for uncovering new problems at the intersection of fields—where human experts, limited by working memory and attention span, may struggle to grasp more than a handful of domains. LLMs, with their ever-expanding knowledge base, offer the potential for cross-pollination of ideas. But when it comes to deep, niche, and fundamental breakthroughs, I’m not buying it—hence my disappointment with that NeurIPS review :( Here’s the thing—mathematically, LLMs are trained to regenerate the data distribution they’ve already seen. So, it's inherently challenging, if not impossible, for them to produce something truly novel. I like to think of LLMs as vast lookup tables, which makes them amazing assistants, but let’s not call them “scientists” just yet!

92

4 Comments

Researchers have developed a novel "Hidden Chain-of-Thought" (HCoT) framework that significantly speeds up large language model inference while preserving multi-step reasoning capabilities. HCoT (Hidden Chain-of-Thought) differs from CoT (Chain-of-Thought) by compressing the multi-step reasoning process into a specialized token representation, allowing for faster inference while maintaining reasoning capabilities. Here are the key steps for implementing the Hidden Chain-of-Thought (HCoT) framework: 1. Data Preparation   - Construct HCoT training samples from original datasets.   - For math reasoning tasks, use GPT-4 to generate separate fields for thoughts and contents.   - For question answering, utilize existing fields from datasets like ScienceQA.   - For agent invocation tasks, employ fields generated through frameworks like ReAct. 2. Auxiliary CoT Model Training   - Extract thought processes from HCoT training samples.   - Insert special [CoT] tokens between content and thought segments.   - Train the model to maximize the likelihood of generating accurate thought representations.   - Incorporate symmetric contrastive loss to enhance thought compression. 3. HCoT Model Training   - Replace thought processes in original samples with [CoT] tokens.   - Fine-tune the model to generate content and [CoT] tokens.   - Freeze parameters of the auxiliary CoT model during this stage. 4. Inference Process   - The HCoT model generates content until encountering a [CoT] token.   - The auxiliary CoT model produces a compressed thought representation.   - The compressed representation is fed back into the HCoT model to continue generation. 5. Performance Evaluation   - Compare the HCoT model against baselines like zero-shot CoT and full CoT training.   - Evaluate accuracy across various tasks (math reasoning, science QA, agent invocation).   - Measure speedup in terms of sequence length and wall-clock inference time. 6. Optional CoT Recovery   - When full reasoning steps are needed, allow the auxiliary CoT model to generate the complete thought process.

11

1 Comment

Good folks from Carnegie Mellon University's WAVLab have released XEUS - A Cross-lingual Encoder for Universal Speech 🌐🗣️ Simple speech representation is hindered by a lack of labelled data. That's where Self-supervised learning (SSL) comes in! 🔍📚 XEUS is a SoTA speech encoder trained in 4000+ languages! 🏆🌍 Before this, I did not even know that there are 4000+ languages. But the paper says this is a small dent in the world's 7000+ languages. 🌐🗺️ More details: - 577M parameters 📊 - 1M+ hours of speech training data across 4057 languages 🗣️⏳ - Uses the E-Branchformer architecture and is trained using HuBERT-style masked prediction of discrete speech tokens 🏗️🔠 - For robustness, the input speech is also augmented with acoustic noise and reverberation 🎙️🔊 Beats both MMS 1B and w2v-BERT v2 2.0 🥇🏆 It requires fine-tuning to be used in downstream tasks such as Speech Recognition or Translation. 🛠️🌐 Its hidden states can also be used with k-means for semantic Speech Tokenization. 🗣️🔑 Release: https://lnkd.in/gqrfvsRF

11