Bryant Chen

Could RAFT (Retrieval-Augmented Fine-Tuning) be the next evolution of RAG architecture? I am following this with great interest. https://lnkd.in/eHDx3rHz

13

One new interesting application of LLMs: CAIS has released a forecasting bot that provides probability estimates for any question you ask. For example, to the question "Will GPT5 be released before 2025?", it answers that it's 70% likely. The bot works by searching for relevant information online, feeding it to GPT-4o, asking a few guiding questions, and then requesting a probability estimate. What's impressive is that this bot outperforms experienced human forecasters, even though GPT-4o wasn't specifically trained for this task. This kind of application shows that AI could be of great help in improving decision-making in the future. Link to the blog post: https://lnkd.in/gjP7JQgz Link the bot if you want to try it yourself: https://forecast.safe.ai/

11

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

Ilya Sutskever confirmed at NeurIPS '24 pre-training scaling law will not be the same. Internet data, the AI fossil fuel, is running out. It is not the end of the AI hype, the rapid rise of foundational AI has outpaced our ability to fully grasp their long-term societal impact. AI agents and applications are poised to transform every facet of our lives, and we will see it happening in coming few years. There will still be improvements in model architecture, and synthetic data creation, so models will improve, just not at the lightning speed they did till now. Looking forward, the single most important milestone in AI is creating systems capable of advancing themselves—AI researchers that can innovate and improve foundational models autonomously. We need to assess how far are we from this stage.

9

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

180

2 Comments

🚨 New paper w/ interesting insights for fine-tuning and GPU poor folks 🚨 🚀 Exploring Fine-Tuning for LLMs: Full Fine-Tuning vs. #LoRA - Are They Truly Equivalent? 🤔 As LLMs grow, fine-tuning them for specific tasks becomes challenging due to high computational costs. Recently, Low-Rank Adaptation (LoRA) has become a popular alternative, promising similar performance with fewer parameters. But are LoRA-tuned models the same as fully fine-tuned ones? MIT’s latest study dives deep into this question, revealing some surprising differences! 🔍 📝Key Takeaways: 1. Spectral Differences 🧬: LoRA introduces “intruder dimensions” not found in fully fine-tuned models. These alter the model’s weight structure, suggesting that LoRA and full fine-tuning leverage different parts of the parameter space, even if they hit similar accuracy on specific tasks. 2. Generalization & Robustness 🌐: Models fine-tuned with LoRA may struggle with new or sequential tasks, forgetting previously learned knowledge more easily. Full fine-tuning, however, shows better retention and robustness in shifting environments. 3. Effective Rank & Optimal Performance ⚙️: LoRA’s rank matters! Mid-range ranks (like r=64) perform well on tasks and retain pre-trained knowledge, while lower or higher ranks can lead to greater forgetting. Smart parameter tuning (like setting α=2r) can help LoRA resemble full fine-tuning more closely. 4. Why It Matters 💡: LoRA offers incredible parameter efficiency, but its unique trade-offs might affect its use in scenarios needing high adaptability and long-term memory. With adjustments, LoRA can bridge some of these gaps, but full fine-tuning remains the go-to for complex or multi-tasking demands.

25

1 Comment

My intern at Schneider Electric and student at Northeastern University, Udisha Dutta Chowdhury, is presenting our work at PyData NYC tomorrow! Her talk is titled "Adopting Open-Source Tools for Time Series Forecasting: Opportunities and Pitfalls".  In addition to introducing sktime and skforecast as case studies, she will also guide the attendees through the broader considerations of tool selection for forecasting. We will emphasize the importance of Data Understanding, Data Preparation, and Backtesting in building effective and reliable forecasting pipelines. If you are in the conference, please come say hi! The talk will also be posted on YouTube later. #techtalk #algorithms #forecasting #machinelearning #datascience

27

1 Comment

ok - given twitter/x is the place where bleeding edge R&D happens (if you can evade the AI hype and crypto posts successfully), I'm trying to figure out how much cross-over there is over here in what i think is more of the real world. 1. have you heard of the 'anon' entropix project? (uses parallel sampling, with a draft model, and a bunch of other inference-level 'hacks' to the tokenizer and decoder) 2. have you tested it? (i did for ~1 hour yesterday, not sure what to think yet) 3. has anyone seen anything neat with speculative decoding + draft models (haven't seen a lot in the open since medusa)? one thing i know i like about entropix is that it uses a base model (without chat/instruct tuning) + an instruct in parallel. base models are powerful - even 1B parameter ones. tldr: if there's any activity/comments on this post, i will be shocked, but genuinely curious to get the real world's perspective. if you can get really fantastic results by running two 1B parameter LLMs on certain tasks, that clears the way for running almost everything locally, on-device, privately, and only offloading to cloud/api services what you can't do locally.

12

7 Comments

🚀 Improving Math Level 5 Benchmark Evaluations for Fairer LLM Assessments! 🚀 I’m excited to share a recent update I’ve submitted to Hugging Face’s LLM Evaluation Harness [https://lnkd.in/dfSa-cq7] This improvement aims to address a critical issue impacting Math Level 5 benchmark evaluations—where some high-performing models were getting zero scores due to formatting mismatches. 📉 As the co-creator of Le-leaderboard with Alexandre Lavallee (https://lnkd.in/dZsA7VPY) and the creator of OpenLLMTurkishLeaderboard v0.1 and v0.2 on Hugging Face, I’ve been deeply involved in evaluating LLMs, and this experience highlighted a recurring issue: the evaluation framework often marks 100% of a model's responses for the MATH-LVL5 Benchmark as invalid due to incorrect extraction of final answers. For example, models like Qwen/Qwen2.5-32B-Instruct were unfairly scored—not because they lacked proficiency, but because their answer format didn’t match the expected regular expression. To solve this, I proposed: ✨ Modifying the Prompt: Models will be instructed to end responses with specific characters (e.g., "###") to ensure consistent output formatting. ✨ Adjusting Extraction Logic: The framework will now capture the answer following the designated characters, similar to previous methods used for the GSM8K dataset. By refining this answer extraction process, we can ensure evaluations are fairer and more accurate. The changes will: ✅ Correctly represent models' performance ✅ Provide more reliable evaluation results ✅ Help the community gain a clearer view of LLMs’ true mathematical capabilities I believe this update will make a meaningful difference in the evaluation process, and I’d love to hear any feedback from the community. 💬 On a related note, I implemented this same fix on Le-leaderboard and observed a significant performance increase for models like Qwen/Qwen2.5-7B-Instruct, where scores jumped from around 6% to 35%. This underscores the need for improved parsing in the original implementation. Le-leaderboard now reflects the true evaluation of the MATH LVL5 benchmark with better parsing. Check it out here: https://lnkd.in/dDxyNAMA #LLM #AI #Evaluation #MachineLearning #HuggingFace #Leaderboard cc: Hugging Face

15

2 Comments

I just read an insightful piece by Eric Broda (published in Towards Data Science) that outlines the Agentic Mesh—a new ecosystem designed for autonomous agents to interact, collaborate, and transact. His work introduces a structured approach for managing the coming surge in autonomous agents, showing how they can function independently yet collectively. Agentic Mesh serves as a networked environment where autonomous agents operate in a safe, transparent, and efficient manner. It allows agents to find each other and securely interact under a framework of policies. Each agent has a specific purpose and autonomy, making it a valuable contributor to the ecosystem without sacrificing security or control. Core Components: 1. Marketplace: A centralized space where users discover agents by tasks and capabilities, offering a trusted environment for engagement. 2. Registry: A metadata repository organizing each agent’s purpose, owner, and policies, allowing agents and users to securely find and connect with each other. 3. DNS Integration: Each agent has a recognizable digital identity, enabling seamless discovery and interaction, just like an internet service. 4. Agent Stack: This layered structure of communication protocols, control services, and interfaces helps agents operate effectively and responsibly. It ensures that agents act independently within controlled parameters. Agentic Mesh’s effectiveness depends on agents sharing consistent attributes, including: • Purpose: A clear objective drives each agent’s actions. • Ownership: Transparency around ownership fosters responsibility and trust. • Autonomy: Agents act independently within set parameters to complete tasks. • Discoverability: Easily locatable, ensuring efficient interaction within the network. • Intelligence: Agents utilize language models to plan and execute tasks, adapting to complexity as needed. Operational Flow: Registration, Discovery, and Execution Agent interactions in the Mesh follow three core flows: 1. Registration: Agents set up purpose, policies, and registration data to gain visibility in the mesh. 2. Discovery: Through the registry, agents and users locate potential collaborators based on defined needs. 3. Execution: Engaged agents use AI models to plan and perform tasks, enlisting collaborators when necessary to complete workflows. Read the full article: https://lnkd.in/eVmq5BFC

16

1 Comment

🤔 Fascinating Insight into LLM Prompting A counter-intuitive observation that lots of members of applied community are making about Large Language Models that's challenging conventional wisdom: The effectiveness of prompting strategies is deeply intertwined with the model's training data exposure. 💡 Key Observation: Take the HumanEval benchmark for code generation. While we might expect sophisticated approaches like reflexion (using multi-stage reasoning) to outperform simpler methods, the reality is surprising. When an LLM has strong knowledge retrieval capabilities, complex prompting strategies can actually impair its performance. 🔍 Why This Matters: It flips our assumptions about "advanced" prompting techniques Suggests simpler might be better when the model has relevant training exposure Forces us to rethink how we approach prompt engineering This reinforces a crucial lesson in AI: Sometimes, getting out of the model's way is more effective than elaborate prompting gymnastics.

2

I just published Vol. 78 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Systematic Review of Language Model Representations, from Zhang et al. 📚 Adaptive RAG Through Knowledge Boundary Detection, from Alibaba 📚 Understanding Context Sufficiency in RAG Systems, from Joren et al. 📚 A Cache-Driven Approach to Scaling Recommendation Systems, from Kuaishou 📚 A General Framework for Content Annotation and Retrieval, from Clarke et al. 📚 A Comprehensive Framework for Multimodal Long Document Understanding, from Alibaba 📚 Specialized Text Embeddings for Financial Document Understanding, from Balyasny Asset Management 📚 A Theoretical Framework for Comparing Recommendation Loss Functions, from Teodoro et al. 📚 Understanding Document Retrieval Trade-offs in RAG Systems, from Intel Labs 📚 A Systematic Evaluation of Commercial LLM Fine-tuning Services, from Stanford University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

33

1 Comment

This recent study on Whisper's hallucination problems is super interesting (and somewhat concerning): https://lnkd.in/gAqRy95U > "in one sound bite, Whisper correctly transcribed a single, simple sentence, but then hallucinated five additional sentences that contained the words “terror,” “knife” and “killed,” none of which were in the original audio" At Rev, accuracy and reliability is non-negotiable. Unlike other transcription services that might ‘hallucinate’ extra sentences, we ensure every word transcribed is exactly what was spoken. #rev #ai #asr #speechtotext #stt #languagetechnologies #speech #whisper #openai

48

8 Comments

For those interested in the intersection of graph theory and machine learning for proof/counterexample/instance finding, here are two papers that leverage machine learning to find Ramsey graphs (and counterexamples): https://lnkd.in/eh3gaFJ9 https://lnkd.in/e2TkJuxD

130

12 Comments

When I joined the University of California, Berkeley LLM Agents MOOC, I didn’t realize how transformative the experience would be. Guided by Dawn Song and Xinyun Chen, and inspired by guest lectures from phenomenal minds like Denny Zhou, Jerry Liu, and Graham Neubig, this course was so much more than just learning—it was a deep dive into the transformative future of Agentic AI. One lecture that truly resonated with me was on AutoGen frameworks by Chi Wang, where we explored how AI agents can collaborate seamlessly and adapt to solve real-world challenges. Motivated by this, I created something: 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗖𝘂𝘀𝘁𝗼𝗺𝗲𝗿 𝗦𝘂𝗽𝗽𝗼𝗿𝘁 An intelligent system designed to reimagine customer support workflows. 𝗞𝗲𝘆 𝗳𝗲𝗮𝘁𝘂𝗿𝗲𝘀: 𝗤𝘂𝗲𝗿𝘆 𝗖𝗮𝘁𝗲𝗴𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Automatically organizes queries into categories like technical, billing, and general. 𝗦𝗲𝗻𝘁𝗶𝗺𝗲𝗻𝘁 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀: Detects customer sentiment (positive, neutral, or negative) to provide tailored responses. 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗩𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Powered by LangGraph, it dynamically visualizes workflows to simplify management. 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗘𝘀𝗰𝗮𝗹𝗮𝘁𝗶𝗼𝗻: Flags critical issues for human intervention when necessary. 𝗚𝗶𝘁𝗛𝘂𝗯 𝗥𝗲𝗽𝗼𝘀𝗶𝘁𝗼𝗿𝘆: https://shorturl.at/DOEIP 𝗠𝗲𝗱𝗶𝘂𝗺: https://shorturl.at/fqrni Seeing this project come to life was quite rewarding. It brought together everything I learned in the course—turning abstract concepts into practical, impactful solutions. To anyone passionate about AI, agentic systems, or pushing boundaries, this course is for you. Grateful for this journey and excited to keep exploring the limitless possibilities of AI! 𝗖𝘂𝗿𝗶𝗼𝘂𝘀 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗰𝗼𝘂𝗿𝘀𝗲? 𝗖𝗵𝗲𝗰𝗸 𝗶𝘁 𝗼𝘂𝘁 𝗵𝗲𝗿𝗲: https://lnkd.in/d7Ch4-Bp

36

3 Comments

I just published Vol. 80 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Comparing Commercial and Open-Source LLMs for Systematic Review Query Generation, from TU Wien 📚 A Systematic Comparison of Ranking Algorithms for LLM Evaluation, from the University of Michigan 📚 A Comprehensive Analysis of LLM Representations for Regression Tasks, from Google 📚 Benchmarking Multi-modal Retrieval-Augmented Generation for Web Content, from Ma et al. 📚 Efficient Large Language Model Scaling for Industrial Dense Retrieval, from Huang et al. 📚 A Survey on LLM-as-a-Judge, from Gu et al. 📚 Reproducing and Adapting 2D Matryoshka Training for Information Retrieval, from Wang et al. 📚 A Fast and Exact Similarity Search Engine for Billion-Scale Data Series, from Schafer et al. 📚 A Review of Visual Question Answering in the Era of Large Language Models, from SYSU 📚 Integrating ID Embeddings and Language Models for Enhanced Sequential Recommendations, from Huawei #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

32

1 Comment

I like Gradio (https://www.gradio.app/). It's better than Streamlit in a way that it supports building user interfaces right in the Jupyter Notebook environment. I built several exploration interfaces and it shows the potential for more advanced dashboard apps in Python. This is a univariate histogram explorer. More below 👇

30

2 Comments

Hallucination is Inevitable in LLMs An important new paper out that shows that even with perfect data and an idealized architecture “LLMs will always hallucinate”. I suspect that this will over time be linked to Gödel’s Incompleteness Theorem [1] and Turing’s Halting Problem [2]. Every technology has trade-offs. We love LLMs but they need to be used in the right way. [1] Good pop science depiction in Oppenheimer (Gödel was Einstein’s walking companion) or also see Rebecca Goldstein’s book Incompleteness [2] Turing is depicted in The Imitation Game and also the progenitor of the famed Turing Test.

37