Xiangyi Chen

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

🚨 New paper w/ interesting insights for fine-tuning and GPU poor folks 🚨 🚀 Exploring Fine-Tuning for LLMs: Full Fine-Tuning vs. #LoRA - Are They Truly Equivalent? 🤔 As LLMs grow, fine-tuning them for specific tasks becomes challenging due to high computational costs. Recently, Low-Rank Adaptation (LoRA) has become a popular alternative, promising similar performance with fewer parameters. But are LoRA-tuned models the same as fully fine-tuned ones? MIT’s latest study dives deep into this question, revealing some surprising differences! 🔍 📝Key Takeaways: 1. Spectral Differences 🧬: LoRA introduces “intruder dimensions” not found in fully fine-tuned models. These alter the model’s weight structure, suggesting that LoRA and full fine-tuning leverage different parts of the parameter space, even if they hit similar accuracy on specific tasks. 2. Generalization & Robustness 🌐: Models fine-tuned with LoRA may struggle with new or sequential tasks, forgetting previously learned knowledge more easily. Full fine-tuning, however, shows better retention and robustness in shifting environments. 3. Effective Rank & Optimal Performance ⚙️: LoRA’s rank matters! Mid-range ranks (like r=64) perform well on tasks and retain pre-trained knowledge, while lower or higher ranks can lead to greater forgetting. Smart parameter tuning (like setting α=2r) can help LoRA resemble full fine-tuning more closely. 4. Why It Matters 💡: LoRA offers incredible parameter efficiency, but its unique trade-offs might affect its use in scenarios needing high adaptability and long-term memory. With adjustments, LoRA can bridge some of these gaps, but full fine-tuning remains the go-to for complex or multi-tasking demands.

25

1 Comment

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

77

Is GPT-4o good at table recognition? Not quite! OCR looks fine, but tripped by spanning cell table structure! 🤩 However, our UniTable, with only 124M param trained by self-supervision, shows great understanding! 👉 UniTable paper: arxiv.org/abs/2403.04822 👉 UniTable code: https://lnkd.in/e4a7YMxJ #MachineLearning #GPT-4o #Table #Document

85

8 Comments

OpenCoder: An Open-Source Code LLM Redefining Model Efficiency 🌟 The OpenCoder team has launched a game-changer in the world of code language models. Unlike many LLMs that focus on sheer size, OpenCoder brings something unique to the table: a data-focused training approach. By putting quality before volume, OpenCoder’s 1.5B and 8B models rival some of the biggest players, like DeepSeekCoder and QwenCoder, with only a fraction of their token count. The core strength of OpenCoder lies in its meticulous data processing pipeline and two-stage fine-tuning. With a training dataset composed of 2.5 trillion tokens, 90% raw code and 10% code-related web data, OpenCoder refines every aspect of its corpus. Advanced filtering and deduplication strategies, applied to 607 programming languages, optimize model efficiency and precision, ensuring high-quality data drives its training. Key Features: - Complete Training Transparency: Open-source datasets and training pipelines enable reproducibility and further research. - Instruction Fine-Tuning: Two-stage tuning sharpens both core CS concepts and real-world coding skills, prepping OpenCoder for production-level code generation. - Available on Ollama: Accessibility in deployment means more avenues for experimentation and improvement. OpenCoder's data-driven strategy showcases the potential of a leaner approach. By focusing on quality data rather than quantity alone, OpenCoder achieves file-level deduplication that outperforms traditional repo-level methods, making it both efficient and robust. 🚀 Read more here: https://lnkd.in/eEnG3yJA #OpenSource #MachineLearning #CodeLLM #AI #ModelEfficiency #DataQuality #Innovation #CodeGeneration #OpenCoder #TechNews #AIResearch #Programming #LargeLanguageModels #ArtificialIntelligence

1

Have you wondered how the decision boundary of in-context learning in LLMs compares to traditional models like Decision Trees and KNN? 🤔 Excited to share our new research, which uncovers unexpected irregularities and non-smoothness in LLMs' in-context decision boundaries! 🔍 📄 Paper: https://lnkd.in/gZAKEJH6 🐦 Tweet summary thread: https://lnkd.in/gY9xkk6d 💻 code: https://lnkd.in/gCsgQk8r Here are some key findings 📌 : - Increasing the number of in-context examples can enhance LLM accuracy on both linear and non-linear classification tasks. But how reliable are these in-context classifiers? We probe their decision boundaries to find out. - By visualizing the decision boundaries, we show that SOTA LLMs, ranging from 1B to large closed-source models such as GPT-3.5-turbo and GPT-4o, all exhibit different non-smooth, irregular decision boundaries, even on simple linearly separable tasks. 🔍 - How do these irregularities arise? 🤔 We study various factors that impact decision boundary smoothness in LLMs, including in-context example count, quantization levels, label semantics & examples order. Then, we identify methods to improve the smoothness of the boundaries. 🛠 - First, increasing in-context examples does not guarantee smoother decision boundaries. While classification accuracy improves with more in-context examples, the decision boundary remains fragmented. - Quantization Impact: Reducing precision from 8-bit to 4-bit can significantly affect areas near the decision boundary, especially in regions with high uncertainties, potentially flipping decisions in these regions. - Label Sensitivity: Decision boundaries are sensitive to label names and example order. Shuffling in-context examples and labels can alter the model’s decision boundaries, indicating a dependency on the semantic prior knowledge of the labels. - 🛠️ Can we improve decision boundary smoothness in LLMs through training? We show that fine-tuning on simple linearly separable tasks can improve the smoothness of decision boundaries and generalize to more complex non-linear, multi-class tasks, enhancing robustness. - Further, fine-tuning the token embedding and attention layers leads to smoother decision boundaries, while fine-tuning the linear prediction head alone does not. - We also explore uncertainty-aware active learning: By adding labels for the most uncertain points to the in-context dataset, we can smoothen the decision boundary more efficiently. - 📚 Effect of Language Pretraining: Transformers trained from scratch on synthetic classification tasks can learn smooth in-context decision boundaries for unseen problems. Thanks for reading about our work! 💡 For a more detailed summary, please check out our tweet thread: https://lnkd.in/gY9xkk6d! 🌟 Work done with amazing collaborators 🙌 Tung Nguyen and our advisor Aditya Grover at UCLA!

63

[research update] use a Transformer's multi-head attention activations to fetch RAG style, to augment multi faceted problem breakdown for LLMs in ."Multi-Head RAG: Solving Multi-Aspect Problems with LLMs": 1. Enhances LLMs by integrating document retrieval (RAG) into the LLM context. 2. Challenge: Existing RAG solutions struggle with queries needing multiple documents with varied contents due to distant embeddings. 3. Multi-Head RAG (MRAG): Introduces a novel approach using Transformer's multi-head attention activations to fetch diverse documents. 4. Activation Utilization: Different attention heads capture varied data aspects, improving complex query retrieval. 5. Improved Relevance: MRAG shows up to 20% improvement in relevance over standard RAG baselines. 6. Evaluation Methodology: Provides metrics, synthetic datasets, and real-world use cases to demonstrate effectiveness. 7. Seamless Integration: MRAG can be integrated with existing RAG frameworks and benchmarking tools like RAGAS. 8. Multi-Aspect Problem Solving: Designed specifically to handle multi-aspect queries more effectively. 9. Data Stores Compatibility: Compatible with various data store classes. 10. Potential Applications: Benefits scenarios requiring diverse information retrieval, enhancing LLM responses. Experiment The paper introduces MRAG and evaluates it using synthetic datasets and real-world data. It compares MRAG's performance against standard RAG models. The experiment focuses on the model's ability to retrieve relevant documents across various query aspects, demonstrating up to a 20% improvement in relevance over baselines. Use Case MRAG is tested on real-world scenarios like customer support, where queries often require information from multiple documents with diverse contents. The model's capability to retrieve and integrate multi-aspect data enhances the quality of responses.

81

5 Comments

About 2 months ago Li Yin didn't hold back in critiquing LLM application frameworks. Fast forward to 7/3/24, and her team has just released the alpha version of LiteRAG, with beta on the horizon(7/9/24). First Impressions? It's a nice balance between how I use instructor and DSPy. Instead of Pydantic, it uses the dataclasses, which is a standard library construct. The library also introduces DataClass as a wrapper to simplify generating schemas, adding examples, and loading JSON AND YAML. If you're familiar with Pytorch, or DSPy, then you'll feel at home composing your Components, which is a core abstraction. My favorite component as of now, Sequential, allows you to chain together your components input -> output. Definitely looking forward to seeing how this library grows! But, you don't need me to give you your opinion, Go check it out and make your own. LightRAG -> https://lnkd.in/g4cWDP88 If you want to see how I applied it to a current project, check out the video! Happy MonYay yall!

81

6 Comments

A new week brings a fresh opportunity to learn something new! If you have worked with more complex PyTorch models, especially when implementing LLMs, you may have encountered PyTorch buffers. To help you understand them better, I've recorded a short 13-minute hands-on coding video this weekend. In this video, I explain what PyTorch buffers are, when we need them, and how to use them effectively: https://lnkd.in/giAVBGAy

711

21 Comments

🚀 Research Paper Highlights: Let's explore the paper- 'Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA' by Minzheng Wang et al. 🔍 Realistic Long-Context Benchmark: Introducing the Loong benchmark, designed to align with real-world long-context applications through extended multi-document question answering (QA). Each document is crucial to the final answer, ensuring relevance and accuracy. 🧩 Diverse Task Types: Loong brings in four task types—Spotlight Locating, Comparison, Clustering, and Chain of Reasoning—across varied context lengths. This diversity offers a comprehensive evaluation of long-context understanding. 📊 Performance Insights: Extensive experiments show that current long-context language models have significant room for improvement. Retrieval-augmented generation (RAG) performs poorly, highlighting Loong's effectiveness in assessing long-context modeling capabilities. 📈 Key Conclusion: This study introduces Loong, a benchmark for real-world multi-document scenarios. Analysis of six advanced language models, including GPT-4o and Gemini-Pro1.5, reveals that even the most powerful models struggle with long-context tasks. In-depth analyses emphasize the need for enhancing long-context modeling by comparing the RAG approach and examining the scaling laws related to context size. Further reading- https://lnkd.in/dRn75XXh 🌟 Stay tuned for more updates on upcoming research and analysis in this rapidly evolving landscape of Generative AI. #LOONG #LLMs #handlinglongcontext #generativeai #research #innovation

12

Excited to have presented our paper on Bayesian Prompt Ensembles at ACL 2024. In this paper, we explore the use of multiple prompt re-formulations to capture model uncertainty in LLMs. We derive the connection between prompt re-formulation and model uncertainty/Bayesian NNs and propose a method to learn the relative importance of different prompts for best performance. As part of this work, we open-sourced a code package and tutorial with which you can a) easily use LLMs to perform classification tasks and b) apply our method, BayesPE, to improve uncertainty estimation through the use of multiple prompts. Hope it is useful to folks out there! Paper: https://lnkd.in/etpRRwJC Code: https://lnkd.in/eDHdQ6k4

26

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

180

2 Comments

Enhancing Multilingual Safety in LLMs with Code-Switching Red-Teaming (CSRT) 📘 What is this paper about? This paper introduces a novel red-teaming technique called Code-Switching Red-Teaming (CSRT) to test and improve the multilingual understanding and safety of large language models (LLMs). 🤖 First key aspect The CSRT dataset consists of 315 code-switching queries combining up to 10 languages to test LLMs' responses. 📊 Second key aspect CSRT significantly outperforms existing multilingual red-teaming techniques, achieving 46.7% more effective attacks in English. 🧠 Third key aspect Extensive experiments and ablation studies with 16K samples analyze harmful responses and various unsafe behavior categories. 🚀 Why is this a breakthrough? ⏱ First reason Provides a comprehensive method to evaluate and improve LLMs' multilingual capabilities and safety. 📈 Second reason Reduces reliance on manual annotations by offering an automated and scalable testing approach. 🌍 Third reason Enhances understanding of LLMs' performance across multiple languages, making them safer and more reliable. 🔬 Key Findings 🔧 First finding CSRT outperforms existing techniques in eliciting undesirable behaviors in LLMs, proving its efficacy in safety testing. 🧩 Second finding Analysis of harmful responses reveals insights into scaling laws, unsafe behavior categories, and optimal data generation conditions. 🛠 Third finding Validates the extensibility and robustness of CSRT for comprehensive multilingual safety evaluations. 🔍 Implications for the Future 🌐 First implication Improved methods for testing and enhancing the multilingual safety of LLMs can lead to more reliable AI systems globally. 🚗 Second implication Automated and scalable red-teaming techniques reduce the need for manual annotations, making safety evaluations more efficient. 📈 Third implication Better understanding of LLMs' behavior across languages can inform the development of safer and more inclusive AI models. 💡 Takeaways 🎯 First takeaway CSRT sets a new standard for evaluating and improving the multilingual safety of LLMs. 🔄 Second takeaway Automated red-teaming techniques like CSRT can significantly reduce the dependency on manual annotations in safety testing. 🌟 Third takeaway Enhancing LLMs' multilingual capabilities and safety through comprehensive testing approaches ensures more reliable and inclusive AI systems.

1

Exploring In-Context Reinforcement Learning in LLMs with Sparse Autoencoders Read the full article: https://lnkd.in/ePyEX-YC Paper: https://lnkd.in/eeE_ErWt

2

An amazing article on using  Low Rank Adaptation (LoRA) and QLoRA (Quantized Low-Rank Adaptation)  for fine-tuning LLMs. https://lnkd.in/gHu7hfAE #lora #qlora #machinelearning #llms

16

From what I had observed since getting into the field of robotics and computer vision, when neural networks gained popularity in Computer Vision tasks in early 2013-2014s, there was a trend of using bigger and bigger networks for performance gains. Then came the years where models were optimized (and still are) in terms size, data type, conpression and deployment in specialized hardware for very impressive real time performances. I think Transformers are probably heading towards the same direction where in the next few years we might see LLMs and its derived variants would start to decrease in size, utilize multi modal data more efficiently and generalize into multiple domains all the while increasing in throughput by leveraging modern GPGPU and other acceleration hardware.

2