Hao Wang

anthropics/anthropic-cookbook: A collection of notebooks/recipes showcasing some fun and effective ways of using Claude. https://bit.ly/3QpOnSV #AI #MachineLearning #DeepLearning #LLMs #DataScience

36

📢 DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads🚀 In this work, they tackle the significant memory and computational challenges associated with deploying long-context large language models (LLMs). The proposed framework, DuoAttention, introduces a novel distinction between Retrieval Heads and Streaming Heads to optimize the use of the KV cache, reducing memory usage by up to 2.55× and improving decoding speeds by up to 2.18×, all while maintaining high accuracy. Technical Highlights: 1. Retrieval Heads vs. Streaming Heads: DuoAttention divides attention heads into two categories. Retrieval Heads are crucial for processing long contexts, whereas Streaming Heads focus on recent tokens, allowing for a more efficient use of memory resources. This division helps in reducing the computational load without compromising model accuracy. 2. KV Cache Optimizatio: The framework uses selective caching strategies, applying full KV cache to Retrieval Heads while maintaining a constant-length cache for Streaming Heads. This results in significant reductions in both memory usage and latency during long-context inference. 3. Scalable Long-Context Inference: With DuoAttention, models like Llama-3-8B can handle up to 3.3 million tokens on a single A100 GPU. This is made possible through careful optimization of memory allocation and the use of lightweight algorithms to identify the critical heads. 4. Performance Improvements: DuoAttention achieves up to 2.55× memory reduction for multi-head attention (MHA) models and up to 1.67× for grouped-query attention (GQA) models. It also speeds up decoding by up to 2.18× for MHA models and up to 1.50× for GQA models, providing significant benefits for real-world applications involving large language models. 5. Compatibility with Quantization: The method is fully compatible with quantization techniques, such as 8-bit weight and 4-bit KV cache quantization. This compatibility further boosts efficiency, enabling the model to handle larger contexts with reduced hardware requirements. With DuoAttention, the team has achieved long-context capabilities for models like Llama-3-8B, enabling handling of up to 3.3 million tokens on a single A100 GPU. This advancement marks a significant step towards more efficient, scalable LLM deployments! 🔍 Paper: https://lnkd.in/dDFHimSA #LLMs #MachineLearning # #EfficientInference

13

I wrote a short tutorial on using Porcupine to check for linearizability (without needing to deal with the JVM). https://lnkd.in/eW6Y-QQQ

64

https://lnkd.in/gUnEeRVe That's why I started supporting Meta Open-source LLMs(Large Language Models) like LLAMA. I got to know about this LLM when LLAMA 2 was launched in 2023 and yesterday AI at Meta has released LLAMA 3 You can know more about what LLAMA is on their website:- https://llama.meta.com/ ( There they have mentioned how one can contribute to this open source LLM ) #artificialintelligence #artificialgeneralintelligence

2

In this article we are providing the major architectural building blocks and blueprint for building end-to-end GenAI-based applications that are ready for production. I’d like to call out several key considerations as you go about implementing and designing these llm-based applications. #llms #llmops #chatgpt4 #aiml #gemini #mlops #data https://lnkd.in/dMev5ncq

8

Paper of the week : Inference Scaling for Long-Context RAG - uses two strategies to investigate scaling laws for RAG: in-context learning (DRAG) and iterative prompting (IterRAG); finds that RAG performance consistently improves with the expansion of the effective context length under optimal configurations; when optimally allocated, increasing inference computation can lead to linear gains in long-context RAG performance; this leads to the development of a computation allocation model that can provide practical guidance for optimal computation allocation in long-context RAG scenarios. https://lnkd.in/dTfTRjp7 #paperoftheweek #machinelearning #llms

13

Finally got back to my "0 to #glTF with #WebGPU" series, this post looks at adding basic material and texture support to get nicer looking renderings (and more interesting workloads to try and batch up later): https://lnkd.in/gjT7bt2U code: https://lnkd.in/g6fgJHW3

46

Thrilled to share our new work about Hymba, a model architecture that fuses Mamba and transformer! It outperforms Llama-3.2-3B by 1.32% average accuracy while being more hardware-friendly(12x cache reductions and 3x throughput)!

12

Insightful, looking forward to building some neural networks next quarter and exploring Sebastian Raschka, PhD's new book "Build a Large Language Model From Scratch." I own a copy of each of Sebastian's books. I highly recommend him to anyone wanting to start their journey in the ML & AI space. #Transformers #AI #LLMs #DeepLearning

244

4 Comments

New Flux-based Personalization endpoint on fal! Based on OminiControl, It allows you to generate images of subjects in various contexts, including branding, stickers, product mockups, and more. Try it out here: https://lnkd.in/gwCHavr5 More examples in the comments:

146

3 Comments

I’ve completed a project on NYC Taxi Trip Duration Prediction using various regression models, including Linear Regression, Random Forest, and Gradient Boosting. The goal was to analyze and predict the duration of taxi trips based on factors like pick-up and drop-off locations, timestamps, and passenger details. Key Highlights: - Data Analysis: Conducted thorough exploratory data analysis (EDA) to understand the dataset and visualize key trends. - Model Development: Implemented multiple regression models and optimized them for better accuracy, achieving an R² score of 0.70 with Gradient Boosting. -Performance Evaluation: Evaluated model performance using various metrics, ensuring reliable predictions. This project enhanced my skills in data analysis, machine learning, and model optimization. I’m looking forward to applying these learnings in future endeavors! You can check out the project on my GitHub: [https://lnkd.in/g8b_awJU] #DataScience #MachineLearning #Python #NYCTaxi #Project

6

2 Comments

To understand how Large Language Models (LLMs) work, the Transformer Explainer tool is a fantastic resource. Here’s a quick breakdown of how LLMs like GPT use the transformer model, with insights from the tool: 🔹 Tokenization: Text is split into small parts (tokens) for processing. 🔄 Embedding: Tokens become numerical vectors to capture meaning. 🎯 Self-Attention Mechanism: The model assesses each token's importance to others for context. 🏗️ Layered Refinement: Stacked layers refine token meanings and context. 🔡 Decoding: The model generates coherent text from refined tokens. The Transformer Explainer tool makes this process interactive, letting you visualize self-attention and layer transformations in real-time. Check out the tool at [poloclub.github.io](https://lnkd.in/d8K7DTuQ), or explore further on [YouTube](https://lnkd.in/dMR37gFP) and [GitHub](https://lnkd.in/dukBJAHq).

4

In this episode, we discuss CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models by @Je-Yong Lee, @Donghyun Lee, Genghan Zhang, Mo Tiwari, Azalia Mirhoseini. The paper presents "Contextually-Aware Thresholding for Sparsity (CATS)," a method intended to reduce the operational costs of Large Language Models (LLMs) by increasing activation sparsity while maintaining high performance levels. Unlike traditional sparsity-enhancing approaches that degrade model performance, CATS uses a novel non-linear activation function that achieves up to 50% sparsity with minimal loss in effectiveness. Furthermore, CATS improves convergence and performance on tasks when fine-tuning, and its implementation via a custom GPU kernel yields about a 15% reduction in inference time specifically on models like Llama-7B and Mistral-7B.

9

Hot take: We should stop calling image+text->text models 'MLLMs' because true multimodal models like Gemini Flash 2.0 are a different beast entirely. Here's my deep dive into the naming debate with citations from 15+ recent papers: https://lnkd.in/e_uzBSxX

41

Qwen team shared a technical report on their new model Qwen2.5-Coder series. Their model performs in some coding tasks even better than chatgpt o1 preview. In the report, they introduce the Qwen2.5-Coder series, a significant upgrade from its predecessor, CodeQwen1.5. This series includes two models: Qwen2.5-Coder-1.5B and Qwen2.5-Coder-7B. As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder demonstrates impressive code generation capabilities while retaining general versatility. The model has been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size. We believe that the release of the Qwen2.5-Coder series will not only push the boundaries of research in code intelligence but also, through its permissive licensing, encourage broader adoption by developers in real-world applications. read the full report: https://lnkd.in/dbMQjjqw

4