Zijia Chen

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

77

🌟 Boost Meta's Llama 3.3 70B inference throughput by 3x with Tensor RT-LLM optimizations including speculative decoding, in-flight batching, KV caching, custom FP8 quantization, and more.🦙 Technical deep dive ➡️

8

Kudos to #Composable_kernel team and please check out the AMD engineer Haocong Wang presents the ROCm Composable Kernel library which allows to write high-performance kernels for machine learning with HIP C++. #Composable_kernel #ROCm

39

1 Comment

Great explanation and visualization, thanks for sharing Alex Razvant! I would add that other graph compilers have a similar flow, but the differentiation is in the implementation details. For modern AI architectures, graph compiler optimizations are not enough to achieve high performance and utilization, and many AI compilers have dedicated implementations. FlashAttention (kernel-fusion) and PagedAttention (memory management) are examples of dedicated optimization techniques that are not achieved by generic compilation (not to mention Speculative decoding and other algorithmic "tricks"). By the way, this is the main reason in my opinion for why we see so many competing inference engines for LLMs (TensorRT, vLLM, TGI, LMDeploy, ONNX Runtime, SGLang, CTranslate2, llama.cpp, MLC-LLM, DeepSpeed, IPEX-LLM, etc.) 😅

19

1 Comment

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

48

2 Comments

We had a great session about NVIDIA NIMs with Anshul Jindal and Mark Moyou, PhD yesterday. 🚀 From Mark: We got lots of info on inferencing with NIMs and different types of NIMs like embedding models, generative models, reranking models.. 🎨 I walked through how you can use NIMs for embedding, retrieval and generation with #Haystack 🚀 And finally, Anshul did a live coding session where he showed how you can deploy and scale your own self-hosted NIMs. If you missed it, or don't even know what I'm talking about but you wanna learn, the recording is up (adding to the comments👇) All resources, including the example repos and guides are in the description. #AI #LLM #Nvidia #opensource

36

2 Comments

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

Check out the latest research on high-performance graph processing from a hardware/software co-design perspective! This paper dives deep into innovative strategies to optimize computational efficiency. https://lnkd.in/gUXumc5x 廖小飞 Hai Jin Long Zheng Zhiyuan Shao

Implementing Reinforcement Learning (RL) in the context of Large Language Models (LLMs) posed challenges, particularly in designing reward engineering and penalization components. In traditional RL, reward engineering is straightforward, often defined through a scalar reward function. However, implementing the same concept in RL with LLMs required a critic model that evaluates the questions and answers, critiquing them in a manner that simulates the reward function. A recent research paper on Natural Language Reinforcement Learning (NLRL) provides an intriguing perspective on the reward problem in RL by translating traditional RL concepts into natural language constructs. Here's a summary of how it addresses the reward problem: Language-Based Value Functions: Unlike traditional RL, which uses scalar rewards, NLRL employs language-based value functions. These functions articulate the effectiveness of policies and actions in a more interpretable, natural language format. This approach addresses the need for richer, more informative feedback than simple numerical scores. Iterative Refinement of Language Policies and Critics: NLRL incorporates a structured training pipeline where language-based policies and critics are iteratively refined using textual feedback. This method allows the system to handle the complexities of defining rewards in natural language, improving its understanding and evaluation of actions over time through textual feedback loops. Use of Large Language Models (LLMs): The framework leverages the advanced capabilities of LLMs to interpret, generate, and evaluate complex language-based information. This is essential for implementing language-based value functions and policies, enabling the system to generate and refine responses that align closely with desired outcomes. Distilling Enhanced Actions and Evaluations: NLRL employs a process where enhanced evaluations from the language-based value function approximator and improved actions from the language policy improvement operator are distilled. This allows for continuous refinement and learning. Compatibility with Existing LLMs: The framework is designed to be compatible with current LLMs, enabling it to leverage pre-existing language models to process and generate linguistic data effectively. This compatibility is crucial for translating the traditional reward engineering process into a language-based framework. The NLRL framework's introduction of natural language as a medium for representing and evaluating policies and rewards offers a novel solution to the reward problem in RL. By shifting from numerical reward systems to a more flexible, expressive, and intuitive system based on language, NLRL enhances interpretability and adaptability, paving the way for more diverse RL applications. Innovation Hacks AI Inc. #llm #refincorcementlearning #ai

4

Advances in Bayesian Deep Neural Network Ensembles and Active Learning for Preference Modeling Quick read: https://lnkd.in/gzZB2VGh Paper 1: https://lnkd.in/gBkwknBZ Paper 2: https://lnkd.in/gNrqnqg9

18

See how RAG, built on NVIDIA GPU architecture and CodeLlama 34B, transforms access to #UnrealEngine5 documentation, saving developers time and reducing the need for costly model training and fine-tuning.

14

A session on "Julia Compiler Secrets - Static Analysis" by Keno Fischer and Shuhei Kadowaki The Julia compiler has evolved significantly, expanding its static analysis capabilities beyond type inference to include effect analysis, escape analysis, and optimized generics. These advancements, introduced in recent releases like Julia 1.8 and 1.11, are paving the way for faster, more efficient, and safer code. The talk delved into how these new capabilities enhance code performance and correctness, offering insights into the future of the Julia compiler. Exciting times ahead for Julia developers #JuliaCon #JuliaLang #CompilerTech #StaticAnalysis #PerformanceOptimization

74

What a great talk from Swapnil Sayan Saha, on Physics-Aware Tiny Machine Learning! https://lnkd.in/g9g4YWs4 There's exciting times ahead as Physics-aware TinyML is becoming increasingly important in merging the power of machine learning with operating in the physical world. By integrating real-world physics models into ultra-low-power devices, this approach dramatically enhances the accuracy and efficiency of TinyML applications. From improving energy efficiency in smart sensors to enabling more accurate anomaly detection in industrial applications, this approach to machine learning at the edge helps machines better understand their environment - without sacrificing power or performance. As we explore the possibilities, this combination of physics and Edge AI opens up new opportunities in fields like healthcare, environmental monitoring, and autonomous systems. It's a huge leap toward making AI more sustainable and impactful, and allows us to build more useful AI-enabled systems that operate in (and on!) the physical world. #TinyML #EdgeAI #PhysicsAwareAI #AIInnovation #Sustainability #MachineLearning

4

Our #icra2024 paper is now available from #ieee dl. We proposed a new Inverse Reinforcement Learning based method for hand trajectory prediction. Mukund Mitra Gyanig K. Partha Pratim Chakrabarti

18

2 Comments

𝗧𝗵𝗲 𝗲𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝗮𝗿𝘆 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗼𝗳 𝗣𝗘𝗙𝗧 To perform task-specific full fine-tuning with LLMs like LLaMA3 70B, a minimum of 5120GB of computational resources may be required. The enormous computational resource requirements are prohibitive for anyone but the superpower players to utilize LLMs for task-specific fine-tuning. To address this challenge, PEFT has emerged as a viable solution to compensate for the tremendous computational cost of full parameter fine-tuning. PEFT updates only a small number of additional parameters or updates a subset of the pre-trained parameters, preserving the knowledge and reducing the risk of catastrophic forgetting. As a result, there has been an increase on PEFT techniques during the last few months. Some of the most popular techniques where many customers are purring lot of effort to improve LLMs for theirs specific purposes are LoRA, QLoRA or Prompt-Tuning; If you want to start easily testing these techniques, I recommend to follow this couple of repositories: 👉 https://lnkd.in/dtvHP_Cn 👉 https://lnkd.in/daKDY4f5 #ai #genai #finetuning

72

3 Comments

🔍 Quantum AI Series: Modeling Sequences with Quantum States Picking up where I left off in my #QuantumAI series, let’s talk about Modeling Sequences with Quantum States, with insights from a fascinating paper on the topic. Classical probability distributions can be modeled using quantum states—particularly entangled states that retain information from subsystems. This approach allows quantum models to efficiently represent sequences without losing essential details. Tensor networks play a key role in optimizing the process, providing a compact, low-rank factorization of data. As I've shared previously, Singular Value Decomposition (SVD) is foundational to these models, as it helps represent the complex correlations present in sequential data. Matrix Product States (MPS) capture these rich interactions by using reduced densities to efficiently compress sequence representations while preserving their structure. This is the essence of keeping representation while building efficient lightweight models. In summary, quantum probability models allow us to go beyond classical limits in sequential tasks, contributing to information retention. Sharing the paper for a deeper dive if you'd like so: https://lnkd.in/d5aqXyyS #QuantumAI #TensorNetworks #SVD #QuantumComputing #AI

5