I-Ta Lee

Happy about our latest paper titled "Deduplicator: When Computation Reuse Meets Load Balancing at the Network Edge", which was recently accepted for publication by IFIP Networking 2024. All the props go to my senior Ph.D. student, Md. Washik Al Azad. This is the 9th (!) publication during his Ph.D., which is a substantial accomplishment for a systems student. The rest of his papers have been published at IEEE PerCom (best paper runner up award), the IEEE Internet of Things Journal (impact factor > 10), IEEE CLOUD, and the IEEE Wireless Communications Magazine (impact factor > 10), among other venues. In this paper, we explore whether we can achieve the contradictory in nature functions of load balancing and computation reuse at the network edge. To this end, we propose an edge middlebox, called the Deduplicator. Our experimental results demonstrate that we can facilitate computation reuse and distribute the load among edge servers at the same time. You can find a preprint of our paper on arXiv: https://lnkd.in/dkQqq9XM Our code is also available on GitHub: https://lnkd.in/dq3HXJSZ #research #cloudcomputing #edgecomputing #loadbalancing #distributedsystems #systems #computernetworks #middleboxes

45

6 Comments

Have you heard about tiny AI accelerators (smaller than 1cm x 1cm) that can even run on microcontrollers? I’m excited to share that I’ll be attending #NeurIPS2024 to present Nokia Bell Labs' initiative on building collaborative on-device AI systems: “DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators.” This work introduces an innovative approach to improving AI model accuracy without increasing latency, leveraging intelligent data channel extensions on tiny AI accelerators. Huge thanks to Taesik Gong and Fahim Kawsar for their invaluable collaboration! This marks the beginning of our vision to enable multi-device, multi-modal, and multi-sensory computing on the body. Stay tuned for more exciting projects in this direction! * When: Wednesday, Dec. 11, 4:30 PM – 7:30 PM * Where: East Exhibit Hall A-C, Booth 1405 * arXiv: https://lnkd.in/emP4_5vg * Code: https://lnkd.in/ecn5yTCB * Broader overview of our department research: https://lnkd.in/enbW6fnW Summary: Tiny AI accelerators enhance TinyML by improving processing power, but their limited memory often reduces input resolution, negatively impacting accuracy. To address this, we introduce DEX—a method that extends input channels by incorporating spatial information through patch-wise sampling and stacking. DEX leverages unused processors and memory for parallel execution, maintaining inference latency while improving accuracy by an average of 3.5 percentage points.

76

4 Comments

Triple arxiv drop from the last couple of weeks: Remember when we were excited with snapfusion running diffusion on device? we now do diffusion-based super res on 4k x 3k images on the phone. And still match the SOTA results. This is how: Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning https://lnkd.in/e6DD-bJG CLIP models struggle with compositionally and manifest strong bag-of-words behaviour. Meanwhile, while benefiting from strong language understanding capabilities, Visual LLMs trail behind contrastively trained models for discriminative tasks like retrieval. We propose a novel training method that enhances the Visual LLMs retrieval ability, achieving SOTA performance for 1-stage models on discriminative tasks. Discriminative Fine-tuning of LVLMs https://lnkd.in/eS4syJvT Want to create high-resolution images but your diffusion model is trained to operate on lower resolutions? We propose a training-free method that does exactly that. And get state-of-the-art results among competitors. And with minimal latency overhead (much lower than most competing methods). Check it out here: FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion https://lnkd.in/eJVEkBVx

55

3 Comments

The Transformer Series: This video explains the concept of positional encoding in the transformer architecture. In our previous videos, we looked at Skip connections, scaled dot product attention and multi-head attention. Now, let’s shift our focus to another important aspect of the transformer architecture: Positional Encoding. Positional encoding is crucial for transformers to understand the order of input data when looking at sequential data, such as the usual english sentences where the order of the words is important. Positional encoding adds unique markers to each input token to provide context of its position in the sequence. This method allows the model to recognize patterns and relationships in data that depend on the sequence order—something essential in tasks like language understanding or time series analysis. Take a look at this video to discover how positional encoding enables transformers to map out complex underlying patterns in sequence data. https://lnkd.in/g4bi4_A3 #transformers #deeplearning #ai #ml #nlp

59

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15

If you are interested in Large Language Models and fine-tuning space, check out our paper where we study different fine-tuning scenarios and the interaction of various fine-tuning optimizations in such scenarios. In collaboration with Arjun Singh, Anup A. Shirgaonkar, Pavan Manoj and Vijay Aski.

27

LLM Research Papers: The 2024 List (Sebastian Raschka, December 2024) "It’s been a very eventful and exciting year in AI research. This is especially true if you are interested in LLMs. I had big plans for this December edition and was planning to publish a new article with a discussion of all my research highlights from 2024. I still plan to do so, but due to an accident and serious injury, I am currently unable to work at a computer and finish the draft. But I hope to recover in the upcoming weeks and be back on my feet soon. In the meantime, I want to share my running bookmark list of many fascinating (mostly LLM-related) papers I stumbled upon in 2024. It’s just a list, but maybe it will come in handy for those who are interested in finding some gems to read for the holidays. And if you are interested in more code-heavy reading and tinkering, My Build A Large Language Model (From Scratch) book is out on Amazon as of last month. In addition, I added a lot of bonus materials to the GitHub repository." https://lnkd.in/e7d6z4Vj

2

🔗 Simulate PyTorch Models with Simulink! 🚀 Engineers, unlock new possibilities by simulating PyTorch and other Python-based models directly in Simulink using co-execution blocks. This integration streamlines your workflow, enabling seamless model testing and validation. 🔍 What's Inside: >> Co-execution of Python models in Simulink << >> Enhanced simulation and validation processes << 👉 Dive into the details: https://spr.ly/6041o1cOX #DeepLearning #Simulink #Python #ModelSimulation #Engineering #MATLAB #MAthWorks #PyTorch

9

LLMs achieve adult human performance on higher-order theory of mind tasks https://lnkd.in/g-A_JFr7 The paper examines the ability of large language models (LLMs) to develop higher-order theory of mind (ToM) - the human capacity to recursively reason about multiple mental and emotional states (e.g. "I think that you believe that she knows"). The authors introduce a new handwritten test suite called "Multi-Order Theory of Mind Q&A" to evaluate five LLMs against a newly gathered adult human benchmark on ToM tasks. The results show that GPT-4 and Flan-PaLM reach adult-level and near adult-level overall performance on ToM tasks, respectively. Notably, GPT-4 even exceeds adult human performance on 6th order ToM inferences. The findings suggest an interplay between model size and finetuning in the realization of ToM abilities in LLMs. The best performing models appear to have developed a generalized capacity for theory of mind reasoning. Given the critical role higher-order ToM plays in human cooperative and competitive behaviors, these results have significant implications for user-facing applications of large language models that need to understand recursive mental state reasoning. In summary, state-of-the-art LLMs like GPT-4 demonstrate human-level or better ability to make higher-order theory of mind inferences, an important milestone with real-world implications.

21

I've been playing with matrix multiplication on GPUs, using a Hilbert curve to improve L2 cache utilization, and seeing if I can get non-embarrassing performance out of it. Seems this approach can outperform Nvidia's cuBLAS for matrix sizes of 16384x16384 and above. Blog post here with open source code on GitHub for anyone who'd like to mess with it too. #gpu #cuda #cplusplus https://lnkd.in/e4QvmSVR

61

4 Comments

This paper https://lnkd.in/eGaCaD7s shows that calibrating LLMs to the distribution of their training set always leads to hallucination, at a probability lower bounded by Good-Turing frequency estimation. Good-Turing frequency estimation https://lnkd.in/ewrh3RZC was developed to estimate the frequency of discovering a new species. Assuming the discovery of new species follows i.i.d., then its frequency can be reasonably estimated by the historical discovery rate, which is N_1 / N​, where N_1​ is the number of species observed only once (a.k.a., new discoveries), and N is the total number of observations. Now, calibrating LLMs forces them to emit only in-distribution facts (known species) and zeros out the probability of out-of-distribution facts (unknown species). Therefore, calibrated LLMs will hallucinate when the question is OOD, at a rate of N_1 / N​​. And there is no cure, as non-calibrated LLMs can be even worse. #artificialintelligence #machinelearning #deeplearning

208

18 Comments

Hey Folks, Another day, another breakthrough in large language models: "iLLM-TSC: Integrating Reinforcement Learning and Large Language Models for Traffic Signal Control Policy Improvement." This innovative approach combines reinforcement learning (RL) with large language models (LLMs) to enhance traffic signal control. Traditionally, RL has been used to make decisions based on observed data. However, real-life events are often not considered. By integrating LLMs, real-time events can now be validated, leading to more effective traffic control. Visit this paper for more information : https://lnkd.in/deE8KsDY #AI #ArtificialIntelligence #ReinforcementLearning #GenAI #LLM

3

Our #icra2024 paper is now available from #ieee dl. We proposed a new Inverse Reinforcement Learning based method for hand trajectory prediction. Mukund Mitra Gyanig K. Partha Pratim Chakrabarti

18

2 Comments

🔍 Quantum AI Series: Modeling Sequences with Quantum States Picking up where I left off in my #QuantumAI series, let’s talk about Modeling Sequences with Quantum States, with insights from a fascinating paper on the topic. Classical probability distributions can be modeled using quantum states—particularly entangled states that retain information from subsystems. This approach allows quantum models to efficiently represent sequences without losing essential details. Tensor networks play a key role in optimizing the process, providing a compact, low-rank factorization of data. As I've shared previously, Singular Value Decomposition (SVD) is foundational to these models, as it helps represent the complex correlations present in sequential data. Matrix Product States (MPS) capture these rich interactions by using reduced densities to efficiently compress sequence representations while preserving their structure. This is the essence of keeping representation while building efficient lightweight models. In summary, quantum probability models allow us to go beyond classical limits in sequential tasks, contributing to information retention. Sharing the paper for a deeper dive if you'd like so: https://lnkd.in/d5aqXyyS #QuantumAI #TensorNetworks #SVD #QuantumComputing #AI

5

Here is a good primer on general relativity. Anyone wondering what bent space time is, please do take a look. The open questions on connecting gravity to QM are my favorite topics of research. It'd be benefit of unimaginable proportions if we could link these two massively successful theories. Because then we'd be also able to link our spooky-action-at-a-distance (entanglement) with gravity. It seems to me that its really our best friend for any galactic space travel, but unfortunately its currently limited to quantum particles. https://lnkd.in/gbBVpMyw

2

Training of LLMs to generate multi-facet structured substances (JSON) using Targeted Denoising. (EMNLP-2024). https://lnkd.in/g66F_S9x

40

1 Comment

Ladies and Gentlemen, here is the new paper from Meta: LCM, Large Convept Models, no longer token predictors but concept predictors! 💡As in the human brain🧠we do not reason by building sentences, word by word, but we start from abstract concepts, ideas that combine without essentially resorting to language!

7

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

Some folks from my team will be in Barcelona later this month to present at the 2024 KDD Workshop on "Causal Inference and Machine Learning in Practice." They will be presenting our paper titled "Practical Marketplace Optimization at Uber Using Causally-Informed Machine Learning." This work delves into the technical and business challenges of budget allocation for marketplace levers at Uber. These levers are instruments such as driver incentives and rider promotions. Understanding the impact of budget changes on the market and estimating cost efficiency are crucial for setting forward-looking budgets. Our paper describes an end-to-end machine learning and optimization procedure to automate budget decision-making across cities where Uber operates. This end-to-end process includes: * An advanced deep learning (DL) estimator based on S-Learner that leverages extensive user experimental and temporal-spatial observational data. * A novel tensor B-Spline regression model to enforce efficiency shape control while maintaining the sophistication of DL models’ response surface. * Solving high-dimensional optimization problems using Alternating Direction Method of Multipliers (ADMM) and primal-dual interior point convex optimization. All of this has significantly improved Uber’s resource allocation efficiency. Kudos to the team for their hard work on this and for capturing it for an outside audience! https://lnkd.in/gFkbAbCY Zhongbo (Bobby) Chen / Siyu Chen / Jason Dowlatabadi / Yu Xuan H. / Vinayak Iyer / Uday Mantripragada / Rishabh Narang / Apoorv Pandey / Zijun Qin / Abrar Sheikh / Hongtao Sun / Jess Sun / Matthew Walker / Kaichen Wei / Chen Xu / Jingnan Y. / Allen Zhang / Guoqing Zhang, CFA, FRM

99

1 Comment