Byron Boots

Human On the Loop Demand Aware Robot Scheduling: A Mixed Reality Based User Study, IEEE International Conference on Robot and Human Interactive Communication (Ro-MAN) 2024 #robotics #mixedreality #multiarmbandit #reinforcementlearning https://lnkd.in/gM7mchke Ajay Kumar Sandula Rajatsurya Madhusudhana Debasish Ghose Scheduling tasks and subtasks for multiple heterogeneous robots poses a significant challenge, particularly in scenarios where human supervision is essential. The complexity is further compounded when considering human factors in the scheduling process, especially in demand-aware task scheduling environments (tasks are generated based on external demand). The primary goal of this research is to understand human decision-making and how it affects their scheduling process, and we undertook two studies. In the first study, we employed a mixed reality based user study to explore how human perception of the scheduling environment influences task scheduling and facilitates personalized resource allocation. Our findings indicate that human task schedulers exhibit enhanced performance when assisted by autonomous agents, compared to scenarios with limited autonomy in robotic systems. To explore the impact of robot planning on human decision-making and its effects on task scheduling and collision-aware scheduling capabilities, We conducted the second study. This study employed a mixed reality-based warehouse environment, where two users controlled different robots with shared objectives. The findings suggest that human operators exhibited improved collision-aware scheduling without compromising their demand-aware scheduling capabilities when visual aids such as collision cones were incorporated.

22

Must seen for who’s interested in recent advances in robot learning for manipulation.

6

Great explanation and visualization, thanks for sharing Alex Razvant! I would add that other graph compilers have a similar flow, but the differentiation is in the implementation details. For modern AI architectures, graph compiler optimizations are not enough to achieve high performance and utilization, and many AI compilers have dedicated implementations. FlashAttention (kernel-fusion) and PagedAttention (memory management) are examples of dedicated optimization techniques that are not achieved by generic compilation (not to mention Speculative decoding and other algorithmic "tricks"). By the way, this is the main reason in my opinion for why we see so many competing inference engines for LLMs (TensorRT, vLLM, TGI, LMDeploy, ONNX Runtime, SGLang, CTranslate2, llama.cpp, MLC-LLM, DeepSpeed, IPEX-LLM, etc.) 😅

19

1 Comment

https://lnkd.in/eXbCsAnD “He who controls the GPUs, controls the universe.” Access to GPUs, and in particular those made by Nvidia, the leading supplier, is vital for any company that wants to be taken seriously in #ai. Nvidia’s dominance has driven its market value to more than $2trn. 'GPUs do the computational heavy-lifting needed to train & operate large #ai models. Yet, oddly, this is not what they were designed for. The initials stand for “graphics processing unit”, because such chips were originally designed to process video-game graphics. It turned out that, fortunately for Nvidia, they could be repurposed for #ai workloads. 'Might it be better to design specialist ai chips from scratch? That is what many companies, small & large, are now doing in a bid to topple Nvidia. Dedicated ai chips promise to make building and running ai models faster, cheaper or both. Any firm that can mount a credible threat will have no shortage of customers, who dislike its lofty prices and limited supplies. 'GPUs contain several thousand processing engines, or “cores”, which let them run thousands of versions of the same simple task at the same time. Running #ai models similarly involves running lots of copies of the same task in parallel. Figuring out how to rewrite ai code to run on GPUs was one of the factors that triggered the current ai boom. 'Yet GPUs have their limitations, particularly when it comes to the speed with which data can be shuffled on and off them... When training very large #ai models, some GPu cores may be idle as much as half of the time. Andrew Feldman, the boss of Cerebras, a startup based in Sunnyvale, CA, likens it to the gridlock in a grocery store on the day before Thanksgiving. “Everybody’s in a queue, so there are blockages in the parking lot, there are blockages in the aisles, blockages at the checkout. That’s exactly what’s happening with a gpu.” 'Cerebras’s response is to put 900,000 cores, plus lots of memory, onto a single, enormous chip, to reduce the complexity of connecting up multiple chips & piping data between them. 'Groq, a startup from Mountain View, CA, is taking a different approach. Its ai chips, called language processing units (lpus), are optimised to run #llms particularly quickly. 'Yet another approach is that taken by MatX, also based in CA. GPUs contain features & circuitry that provide flexibility for graphics, but are not needed for llms. The firm is working on getting rid of such unnecessary cruft.

14

3 Comments

In this paper, Linyu Liu etal’s study the problem of uncertainty estimation and calibration for LLMs. We first formulate the uncertainty estimation problem for LLMs and then propose a supervised approach that takes advantage of the labeled datasets and estimates the uncertainty of the LLMs’ responses. Based on the formulation, they illustrate the difference between the uncertainty estimation for #LLMs and that for standard #ML models and explain why the hidden neurons of the LLMs may contain uncertainty information. They designed an approach which demonstrates the benefits of utilising hidden activations to enhance uncertainty estimation across various tasks and shows robust transferability in out-of-distribution settings. They distinguish the uncertainty estimation task from the uncertainty calibration task and show that a better uncertainty estimation mode leads to a better calibration performance. The method is easy to implement and adaptable to different levels of model accessibility including black box, grey box, and white box. The key is to develop an LLM which provides the user with a level of understanding of the likelihood that the results are good enough to make decisions, or not. https://lnkd.in/eYYHRZxz #Uncertainty #Estimation #LLMs #AI

9

Berkeley team have developed an enhanced method, called LoRA+, to efficiently fine-tune large language models using Low Rank Adaptation (LoRA). LoRA reduces the number of trainable parameters by introducing low-rank matrices A and B to adapt pretrained weights. However, the standard LoRA setup, which applies the same learning rate to both A and B, leads to suboptimal feature learning for very wide models.     The authors show mathematically that the optimal learning rates for A and B should increase differently with model width. LoRA+ addresses this by using different learning rates for the two matrices, with the learning rate of matrix B being a fixed multiple λ of the learning rate of matrix A. Guidelines provide recommendations for choosing λ based on model architecture and task.   Experiments validated that LoRA+ improves both fine-tuning speed and performance compared to standard LoRA for GPT-2, RoBERTa, and LLaMA models on various benchmarks. Benefits were greater for tasks requiring efficient feature learning to align the model. Integrating LoRA+ is straightforward, requiring only setting the learning rate ratio hyperparameter λ, with a recommended default of λ=16.    This work provides guidance on hyperparameter settings for LoRA, a standard for efficiently specializing large language models. By showing how to "scale up" LoRA for today's massive models, LoRA+ is a useful tool for practitioners wanting to leverage powerful foundation models while controlling costs.    Arxiv: https://lnkd.in/ep6MHyWh

1

It was nice to interview with Karen for the following article in Semiconductor engineering on how deep learning transformed computer vision. https://lnkd.in/gpweppRx

14

I've been working on dynamics model learning in robotics for >6 years (from the Neural-Lander paper). Training a small DNN + some regularization + some online weight adaptation has proven to be effective for specific robots in specific tasks. LLMs have shown the power of transformer, large-scale pertaining, fine-tuning, and in-context adaptation. Can we do the same thing for dynamics model learning? Excited to introduce AnyCar, a transformer-based universal vehicle dynamics model that adapts to various cars & envs & tasks & state estimators, via in-context adaptation. It has some interesting designs akin to LLMs: 1. A seq2seq model aware of history. Input: K-step noisy state/action history, H-step future action. Output: H-step future state prediction; 2. Large-scale pertaining using 100M sim data collected in diverse sims, including Isaac Sim, MuJoCo, and F1-level Assetto Corsa; 3. Real-world "alignment" via fine-tuning on 20K real data; 4. Online "reasoning" by sampling-based MPC that rolls out action sequences using the learned model. AnyCar shows strong adaptability in many scenarios, such as drifting in the wild with 3D-printed plastic tires. Fully open-sourced (including the 1/16 racing car design): https://lnkd.in/e8wZRNPJ

1,203

15 Comments

Pretrained large networks, sometimes called foundational models, are becoming increasingly useful in our research these days. Many works are pointing out how these models are advanced and can be generalized to many different tasks. I generally take the more humble view and accept that these methods fall somewhere between far better than starting from a randomly initialized model to can be helpful for planning. I gave a lecture in my class to piece together the very recent progress on using large or pretrained models for robot control. There are likely more works to include, but I suspect this outline will help others catch up to the progress made in using large models for robotics. More source information on the lecture is available here. https://lnkd.in/gcX8vgr7 I also look forward to presenting this recent research in a more condensed format at #WorldAiSummit this week, where I will talk about how we can use generalist robots to learn the tech tree of the universe. https://lnkd.in/gEEB_5kZ

65

1 Comment

Are you facing hurdles in scaling up your computational tasks or fine-tuning deep learning models on distributed memory systems with GPU acceleration? Join us at JuliaCon 2024 for the Dagger.jl Workshop! Discover how Dagger's parallel task-based runtime can empower you to efficiently parallelize across threads, nodes, and GPUs. Secure your spot now! 🌟 JuliaCon 2024 #JuliaLang #JuliaCon2024 #ParallelComputing #Daggerjl #GPUComputing

11

Big news from Meta: llama3.2 is here along with LlamaStack making development of AI systems on edge systems move accessible. This will change the way we use and access open-source AI since the 1B and/or 3B parameter models don't need large amounts of compute. But what about the 11B and 90B LLM models!? Well..... "... Llama 3.2 11B and 90B can’t be accessed in Europe. As a result, several Meta AI features available elsewhere, like image analysis, are disabled for European users. Meta once again blamed the “unpredictable” nature of the bloc’s regulatory environment." I'm looking at you European Parliament and European Commission #ai #aiineurope #tech #llms

1

Wake up - new scaling laws paper just dropped! "Scaling Laws for Precision" Multiple interesting findings: * Training larger models in lower precision can be compute optimal - i.e. for a given compute budget you want to train larger models in lower precision rather than smaller models in higher precision in order to achieve the optimal loss * Too much pretraining data can actually be harmful as you quantize such models to lower precision for inference. Llama models popularized the data saturation regime (training on many more tokens / param than recommended by Chinchilla scaling laws (20 tok/param)) as their goal was not to train a compute optimal model but instead to optimize for inference. These models are more suceptible to degradations due to aggressive quantization. * When number of params is held fixed, compute optimal precision increases approximately logarithmically with data, i.e. for larger models you probably don't want to train in low precision even if your GPUs/training stack supports it! We're hitting the quantization limits that have historically brought us a lot of speed improvements (going from fp32 -> bf16 -> fp8). Lots more details in the paper: https://lnkd.in/dC3_EVrs --- 🔔 If you're looking to hire best ML/tech talent check out my platform here: https://lnkd.in/duJZ3D9T post a job here: https://lnkd.in/df-Ukm8E

169

4 Comments

🌭🌭HOP: Hand-Object interaction Pretraining🌭🌭 👉Berkeley unveils a novel approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. Using in-the-wild videos to generate sensorimotor robot trajectories. 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬: ✅General manipulation prior from in-the-wild ✅Simulator as videos-sensorimotor intermediary ✅Fine-tuning via RL and behavior cloning (BC) ✅Sample-efficient adaptation to downstream tasks #artificialintelligence #machinelearning #ml #AI #deeplearning #computervision #AIwithPapers #metaverse 👉Discussion https://lnkd.in/dMgakzWm 👉Paper https://lnkd.in/dhRFQara 👉Project https://lnkd.in/dCa77u7P

168

3 Comments

Check out how the world’s first #generativeAI models for #OpenUSD development will be available as NVIDIA NIM microservices.

9