Deep Doshi

🎇 On National Day, I went for a leisurely drive in San Francisco and ended up "stress-testing" a Waymo self-driving car on the road. 🚗 As an autonomous driving practitioner, who wouldn't be curious about the real-time robustness of the cutting-edge Waymo One driving system? While cruising downtown, I accidentally noticed a Waymo car tailgating me. While this isn't unusual for SF citizens, a wild, "evil" idea suddenly hit me: Why not directly "adversarially attack" the world's autonomous driving status quo? I executed an unexpected maneuver—suddenly reversing—to see how it would react. 🌟 The response was stellar! At the moment I reversed, Waymo One honked instantly—quicker than any human could—activated its hazard lights, and backed away to maintain a safe distance. This reckless move on my part served as an edge case to test the algorithm's robustness under extreme conditions and, potentially, could be a challenging training sample to enhance Waymo's future autonomous systems. 💫 Thrilled to personally experience how current cutting-edge autonomous algorithms handle rare driving behaviors—and how stable and safe Level 4 autonomy is in dealing with diverse scenarios. However, it also prompted deep reflection as an AI researcher in this field: 🤔 In an industry with little room for error, how can we ultimately avoid or minimize issues that AI fails to handle? 💡 I believe two research directions are particularly promising for achieving Level 5 level autonomy in future mobility systems: - 1️⃣ Development and deployment of vehicle-to-everything (V2X) cooperative systems (including V2V, V2I, V2P, etc). Our initial studies (e.g., V2X-ViT, ECCV'2022 arxiv.org/abs/2203.10638) show that in scenarios with severe occlusions or noise, such cooperative systems can significantly enhance the robustness of perception systems, thereby eventually improving traffic safety. - 2️⃣ Adversarial scenarios generation (including Sim-to-Real, generative modeling). Research done by my colleagues at UCLA (V2XP-ASG, ICRA'2023 https://lnkd.in/gu5nKVHD) shows that adversarial learning techniques can effectively simulate adversarial scenarios, greatly improving model robustness in complex "corner case" situations. Of course, it's often infeasible to collect such collision scenarios. 👨‍🏫 As a new Assistant Professor of CS at Texas A&M University, I will lead a group focusing on these exciting research directions, which can be a proactive approach to reducing accidents and improving safety for all. 🔥 I look forward to future collaborations with governments, academics, and companies to research and develop data and algorithms that can help enhance the safety of vulnerable road users, especially seniors and children. We envision a people-centered intelligent transportation system in the future. Interested in these topics? Let's connect and discuss further! #AutonomousVehicles #AI #MachineLearning #SmartCities #Transporation #Humanity #Mobility

143

3 Comments

 A pre-print of my summer intern's recent paper, entitled “Domain decomposition-based coupling of operator inference reduced order models via the Schwarz alternating method”, is available on ArXiv here: https://lnkd.in/g8qZ94CA. The paper presents and evaluates an alternating Schwarz-based approach for coupling together subdomain-local reduced order models constructed via non-intrusive Operator Inference (OpInf) with each other and with subdomain-local full order models (FOMs).

53

Amazing findings by Hugh Zhang, Michele L., Summer Yue, and others at Scale AI, addressing a critical concern regarding Large Language Models (LLMs) evaluation and the challenge of data contamination! With LLMs achieving remarkable success on various benchmarks, concerns have arisen about the extent to which this performance reflects true reasoning ability versus exposure to similar benchmark questions. In response, the paper https://lnkd.in/gttAW-g2 undertook an investigation, introducing Grade School Math 1000 (GSM1k) to closely mirror established benchmarks like GSM8k. Their findings shed light on the prevalence of overfitting, particularly among models like Mistral and Phi, which exhibit accuracy drops of up to 13% on GSM1k compared to GSM8k. However, models at the forefront, such as Gemini, Claude, and GPT, demonstrate minimal signs of overfitting. This research not only underscores the importance of robust benchmarking but also prompts critical reflections on the evaluation of LLM performance. Kudos to the team for their work! #llm #research #genai #machinelearning #ai

15

5 Comments

Its sometimes hard to believe that we live in an era where driverless vehicles aren't just a vision anymore but a reality. Waymo driverless cabs for example, are transforming the San Francisco streets forever! It was a wonderful experience working on an application of autonomous vehicles as part of my final project in the course CS267: Applications of Parallel Computing at the University of California, Berkeley. Our team, comprising Terence N., Zeid Solh, and myself, focused on parallelizing stereo vision for driverless applications. Stereo vision is crucial for self-driving cars as it builds a 3D understanding of the environment by using two cameras. However, real-time processing of these images is computationally expensive, posing a significant challenge for autonomous vehicles. We tackled this by exploring various parallel computing techniques to improve the speed of calculating the Disparity Map algorithm. We used popular techniques such as #SIMD (Single Instruction, Multiple Data), #OpenMP (multi-threading), and CUDA (NVIDIA GPU acceleration), and benchmarked the performance. By distributing the computational load across multiple cores and processors, we achieved substantial performance improvements, enabling real-time processing of high-resolution images and videos. The results were promising! Utilizing CUDA, we saw significant speedups, demonstrating the potential for efficient, real-time depth perception in autonomous systems. This project signified the importance of parallel computing in advancing autonomous driving technologies. I'm incredibly proud of our team's hard work and the insights we gained through this project. Check out our GitHub repo: https://lnkd.in/gYHrEU9z for more details on our implementation! #UCberkeley #CS267 #ParallelComputing #StereoVision #AutonomousDriving #DeepLearning #ComputerVision

49

3 Comments

AI NEWS SERIES Applied Intuition’s Secures $300m for AI in Vehicle Manufacturing Product: - Applied Intuition offers a comprehensive ADAS/AD development toolchain, integrating simulation and data products for rapid, safe automotive software development. - Key products include off-road autonomy stack, generative AI for simulation scenarios, and hardware-in-the-loop testing software. Company: - Founded in 2017, headquartered in Mountain View, CA, Applied Intuition has raised $901.5M from investors like Andreessen Horowitz and Porsche Investments. Impact: - The company's advanced simulation technology enables more efficient, safer development cycles in the automotive industry, shaping future transportation and influencing industry standards. sources: https://lnkd.in/e6VBnZYG https://lnkd.in/e5naFTXu #technology #innovation #future #artificialintelligence #business

1

AI as the Front-End for a Backend Heavily Technical Course — says Andrej Karpathy in the No Priors podcast with Sarah Guo and Elad Gil. I often listen to podcasts at 1.5x speed, but with Andrej, I slow down to catch every detail of his fascinating insights. Key moments from the podcast: 00:02:00 🚗 Advances in self-driving tech and AGI implications. 00:03:21 🤖 Tesla’s lead over Waymo in self-driving capabilities. 00:04:03 🌐 Tesla’s cost-effective sensor strategy. 00:06:23 🏭 Viewing Tesla as a robotics company. 00:08:45 🔄 Early applications for humanoid robots. 00:10:37 🌱 Practical uses like leaf blowing. 00:11:36 🧠 Transfer learning and a unified humanoid platform. 00:15:22 ✨ Impact of Transformers on neural networks. 00:16:33 🔍 Shifting research bottlenecks. 17:15 📊 Focus on synthetic data and enhanced collection methods. 17:57 🧠 Brain reasoning and AI development. 19:10 ⚠️ Importance of diversity in synthetic data. 21:16 🧬 Transformers and human cognition. 22:40 🤖 Evolution of human-AI augmentation. 25:30 🔄 Concerns about closed AI platforms. 27:09 📏 Efficiency of smaller AI models. 29:55 🌍 AI’s role in democratizing education. 32:06 🎓 AI revolutionizing personalized learning. 34:26 🏆 Enhancing individual performance through better educational tools. 34:54 🌍 AI translations and language adaptability. 35:50 🔍 Tailoring learning experiences to individuals. 36:44 🏛️ Minimizing gatekeeping in AI research. 37:12 🤝 Community clustering and knowledge sharing. 39:06 🎓 Making education accessible and enjoyable. 40:04 🧠 Learning as mental exercise. 41:29 👩‍🎓 Focus on continuous learning for undergraduates. 42:26 📚 Emphasizing math, physics, and computer science for young learners. Karpathy’s thoughts on integrating AI into education offer a compelling vision for the future of learning and teaching. Full Interview - https://lnkd.in/dGZM7_k9 #AI #Education #EdTech #DeepLearning #Transformers #MachineLearning #SelfDrivingCars #Innovation #TechPodcast #AIinEducation #PersonalizedLearning #FutureOfLearning #NoPriorsPodcast #AndrejKarpathy #AIInsights #TechTrends

3

✨ Exciting insights from Yann Dubois's recent lecture at Stanford University on building Large Language Models (#LLMs) 🟢 Components of LLMs: Focus on architecture, training algorithms, data collection, evaluation methods, and system optimization for efficient model deployment. 🟢 Pre-training vs. Post-training: Understand the classical language modeling paradigm and the emerging trend of fine-tuning models like ChatGPT to function as AI assistants. 🟢 Importance of Tokenization: Effective tokenization strategies are crucial for handling diverse languages and ensuring model performance. 💎 This lecture is a must-watch for anyone interested in #AI advancements! 👉 https://lnkd.in/dcevp3cJ

44

1 Comment

Please check our new paper about using generative world models to train end-to-end neural stacks for autonomous driving, robotics, embodied AI. Video explaining our work: https://lnkd.in/g73q4uwZ Paper pre-print: https://lnkd.in/gBF2kWDF "Mitigating Covariate Shift in Imitation Learning for AV Using Latent Space Generative World Models"

62

2 Comments

Interesting article about Generative AI! Subscribe to Rishabh Yadav's blog for many more!

10

Is it just me, or is "leading" the autonomous vehicle race determined by the irresponsibility of your claims about it? Seeing a commercial today was the first time I had heard of Ford Motor Company's BlueCruise, which is advertised as "hands-free highway driving". Instead of requiring the driver to keep their hands on the wheel, it supposedly uses eye-tracking to check that the driver is at least ostensibly paying attention to the road, and their marketing is actively encouraging taking their hands off the wheel. What could go wrong? Well, inattentional blindness for one thing. It also didn't take much searching to find incidents under NTSB investigation involving the system, including hitting--and killing--the drivers of at least three stationary vehicles on interstate highways. Links in comments. #criticalthinking #notsoautonomousvehicles #marketinglevel2likeitslevel5

2

1 Comment

"The Alphabet Inc. unit, which operates its commercial ride service in three major cities and is adding a fourth, is starting to generate significant revenue. But safety concerns persist." This article declares Waymo the winner in the robotaxi game. Waymo has so far avoided high-profile mishaps, such as those that caused big problems for Cruise and before them Uber ATG. I provide some safety context for the article: “Waymo has made impressive progress in scaling up operations,” said Philip Koopman, an autonomous vehicle safety researcher and associate professor at Carnegie Mellon University. “Some had previously criticized their ramp-up for being slower than other companies, but they are looking a lot smarter these days for having done that.” "Still, the company has more work to do. “We've seen some recent incidents – for example, a utility pole crash – and an investigation from NHTSA. I hope those aren’t signs they’ve started scaling up faster than they should due to funding pressure,” Koopman said. “Safety still remains the biggest question.” Also describes a poor outlook for Tesla robotaxi technology (which I agree with, but did not contribute to). Worth a read of that part of the article for yourself. https://lnkd.in/e-dgzcKb (paywall; I don't have gift link to this one) Nice article by Alan Ohnsman

37

6 Comments

LLM-based GUI Agents (Survey) LLM-based GUI agents are capable of interpreting complex GUI elements and autonomously executing actions based on natural language instructions. This paper presents a comprehensive survey of LLM-brained GUI agents, exploring their historical evolution, core components, and advanced techniques. The paper also discusses - existing GUI agent frameworks, - the collection and utilization of data for training specialized GUI agents, - the development of large action models tailored for GUI tasks, and - the evaluation metrics and benchmarks necessary to assess their effectiveness. #llms #guiagents #agents #nlproc

166

4 Comments

I spent Labor Day weekend in San Francisco where Waymo has taken over the streets! I saw no major issues except a lot of traffic created from looky-loo's checking out the driverless vehicles. While Waymo is expanding it's robo-taxi service throughout the Bay Area and parts of Los Angeles it is still facing challenges. Last week, a Waymo vehicle re-routed into oncoming traffic in Phoenix 😳 How do you feel about autonomous vehicles and robotaxis? Will they go the way of the horse and buggy or will it be the future of transportation? #mobilitynews #mobilityupdate #mobility #urbanmobility #transportation #waymo #google #alphabet #parking #parkingnews #robotaxi #avs #autonomousvehicles #nodriver #rideshare #technology #future #cars International Parking & Mobility Institute (IPMI) Parking Today Media National Parking Association

17

1 Comment

GEN AI Our generative AI extension connects Altair RapidMiner users with over 450,000 open-source LLMs on Hugging Face. The extension also has access to public versions of the biggest models on the market, like ChatGPT. Pick from pre-trained models for specific tasks or go for the biggest on the block. Altair RapidMiner can do both, and more, without users needing to code. Unlike building generative AI functions from scratch, users can immediately deploy Altair’s extension for all Altair RapidMiner users in the organization – without drawing additional Altair Units or writing a line of code. We’ll manage the tech so you can manage your work. In addition to using models on Hugging Face and ChatGPT out of the box, Altair RapidMiner makes fine-tuning existing LLMs swift and painless. Fine-tuning increases relevance of results and accuracy of outputs, key for enterprise-scale applications. Protect your IP and improve security by running branched copies of models stored locally in Altair® AI Hub™. Machine learning excels at classification, regression analysis, and clustering. Machine learning algorithms are powerful tools, but their outputs are limited. Not so with generative AI. Altair’s generative AI extension can create diverse text-based content types that users can customize for a wide array of tasks. Altair RapidMiner suite users can utilize our tool for AI #snsinstitution #snsdesignthinkers #designthinking

3

2 Comments

As Waymo enters an era of scaling they are making a big show of publishing safety data. But not all is as they want it to seem. Scaling, especially to serve new cities in addition to adding vehicles to the fleet, is where the risk of edge cases to safety is going to be the most challenging. Higher operational speeds (e.g., for airport service) additionally introduce a higher fatality risk to any crash they do have. The safety statements being made by their public and government relations apparatus are misleading. Maybe this will work out, and maybe it won't. But this data is in many ways just a distracting sideshow from the core safety question of how fatalities will turn out. Waymo hopes that will turn out well (and so do I), but it is irresponsible to practice what amounts to transparency theater and make misleading safety statements in the pursuit of fast scale-out. Details here in this week's substack posting:

67

10 Comments

This is #icra2024 Week, and I want to tell you about a very cool paper I spotted from Waymo's research team: Stateful Tracking with Transformers for Autonomous Driving! It's a 3D Multi-Object Tracker... but with Transformers rather than Kalman Filters. It's very "edgy", and I'm doing a complete breakdown in this video. Dragomir Anguelov (Drago) Qinru Li How did I do? Did I miss anything? #deeplearning #computervision #lidar #selfdrivingcars #robotics

119

6 Comments