Minglei Chen

Sharing my thoughts on How Waymo (formerly known as the Google Self-Driving Car Project) is Pioneering Autonomous Driving with AI. Waymo has become a trailblazer in self-driving technology, and at the core of it all is AI. Leveraging advanced machine learning and deep neural networks, Waymo’s AI-powered system continuously improves through real-world data, making autonomous vehicles smarter, safer, and more reliable. Perception: AI allows Waymo’s cars to "see" and understand their surroundings by processing data from sensors like cameras, LIDAR, and radar. This helps detect pedestrians, other vehicles, and objects in real time. Prediction: The AI system predicts the behavior of everyone around the vehicle—whether it’s cars changing lanes or pedestrians crossing the street—ensuring smooth and safe driving decisions. Planning & Control: AI makes split-second decisions about accelerating, braking, and steering based on the analysis of complex traffic patterns, enabling Waymo vehicles to safely navigate roads. With millions of autonomous miles driven, Waymo’s AI is constantly learning and improving, helping create the future of safer and smarter transportation. The road ahead is AI-driven! www.rxsmeta.com #AI #AutonomousVehicles #Waymo #SelfDrivingCars #MachineLearning #SmartMobility #TransportationTech #FutureOfTransport #AIinAction #Innovation

2

Three insights from the TEDAI San Francisco X Factory X AI at Meta CodeGen Hackathon. Had the pleasure of judging a fantastic hackathon in SF yesterday. Here's the biggest things I took away from iteration on the cutting edge of GenAI for code: 1) 𝙂𝙚𝙣𝘼𝙄 𝙘𝙤𝙙𝙞𝙣𝙜 𝙨𝙩𝙖𝙧𝙩𝙚𝙙 𝙬𝙞𝙩𝙝 𝙖𝙘𝙩𝙞𝙫𝙚𝙡𝙮 𝙖𝙨𝙨𝙞𝙨𝙩𝙞𝙣𝙜 𝙙𝙚𝙫𝙨 𝙖𝙩 𝚠̲𝚛̲𝚒̲𝚝̲𝚎̲ 𝙩𝙞𝙢𝙚, 𝙗𝙪𝙩 𝙂𝙚𝙣𝘼𝙄 𝙖𝙜𝙚𝙣𝙩𝙨 𝙣𝙖𝙩𝙪𝙧𝙖𝙡𝙡𝙮 𝙛𝙞𝙩 𝙖𝙩 𝙙𝙞𝙛𝙛 𝚛̲𝚎̲𝚟̲𝚒̲𝚎̲𝚠̲ 𝙩𝙞𝙢𝙚. Iterative agenic workflows can run at review with automated vetting of generations, and only serve edit suggestions shown to improve a tested outcome (i.e. memory efficiency). Bonus is that this can scale with inference time compute to try more or less variations or revision cycles. 2) '𝘾𝙤𝙙𝙚' 𝙖𝙨 𝙖𝙣 𝙖𝙗𝙨𝙩𝙧𝙖𝙘𝙩 𝙘𝙤𝙣𝙘𝙚𝙥𝙩 𝙤𝙥𝙚𝙣𝙨 𝙖 𝙬𝙤𝙧𝙡𝙙 𝙤𝙛 𝙥𝙤𝙨𝙨𝙞𝙗𝙞𝙡𝙞𝙩𝙞𝙚𝙨 𝙛𝙤𝙧 𝙣𝙤𝙣-𝙨𝙤𝙛𝙩𝙬𝙖𝙧𝙚-𝙙𝙚𝙫 𝙚𝙣𝙜𝙞𝙣𝙚𝙚𝙧𝙞𝙣𝙜 𝙙𝙞𝙨𝙘𝙞𝙥𝙡𝙞𝙣𝙚𝙨. Many tools in these areas involve cumbersome UI's that ultimately translate points and clicks into complex config files, and LLM's are pretty good at editing these underlying files directly. 3) 𝘾𝙤𝙙𝙞𝙣𝙜 𝙖𝙗𝙞𝙡𝙞𝙩𝙞𝙚𝙨 𝙞𝙣 𝙢𝙤𝙙𝙚𝙡𝙨 𝙖𝙧𝙚 𝙜𝙚𝙩𝙩𝙞𝙣𝙜 𝙢𝙤𝙧𝙚 𝙘𝙤𝙢𝙥𝙖𝙘𝙩 𝙩𝙝𝙖𝙣 𝙚𝙫𝙚𝙧. One hackathon team was able to use a combination of synthetic data creation/refinement and model distillation (in this case from my near and dear Llama 405B) to fit highly advanced code translation ability in a 14B model. I have a feeling this trend will continue, which will open up more inference compute heavy coding opportunities like broad repo summarization. Bonus here - the resulting model was open sourced by Jason Lu: https://lnkd.in/gswMbCR3 Shout out to SHACK15 and Cerebral Valley for hosting. Incredible views and vibes.

19

3 Comments

I found this post pretty insightful (in terms of qualifying the differences), though I think it misses what the difference actually is When you're interviewing someone for 45 minutes, what kind of data can you get? Can you really conclude this person is going to be a good leader? Or are you simply measuring how well the person can bullshit? Coding & problem solving, on the other hand, cannot be bullshit They are a good predictor & signal of actual real-world performance (at least, how good of a coder & problem solver the person is) Whereas if you interview someone & they "seem to be a good leader", what you're actually probably measuring is that they're a really good bullshitter If I had to assign probabilities: A) Probability that a "leadership interview" leads to a hire which is: A good bullshitter: 90% Competent: 20% Hardworking: 20% A good leader: 20% B) Probability a technical/problem-solving interview leads to a hire which is: Competent: 90% Hardworking: 50% A good leader: 20% A good bullshitter: 20% I'll take (B) any day

Chip Huyen on Twitter today below. I 100% agree. Metric hygiene is system hygiene. LLM-Generated metrics require a higher standard of care than they're currently receiving, and much more scrutiny. Mind the compasses. 🐦  A big issue I see with AI systems is that people aren't spending enough time evaluating their evaluation pipeline. 1. Most teams use more than one metrics (3-7 metrics in general) to evaluate their applications, which is a good practice. However, very few are measuring the correlation between these metrics.If two metrics are perfectly correlated, you probably don't need both of them. If two metrics strongly disagree with each other, either this reveals something important about your system, or your metrics just aren't trustworthy.2. Many (I estimate 60 - 70%?) use AI to evaluate AI responses, with common criteria being conciseness, relevance, coherence, faithfulness, etc. I find AI-as-a-judge very promising, and expect to see more of this approach in the future.AI-as-a-judge scores aren’t deterministic the way classification F1 scores or accuracy are. They depend on the judge's model, the judge's prompt, and the use case. Many AI judges are good, but many are bad.Yet, very few are doing experiments to evaluate their AI judges. Are good responses given better scores? How reproducible the scores are -- if you ask the judge twice, do you get the same score? Is the judge's prompt optimal? Some aren’t even aware of the prompts their applications are using, because they use prompts created by eval tools or by other teams.Also fun fact I learned from a (small) poll yesterday: some teams are spending more money on evaluating responses than on generating responses 🤯 https://lnkd.in/gYnTRZW3

2

My team at Meta (In-Feed Recommendations) is hiring a number of ML engineers in the US We are building one of the most advanced recommendation engine in the industry to power discovery experiences on Facebook newsfeed with a diverse set of public content (Reels, videos, photos, web links, conversations, etc) available on our platform. You can make a big difference by working on a wide range of projects, including but not limited to user/content interests understanding, next-gen recommendation systems and modeling technologies, new social discovery experiences to help create communities of like-minded people. Check out the following links to learn more: The AI Behind Unconnected Content Recommendations on Facebook and Instagram https://lnkd.in/gZ7XV_am Meta Rethinks the Philosophy of the Facebook Feed https://lnkd.in/ggJzk3g3 If interested, please DM me. ==================== Team / Role Summary We are part of the FB Feed Relevance Org, and our team’s mission is to create joyful discoveries and meaningful connections via personalized recommendations on people’s home feed with deep understanding of their interests. We are responsible for building the E2E recommendation system to power all the interest-based public content discoveries across FB Feed. Why is This Impactful IFR is a central product in Facebook’s long term vision, Discovery Engine. Each day we are sending more than 50B posts to our users. From Mark Zuckerberg's Q2 earnings call: "AI-recommended content from accounts you don't follow is now the fastest growing category of content on Facebook's feed. Since introducing these recommendations, they’ve driven a 7% increase in overall time spent on the platform. This improves the experience because you can now discover things you might not have otherwise followed or come across." Your Role As an ML Engineer, you will help build one of the world’s largest recommendation engine, implement state of the art ML algorithms and modeling techniques, directly influence billions of users’ News Feed Experience. You will also learn how to build an ultra large scale Recommender System from end to end and grow your career as a domain expert, and lead our efforts to build emerging new experiences on the Newsfeed Recommendations platform.

64

1 Comment

Come join us tomorrow in SF to learn more about the latest updates to run AI workloads on Google Cloud!

11

1 Comment

Xian Ke, this is incredible! The speed at which you and Vishal Punjabi leveraged crewAI, Streamlit, and various LLM models to create Finbot, a powerful company analysis tool for investors, was mind-blowing. ICYMI, Finbot is here: https://lnkd.in/eSQD69qA I couldn't help but chuckle when I saw the Finbot name - it reminded me of the Wall Street Bets crowd. Maybe you can get them to yolo their life savings based on Finbot's analysis? (Kidding, of course. Or am I? 💎🙌) But in all seriousness, your enthusiasm and passion for AI and LLMs is contagious. It's engineers like you who are pushing the boundaries of what's possible with AI and paving the way for a brighter future.🏆 I'd love to extend an invitation to any other software engineers from the bootcamp to join our Devs Do AI Discord server. It's a great place for engineers without an ML / AI background to connect with like-minded devs, share knowledge, and collaborate on LLM projects. DM me for the invite link. 🤝

13

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

I can't believe the key takeaway I had for middle and high school girl students this weekend in a programming camp was something even expert software engineers forget sometimes: "software engineering is much more than coding". I had the chance to talk to these young bright minds at Santa Clara university. I explained, hopefully in an intuitive way, how humans coordinate at large scale to build products like YouTube. And mainly I wanted to bust the myth that software engineering is a job where you code heads down all day and don't need to talk to people. Three main qualities I highlighted in a good engineer: - knows their users - team player - problem solver That made me think though. The meetings and all those office hours, and 1:1s spent "socializing", while on the surface seem like unproductive hours, they're indeed my job now. And that's true of anybody who's growing up the ladder in organizations. Human coordination is a beautiful problem :)

139

3 Comments

This is quite a change - a major one, for the C++ community. Chandler has been responsible for the direction of C++ at Google for many years. He's a very smart engineer who has been responsible for moving the ball forward in a lot of very impressive ways. He's the kind of person who doesn't take "that's impossible" for an answer - and is usually right. Way back in the 2000s, C++ engineers developed what could snidely be called "Java envy". They saw their compatriots with IDEs that could find the definition for a call and refactor code automatically, and runtimes that could guarantee memory safety, and rich support for multithreading, and various other things that made them productive. They wanted those things, too! The primary reason that C++ engineers swore by C++, was, of course, performance. The average, mediocre Java program could typically get similar performance to the average, mediocre C++ program compiled with -O2, give-or-take GC and JIT costs. But it is harder to squeeze the last ounce of performance out of Java, and C++ engineers were convinced that there would be huge performance costs if they switched. So, instead of using Java, they started replicating Java's productivity features as best as they could for C++. And they've done a great job! They have IDEs that more-or-less work most of the time, they have heavyweight sanitizers that can enforce memory safety in test code, they've brought threading into the language. All in all, they did about as much as you could imagine to try to bring some of the improvements in tooling and runtime support to a language that doesn't accommodate them easily. But there was always a kind of "final straw" - compilers and runtimes wouldn't compromise production performance for safety. One tradeoff Java makes is range checking - it checks to make sure that you don't access memory outside the bounds of an array. Now, the fact is, range check elimination support in Java has been able to remove 95+% of range checks for decades. The overall cost of range checking in a modern language with good compiler support is negligible. Even so, for years, it was kept out of C++, for reasons that Chandler describes below. C and C++ developers didn't write range checks, so there was no reason to optimize them. And, without experience, they didn't believe those checks could be cheap by default. But, in the mean time, other languages started to use llvm. Other languages that *had* range checking. So support got better. And the security issues from out of bounds accesses became worse. So the C++ community at Google tried it out. And the cost? 0.3% cost across Google's services for bounds checks in the standard library types - a cost that's likely to come down over time. My hope is that we will see a lot more dynamic safety checks in production C++ code, and that these kinds of efforts will see near-universal adoption.

44

What does it feel like to be an engineering manager at Amazon: Since the RTO announcement it became popular to write shit about Amazon, but it's still one of the best companies around the world. A few weeks ago, Gilad Naor shared with the 'Leading Developers' readers 3 lessons from his time as an engineering manager at Amazon. Here is the first one, read the full article here: https://lnkd.in/dHBxv2Ha 𝟭. 𝗦𝗼𝗹𝘃𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺𝘀 𝘄𝗶𝘁𝗵 𝗺𝗲𝗰𝗵𝗮𝗻𝗶𝘀𝗺𝘀 My first meeting at Amazon was baffling. I entered a room with about a dozen Software Development Managers (SDMs) and Product Managers (PMs). Everyone sat in silence and looked at their laptops or the printed papers they had before them. It was my second day at Amazon, and I picked up one of the extra sheets and started reading. After those first ten awkward minutes, everyone finished reading and the meeting began. Why do Amazon meetings always start this way? This is a specific example of how Amazon approaches problem-solving. It is the "Good Intentions Don't Matter" approach. In a famous internal video, Jeff Bezos talks about why good intentions are irrelevant. When things don't work, it does not matter what people wanted. All that matters is fixing the problems with mechanisms. For example, many busy managers come to meetings without first reading the prep material. They all intended to read the 6-Pager before the meeting. They just got sidetracked by the latest emergency. Scolding the managers won't solve the problem. A mechanism will. This is why all Amazon meetings start with a study period. The remaining meeting time is then spent on high-quality decision-making. Meta solves the same problem by creating cultural norms. I can't count the number of times that I heard a senior leader cut off the presenter: "Stop, assume that everyone here already read the slides. Let's get to the point." This is a culture-first approach. I have seen this work well in some teams, and not work at all in other teams. At Amazon, the solution is part of the company's "operating system". ------------------------- Anton here - by now everyone is familiar with the 6-pager concept, but that "operating system" thinking can be used almost anywhere. Think about something annoying that your developers do, and how can you solve it. For example, I remember being annoyed that people don’t update the Jira tickets. Finally, after requesting multiple times, now we just take 20 seconds from the standup to update them.

56

8 Comments

The future is here

5

I first joined Amazon in December of 2005. Even though I left for two years, my "Amaversary" still coincides with the end of the year and a natural moment for reflection. As I reached my 17 year mark this December, I also had the privilege of working with Amazon Careers to share some of my journey and thoughts about individual contributor technical leader and people manager career paths. In my ruminating, I put together far more content than could be used for a single article. So, I've decided to post here as a series of Q&As - in case others might find my observations, experiences, and perspectives interesting or useful. Below is the first of 6. 1. 𝘽𝙧𝙞𝙚𝙛𝙡𝙮 𝙨𝙝𝙖𝙧𝙚 𝙮𝙤𝙪𝙧 𝙘𝙖𝙧𝙚𝙚𝙧 𝙟𝙤𝙪𝙧𝙣𝙚𝙮; 𝙬𝙝𝙖𝙩 𝙢𝙤𝙩𝙞𝙫𝙖𝙩𝙚𝙙 𝙮𝙤𝙪 𝙩𝙤 𝙩𝙖𝙠𝙚 𝙤𝙣 𝙙𝙞𝙛𝙛𝙚𝙧𝙞𝙣𝙜 𝙘𝙖𝙧𝙚𝙚𝙧 𝙥𝙖𝙩𝙝𝙨, 𝙄𝘾 𝙫𝙨 𝙢𝙖𝙣𝙖𝙜𝙚𝙧? I started my career as a software engineer 25 years ago. I've worked for startups, enterprise companies, and 7 different organizations over 17 years and two tours at Amazon. I've written firmware for robotic camera mounts; built highly scalable distributed systems; designed restaurant marketing, ordering, logistics, and operations software; created automated test harnesses for mobile games; tested HD DVD encoding; helped launch grocery delivery to customers around the globe - just to name a few of the many cool things I've had the privilege to work on. I've been an entry-level software engineer, a technical lead of teams, a software manager on a couple of occasions, chief software architect reporting to a CTO, and (for the last few years) Amazon Principal Engineer directly supporting a VP with about 200 engineers within my remit. I first became a manager because I mistakenly thought it was the only way to continue growing as a leader, and in many ways I failed. I learned some very humbling lessons about leadership through that experience, which I used to be a better individual contributor technical leader on my path to Senior Engineer. My second time as manager was because my organization had a need that I could fill. I had grown enough as a leader that this time I was much more successful. I said yes to an opportunity that started with managing a small team through delivery of a complex and critical project. Once I grew a high performing team that delivered that initial project, I took on more scope and expanded into multiple teams. After a couple of years of success growing and investing in my teams so that we could deliver greater value to our customers and business, I decided to prioritize being close to my family. For me, at that time, it meant giving up my people management role and returning to the role of an individual contributor technical leader that supports a globally distributed organization. Even though I really enjoyed my time as a people manager, there is no better job for me than that of a Principal Engineer. #amazon #careerdevelopment #technicalleadership #peoplemanagement

91

3 Comments

One of the best things about being in San Francisco 🌉 is the ability to drop in on great talks. Last week I had a chance to see NVIDIA's CEO Jensen Huang (JH) being interviewed by Stripe's CEO Patrick Collison. Jensen is among the longest-serving founder-CEOs in Silicon Valley at the moment (30+ years). With that comes a special clarity of mind and some hot takes. 🌶👇 🗣 Jensen doesn't believe in 'praise in public and criticism in private'. Instead, he reasons that most information in the organization should not be privy to certain people. By giving feedback in public, you allow everyone to learn. 'What is better than learning from your own mistakes? It's learning from someone else's mistakes!!' 🔥 JH also doesn't like to fire people. He would rather "torture them into greatness". He invests in people and he doesn't like to write off his investments. 👨‍💻 Asked whether he likes his job every day, he answers "no". The perfect job doesn't give you happiness 100% of the time. "No great things were done, that were easy all the way." ... "Even if I come home from a day where I did things I didn't enjoy, it could still be have been a great day." 👩‍👧‍👦 Jensen famously has 60 direct reports. "This allowed us to cut out probably 7 levels of management". His subordinates don't necessarily have such large teams. "It doesn't scale down, less senior people need more guidance". Looking at the numbers, this approach allowed NVidia to reach ~2T market cap with ~28.000 people vs. Microsoft's 3T with ~220.000 people and Apple's 2.6T with 160.000 people. ❔ Jensen answers almost every question in the same way: "Let's break it down, let's reason about it". This is a great response on so many levels. First, it allows him to not be put on the spot. Second, it allows him to teach his workforce how he reasons through a problem so that they can emulate it and create alignment. Third, it gives people a chance to point out flaws in his reasoning. He admits he is not right all the time. ON AI 👨‍🔧 "You will not lose your job to AI, your company will go out of business because another company used AI." "If you're not using AI you're doing it wrong." ⏱ The extent to which AI will enter many processes is largely underestimated. Right now we are focused on language - sounds in a time sequence. "Any time sequence can be tokenized." This will include robotic movement. 🥛 JH also expects that we won't be satisfied with foundation models "You're going to want to tune it into perfection because you care so deeply. Say if it's already at 99% you'll want to get it to say 99.3%" This last piece is also why we created Query Vary (YC W22). AI will never feel truly intelligent until it completely understands your data and context. It needs to be taught, regardless of how powerful the foundation model is. Finally, besides insights, it was remarkable how funny Jensen was, cracking a joke at roughly a 1 per 3 min rate. It was a very entertaining talk. #ai #sf

61

8 Comments

Should you move to Silicon Valley as an engineer? Here's the perspective I gained after 20+ years in the Bay Area that I wish I could tell my younger self: When I moved to the Bay Area in the late '90s, it was amazing. Great weather, lots of opportunities, and a community full of passionate tech people. It was a great decision for me. But for you? I don’t think there’s a simple Yes or No. Because here’s the thing… It’s not just about the allure of the location. It’s not about the weather, or the opportunities. Yes, they’re factors to consider, but what matters most is you. What helped me decide: “What’s this location like on a bad day? Am I okay with that?” Smaller cities have their appeal – lower cost of living, being close to family and friends. ...But on a bad day, a smaller city is not a tech hub. And I’m a tech guy! If I had to look for a new tech job in a smaller city on a bad day, it’d be tough. But in Silicon valley I can throw a stone and find 10 companies. That’s where I want to be. Living in an innovation hub matters to me, but it might not make sense for you. It’s on you to self-reflect and find answers for yourself. Journal, and ask yourself: • What do I want? • Which roles do I want to try out? • What are my goals for the next three years? The perspective you gain will help you choose.

26

3 Comments

Recently, there has been much discussion about Nvidia's CEO's approach of not having 1:1 meetings while leading 60 direct reports, emphasizing the benefits of a flat organizational structure and portraying 1:1s as unnecessary. Like with many trends, the focus has been primarily on these two aspects. Leaders have attempted to replicate this model by increasing the number of direct reports per manager or even eliminating 1:1s altogether. It's refreshing to learn about another side of the story that many are trying to emulate, one where valuing employees and placing them at the center of the organization is crucial for the company's success.

14