Xianying (Steven) Liu

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

Three insights from the TEDAI San Francisco X Factory X AI at Meta CodeGen Hackathon. Had the pleasure of judging a fantastic hackathon in SF yesterday. Here's the biggest things I took away from iteration on the cutting edge of GenAI for code: 1) 𝙂𝙚𝙣𝘼𝙄 𝙘𝙤𝙙𝙞𝙣𝙜 𝙨𝙩𝙖𝙧𝙩𝙚𝙙 𝙬𝙞𝙩𝙝 𝙖𝙘𝙩𝙞𝙫𝙚𝙡𝙮 𝙖𝙨𝙨𝙞𝙨𝙩𝙞𝙣𝙜 𝙙𝙚𝙫𝙨 𝙖𝙩 𝚠̲𝚛̲𝚒̲𝚝̲𝚎̲ 𝙩𝙞𝙢𝙚, 𝙗𝙪𝙩 𝙂𝙚𝙣𝘼𝙄 𝙖𝙜𝙚𝙣𝙩𝙨 𝙣𝙖𝙩𝙪𝙧𝙖𝙡𝙡𝙮 𝙛𝙞𝙩 𝙖𝙩 𝙙𝙞𝙛𝙛 𝚛̲𝚎̲𝚟̲𝚒̲𝚎̲𝚠̲ 𝙩𝙞𝙢𝙚. Iterative agenic workflows can run at review with automated vetting of generations, and only serve edit suggestions shown to improve a tested outcome (i.e. memory efficiency). Bonus is that this can scale with inference time compute to try more or less variations or revision cycles. 2) '𝘾𝙤𝙙𝙚' 𝙖𝙨 𝙖𝙣 𝙖𝙗𝙨𝙩𝙧𝙖𝙘𝙩 𝙘𝙤𝙣𝙘𝙚𝙥𝙩 𝙤𝙥𝙚𝙣𝙨 𝙖 𝙬𝙤𝙧𝙡𝙙 𝙤𝙛 𝙥𝙤𝙨𝙨𝙞𝙗𝙞𝙡𝙞𝙩𝙞𝙚𝙨 𝙛𝙤𝙧 𝙣𝙤𝙣-𝙨𝙤𝙛𝙩𝙬𝙖𝙧𝙚-𝙙𝙚𝙫 𝙚𝙣𝙜𝙞𝙣𝙚𝙚𝙧𝙞𝙣𝙜 𝙙𝙞𝙨𝙘𝙞𝙥𝙡𝙞𝙣𝙚𝙨. Many tools in these areas involve cumbersome UI's that ultimately translate points and clicks into complex config files, and LLM's are pretty good at editing these underlying files directly. 3) 𝘾𝙤𝙙𝙞𝙣𝙜 𝙖𝙗𝙞𝙡𝙞𝙩𝙞𝙚𝙨 𝙞𝙣 𝙢𝙤𝙙𝙚𝙡𝙨 𝙖𝙧𝙚 𝙜𝙚𝙩𝙩𝙞𝙣𝙜 𝙢𝙤𝙧𝙚 𝙘𝙤𝙢𝙥𝙖𝙘𝙩 𝙩𝙝𝙖𝙣 𝙚𝙫𝙚𝙧. One hackathon team was able to use a combination of synthetic data creation/refinement and model distillation (in this case from my near and dear Llama 405B) to fit highly advanced code translation ability in a 14B model. I have a feeling this trend will continue, which will open up more inference compute heavy coding opportunities like broad repo summarization. Bonus here - the resulting model was open sourced by Jason Lu: https://lnkd.in/gswMbCR3 Shout out to SHACK15 and Cerebral Valley for hosting. Incredible views and vibes.

19

3 Comments

How you can make your LLM applications better!

5

Today I'm sharing this 🔥 video from our recent Engineering onsite at Demandbase HQ in SF! https://lnkd.in/eFAFR6QG We were so fortunate to host Zhengxuan Wu and Aryaman Arora, two upcoming AI researchers from Stanford University Department of Computer Science. Their approach to fine tuning, called ReFT, targets the representations emitted by neural net layers rather than the weights of the layers themselves. This makes it very flexible when using the same base model for swapping multiple fine tunes. The memory overhead is also very low for their intervention weights. Another highlight is how powerful it can be. One POC they showed forced the LLM to speak in only emojis exhibiting ReFT's power to control output. Control is crucial when, for example, you want to prevent your bot from offering customers unauthorized refunds!! If you're already familiar with LoRA, it should be pretty simple to use with only a few extra hyperparameters -- details in the video. Here's a link to the repo. Give them a star! https://lnkd.in/eJScNbzK Thanks again for letting us host your first external talk Aryaman and Zen. It's very exciting tech!!

17

4 Comments

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

16

3 Comments

LLMs and LangChain: The Dynamic Duo 🦸‍♀️🤖 Just finished a fun project where we built a Q&A bot using these two powerhouses. It's like having a personal AI assistant that's always ready to answer your burning questions! 🔥 Here's what I learned: LLMs are wizards: They can conjure up answers from thin air (or, uh, the internet). 🧙‍♂️ LangChain is the glue: It holds everything together and makes it easy to use. 🔗 Endless possibilities: This tech can be used for anything from customer support to writing catchy social media posts. 📝 Want to see it live (obvio a test server link 😛 ): https://lnkd.in/dZNAVCSV Note: it's a render link, they will take down the instance because of no use, you can hit the link and wait for two minutes, they will spin up the instance and you can try it out. Want to see the code? Check it out here: https://lnkd.in/dzjh4r47 #AI #MachineLearning #NLP #LangChain #LLM #Python #Developer #FunProject"

22

In this episode, we discuss Learning Task Decomposition to Assist Humans in Competitive Programming by Jiaxin Wen, Ruiqi Zhong, Pei Ke, Zhihong Shao, Hongning Wang, Minlie Huang. The paper presents a method to enhance human understanding and repair of language model (LM)-generated solutions by automatically breaking down complex solutions into simpler subtasks. They introduce a novel objective called assistive value (AssistV) to measure how easily humans can repair these subtasks and validate their method through a dataset of human repair experiences. The approach significantly improves the problem-solving ability and speed of non-experts in competitive programming, allowing them to solve more problems and match the performance of unassisted experts.

2

Chip Huyen on Twitter today below. I 100% agree. Metric hygiene is system hygiene. LLM-Generated metrics require a higher standard of care than they're currently receiving, and much more scrutiny. Mind the compasses. 🐦  A big issue I see with AI systems is that people aren't spending enough time evaluating their evaluation pipeline. 1. Most teams use more than one metrics (3-7 metrics in general) to evaluate their applications, which is a good practice. However, very few are measuring the correlation between these metrics.If two metrics are perfectly correlated, you probably don't need both of them. If two metrics strongly disagree with each other, either this reveals something important about your system, or your metrics just aren't trustworthy.2. Many (I estimate 60 - 70%?) use AI to evaluate AI responses, with common criteria being conciseness, relevance, coherence, faithfulness, etc. I find AI-as-a-judge very promising, and expect to see more of this approach in the future.AI-as-a-judge scores aren’t deterministic the way classification F1 scores or accuracy are. They depend on the judge's model, the judge's prompt, and the use case. Many AI judges are good, but many are bad.Yet, very few are doing experiments to evaluate their AI judges. Are good responses given better scores? How reproducible the scores are -- if you ask the judge twice, do you get the same score? Is the judge's prompt optimal? Some aren’t even aware of the prompts their applications are using, because they use prompts created by eval tools or by other teams.Also fun fact I learned from a (small) poll yesterday: some teams are spending more money on evaluating responses than on generating responses 🤯 https://lnkd.in/gYnTRZW3

2

You want to do Machine Learning on large scale Marketing solutions ? Check below 🚀

3

🎉 Big congratulations to our incredible team (Haoyu Chen, Nick Greenquist,Yuxing Wang, Sridevi Maharaj, Ruiyang Wu and many more) for delivering several quarters-long impactful features on the Play Store! 🚀 We’re thrilled to have launched key updates like Search Home, Sports Moment, and WearOS recommendations, all designed to enhance personalization and help billions of users worldwide search and discover content they love. 🌍✨ Check out the Play Store Q3 updates: https://lnkd.in/dn5wx8Mi #GooglePlay #Personalization #Recommendation

74

1 Comment

📢 DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads🚀 In this work, they tackle the significant memory and computational challenges associated with deploying long-context large language models (LLMs). The proposed framework, DuoAttention, introduces a novel distinction between Retrieval Heads and Streaming Heads to optimize the use of the KV cache, reducing memory usage by up to 2.55× and improving decoding speeds by up to 2.18×, all while maintaining high accuracy. Technical Highlights: 1. Retrieval Heads vs. Streaming Heads: DuoAttention divides attention heads into two categories. Retrieval Heads are crucial for processing long contexts, whereas Streaming Heads focus on recent tokens, allowing for a more efficient use of memory resources. This division helps in reducing the computational load without compromising model accuracy. 2. KV Cache Optimizatio: The framework uses selective caching strategies, applying full KV cache to Retrieval Heads while maintaining a constant-length cache for Streaming Heads. This results in significant reductions in both memory usage and latency during long-context inference. 3. Scalable Long-Context Inference: With DuoAttention, models like Llama-3-8B can handle up to 3.3 million tokens on a single A100 GPU. This is made possible through careful optimization of memory allocation and the use of lightweight algorithms to identify the critical heads. 4. Performance Improvements: DuoAttention achieves up to 2.55× memory reduction for multi-head attention (MHA) models and up to 1.67× for grouped-query attention (GQA) models. It also speeds up decoding by up to 2.18× for MHA models and up to 1.50× for GQA models, providing significant benefits for real-world applications involving large language models. 5. Compatibility with Quantization: The method is fully compatible with quantization techniques, such as 8-bit weight and 4-bit KV cache quantization. This compatibility further boosts efficiency, enabling the model to handle larger contexts with reduced hardware requirements. With DuoAttention, the team has achieved long-context capabilities for models like Llama-3-8B, enabling handling of up to 3.3 million tokens on a single A100 GPU. This advancement marks a significant step towards more efficient, scalable LLM deployments! 🔍 Paper: https://lnkd.in/dDFHimSA #LLMs #MachineLearning # #EfficientInference

13

Happy holidays! We are kicking off 2025 with the next AI/ML infra meetup in San Francisco, co-hosted by Alluxio and Anyscale, at Anyscale's office on Thu, Jan 23. 🌟 Robert Nishihara, Co-Founder of Anyscale, will share invaluable insights on mastering AI deployment with Ray. Learn how to overcome the last-mile challenges in bringing AI models to production and discover best practices for scaling your AI infrastructure. 🌟 Prof. Junchen Jiang from University of Chicago will reveal cutting-edge strategies for optimizing LLM inference. Get practical insights on building faster, more cost-efficient inference stacks that can handle production workloads. 🌟 Bin Fan, VP of Technology at Alluxio, will dive deep into the critical balance of cost, performance, and scale when running GPU/CPU workloads in the cloud. Learn how leading organizations are optimizing their AI infrastructure for maximum efficiency. Register now to secure your spot! https://lu.ma/swc2imah #AIInfrastructure #MachineLearning #TechMeetup #SanFrancisco #AI #Ray #Pytorch

33

A curated collection of the papers and models consisting of 5 selections across 10 categories... The 2025 AI Engineer Reading List (January 2025) "We picked 50 paper/models/blogs across 10 fields in AI Eng: LLMs, Benchmarks, Prompting, RAG, Agents, CodeGen, Vision, Voice, Diffusion, Finetuning... The picks from all the speakers in our Best of 2024 series catches you up for 2024, but since we wrote about running Paper Clubs, we’ve been asked many times for a reading list to recommend for those starting from scratch at work or with friends. We started with the 2023 a16z Canon, but it needs a 2025 update and a practical focus. Here we curate “required reads” for the AI engineer. Our design goals are: - Pick 50 papers , optional extras. - Tell you why this paper matters instead of just name drop - Be very practical for the AI Engineer We ended up picking 5 “papers” per section for: Section 1: Frontier LLMs Section 2: Benchmarks and Evals Section 3: Prompting, ICL & Chain of Thought Section 4: Retrieval Augmented Generation Section 5: Agents Section 6: Code Generation Section 7: Vision Section 8: Voice Section 9: Image/Video Diffusion Section 10: Finetuning https://lnkd.in/erxnwUpB

3

Thanks to Youngjin Kim and Moweon Lee, two experienced Naver engineers with years of expertise in big data analytics, for sharing how they use StarRocks in complex real-time data analytics scenarios. Youngjin highlighted the reasons why Naver chose StarRocks: • 🚀Strong JOIN Performance • 🚀Smart Cache Strategy • 🚀Cloud Native Storage • 🚀Analytics with Flexibility • 🚀Materialized View To learn more, check out this impressive video: https://lnkd.in/gHu-WZND

7

I recently had the opportunity to attend 3 major AI events in the last 30 days, where I gained valuable insights into the future of artificial intelligence. Here are some key takeaways: Amazon AI Day: - Amazon CTO (Werner Vogels) highlighted the potential of generative AI and large language models in driving innovation and societal impact, particularly in healthcare and sustainability. - Explored how generative AI is revolutionizing development processes and transforming businesses across industries. MagicBall - AI Week: - Showcased advancements in AI infrastructure by over 24 startups from Bengaluru, emphasizing the city's significance in AI development. - Engaged in discussions on responsible AI development and ethical considerations for the future. Microsoft AI Day: - Microsoft's Innovation Hub India CTO (Sandeep Alur) discussed AI's critical role in digital transformation and the company's commitment to ethical AI development. - Showcased the revolutionary potential of Generative AI and its integration into Microsoft Azure. These events, combined with insights from the ISB CTO program, emphasize that AI is here to create new opportunities and possibilities, not to take away jobs. Embracing a growth mindset is crucial to thrive in the evolving AI landscape. The AI revolution has just begun, and it's essential to adapt and innovate for a sustainable future. #GenAI #AI #BrightFuture

29

6 Comments

🎯 How teams are cutting Machine Learning Engineer hiring time from 90+ days to 28 days: 1. Role-specific technical assessments that mirror real projects • Real-world problem-solving instead of whiteboard exercises • Actual code reviews from your codebase • System design challenges based on your architecture • Practical ML model optimization tasks 2. Direct access to tech leads in first interview • No HR screening delays • Technical discussion from day one • Immediate architecture alignment • Quick technical fit assessment 3. 48-hour feedback commitment from all parties • Same-day interview feedback • Next-steps within 24 hours • Offer decisions in 48 hours • Clear communication timeline 4. Streamlined technical evaluation process • One comprehensive technical round • Parallel evaluations when possible • Combined culture/technical assessments • Quick technical team consensus Teams implementing even one of these cut hiring time by 40%. "Especially effective for fast-growing startups and scale-ups" #TechHiring #MLEngineering #TechRecruitment

3

It’s time for Social Saturday #1 (Stories Beyond Achievements) 🌟 Meet Diksha Gohlyan, a Senior Software Engineer at NVIDIA and Technical Interview Coach at Interview Kickstart, and a former software engineer at Google. 🌟 What inspired you to pursue a career in tech? -> I always wanted to be an engineer growing up, the definition of engineering being building homes. As I grew older, software engineering became a luring field and my interest in computers and exposure to its power led me to pursuing a career in tech. 🤔 How would you describe a typical day at your job? -> A typical day for me would be a few meetings with my team and other stakeholders, a lot of hands-on programming solving problems and resolving bugs, and great discussions with teammates over lunch on any topic really. 🚀 What did you want to be when you were a kid? -> Engineer - Not sure if it was my parents who instilled that on me or the ambition in me wanted to do something different compared to what my family did, but I was clear I wanted to be an engineer, what kind, still unsure. 🌈 Share your favorite childhood memory.  -> This aces all the fun I had with my siblings. We were throwing balloons from our terrace during Holi, and it hit an aunty. She was furious and started ascending up the stairs. My sisters, who were older, jumped over the fence to the other side to hide, but I couldn’t reach and had nowhere to go but to descend down the stairs. The lady caught me and thrashed me. Even with the thrashing its still my favorite moment because it reminds me of all the fun we used to do as kids. 🤔 Your friends describe you as…  -> Hard working, always in a hurry, and sometimes helpful. 🕰️ Describe yourself when you were 22 years old. 🏆 An accomplishment you're particularly proud of? 😓 Reflect on a moment or time when you felt like you were struggling a lot.  🎓 What's the biggest lesson you've learned in your career so far?  ⚖️ How do you balance work and personal life?  🎲 A random fact about yourself. 🎬 What is your favorite movie or series of all time? 💬 A favorite quote or mantra that guides you in your career.  💼 What non-technical skills do you believe are crucial for success in tech? 🚀 A message to young people looking to pursue their career in tech.      Edit: Swipe through the images below to find out more! 👇📸 🙌 A shoutout to your mentor during your early career.  -> Devesh - My ex-manager from Google - I am still in touch with him. Connect with Diksha di by expressing what inspired you most about her journey. And if you're eager for more stories, stay connected! Sharing experiences is how we all grow! 🌟 #YourNewConnection #SocialSaturday #StoriesBeyondAccomplishments #TechInNepal #TechInspiration #NepaleseInTech #WomenInTech

96