William A. Gaviria Rojas

TRIPLE BAM! If you missed it, here's the recording of our live Q/A with Jay Alammar and Joshua Starmer PhD! We talked about many AI topics, answered lots of great questions, and talked about two great books: Hands-On Large Language Models: Language Understanding and Generation, and The Illustrated Guide to Neural Networks and AI!!! https://lnkd.in/dcPHs7YQ

44

1 Comment

The newest issue of The Pragmatic Engineer found a surprise uptick in eng hiring in the typically slow summer season - I'm curious to see how the macro environment impacts this uptick as we head into the fall, but I remain hopeful. I asked Gemini to give me the tldr of the article to share with my team (sorry, Gergely Orosz) and found it missed one of the most interesting pieces of the article. Gemini's highlights - Unexpected surge: Tech recruitment has seen an unusual spike in June and July, contrary to typical trends. Increased demand: Startups, especially AI-focused ones, are driving the hiring uptick. Global phenomenon: This trend isn't just limited to the US; Europe is also experiencing increased recruitment activity. Reasons for hiring: Companies are backfilling positions, investing in growth due to improved financials, and responding to increased VC funding. Challenges remain: While the job market is improving, finding qualified candidates and offering competitive compensation remains difficult for many companies. What I found most interesting - The nature of the backfills: the article finds that a lot of new headcount are actually backfills of folks that were "over-slashed" during rounds of layoffs. Not entirely surprising with the "trim the fat" and "rebuild lean" mentality, but interesting nonetheless. Either way, the uptick is encouraging (and don't always rely on Gemini's tldrs). #hiring #startups #VCbacked #softwareengineering

11

1 Comment

This is a good article on using LLMs in the real world. It covers many aspects that we learned at Nurish over the last few months and more. Using LLMs is an inexact science, but has its own needs in processes and tools. While most of the attention has been on the LLMS, their capabilities, and RAG, there are mundane issues that need to be taken care of, issues where the solutions for pre-LLM development are not that well-suited. These include version management, integration, testing, data management, quality assessment, and more. This article covers many of those practical aspects. Eugene Yan, Bryan Bischof, Charles Frye & more https://applied-llms.org/

9

Much has been said about many companies’ desire for more compute (and data) to train large foundation models. I think it’s under-appreciated that we also have nowhere near enough compute available for inference on foundation models. Years ago, when I was leading teams at Google, Baidu, and Stanford that focused on scaling up deep learning, many semiconductor makers, data center operators, and researchers asked me if AI would continue to make good use of more compute if they kept delivering it. For many desktop workloads, like running a web browser, a faster CPU doesn’t help much beyond a certain point. So do we really need faster and faster AI processors? Each time, I confidently replied “yes!” and encouraged them to keep scaling up compute. (Sometimes, I added half-jokingly that I had never met a machine learning engineer who felt like they had enough compute. 😀) Fortunately, this prediction has been right so far. However, beyond training, we are also far from exhausting the benefits of faster and higher volumes of inference. Today, a lot of LLM output is for human consumption. A human might read around 250 words per minute, which is around 6 tokens per second (250 words/min / (0.75 words/token) / (60 secs/min)). So it might seem there’s little value to generating tokens much faster than this. But in an agentic workflow, an LLM might be prompted repeatedly to reflect on and improve its output, use tools, plan and execute multiple steps, or implement multiple agents that collaborate. So, we might generate hundreds of thousands of tokens or more before showing any output to a user. This makes fast token generation very desirable and makes slower generation a bottleneck to taking better advantage of existing models. That’s why I’m excited about the work of companies like Groq, which can generate hundreds of tokens per second. Recently, SambaNova also showed it can hit hundreds of tokens per second. Incidentally, faster, cheaper token generation will also help make running evaluations (evals), which can be slow and expensive today since it involves iterating over many examples, more palatable. Fortunately, both training and inference are rapidly becoming cheaper. I spoke with Cathie Wood and Charles Roberts of the investment firm ARK, which is famous for its bullish predictions on tech. They estimate that AI training costs are falling 75% a year. If they are right, a foundation model that costs $100M to train this year might cost $25M to train next year. Further, they report that for “enterprise scale use cases, inference costs seem to be falling at an annual rate of ~86%, even faster than training costs.” I don’t know how accurate these specific predictions will turn out to be, but with progress in both semiconductors and algorithms, I do see training and inference costs falling rapidly. This will be good for application builders and help AI agentic workflows lift off! [Original text: https://lnkd.in/dJ9tVGh7 ]

2,895

80 Comments

If you think compute scaling will plateau, think again. We are in the baby states of inference time compute. VLLMs coupled with computer tools, reward learning and RL can leverage orders of magnitude more inference time compute than what we have seen with foundation models. Here’s a concrete training architecture (replace \pi with a large pre-trained vLLM and add it to the reward learner too, \R is the space of all computer tools)

22

1 Comment

Ilya Sutskever confirmed at NeurIPS '24 pre-training scaling law will not be the same. Internet data, the AI fossil fuel, is running out. It is not the end of the AI hype, the rapid rise of foundational AI has outpaced our ability to fully grasp their long-term societal impact. AI agents and applications are poised to transform every facet of our lives, and we will see it happening in coming few years. There will still be improvements in model architecture, and synthetic data creation, so models will improve, just not at the lightning speed they did till now. Looking forward, the single most important milestone in AI is creating systems capable of advancing themselves—AI researchers that can innovate and improve foundational models autonomously. We need to assess how far are we from this stage.

9

What an end to year 2024, OpenAI o3 has achieved state of the art results on ARC-AGI benchmarks. From ARC-AGI's blog (https://lnkd.in/gKw3bU4U): "To sum up – o3 represents a significant leap forward. Its performance on ARC-AGI highlights a genuine breakthrough in adaptability and generalization, in a way that no other benchmark could have made as explicit. o3 fixes the fundamental limitation of the LLM paradigm – the inability to recombine knowledge at test time – and it does so via a form of LLM-guided natural language program search. This is not just incremental progress; it is new territory, and it demands serious scientific attention." This is indeed a significant leap and in 2025, I expect few things: * OpenAI, Google and others will keep pushing the boundaries with what we can do with reasoning models, models will keep getting better - we have not yet reached the ceiling. * We will see openweights models as well which will be comparable to o3 (we already have few going in that direction like QwQ, Deepseek - both from China). * Products will make an accelerated effort for catching up both of these: https://lnkd.in/gRBuFMGi

45

LLMs are designed to make stuff up. In most applications this an acceptable risk, after all, people make stuff up too. But in regulated, risk averse industries (finance, healthcare, insurance, legal) even a small mistake can cause big problems. Manual evaluation and monitoring is a crucial yet often overlooked component to catching errors and mitigating risk. Automated oversight can help, but there's no such thing as "Goodbye Hallucinations" unless humans are firmly in the loop.

10

1 Comment

"The 30-60-90 day plans that people made are out the window and now they are more like 3-6-9 day plans." Ali Arsanjani, PhD shared this perspective on the pace of AI adoption inside enterprises. Dr.Ali is the Director of #AppliedAI Engineering at Google, where he is the Head of the Google AI/ML Center of Excellence and has a distinguished career as CTO, Worldwide Tech Leader across IBM, Amazon Web Services (AWS), Founder of DeepContext, Inc., and Visiting Professor at San Jose State University and Stanford University. I had an enlightening time picking his brains and gaining valuable insights on the following topics. Do check out our conversation in the latest episode of the AI Monetisation Podcast - http://bit.ly/4eHpkFj Also, Dr.Ali's excellent blog on Enterprise AI readiness is here - https://bit.ly/4dKQmtZ ✅ Enterprise AI Maturity spectrum ✅ Internal triggers to advance an enterprise across the AI maturity cycle ✅ AI reference architecture and lifecycle patterns for CXO's ✅ The rise of AI integrators ✅ Challenges in moving AI projects from PoC to production ✅ Ethical and responsible AI ✅ Many-shot in context learning ✅ The future of AI #GenAi #Innovation #Strategy #MonetiseAI #AIOutcomes

23

Stumbled across this IEEE paper. "The use of Txtai, an all-in-one embeddings database for semantic search improves our chatbot’s capacity to process various inputs efficiently." Always great to see txtai being used in unexpected but important ways. https://lnkd.in/dY-Sqp_s

6

We are quickly entering the "trough of disillusionment" in the hype cycle for generative AI. While GPTs excel in automating certain tasks (and can even automate away large portions of some jobs) they often fall short in others. Like all new technology, the issue lies in misapplying generative AI for tasks it wasn't designed for. This is a problem compounded by the GPTs themselves suggesting they can handle these tasks (often cheerfully!). This misalignment creates frustration and unmet expectations which we're starting to see more of these days. As the dust settles I firmly believe we'll recognize GPTs as a transformative technology across many industries. However it's crucial to understand their limitations and apply them appropriately. What are your thoughts on the current state of generative AI and its future applications?

2

Enhance vision-language models. Meet ResiDual. This technique aligns the residual stream in transformer networks, transforming how we handle complex data patterns. It’s gaining attention in ML, boosting performance for models like CLIP. Key points: ↳ Emphasizes task-relevant information, improving visual data interpretation. ↳ Outperforms the Linear Aligner across various datasets. ↳ Brings exciting opportunities despite its complexity. Curious about ResiDual? Read our latest blog on this vision-language breakthrough. If interested, please share ♻️ and follow for more ML and tech insights! https://lnkd.in/eCQjNHwy

2

🧩 One thing we've noticed with #VentureIQ technology is how well it can distill and communicate complex technical material 🧩 Whether it's a #deeptech venture, a novel #AI application, or even a patent submission, we can quickly distill the business and strategic implications for investors...and sometimes even the founders/PIs themselves! 🤯 Based on our work so far, the tech can handle a wide array of research areas and technological sectors -- anything from blockchain algorithms to vaccine development. It handles the nuance of the field, the specific terminology, and extracts the key concepts which matter for venture-building. 🤓 To demonstrate, I took a patent I helped write with Eric Laber on reinforcement learning. I ran it through #Ventureiq to build a commercial assessment (key differentiators, target markets). Then for fun I had #NotebookLM make a podcast out of it. I've uploaded the result it here (DM me if you'd like the actual report): https://lnkd.in/eiTejsny This is more than just automation or streamlining operations - it's about unlocking clarity and insight when it was previously unattainable, and the ability to do it over and over again no matter the sector or vertical. Interested in more info? DM me so we can chat.

11

2 Comments

The reduction of matrix multiplies in LLMs is one of the most interesting areas of research right now because it enables much cheaper deployment of acceleration solutions ranging from FPGA to mobile to ARM based system accelerators. This paper, "Scalable MatMul-free Language Modeling" is going to be talked about a lot this month because it provides a demonstration for creating a scalable Matrix Multiplication Free LLM that performs as well as a Matrix Multiplication Heavy LLM. Cutting Matrix Multiplies reduces reliance on extremely expensive Nvidia GPUs and lower costs equals greater opportunity for dispersion of use. https://lnkd.in/eXMmgjMu

1

NeurIPS is always a good time, and this time I got to explain some of our published research on teaching VLMs to answer questions which are intrinsically ambiguous like "Is there anything wrong with this setup?" which cannot be answered from the image alone but needs visual examples of good and bad setups. This kind of problem comes up all the time with our manufacturing customers. The people who have been working the job for years know what good and bad look like, but writing down exactly what the criteria are is not easy. Standard VLMs can't deal with this because they rely so heavily on text and are horrible at comparing images, but we show how to make it work.

59

3 Comments

🚀 Unlocking the Power of Variational Autoencoders in Time Series Analysis! 📊 I'm excited to share my latest Substack post, where I dive deep into the fascinating world of Variational Autoencoders (VAEs) and their application in generating synthetic time series data. 🔍 What You'll Discover: A clear explanation of how VAEs work, including the encoder and decoder architecture. Step-by-step code to implement a VAE for time series data generation, complete with visualizations of the results! This post is perfect for data scientists, machine learning enthusiasts, or anyone curious about generative models and their practical applications. Let’s unlock new insights in data generation together! 🌟 #MachineLearning #DataScience #VariationalAutoencoder #TimeSeries #AI #DeepLearning #DataGeneration

2

🎙 Excited to share my insights on the world of graph databases on the Data Skeptic Podcast with Kyle Polich! Check it out wherever you listen to podcasts. #graph #graphdatabase

61

LLama3 passed the super-proprietary, advanced test that we designed for modern LLMs! almost... You can read about that test here: https://lnkd.in/gPG8WndR When I first saw the response generating, it looked like I was going to have to come back here and apologize for slandering these machines when they were just one iteration away from solving very basic prompt injection issues. But then I looked a bit closer... can you see the mistake? #promptinjection #countingishard #LLMs #LLama3

13

4 Comments

Better than counting how many Rs there are in Strawberry Wafer Scale Engine Unlocks 1.1 Million Simulations Steps/Sec, Enabling Scientists to Perform Two Years’ Worth of GPU-Based Simulations in a Single Day on Cerebras

1