Ben Epstein

The cost of training top models is increasing 2-3X per year, and at this rate will reach $1 billion per model by 2027. The findings are from Epoch AI and follow their recent analysis of increases in compute for foundation models running at 4-5X per year. They've also provided some interesting data on the relative breakdown of cost components like staff, hardware, and energy. A few thoughts after reading this: - I would assume staff costs won't increase as much as hardware and energy costs, so the relative contribution of human labor to future AI models will decrease, in yet another sign of capital's increasing dominance over labor in the industry (just look at NVIDIA's ever-rising stock price). - If compute is increasing 4-5X per year but cost is increasing 2-3X, is it safe to assume that training runs are getting more efficient? - Some people, like Anthropic CEO Dario Amodei, have said we'll see $1 billion training runs in the next year. Does this mean the 2-3X pace is about to increase? That Epoch is wrong in its estimate? Or that Amodei is overstating the cost, possibly to put fear into competitors who can't keep pace? - Only a few companies are going to be able to train future models. We're likely going to see further consolidation. Perhaps we get to a future with only two large general purpose models, like the Windows/MacOS and Android/iOS duopolies we've seen in the past. Article here: https://lnkd.in/g5SejXen

7

OpenAI releases "Strawberry" -01 preview model focused on reasoning by incorporating an Ai agent into multi-step complex task execution. Great article from Ethan Mollick with experiments and results. He writes: "Using o1-preview means confronting a paradigm change in AI. Planning is a form of agency, where the AI arrives at conclusions about how to solve a problem on its own, without our help." Incorporating Ai agents is an inevitable step in the evolution of LLMs and Generative Ai. We will indeed be facing some interesting conversations on the future of work as Ai continues to take on more and more complex activities. #Ai #artificialintelligence #generativeai #genai #chatgpt #strawberry #reasoning #futureofwork https://lnkd.in/gcFrinis

7

This is a good article on using LLMs in the real world. It covers many aspects that we learned at Nurish over the last few months and more. Using LLMs is an inexact science, but has its own needs in processes and tools. While most of the attention has been on the LLMS, their capabilities, and RAG, there are mundane issues that need to be taken care of, issues where the solutions for pre-LLM development are not that well-suited. These include version management, integration, testing, data management, quality assessment, and more. This article covers many of those practical aspects. Eugene Yan, Bryan Bischof, Charles Frye & more https://applied-llms.org/

9

TL;DR: I'm back doing consulting I’ve just placed Restobot in liquidation. After it turned out that one of the main hypotheses was just a corona-induced transient effect, we never found anything resembling product-market fit and the team naturally disintegrated after that. Failing fast beats failing slow. This journey taught me a lot about markets, why certain products are the way they are &c. Hindsight is, as usual, 20/20. Next year I’m diving back into consulting through Hipcode. Hit me up for anything related to AI, math-heavy programming, machine vision, advanced robotics/physics or general systems analysis.

52

2 Comments

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

Notion vs Glean reminds me a lot of Tesla vs Waymo... Just like I think Tesla is going to solve the 'software' problem of autonomous vehicles before Waywmo solves the 'hardware' problem... I think that Notion is going to solve 'enterprise knowledge search' before Glean solves the problem of document and knowledge creation... Why? A couple reasons: 1) Just like Tesla, Notion has a huge advantage when it comes to the amount of underlying data and documents to train on... They already have 100M+ users, and likely 10x that in terms of the number of documents, files, etc. in team's workspaces. 2) It's very hard to change people's behaviour re: where/how they create 'work'. It's why a lot of companies have come and gone in that category... As exciting as Glean's AI writing assistants might seem, most people are just fine (or even prefer) a good old fashioned editor/writer. I could be way off here, but this will be an intersting one to follow. Thoughts? Where am I getting it wrong? #ai #enterprise #knowledge #management #notion #glean

3

1 Comment

Some features take almost no time once you get down to implementing it! It's usually the other way round, but this happened to me last week making Ostinato's variable fields feature more flexible using an additional user-specified parameter! With this new parameter, generating scaled NAT flows becomes much easier!

17

2 Comments

If you thought software engineers were grumpy before, just wait until they start working with open-ended AI systems. One of the biggest challenges in designing AI systems is establishing guardrails to the behaviors. For example, in traditional software, you can click one of two buttons...two options developers have to deal with. In language-based systems, the inputs are theoretically infinite, which means I have to try to constrain the user to a semi-finite set of options. Without limits, users get frustrated by bad responses, hallucinations, and unintended consequences. In short, engineers now need to enforce policies instead of explicit rules and that is a substantial increase in complexity (and therefore grumpiness). https://buff.ly/3KvNOnq

3

AI's hallucination problem is more than just a glitch - it's a 41% error rate that's holding back real-world applications. Enter knowledge graphs and GraphRAG: powerful tools grounding AI in facts and relationships, making outputs more accurate and explainable. For businesses seeking to harness AI's potential, this could be the key to unlocking trusted, practical solutions. #AIHallucinations #KnowledgeGraphs #AIAccuracy #EnterpriseAI #InnovationInTech What steps is your organization taking to improve AI reliability?

6

3 Comments

Today is like yesterday, and today is like a year ago. When we talk to customers here at Recast about what assumptions we use, we say we really have those two core assumptions for things that change over time. These come down to Gaussian processes, which I know sound scary, but let me explain: A Gaussian process says the following – if you take any two slices of the domain on which it’s defined, the distribution of the data living on those slices is going to be a multivariate normal distribution. By “domain” and “slice”, what we mean at Recast is “time” and “a day” - meaning that we’re talking about data two — or three, or eight hundred — days apart. But in general Gaussian processes can be defined across locations or any other sort of structure you care to define. Now, that’s simple enough. What makes it magical is that you can define really interesting correlation structures between those two points. In a typical time series case, two points in time are going to be very similar to each other. But also two points in time separated by 365 days are also going to be very similar because there's a seasonal pattern. There are all sorts of different Gaussian processes that exist with different assumptions laid into them. But those are the ones that we use and everything that varies over time – the level of organic demand, the ROI curve for a channel, etc.

11

2 Comments

#RAG DOES NOT REDUCE HALLUCINATIONS IN #LLMS: Think I've called this out a few times, but the validation is nice - https://lnkd.in/eAi6zxW2. The article makes two well established points: 1) "Hallucination is a problem of reasoning and not relevance! Any amount of relevant text fed through RAG to a machine will retain the original perplexity and entropy in the system to hallucinate independent of the text"; and 2) "The initial challenge arises from the retrieval phase’s reliance on semantic similarity, which is quantified by inner product spaces or cosine similarities. This metric, while capturing semantic closeness, does not differentiate between factually accurate and inaccurate information." This should be clear to everyone, but the industry still shills flimsy RAG scaffolding anyway (shovels to gold miners).

104

32 Comments

Gemini 2.0 handles complex math and planning tasks previously out of reach of LLMs. You can turn on "code generation" in AI Studio and in the APIs. For context, let's talk about function calling and voice AI agents. (I spend a lot of time working on voice AI.) Voice AI agents use function calls for: - Interacting with external systems specific to particular use cases and deployment contexts. - A wide range of RAG-like information lookup tasks. - Conversation persistence, including complex actions like saving a privacy-redacted version of a conversation. - Script following. Some of these uses were obvious before we had built and deployed production voice AI systems. But some of them weren't. (Script following, for example.) We had to build experiments, and then production systems, to figure out what features and tooling these new systems really need. I'm convinced that code generation will be as important as function calling, once we've accumulated some experience building with Gemini 2.0 and similarly capable new LLM versions. Credit for this insight goes entirely to Shrestha Basu Mallick, who shepherds work at Google on the Gemini APIs and AI Studio. When Shrestha first told me that really good code generation was going to make a lot of new things possible, I didn't really get it. But after playing with code generation for a few weeks ... I still don't fully get it, but in a good way, now. It's clear that there are some obvious things to explore: - Context-aware math on demand driven by natural language. - Generative UI. - Better "binding" behavior with function calling. But I'm also convinced now that there is a lot of unexplored, non-obvious (at least to me, yet) territory. Let's take that last one — improved function calling performance. Function calling and code generation go together like peanut butter and jelly. Functions often take arguments that are fairly complicated. With code generation enabled, Gemini has access to this new pattern: natural language ➡️ code that creates argument structs for functions ➡️ multiple function calls Think about how powerful this is. The ability to use even very basic code structures — loops, for example — massively expands what LLMs can do. In fact, my impression is that function calling in Gemini 2.0 is now trained as a subset of the model's code generation capabilities. This feels quite different than how function calling works in other models that I have access to. In other models, function calling feels like a close cousin to structured output. Which is great. But this new "modality" is even better. (More on the prompt in this post, and the meta-prompt for this post, here: https://lnkd.in/gycKtnNX)

49

10 Comments

Some time ago, I was ruminating about the epiphany that NLP's evolution has been the continuous refinement and enhancement of the "strings in, strings out" paradigm. I thought perhaps I could write a series of articles or even a book – ah, the best-laid plans of mice and men! Who has the time? I can offer you at least an article. Join me as I explore the major milestones in NLP history using the “strings in, strings out” lens. Here's a not-behind-the-paywall-link for friends. https://lnkd.in/d5nK8FNK

33

With the coming of artificial super intelligence (ASI), there are many unknown unknowns. So we used HelloScribe A.I to develop a scenario plan to respond to all possible outcomes. These are the results: https://lnkd.in/eamzTBMg #helloscribe #planforanything

1

1 Comment

Increasingly AI applications invoke tools, leverage resources/data and have multiple steps and models involved in a complex pipeline. The boundaries between "RAG", "agents" and other names we have given these compound AI systems are blurring. For example, one thing that would be super useful is hyperparameter-style optimization for RAG pipelines. You define the knobs of your system, and the search space, and we run many sweeps to find the best configuration. Some knobs could be document chunking, embedding model or retrieval-time LLM used. But other knobs could be whole algorithms or techniques, such as contextual retrieval (https://lnkd.in/ejH9vCXZ), graph RAG, etc. What's clear is there are so many permutations for increasingly complex AI systems that we need tools to help systematically measure what works and doesn't work for *your* application. Evals is just the starting point -- to take those learnings and improve your applications with it requires methodical experimentation.

23

4 Comments

TRIPLE BAM! If you missed it, here's the recording of our live Q/A with Jay Alammar and Joshua Starmer PhD! We talked about many AI topics, answered lots of great questions, and talked about two great books: Hands-On Large Language Models: Language Understanding and Generation, and The Illustrated Guide to Neural Networks and AI!!! https://lnkd.in/dcPHs7YQ

44

1 Comment

I can’t believe we used to do this to get structured outputs from LLMs. That was before DSPy, Instructor and anonLLM. It made me want to reflect on the best practices going forward. This is super important because of the rise of compound AI systems that require structured outputs to streamline communication between components. Check out my article below for more info! https://lnkd.in/ejCunhQ5

3

LLMs are terrible at basic math. Why? Here's why: Unlike people, LLMs don’t conceptualize mathematics. They simply recognize patterns in the training data (text) they’ve learned from. When asked what 2 + 2 is, an LLM recalls instances of seeing ‘2’, ‘+’, ‘2’ and so on, allowing it to approximate solutions but not match the precision of a calculator. That’s how we end up with LLMs asserting that 2 + 2 = 5. But for analytical teams, like finance, that makes LLMs really hard to adopt in day-to-day workflows. The productivity gains of having AI do calculations are negated by needing to fact-check and audit every calculation and number. LLMs are also not deterministic, so no amount of prompting is going to guarantee you’ll get the correct answer every time. And that’s just for basic math – the outputs I’ve seen from LLMs on more complex calculations and analyses can get very wonky. Add thousands of rows of data and you’re squarely in hallucination land. So what’s the solution? Calculate things manually? No. The good news is that this is changing quickly and there are options: 1 - AI is getting better at math. Researchers continue to train LLMs with more specialized datasets that include a much wider array of mathematical problems and solutions. The newest models, like OpenAI’s latest family of models, are incorporating things like symbolic reasoning and “chain of thought” methods to break complex problems into smaller, logical steps the LLM can more easily process. Results have been really promising and we’ve seen each new iteration of models continue to beat previous benchmarks. 2 - Leverage AI agents For intensive workflows, and to really make sure your outputs are correct, working with companies that are building high-performing AI agents is likely your best bet. Agentic workflows allow LLMs to be used for what they are great at (inferring user intent, making decisions, etc.) while outsourcing the things they struggle with (complex calculations) to other parts of a broader system. Text-to-SQL, “tool-use”, few-shot prompting, and other strategies can be combined with real-time data integrations to dramatically improve performance on analytical operations. At Concourse we’ve spent the better half of 2024 getting our AI Agents to be really good at math, and I’m so excited to continue growing our capabilities to tackle even more complex mathematical problems for our customers in 2025. Reach out if you’re interested in learning more about how we’ve built Concourse and the impact we’re already delivering for finance teams.

42

5 Comments

A game-changer in AI evaluation just dropped. Scale AI has unveiled its first-ever SEAL Leaderboards. This isn't just another ranking; it's a pivotal move towards more transparent, domain-specific assessments of large language models (LLMs). Why does this matter? For starters, it shifts the narrative from a one-size-fits-all evaluation to nuanced, domain-specific insights. OpenAI, Nvidia, and other giants already rely on Scale AI for training data. Now, they—and we—have a map to navigate the strengths and weaknesses of various LLMs across different sectors. This initiative couldn't come at a better time. As AI continues to permeate every industry, understanding which models excel in specific domains isn't just beneficial; it's crucial for strategic deployment and development. I'm all in on this approach. By fostering a competitive yet transparent ecosystem, Scale AI isn't just ranking models; it's pushing the entire industry towards more specialized, effective AI solutions. This is exactly the kind of innovation that accelerates progress and ensures AI technologies meet the real-world needs of businesses and consumers alike. What's your take on the impact of domain-specific leaderboards for LLMs? [Link to the article](https://lnkd.in/g-TqVMEE) Check this out: https://lnkd.in/g-TqVMEE

8

2 Comments