AI21 Labs

A rented frontier model makes you one of many. But an open model post-trained on your data, built for your workflows, and compounding with every interaction you feed back into it? That’s intelligence your competitors can never get. Talk to Or Dagan, Daniel Gissin, Sharon Maarek and Roi Lavan at NVIDIA GTC Berlin this week to learn how specialized intelligence becomes your AI edge.

Over the past three weeks, our Forward-Deployed Researchers [FDRs] collaborated with NVIDIA, Gong, Deel, and Alice (Formerly ActiveFence) to post-train NVIDIA’s recently released Nemotron 3.5 Super VL open model to deliver specialized intelligence across three different use cases. We believe that the companies that get the most out of AI won't all be using the same intelligence. More and more companies are moving towards AI independence, aiming to own their models for the right reasons: lower cost, control over their data and models, and a proprietary edge a rented model can't offer. Few get there alone. Post-training that holds up in production still takes expertise that sits in a handful of labs. We were glad to partner on this path with Gong, Deel and Alice, training specialized models that match or surpass frontier quality – at a fraction of the cost – across data extraction, policy QA and attack resilience use cases. Full results and methods: https://lnkd.in/e3zjm6PT

For the past few weeks I've worked on one question: how do you build a benchmark that reflects how developers really use coding agents (or how people use any AI system built for a specific job)? It was harder than I expected. These are the steps we went through: 𝟭. 𝗪𝗲 𝗹𝗼𝗼𝗸𝗲𝗱 𝗮𝘁 𝗿𝗲𝗮𝗹 𝘂𝘀𝗮𝗴𝗲. Our own developers' sessions and public datasets like SWE-chat. Most of the work was questions, debugging and short follow-ups, not new code. 𝟮. 𝗪𝗲 𝗰𝗼𝗺𝗽𝗮𝗿𝗲𝗱 𝗶𝘁 𝘄𝗶𝘁𝗵 𝗽𝘂𝗯𝗹𝗶𝗰 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀. They are useful, but they mostly test well-defined tasks with a test at the end. 𝟯. 𝗪𝗲 𝗺𝗮𝘁𝗰𝗵𝗲𝗱 𝘁𝗵𝗲 𝘁𝗮𝘀𝗸 𝘁𝘆𝗽𝗲𝘀 𝘁𝗼 𝗿𝗲𝗮𝗹 𝘂𝘀𝗮𝗴𝗲. Each type has the same share as in real sessions, including tasks from our own code. 𝟰. 𝗪𝗲 𝗿𝗲𝘄𝗿𝗼𝘁𝗲 𝘁𝗵𝗲 𝗽𝗿𝗼𝗺𝗽𝘁𝘀 𝘁𝗵𝗲 𝘄𝗮𝘆 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀 𝘄𝗿𝗶𝘁𝗲. Short and conversational, with the same code and tests underneath. 𝟱. 𝗪𝗲 𝗺𝗮𝗱𝗲 𝘀𝘂𝗿𝗲 𝗲𝘃𝗲𝗿𝘆 𝘁𝗮𝘀𝗸 𝗶𝘀 𝗳𝗮𝗶𝗿 𝘁𝗼 𝗴𝗿𝗮𝗱𝗲. The reference solution passes and an empty attempt fails. 𝟲. 𝗪𝗲 𝗿𝗲𝗽𝗹𝗮𝗰𝗲𝗱 𝘄𝗲𝗮𝗸𝗲𝗿 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻 𝘁𝗮𝘀𝗸𝘀. New questions on widely used open-source repositories, which can't be answered without reading the repository's code. 𝟳. 𝗪𝗲 𝘀𝘁𝗮𝗿𝘁𝗲𝗱 𝘁𝗮𝘀𝗸𝘀 𝗺𝗶𝗱-𝘀𝗲𝘀𝘀𝗶𝗼𝗻. The agent first answers a question about the code, then gets the task in the same session. We also checked the benchmark with our gateway's router: • The router saves cost by sending each request to a cheaper model when that's good enough, and to the premium model only when needed. • We tuned it on a public benchmark to send 25% of requests to the premium model. • On real developer sessions it sent 48%, nearly double. • We tuned it again on our benchmark. • On the same sessions it sent about 25%, right on target. AI21 Labs Joanna Kramer Oded Avraham Lior Benita Yuval Globerson Niv Granot Eran Gat Daniel Gissin

Our team is headed to NVIDIA GTC Berlin next week. We’re excited for a full week of events, including our Co-founder and CEO Ori Goshen at Tuesday’s Founders Panel (Oct 20). We’re keen to hear how organizations are navigating the choice between renting and owning intelligence, and to share some news of our own. See you there! Drop a message to Ori Goshen Daniel Gissin Or Dagan Sharon Maarek or Roi Lavan if you’ll be in town and want to talk about building specialized intelligence for your organization.

When an agent isn't good enough, a common instinct is to scale it. This assumes the architecture is right but just underpowered. While it usually works, it's also usually expensive 💸 After more than a year of researching agentic systems, we found holes in that assumption. Most of the time, the architecture was the problem. We found that picking the right one comes down to a simple question: is the task verifiable? If you can't verify → aggregate If you can verify → generate cheaply and spend your budget on the check Our new Labs in Front post, written by Yuval Belfer, explores how asking ‘is the task verifiable?’ led us to efficient agent architectures across four tasks (agentic search, deep research, RAG indexing and coding. Draws on research done by Adi Elbaz, Eran Goldstein, Tamar Levy-Loboda, Niv Granot, Assaf Gerner, Eli Lepkifker, Oded Avraham, Guy Rephaeli, Guy Freund, Alan Arazi, Ido Weiss and Roee Hendel.👏 🔗 Read the full piece: https://lnkd.in/d6tTeuPF

"You're only as good as your test set״ - That's Or Dagan, our COO, talking with Ran Tavory on the Reversim podcast. Your evals make quality something you can move - but by how much? Tuning agents and harnesses by hand chases a moving target: models update, prices change, your traffic shifts. At scale, you can easily end up paying frontier rates for general intelligence your task doesn't use. On the podcast, Or breaks down why hitting frontier quality (without the frontier price tag) requires pulling two levers. 1. Harness design: Automatically search against your own evals for a Pareto-optimal config to reduce weeks of tuning to hours. 2. Model weights: Open models post-trained on your data and in your environment reach task-specific excellence at a fraction of the cost - and are the only models your competitors can't rent. Thanks Ran Tavory for having us! Listen here - https://lnkd.in/dyMdARDV

A general-purpose model is, by definition, meant to be good at a lot of different tasks. Achieving excellence on your specific task, on the other hand - that’s something else. Task-specific excellence starts with trajectory-aware routing, continues to the harness layer, and, when needed, to post-training a model on your own data and within your own environment. Our co-Founder Prof. Yoav Shoham covers both ends of that spectrum, and more, with John Werner and Dr. Alex Wissner-Gross on Conversations in Action. Worth a listen 🎙️

Last week at the AI Infra Summit, our co-Founder & CEO Ori Goshen sat down with Michael Stewart from M12, Microsoft's Venture Fund to talk about running AI agents in production and the cost-quality tradeoff. The TL;DR: A fleet of smaller open models, with routing logic that directs each call to the one that fits, can reach frontier-level quality for a lower price. After tackling the routing layer, the next round of cost-quality gains sit in the harness, around the model. And when that isn’t enough also, you stop working around the model and change it: take an open one and post train it on your own data, in your own environment. Those 3 levels are what we work on at AI21: the gateway that routes each call, the harness around the model, and a dedicated post-trained model when the workload justifies it. BTW - most teams need more than one. Thanks Michael Stewart for this sharp conversation!

Glad to have taken part in the IIA Silicon Valley AI Summit at Google Bay View last week, where our co-founder Prof. Yoav Shoham participated in ״The Human Threshold״ panel, a conversation that got into the question on so many of our minds these days: Not just what AI is capable of, but what do we become alongside it? Every company putting agents into production is grappling with some version of that question. Thanks to Imagination in Action and the Stanford Institute for Human-Centered Artificial Intelligence (HAI) for having us, to John Werner for hosting, and to Vanessa Parli of Stanford HAI and Satish Iyer of Dell Technologies for the good conversation.

לעידן ה-AI יש חוקי משחק אחרים מאלה שההייטק הישראלי נבנה עליהם.  אתמול בערב, יחד עם Vintage Investment Partners, אירחנו אצלנו במשרדי AI21 ערב שהוקדש לשיחה פתוחה על מקומה של ישראל במרוץ ה-AI הבינלאומי, היתרונות שלנו, המקומות שבהם אנחנו עדיין מאחור - ובמה כרוכה הדרך לעצמאות AI אמיתית. על הבמה עלו בסדר הזה: Anne Neuberger שותפה ב-Andreessen Horowitz ו-Head of Global Affairs יחד עם Abe Finkelstein שדיברו על איך ישראל נראית מהוואלי, Asaf Horesh עם סקירת מאקרו מזווית ההון סיכון, Eyal Waldman, מגדולי החלוצים של ההייטק הישראלי, בשיחה עם פרופ' Yoav Shoham על הפרספקטיבה הישראלית; ושני פאנלים לאורך הסטאק - הראשון על שכבות התשתית והמשאבים בו הצטרפו ל-Ori Goshen המנכ״ל שלנו Omri Geller מ-NVIDIA ו-Alon Yariv מ-Crusoe, והשני על האפליקציות והביזנס שנבנים מעל, והאתגרים האופייניים להם: מודלים עסקיים, moats, cost ועוד, בהשתתפות Eilon Reshef מ-Gong, וAlon Schindel מ-Wiz ו-Guy Nadav מ-Booking.com. בין הדברים המשמעותיים שנידונו, כמה נקודות שחשוב לשים עליהן את הזרקור: 1. ריכוזיות ההשקעות בשיא היסטורי. קומץ קרנות-ענק מגייס את רב ההון, וקומץ חברות מקבל את רובו. עבור אקוסיסטם קטן כמו שלנו זה אומר שבעוד שחלק יכולים לגייס הרבה, הרבה אחרים יכולים למצוא את עצמם מתקשים מאוד לגייס.  2. ריבונות טכנולוגית בתחום ה-AI היא לא מותרות. לאותה יכולת אזרחית יש גם השלכות לאומיות ובטחוניות, ומדינה לא יכולה להניח שתמיד תוכל לקנות את המודלים הטובים בעולם. יכולת עצמית, לאורך כל הסטאק (data centers, סיליקון, אנרגיה ומודלים) היא צורך אסטרטגי. 3. הביקוש ל-compute הוא עצום (GPUs בdata centers כבר מכורים מראש עד סוף 2027 עוד לפני שהבטון במתחם ניצוק), אך מרוכז בידי המעבדות הגדולות. 4. הפער בין מודלים פתוחים לחזית התכווץ לחודשים בודדים, דבר המאפשר ליותר שחקנים, כולל בישראל, לפרוץ עם אינטליגנציה מתמחה בתחומים בהם יש לנו יתרון יחסי. תודה גדולה לדוברים המרתקים, למראיינים Ilan Leiferman Abe Finkelstein וIren Reznikov ומאות האורחים שהגיעו. נרים כוס (קמפרי-תפוזים, ברוח החלוציות🍊) לשנה טובה, מוצלחת ומלאת חדשנות!