Dave Burgess

The Economist's article “Large language models are getting bigger and better” on April 17, 2024 on the challenges of scaling large language models (LLMs) rightly identifies the limitations of current AI architectures. While the relentless pursuit of larger LLMs suggests a future of limitless progress, a deeper dive reveals significant flaws. The "scaling hypothesis"—that more data and power equals human-like intelligence—misses crucial aspects. LLMs face serious hurdles: the dwindling supply of high-quality data, the inefficiency of scaling, and the simple fact that size doesn't equate to cognitive complexity. Here, neurosymbolic AI presents a promising alternative, blending the data-handling prowess of neural networks with the logical precision of symbolic AI. This combination not only promises enhanced efficiency and structured reasoning but also aligns closer with human cognitive processes. Neurosymbolic AI excels in areas where traditional models falter, offering a path to more sustainable, intelligent systems capable of real-world applications like airplane design and financial risk assessment. It’s time for the AI community to pivot towards technologies that not only scale but think smarter. https://loom.ly/96rDCD0

21

4 Comments

I just read an insightful piece by Eric Broda (published in Towards Data Science) that outlines the Agentic Mesh—a new ecosystem designed for autonomous agents to interact, collaborate, and transact. His work introduces a structured approach for managing the coming surge in autonomous agents, showing how they can function independently yet collectively. Agentic Mesh serves as a networked environment where autonomous agents operate in a safe, transparent, and efficient manner. It allows agents to find each other and securely interact under a framework of policies. Each agent has a specific purpose and autonomy, making it a valuable contributor to the ecosystem without sacrificing security or control. Core Components: 1. Marketplace: A centralized space where users discover agents by tasks and capabilities, offering a trusted environment for engagement. 2. Registry: A metadata repository organizing each agent’s purpose, owner, and policies, allowing agents and users to securely find and connect with each other. 3. DNS Integration: Each agent has a recognizable digital identity, enabling seamless discovery and interaction, just like an internet service. 4. Agent Stack: This layered structure of communication protocols, control services, and interfaces helps agents operate effectively and responsibly. It ensures that agents act independently within controlled parameters. Agentic Mesh’s effectiveness depends on agents sharing consistent attributes, including: • Purpose: A clear objective drives each agent’s actions. • Ownership: Transparency around ownership fosters responsibility and trust. • Autonomy: Agents act independently within set parameters to complete tasks. • Discoverability: Easily locatable, ensuring efficient interaction within the network. • Intelligence: Agents utilize language models to plan and execute tasks, adapting to complexity as needed. Operational Flow: Registration, Discovery, and Execution Agent interactions in the Mesh follow three core flows: 1. Registration: Agents set up purpose, policies, and registration data to gain visibility in the mesh. 2. Discovery: Through the registry, agents and users locate potential collaborators based on defined needs. 3. Execution: Engaged agents use AI models to plan and perform tasks, enlisting collaborators when necessary to complete workflows. Read the full article: https://lnkd.in/eVmq5BFC

16

1 Comment

Apple announces ToolSandbox A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities abs: Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at github.

11

Crossing the Chasm: 1 million #token context window for a #LLM #AI LLM's take in tokens then predict the next best tokens. So asking an LLM "what is the job of a "radiologist"?, will include a response with "reading X-rays and MRI images...." Where the above context window is 6 words which for arguments sake can be considered 6 tokens. Regardless, the longer the context window the better the memory & recall during a conversation with a LLM. If you also provide useful context in the prompt then you can get useful answers in "one shot" without new tuning or training of a LLM. Why this matters? #ChatGPT etc don't know what they don't know. But they can (by design) reason on "anything" if you package up what you want to reason on as part of the prompt. Definition: Context Window A context window measures how many tokens — the smallest building blocks, like part of a word, image or video that a model can process at once. Skipping to the chase Google announced its LLM #Gemini Pro 1.5 has a context window of 1m tokens. With near-perfect next-token prediction & retrieval rate of more than 99% for up to 10m tokens." "In one test, we dropped in an entire code base and it wrote documentation for it, which was really cool,” "...it was able to accurately answer questions about the 1924 film Sherlock Jr. after we gave the model the entire 45-minute movie to ‘watch.’” "....this rare language — Kalamang — that fewer than 200 people worldwide speak..you can put the entire grammar manual and some examples of sentences into context, and the model was able to learn to translate from English to Kalamang” "....Customers can process vast amounts of information in a single stream including 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code, or over 700,000 words." Researchers into LLM's are making leaps and bounds on multiple fronts. For reference: n-gram models of the 1990s & 2000s typically constrained to 5 tokens...ChatGPT: 16K tokens... Claude 2.1: 200k tokens Notes: NB: A larger context window allows LLM's to maintain coherence over longer conversations or documents NB: Each token is converted into a numerical representation (vector) through a process called embedding. If a model has an embedding size of 768, then you can think of it as having 768 input neurons for each token. NB: RAG-based LLM's were put in place to bring additional context into a query (maybe because the LLM was not trained on a subject or current to the day). RAG frameworks (e.g. Llama-Index) transform input data into vector embeddings, stored in a database then retrieve the most relevant data for each query, theoretically bypassing a token limit. NB: With very large context widows you can submit entire books, original music, intellectual property such as code, legal proceedings that were never part of the training corpus of a foundation model. NB: Gemini 1.5 Pro reason across image (frames) and audio (speech) and video. Khurram Latif

7

Diagram of Thought (DoT): An AI Framework that Models Iterative Reasoning in Large Language Models (LLMs) as the Construction of a Directed Acyclic Graph (DAG) within a Single Model https://flip.it/Mh_Mhu

1

LLM Data Engineering Pattern 3: Aggregating Unstructured Insights at Scale Modern financial analysis often requires synthesizing information from a vast array of unstructured sources—regulatory filings (e.g., 10-K, 10-Q), research reports, market commentary, and even audio transcripts of earnings calls. In many cases, these sources must be combined to form a comprehensive view of an industry’s risk profile. Historically, analysts have relied on manual selection and review of documents, a labor-intensive process that does not scale well in today’s data-rich environment. Consider analyzing credit risk in the U.S. commercial real estate sector: 1) Regulatory Filings: An LLM can systematically parse through hundreds of 10-K and 10-Q filings to extract commentary on asset valuations, loan-to-value ratios, or refinancing challenges. 2) Research Reports: It can synthesize qualitative assessments from research houses covering property market trends, default probabilities, or vacancy rates. 3) Earnings Call Transcripts: It can highlight executives’ forward-looking statements regarding rental income stability, interest rate exposures, or occupancy forecasts. By integrating these unstructured data points, an analyst can address questions like: “Given the commentary on sector vulnerabilities in these filings, research reports, and macroeconomic indicators, what are the key risk factors influencing the commercial real estate market?” Financial people, how are you doing this today? #llm #dataengineering #llmdataengineering #financialdata #finance

20

1 Comment

A seismic shift from Software-as-a-Service to Service-as-Software is eminent, introducing the notion of Systems-of-Work ("AI Workers"), in contrast to Systems-of-Record and Systems-of-Engagement, with outcome-based monetization models! Traditional data stores are being replaced by knowledge graphs that aggregate information from various sources, enabling AI agents to access a comprehensive, interconnected body of knowledge.

4

Generative AI and machine learning: two titans of tech, often confused but distinctly powerful. One creates, the other predicts. Both are reshaping industries at breakneck speed. Which will have the greater impact on your business? The answer might surprise you. #AI #MachineLearning #TechInnovation #BusinessStrategy #FutureOfWork What's your take on the AI revolution? Share your thoughts below.

3

AI memory is fundamental for conversational AI and AI agents. It is critical for building personalized experiences and improving reasoning. Check out Letta’s course to understand how to use AI memory to enhance the customization and quality of your AI app.

15

Gemini 2.0 handles complex math and planning tasks previously out of reach of LLMs. You can turn on "code generation" in AI Studio and in the APIs. For context, let's talk about function calling and voice AI agents. (I spend a lot of time working on voice AI.) Voice AI agents use function calls for: - Interacting with external systems specific to particular use cases and deployment contexts. - A wide range of RAG-like information lookup tasks. - Conversation persistence, including complex actions like saving a privacy-redacted version of a conversation. - Script following. Some of these uses were obvious before we had built and deployed production voice AI systems. But some of them weren't. (Script following, for example.) We had to build experiments, and then production systems, to figure out what features and tooling these new systems really need. I'm convinced that code generation will be as important as function calling, once we've accumulated some experience building with Gemini 2.0 and similarly capable new LLM versions. Credit for this insight goes entirely to Shrestha Basu Mallick, who shepherds work at Google on the Gemini APIs and AI Studio. When Shrestha first told me that really good code generation was going to make a lot of new things possible, I didn't really get it. But after playing with code generation for a few weeks ... I still don't fully get it, but in a good way, now. It's clear that there are some obvious things to explore: - Context-aware math on demand driven by natural language. - Generative UI. - Better "binding" behavior with function calling. But I'm also convinced now that there is a lot of unexplored, non-obvious (at least to me, yet) territory. Let's take that last one — improved function calling performance. Function calling and code generation go together like peanut butter and jelly. Functions often take arguments that are fairly complicated. With code generation enabled, Gemini has access to this new pattern: natural language ➡️ code that creates argument structs for functions ➡️ multiple function calls Think about how powerful this is. The ability to use even very basic code structures — loops, for example — massively expands what LLMs can do. In fact, my impression is that function calling in Gemini 2.0 is now trained as a subset of the model's code generation capabilities. This feels quite different than how function calling works in other models that I have access to. In other models, function calling feels like a close cousin to structured output. Which is great. But this new "modality" is even better. (More on the prompt in this post, and the meta-prompt for this post, here: https://lnkd.in/gycKtnNX)

49

10 Comments

🚨BREAKING BRAINS: the lobotomizing of large language models and the paradox of control. https://lnkd.in/eBtUfM3G #LLMs #AI #AGI #GPT4 #GPT5 #Claude3

1

Despite the disdain for "GPT-wrappers", LLM-driven apps is the layer where value is accruing. This is cuz evals are the differentiator & not easily replicable, unlike AI models. The reason why OpenAI pursues regulatory capture is cuz the moat is shallow, not cuz of existential risk of humanity. In Clay Christensen's lesser known "Law of Conservation of Modularity", he spells this out, summarized and outlined by Stratechery. https://lnkd.in/gt-2isr9 So if you don't want your lunch eaten. Get started on your evals today, and pick up a copy. https://lnkd.in/gxwnHVND

1

The Nature of Large Language Models (#LLM): Discovery or Invention? Not just a Socratic debate, or philosophical discussion. If LLMs are an invention, then we really should know more about how they work, right? If Discovery, then what else do we need to uncover that is right here in front of our eyes? Arguments for Discovery: * LLMs leverage fundamental patterns and structures inherent in human language and knowledge, which we are uncovering rather than creating from scratch. * The emergent capabilities of LLMs were not fully anticipated or designed, but rather discovered through experimentation. There is an entire discipline needed to communicate with LLMs called #PromptEngineering Arguments for Invention: * The specific architectures, training methods, and implementations of LLMs are human-designed , human considered, and prone to human errors. * LLMs required significant engineering effort and novel technical approaches to create; yes, they are created. Newton and Leibniz both developed calculus independently in the late 17th century. Was Calculus discovered (as a fundamental mathematical truth) or invented (as a novel mathematical system). A fun diversion to consider this one, no? Understanding the answer will help us to chart the path forward. However, there may not be an answer, we need to consider that as well. Are LLMs on the path to #AGI ? Suggesting some reasons here, I will be 'out over my skis', and I am a good skier... That said, * LLMs demonstrate strong zero-shot and few-shot learning abilities, indicating a capacity for generalization that's crucial for AGI. * LLMs sometimes display unexpected capabilities not explicitly trained for, hinting at the potential for further emergent intelligence. I am only scratching the surface, barely - I do think about such things, call me strange, bored, or innovative, not sure which... For me, #GenerativeAI still required Human-in-the-Loop and we are still really early and only at the beginning. We all have a lot to learn. For me, I am going to enjoy the ride! #Innovation #AI

1

9 Comments

Interesting discussion and episodes on the use of the AutoGen framework for Agentic Workflows at the Microsoft Research Forum. The #Agenticframework allows LLMs to allow to perform more complex and intelligent tasks autonomously through one of the key concepts of reflection and autonomy. It has significant impact on practical applications such as personalized medicine, automated travel concierge, autonomous navigation etc.. It was also very interesting to read blogs posted on deeplearning.ai by Andrew Ng. https://lnkd.in/daa28seD

10

2 Comments

My interview on The Post's AI collaboration with VTech in this article:

103

3 Comments

🚀 Can large language models (LLMs) predict social science experiment outcomes? This latest research explores this groundbreaking question with an archive of 70 pre-registered, nationally representative survey experiments, including 476 treatment effects and over 105,000 participants. The results are striking: predictions from an advanced LLM (GPT-4) closely mirror actual outcomes, even outperforming human forecasters in accuracy. This study highlights the potential of LLMs to revolutionize experimental methods, but it also uncovers crucial limitations and risks. Dive into the full paper to explore these findings and the future of AI in social science. #AI #SocialScience #LLMs #MachineLearning #DataScience #Research #Innovation #PredictiveModeling Credits: Ashwini Ashokkumar*(1) Luke Hewitt*(2) Isaias Ghezae(2) Robb Willer(2) (1) New York University (2) Stanford University * Equal contribution, order randomized August 8, 2024 Web demo: https://lnkd.in/gDr8-n23

6

1 Comment

Don't Be Driven to Complacency With a POC: Enterprise GenAI Business Transformation is a Different Beast Building a cool LLM application to demo is exciting! But scaling it for real-world enterprise use and getting tangible business value? That's a whole new challenge, regardless of whether you use a vendor LLM or a custom on-prem LLM . Here's why: POC vs. Production-Ready: Sure, your LLM can demonstrate promising results on a small clean, sanitized sandbox. But can your end-to-end systems sustain performance on real-world scenarios, and with messy enterprise data with the rigor, accuracy, robustness and consistency that is necessary for enterprise use ? LLMs are a small part of a much larger equation: Many of us are enthralled by the shiny LLMs, but enterprises are about many factors including data, platforms, heterogeneous systems with complex processes, and human stakeholders (back-office, middle-office functions, and clients).  Are your data assets consumable and integrated with the vendor or custom GenAI solutions? Security, Governance & Safety: Bias, fairness, and responsible AI practices take center stage. Are you ready to monitor outputs and mitigate potential risks? Can it handle sensitive data, integrate with existing enterprise systems, and meet rigorous security standards? Extensible: Are your e2e platforms modular (vs monolithic) to accommodate the rapid tech advances to ensure your products and solutions dont become “stale” in 3-6 months?! Continuous Improvement: Can your AI platforms leverage the virtuous cycle of continuous improvement by learning from client usage and get better over time? This is overlooked in many AI initiatives and I cannot emphasize enough the business benefits that can be derived by data-driven AI customization and tuning Transitioning from POC to production requires careful strategy, planning, execution, and e2e enterprise-ready platforms, and change management. This takes skills, experience and a cultural mindset shift. The blog from Jerry Liu at Llamaindex expressed this very well:  https://lnkd.in/eUM4H6Af Quote: An issue I see in enterprise LLM app development is that the “build a chatbot in 5 minutes” appeal of LLMs becomes a hindrance when you try to productionize anything. Setting up a demo is easy. But over half a developer’s time is spent on tuning and iterating rather than building. #enterpriseAI #LLMs #AIresponsibility

28

9 Comments