Stefano Pacifico

#RAG DOES NOT REDUCE HALLUCINATIONS IN #LLMS: Think I've called this out a few times, but the validation is nice - https://lnkd.in/eAi6zxW2. The article makes two well established points: 1) "Hallucination is a problem of reasoning and not relevance! Any amount of relevant text fed through RAG to a machine will retain the original perplexity and entropy in the system to hallucinate independent of the text"; and 2) "The initial challenge arises from the retrieval phase’s reliance on semantic similarity, which is quantified by inner product spaces or cosine similarities. This metric, while capturing semantic closeness, does not differentiate between factually accurate and inaccurate information." This should be clear to everyone, but the industry still shills flimsy RAG scaffolding anyway (shovels to gold miners).

104

32 Comments

LLMs cannot reason and don't think Agents built on LLMs do not have agency In the attached blog, Melanie Mitchell deconstructs the so-called emerging properties like "reasoning" and "chain of thought". It's very well written and you will find some excellent examples and references. I harbour no doubts: LLMs are not intelligent, they don't reason and they don't have thoughts. There is no chain of thought: it's simply a chain of tokens. Why does GPT4-o1 says "Thinking" in the UI? Because the OpenAGI programmers wrote it in there. It doesn't come from the model. The delay is no doubt caused by inference batching across multiple models. That's not thinking, it's simply cost optimisation in the back end. Putting "Thinking" in the UI is wantonly trying to mislead the user. Which brings me to another point: watch out because agents do not have agency. You can think of LLM agents as a method to decompose a task into subtasks using statistical methods, with a layer of orchestration. There's no reasoning, and it is not reliable: you are still problem-solving on a wobbly statistical model that needs to be supervised by a highly skilled human. It is intelligent? No Can it be useful? YES !!! https://lnkd.in/d8isCjYz

126

29 Comments

I just read an insightful piece by Eric Broda (published in Towards Data Science) that outlines the Agentic Mesh—a new ecosystem designed for autonomous agents to interact, collaborate, and transact. His work introduces a structured approach for managing the coming surge in autonomous agents, showing how they can function independently yet collectively. Agentic Mesh serves as a networked environment where autonomous agents operate in a safe, transparent, and efficient manner. It allows agents to find each other and securely interact under a framework of policies. Each agent has a specific purpose and autonomy, making it a valuable contributor to the ecosystem without sacrificing security or control. Core Components: 1. Marketplace: A centralized space where users discover agents by tasks and capabilities, offering a trusted environment for engagement. 2. Registry: A metadata repository organizing each agent’s purpose, owner, and policies, allowing agents and users to securely find and connect with each other. 3. DNS Integration: Each agent has a recognizable digital identity, enabling seamless discovery and interaction, just like an internet service. 4. Agent Stack: This layered structure of communication protocols, control services, and interfaces helps agents operate effectively and responsibly. It ensures that agents act independently within controlled parameters. Agentic Mesh’s effectiveness depends on agents sharing consistent attributes, including: • Purpose: A clear objective drives each agent’s actions. • Ownership: Transparency around ownership fosters responsibility and trust. • Autonomy: Agents act independently within set parameters to complete tasks. • Discoverability: Easily locatable, ensuring efficient interaction within the network. • Intelligence: Agents utilize language models to plan and execute tasks, adapting to complexity as needed. Operational Flow: Registration, Discovery, and Execution Agent interactions in the Mesh follow three core flows: 1. Registration: Agents set up purpose, policies, and registration data to gain visibility in the mesh. 2. Discovery: Through the registry, agents and users locate potential collaborators based on defined needs. 3. Execution: Engaged agents use AI models to plan and perform tasks, enlisting collaborators when necessary to complete workflows. Read the full article: https://lnkd.in/eVmq5BFC

16

1 Comment

One new interesting application of LLMs: CAIS has released a forecasting bot that provides probability estimates for any question you ask. For example, to the question "Will GPT5 be released before 2025?", it answers that it's 70% likely. The bot works by searching for relevant information online, feeding it to GPT-4o, asking a few guiding questions, and then requesting a probability estimate. What's impressive is that this bot outperforms experienced human forecasters, even though GPT-4o wasn't specifically trained for this task. This kind of application shows that AI could be of great help in improving decision-making in the future. Link to the blog post: https://lnkd.in/gjP7JQgz Link the bot if you want to try it yourself: https://forecast.safe.ai/

11

Recently I learned that LLMs show ability to acquire rare linguistic competence (e.g. "a beautiful five days" is acceptable, but "a blue five pencils" not so much), from a tech talk by Prof. Kyle Mahowald from UT Austin (thanks Dr. Steve Kramer for organizing the guest speaker series). A small, high quality dataset BabyLM was sufficient to study the effect of training data on the model's linguistic ability. By surgically removing different types of phases from training data and observing the degradation model's ability, it can be revealed which part of data is responsible for the particular linguistic ability. This is a great example of doing AI research without breaking a bank. As someone not familiar with linguistics, I take linguistic competence to be a linguistic "value function" or "reward function". Can similar emergence of rare competence (not in training data but model learned it anyway) be shown in other domains? Reference: https://lnkd.in/g4cAPUYk

16

1 Comment

crawshaw – 2025-01-06 https://buff.ly/4gLucu2 This document is a summary of my personal experiences using generative models while programming over the past year. It has not been a passive process. I have intentionally sought ways to use LLMs while programming to learn about them. The result has been that I now regularly use LLMs while working and I consider their benefits net-positive on my productivity. (My attempts to go back to programming without them are unpleasant.)Source: crawshaw – 2025-01-06 (https://buff.ly/407DHwr)

1

Love this paper that tests correlation between humans and evaluation models (LLM as a judge). Few big takeaways 1. Cohen's Kappa score is a better measure of accuracy than pure percentage agreement because it takes into account the bias of random agreement/correlation. 2. Humans only agree 97% of the time! This shows how alignment isn't an all or nothing game but rather a distribution where the goal is to get as close as possible without being perfect cause then you risk overfitting against the human labeler who produce the dataset. 3. Open source models are the way to go for evaluation models - performance is comparable (ex Llama3-70B vs GPT-4) and there are so many more benefits from a transparency, explainability, and reliability perspective. Really great stuff from the team at UMass. If you guys have any other cool papers regarding the LLM-as-a-judge space, would love to hear more! Paper Link: https://lnkd.in/edT8P8Tr

85

My interview on The Post's AI collaboration with VTech in this article:

103

3 Comments

We need to talk about Agents! Inspired by LlamaIndex's recent tutorials on Agents (Ravi Theja Desetty) and Jerry Liu's talk on Advanced Context-Augmented LLM Applications, this post looks at the Planning component of an agentic workflow. We go through Parallelizable sub-queries, Task Decomposition and Planning with Feedback in detail to help LLMs "reason" vs "generate". Happy Reading! #ai #planning #agents #workflowautomation #tech

44

One of the hardest problems in generative modeling today is around #evaluation. How do you know if the output of your diffusion model is actually good? 🤔 If this sounds interesting, come join us as we tackle these hard problems in #genAI! Jobs linked in comments 👇 At Advex AI , we evaluate synthetic samples on two dimensions: 1. Fidelity - is the overall image realistic? 2. Faithfulness - does the generated image align with input instructions? A common challenge we face is the trade-off between fidelity and faithfulness. Current evaluation metrics such as CLIP, Aesthetic score, and RLHF often fail to give reliable results. Here's why: 1. Aesthetic scores (Q-Align, ICML '24) focus on how visually appealing an image is, but for tasks like generating synthetic data to improve downstream model performance, utility matters more than aesthetics. Similar to how augmentations introduce noise to improve robustness and generalization, synthetic images can be valuable even if they don’t score high on aesthetic metrics. 2. Multimodal models such as CLIP (OpenAI) and BLIP (Salesforce) excel at high-level semantics but struggle with low-level details like texture or sharpness. Additionally, these models are also biased towards the distribution it was trained on. 3. RLHF (ImageReward, NeurIPS '23) shows promise, but collecting large volumes of preference data is tough—especially in skilled domains like medical imaging where even the expert judgements are subjective. These limitations hinder full automation of genAI evaluation, but it doesn't have to be this way. At Advex, we’re pioneering new solutions to generative model evaluation! #SolveDataSolveAI

72

8 Comments

Gift link. Two old contacts on LI -- Steve Rosenbush interviews Gary Marcus about the risks in LLMs and need for regulation. Gary and I have long had similar views on LLMs for much the same reason -- decades of evidence. Please read the article. I'll summarize my views. 1) The first law Congress should pass is to make it a serious felony on the level of terrorism to knowingly release high-risk tech to the public that contains catastrophic risks. That's what occurred with ChatGPT and all LLMs since. It should cover any type of tech and/or innovation, and be at the personal level, including CEOs, scientists and engineers who enable it. 2) The second law Congress should pass is to outlaw training on data without explicit, upfront permission in the form of a license. Some members of Congress clearly understand and have described it accurately as theft, but few understand that the entire knowledge economy is being illegally transferred to a few companies, representing the majority of modern economies like the U.S. This MUST END -- yesterday. 3) The third law Congress should pass is how to regulate and monitor LLMs. While most of the immediate risks will be mitigated by outlawing training without license, it won't eliminate cat risk as some publishers of scientific work will still be trained on and made available. I can't emphasize enough how dangerous this is. Those who compare the risk to search don't understand what's possible. The volume of data combined with unprecedented compute power increasingly makes centuries of search engine research possible in seconds. Beyond that, our SGM contains new methods that compresses scientific work from the best minds who ever lived. To even consider making that available to the general public without top-tier security is literally insane and suicidal, but that's precisely where the LLM arms race is headed -- very rapidly. 4) The fourth law Congress should pass is to reinforce SCOTUS rulings that stated that an implied (enforceable) right to privacy exists in several constitutional amendments. The problem is it's never been enforced in the U.S. It's 30 years late but Congress should make privacy an explicit right with zero ambiguity. All of our rights are reflected in data today. 5) Regulations in AI should be limited to LLMs and other 'self-generating' algorithmic models, as that's where the majority of risk lives -- particularly serious cat risks and risks to human rights. The EU had the right idea about this -- in consultation with leading scientists. Regulate according to the actual risk profile, not what some noisy VC in SV says who over-invested in tech they didn't understand, or worse -- don't care. Bottom line. We live in a rules-based civilization. We MUST employ rules-based AI systems. That requires architecture like our KOS (first), not LLM chatbots.

13

LLMs are not so good at generating summaries. Just read an article recently from The Neuron - AI News about the "Lost Middle Phenomenon" in LLMs which further reinforces my beliefs about LLMs. They're bad at summarizing whole documents (especially long texts) even if you could increase the context window. It'll be much better to break large texts down into chunks and utilize a combination of good semantic search + good prompt engineering to generate AI summaries that can mitigate this phenomenon. Glad to say that's how we generate summaries here at Beloga as well. We were pretty close to getting lazy and dumping whole documents into Gemini 1.5's large 1M tokens context model at the beginning 😬 but we stuck through and repurposed our powerful search generative experience for summaries instead. 🚀 #LLM #AI #buildinpublic

11

3 Comments

So - are LLMs Universal? This guy looked at a lot of proofs that LLMs are universal and concluded that they are all wrong or misleading. I didn't check all the proofs and I didn't check all the errors or misunderstandings that he mentions. But I would say that it is not confirmed that any realistic model of LLMs is universal. Specifically about https://lnkd.in/da4qHFgh I did not make up my mind if the model is realistic or not, but I think it makes improvement by 1) Getting the outer while loop into the LLM by emphasizing auto-regression 2) Getting the ability to skip parts of the input with the bounded context window, whether a realistic aspect of a theoretical model or not.

4

1 Comment

📚 "LLMs Will Always Hallucinate, and We Need to Live With This" by Sourav Banerjee and the team. 🧠 This is a foundational paper - if you're an AI / LLM practitioner or champion (or naysayer), this is worth reading - many of you will know this but the evidence is vital. 🔍 Summary: As Large Language Models become more ubiquitous across domains, it becomes important to examine their inherent limitations critically (hence my work on the AI Trust / Verisimilitude Paradox). 🤖 This work argues that hallucinations in language models are not just occasional errors but an inevitable feature of these systems. 🎭 The researchers demonstrate that hallucinations stem from the fundamental mathematical and logical structure of LLMs. It is, therefore, impossible to eliminate them through architectural improvements, dataset enhancements, or fact-checking mechanisms. 🧮 As I have said before - we are playing a game of error minimization, so we need to understand risk and risk mitigation. 🎯 There is still utility in LLMs, but they need to be handled and managed with care. ⚠️ We can save you time, money and help you safely navigate the ‘Age of AI’ #AI Risk Guy Digital Human Assistants Paul Edginton Ricky Sydney https://lnkd.in/gRPDET6x

3

2 Comments

If you're working in ed-tech or building AI tools, making sure language models (LLMs) give accurate info is a big deal. One challenge is "hallucinations," where models sometimes make stuff up or get things wrong. Big Kudos to Ryan for creating this innovative solution : using language-based methods to spot errors in LLMs. It’s an exciting way to investigate AI tools and how to create more reliable and effective solutions for students and teachers. #ai #edtech #logprobs

4

Cost-aware inference is important, but currently LLM prompts are not interoperable. Prompts that work good on one LLM may show worse performance on the other, even more advanced LLM or even on the different versions of the same LLM. Once interoperability problem is solved it will also be possible to implement cost-aware inference that strategically manages computational resources for prompt processing, ensuring sustainable costs. This approach includes model routing https://www.notdiamond.ai/ - automatically selecting the most appropriate model for each task. Startups in this area are actively supported by the industry veterans.

4

2 Comments

This is pretty major. Anthropic is releasing a standard framework for connecting LLMs to the data they need, basically allowing them to control "MCP" servers. Today is going to be exciting. Read all about it: https://lnkd.in/dhCqYGWn

8

2 Comments

After Apple’s research saying LLMs can’t think, it’s funny that now all I’m seeing is research on teaching them to think – something most humans still struggle with. Papers: Looking Inward: Language Models Can Learn About Themselves by Introspection https://lnkd.in/gf6rgnUP On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalisability https://lnkd.in/gNsPtDbi Thinking LLMs: General Instruction Following with Thought Generation https://lnkd.in/guQtRnsD #AIResearch #MachineLearning #ArticificalIntelligence

24