Bahar Zarin, PhD

A substantial piece of work from Apple led by Iman Mirzadeh, keivan alizadeh vahid, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar and Hooman Shahrokhi supports the thesis that transformer-based LLMs do not reason--they perform pattern-matching. The team from  also provides a great benchmarking tool--GSM-Symbolic for assessing mathematical and formal reasoning. This supports my anecdotal findings when I evaluated o1 https://lnkd.in/ec72fecb as well as my thoughts on the incompleteness of our assessments of AGI https://lnkd.in/ehBT3HB3

16

2 Comments

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

Chaos Report: Iran's Response Matrix: A Game Theory Analysis Iran is in a classic decision-making bind: it has to retaliate but it must also be able to survive. However, there is already a problem: it is thinking too much and acting too late. More here on my Substack: https://lnkd.in/gQzQnAKj

10

1 Comment

🚀 New paper from our Llama team AI at Meta! We discuss "cross capabilities" and "Law of the Weakest Link" of large language models (LLMs): 🔹 Cross capabilities: the intersection of multiple distinct capabilities across different types of expertise necessary to address complex, real-world tasks. 🔹 Law of the Weakest Link: cross-capability performance is limited by the weakest underlying capability. Identifying and strengthening these weakest links is key to tackling complex challenges. We’re also releasing a comprehensive taxonomy for LLM capabilities—spanning 76 primary categories and 332 subcategories—plus a CrossEval benchmark of 7 individual capabilities and 7 cross capabilities with 8,400 expert ratings and detailed explanations: 🔹 Individual Capabilities: Core skills like English, reasoning, coding, image recognition, tool use, long context, and multilinguality. 🔹 Cross Capabilities: Complex combinations such as coding & reasoning, image recognition & reasoning, tool use & coding, tool use & reasoning, long context & coding, multilingual & image recognition, and more. Example: “What’s the 10-year trend for total rainfall in Tokyo?”—requires both tool use & reasoning. 🔹 Taxonomy & CrossEval Benchmark: Includes 1,400 expert-annotated prompts based on the taxonomy and 4,200 model responses, each rated by two experts with explanations across easy, medium, and hard levels. Designed by a diverse team of researchers, engineers, data scientists, product managers, and content designers. ➡️ Paper, Data, Benchmark, & Code: https://lnkd.in/g2m-DYjH

173

📣 Calling All Persian-Speaking Researchers and Academics! 🎓 I am excited to introduce a new GitHub repository that contains a LaTeX template for your thesis, specifically designed to be compatible with the Farsi language! 📚 This template is designed to help you focus on the content of your work, while we take care of the formatting and structuring. It's user-friendly, customizable, and complies with academic standards. Whether you're working on your Bachelor's, Master's, or Ph.D. thesis, this LaTeX template can save you hours of work. It comes with pre-defined sections such as Abstract, Introduction, Methodology, Results, Discussion, and References, all in Farsi. The best part? It's open-source! You can modify it as per your needs, and even contribute to the repository to help improve it further. Check out the repository, star it, fork it, and spread the word. Let's make academic writing easier and more efficient for everyone! Link to the repository: [https://lnkd.in/dbxpsZWH] #AcademicWriting #Thesis #LaTeX #OpenSource #GitHub #Farsi

31

4 Comments

This is really awesome. I used notebookLM to summarise a recent paper that shows how o1 like model can be reproduced. After providing the pdf from Arxiv, I noticed that the generated podcast was almost 49 minutes long. Very different from the earlier 10-12 minute version. And another impressive feature was the interactive mode where you can interject the two hosts and ask a question which they very graciously respond to. I wanted to share the audio summary but YT didn’t allow me to upload the WAV file. YT expects a video mp4 file. So I found free videos from Pexels and stitched them together with ffmpeg. The videos are meditative acquarium shots of fish swimming around. And with the audio discussing reproducing o1 as an engaging podcast episode, the video keeps the distraction to the minimum. The paper itself is a great read but if you have a 30-40 minute commute, or want to watch some cool fish swimming around and learn about o1, check out

12

Two new papers this week with my colleagues at Visa: 1. Measuring machine learning model robustness in production using properties of harmonic functions (https://lnkd.in/gnhbseY6) 2. Measuring LLM hallucination based on same, unsupervised and automated! (https://lnkd.in/geMGX7HV)

34

6 Comments

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

5

1 Comment

⚛️ It’s #Physics Time: The Cooper Problem – The Birth of Electron Pairing in Superconductivity ⚛️ 🔍 Historical Context In 1956, physicist Leon Cooper made a groundbreaking observation that would lay the foundation for the BCS theory of superconductivity. He discovered that even a weak attractive interaction between electrons in a lattice could lead to something extraordinary: an instability in the conventional Fermi sea of electrons. This phenomenon became known as the Cooper problem, and it set the stage for explaining superconductivity at a microscopic level. ✨ What’s the Cooper Problem? Imagine adding two electrons to a sea of free electrons (the Fermi sea). Normally, electrons repel each other due to their negative charges, but in a lattice, an attractive interaction can emerge, mediated by lattice vibrations called phonons. Cooper realized that this tiny attraction changes everything. These electrons can pair up to form a Cooper pair, reducing the energy of the system. This pairing happens because the electrons form a quantum state with integer spin, turning them into bosons, which can condense into the same quantum state without scattering off lattice atoms. The result? Superconductivity – the ability of a material to conduct electricity without resistance. 📸 The Mathematical Derivation I’ve attached a mathematical derivation of the Cooper problem, where the Schrödinger equation for two electrons in the Fermi sea reveals how they reduce the system's energy by pairing. Check it out! 💡 Real-World Applications The concept of Cooper pairs underpins the entire field of superconductivity, which has transformed science and technology: 1️⃣ MRI Machines: Superconducting magnets provide high-resolution imaging in healthcare. 2️⃣ Particle Accelerators: Superconducting materials create the powerful magnets that guide particle beams. 3️⃣ Quantum Computing: Superconducting qubits are at the heart of cutting-edge quantum processors. 4️⃣ Lossless Power Transmission: Superconductors enable efficient energy transport with zero resistance. 🔭 Outlook From Theory to Revolution The Cooper problem was just the beginning. Building on this insight, Bardeen, Cooper, and Schrieffer formulated the BCS theory, explaining superconductivity in full detail and earning a Nobel Prize. Today, researchers are exploring high-temperature superconductors and their mysterious mechanisms, with the goal of bringing superconductivity to room temperature – a discovery that could revolutionize energy and computing. 🔗 The Cooper problem demonstrates how a small insight can transform our understanding of the universe. It’s a reminder of the beauty and power of theoretical physics! #Physics #Superconductivity #CooperProblem #QuantumPhysics #BCSTheory #Innovation

33

👏CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks

25

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

Good detailed overview of Retrieval-Augmented Generation (RAG), one of leading techniques to improve accuracy of LLMs. When developing a production-grade LLM-based system, it's essential to understand the advantages and limitations of different techniques (RAG, prompt engineering, fine-tuning etc), across training and serving/inference phases, to improve LLMs for specific applications (see my previous post https://lnkd.in/giuJdJJt) A balanced approach to LLM applications that takes into account accuracy, latency, and cost is crucial for successful productization at scale. #artificialintelligence #largelanguagemodels #GenerativeAI #airevolution

10

I'm thrilled to announce the release of two new fine-tuned large language models (LLMs) and a groundbreaking dataset for Persian NLP. 🚀 Shotor (Llama 3 8B): Fine-tuned on the Persian Alpaca Deep Clean dataset. Paya (Aya 23 8B): Also fine-tuned on the same comprehensive dataset. These models are crafted for high-quality performance in various Persian language applications, leveraging the Alpaca Deep Clean dataset which I meticulously processed. The fine-tuning was achieved using Dora for efficiency. Explore the models: Shotor: https://lnkd.in/ddge8CAh Paya: https://lnkd.in/dT55qCH4 Additionally, I’ve released the first Persian DPO dataset for LLM alignment, aiming to set a new standard in Persian language processing. Check out the datasets: Persian Alpaca Deep Clean: https://lnkd.in/d8ejpYRf Persian DPO: https://lnkd.in/dMF2sihF Your feedback and insights are invaluable as we strive to advance Persian NLP. I invite the community to test these models and datasets, and share your thoughts. Feel free to engage and share this post to spread the word! Looking forward to your feedback and collaboration. #NLP #PersianNLP #MachineLearning #AI #LanguageModels #DataScience #HuggingFace

36

6 Comments

🚀 Exciting News: UC Berkeley Introduces Sky-T1-32B-Preview for $450! UC Berkeley has introduced the Sky-T1-32B-Preview, an open-source reasoning model that is causing a stir in the AI community. This innovative model competes with OpenAI's o1 model, showcasing remarkable performance capabilities. This groundbreaking development highlights the potential for advanced AI functionalities at an affordable price point of just $450. The Sky-T1-32B-Preview not only matches the o1 model in benchmarks like Math500 (scoring 82.4%) but also excels in LiveCodeBench-Easy with an impressive score of 86.3%, demonstrating its proficiency in both mathematical and coding tasks. What makes this model truly stand out is its adherence to open-source values. The NovaSky team at UC Berkeley has generously shared all essential resources for replicating and improving the model, including training data, code, and model weights. Furthermore, they have provided training infrastructure scripts and a detailed technical report with wandb logs, creating a valuable resource for researchers, developers, and AI enthusiasts looking to explore or contribute to the realm of AI reasoning models. #AI #UCBerkeley #OpenSource #ArtificialIntelligence

2