Mohamad Nasr-Azadani

Chaos Report: Iran's Response Matrix: A Game Theory Analysis Iran is in a classic decision-making bind: it has to retaliate but it must also be able to survive. However, there is already a problem: it is thinking too much and acting too late. More here on my Substack: https://lnkd.in/gQzQnAKj

10

1 Comment

A substantial piece of work from Apple led by Iman Mirzadeh, keivan alizadeh vahid, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar and Hooman Shahrokhi supports the thesis that transformer-based LLMs do not reason--they perform pattern-matching. The team from  also provides a great benchmarking tool--GSM-Symbolic for assessing mathematical and formal reasoning. This supports my anecdotal findings when I evaluated o1 https://lnkd.in/ec72fecb as well as my thoughts on the incompleteness of our assessments of AGI https://lnkd.in/ehBT3HB3

16

2 Comments

In computational linguistics, researchers are focused on enhancing language models' ability to extract precise information from extensive textual data. Recent studies introduce "retrieval heads," specialized attention mechanisms that selectively focus on critical segments of large texts, improving targeted data retrieval. These heads, integrated into transformer-based models like LLaMA and Yi, significantly enhance accuracy and efficiency, particularly in handling long-context scenarios. The methodology involves thorough experiments across various models, including the Needle-in-a-Haystack test, which embeds specific information to measure precision. Results show models with active retrieval heads outperform those without, with accuracy dropping notably when these heads are masked. The empirical evidence underscores the effectiveness of retrieval heads in improving information retrieval precision and reliability. This research deepens our understanding of attention mechanisms in large-scale text processing, paving the way for more efficient and accurate language models. https://lnkd.in/dNNPvMXw https://lnkd.in/ddbHX6EA Innovation Hacks AI #aiinnovation #generativeai

1

I tried to explain LLM hallucination intuitively using the concept of semantic entropy. I treated the two different scenarios of short response and long response separately. Hallucination mechanism with long response is more complex since the output typically weaves through multiple concepts. #ai #llm #hallucination #semantic #entropy https://lnkd.in/gF2_2H6w

35

5 Comments

🚀 Exciting News: UC Berkeley Introduces Sky-T1-32B-Preview for $450! UC Berkeley has introduced the Sky-T1-32B-Preview, an open-source reasoning model that is causing a stir in the AI community. This innovative model competes with OpenAI's o1 model, showcasing remarkable performance capabilities. This groundbreaking development highlights the potential for advanced AI functionalities at an affordable price point of just $450. The Sky-T1-32B-Preview not only matches the o1 model in benchmarks like Math500 (scoring 82.4%) but also excels in LiveCodeBench-Easy with an impressive score of 86.3%, demonstrating its proficiency in both mathematical and coding tasks. What makes this model truly stand out is its adherence to open-source values. The NovaSky team at UC Berkeley has generously shared all essential resources for replicating and improving the model, including training data, code, and model weights. Furthermore, they have provided training infrastructure scripts and a detailed technical report with wandb logs, creating a valuable resource for researchers, developers, and AI enthusiasts looking to explore or contribute to the realm of AI reasoning models. #AI #UCBerkeley #OpenSource #ArtificialIntelligence

2

The introduction of the Transformer architecture in 2017's "Attention Is All You Need" revolutionized natural language processing by enabling models to detect and reproduce text patterns more effectively. By incorporating self-attention mechanisms, these foundational language models moved beyond Recurrent Neural Networks, allowing them to "remember" the placement of words across large datasets and generate human-like language. However, we still face two significant hurdles: context and reasoning. While these models excel at pattern recognition, they often lack true understanding of context, leading to outputs that can be completely off—what we sometimes call "hallucinations." There's also no genuine reasoning in a statistical pattern recognition engine. Techniques like Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting aim to address these issues by enhancing context and reasoning capabilities, but they are still built upon the same foundational limitations. And now for something completely different (wink wink): brute-forcing statistical AI with larger models and more data is not a viable path to achieving Artificial General Intelligence—it doesn't make sense. We need a Cognitive AI approach—something that can learn incrementally like a human and doesn't require trillions of training words or massive computational resources. One such option could be Integrated Neuro-Symbolic Architectures (INSA), potentially the coming hype of 2025. Who knows? https://lnkd.in/eh65TyFY

13

2 Comments

🔥 New Blog Alert: "The Rise of Autonomous AI Agents: Are We Ready for AI That Thinks (and Acts) for Itself?" 🔥 Imagine a world where AI doesn’t just assist but learns, adapts, and even makes decisions without human intervention. This is the promise—and challenge—of Autonomous AI Agents, the next big wave in #GenerativeAI! 🌐💡 In my latest blog, I dive into the transformative power of autonomous AI, covering: ✅ Game-changing applications in finance, customer service, and project management ✅ How these agents evolve and learn autonomously ✅ The critical ethical questions surrounding AI independence Whether you're an AI enthusiast, a tech professional, or just curious about the future of work, this piece offers valuable insights into what the rise of autonomous AI means for us all. Ready to explore the future? Check it out here 👉 Read on Medium - https://lnkd.in/g33DbzHA 🔖 Special shout-out to leading voices in #AIethics, #MachineLearning, and #FutureOfWork: @OpenAI, @DeepMind, @Tesla, @Microsoft, @GoogleAI, and @ThePartnershipOnAI. Let's discuss what’s next! #AutonomousAI #Innovation #Automation #TechTrends Amrat Kapoor Geet T. Telang Md Suaib

9

Foundational models, or large language models (LLMs), train at the scale of the entire internet to acquire the vocabulary, syntax, and narrative structure of human language. Fine-tuning involves using a smaller, domain-specific dataset to grant additional specificity to a pre-existing foundational model that has already acquired patterns and features from massive underlying training datasets. By utilizing the existing knowledge embedded in a pre-trained language model, we unlock high-performance language understanding, but fine-tuning helps to get the right answer - not just an answer that sounds right - with a minimum of hallucinations or incorrect info. In healthcare, out-of-the-box, foundational models are often too generic to sufficiently answer the precision of a healthcare query. Foundational language models are trained on a massive scale of data, which includes jokes, incorrect information, misunderstandings, and misinformation, along with the sum total of human knowledge. The problem is, if you ask a large language model to give you factually correct information when it has been trained on BOTH accurate and inaccurate information - you’re playing a bit of a dangerous game. Enter fine-tuning language models. In the old days, we used to build a single model for a single task - for example, classifying procedures into evidence-based or not as laid out by clinical practice guidelines, for patients with particular disease markers. Each disease would have its own model, which took a long time to build and didn’t scale particularly well, and we struggled with the sparsity and inherent high-dimensionality of healthcare data. By fine-tuning LLMs, however, we're able to specialize those foundational models to achieve greater domain specificity and specialization, leverage the unique predictive and planning capabilities of LLMs, and cut down on the incorrect information that such models inevitably produce, while still preserving the underlying language prowess of the LLM. If improved accuracy and specificity wasn't enough of a selling point, there's another notable upside to fine-tuning foundational models for use in a clinical context. In terms of computational efficiency, using LLMs out of the box on specific tasks can require massive compute (and spend) that amounts to computational overkill for the problem. Some LLMs charge per query, some by compute load or time - either way, as a business running LLMs in a production environment, minimizing the number of LLM calls means optimizing your compute expenditures, too. When LLMs fail at scale in healthcare, it's usually along the fault line of failure to specify adequately. Fine-tuning large language models along domain-specific lines is one way to effectively productize genAI for healthcare use cases. #genAI #LLM #ai #healthcareAI #healthtech #largelanguagemodels #generativeAI #chatGPT

61

2 Comments

Could RAFT (Retrieval-Augmented Fine-Tuning) be the next evolution of RAG architecture? I am following this with great interest. https://lnkd.in/eHDx3rHz

13

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15

Announcing our latest paper sPhinX: Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Prompting TL;DR: We propose a recipe to create a new synthetic IFT dataset covering 50 languages, and finetuned phi-3-small and Mistral with significant improvements over several multilingual benchmarks spanning reasoning, reading comprehension and QA without compromising on English performance! Preprint: https://lnkd.in/gFNiTqfX Abstract: Despite the remarkable success of LLMs in English, there is a significant gap in performance in non-English languages. In order to address this, we introduce a novel recipe for creating a multilingual synthetic instruction tuning dataset, sPhinX, which is created by selectively translating instruction response pairs from English into 50 languages. We test the effectiveness of sPhinX by using it to fine-tune two state-of-the-art models, Phi-3-small and Mistral-7B and then evaluating them across a comprehensive suite of multilingual benchmarks that test reasoning, question answering, and reading comprehension. Our results show that Phi-3-small and Mistral-7B fine-tuned with sPhinX perform better on an average by 4.2%pt and 5%pt respectively as compared to the baselines. We also devise a strategy to incorporate N-shot examples in each fine-tuning sample which further boosts the performance of these models by 3%pt and 10%pt respectively. sPhinX also outperforms other multilingual instruction tuning datasets on the same benchmarks along with being sample efficient and diverse, thereby reducing dataset creation costs. Additionally, instruction tuning with sPhinX does not lead to regression on most English benchmarks. with Kumar Tanmay, Sanchit Ahuja, Hardik Chauhan, Barun Patra, Kriti Aggarwal, Luciano Del Corro, Arindam Mitra, Tejas Indulal Dhamecha, Ahmed Awadallah, Monojit Choudhury and Sunayana Sitaram

109

𝐌𝐮𝐬𝐭-𝐫𝐞𝐚𝐝, 𝐟𝐫𝐞𝐞 𝐛𝐨𝐨𝐤 𝐭𝐡𝐚𝐭 𝐩𝐫𝐨𝐯𝐢𝐝𝐞𝐬 𝐚 𝐩𝐚𝐭𝐡𝐰𝐚𝐲 𝐟𝐫𝐨𝐦 𝐆𝐀𝐈 𝐭𝐨 𝐀𝐆𝐈! ​ Yann LeCun has expressed skepticism about the potential of LLMs to achieve Artificial General Intelligence (AGI), citing limitations in memory, planning, and grounding in real-world understanding. However, we argue that 𝗟𝗟𝗠 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝘃𝗲 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 (𝗟𝗖𝗜)—particularly with multiple multimodal LLMs working together—offers a promising architecture to address these challenges and progress toward AGI. ​ Two key architectural innovations underlie LCI: ​ 𝐒𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐔𝐧𝐜𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐚𝐧𝐝 𝐂𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐑𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐏𝐥𝐚𝐧𝐧𝐢𝐧𝐠: Inspired by the human mind’s dual-layer structure, LCI divides its architecture into foundational and adaptive layers. The foundation layer, akin to unconscious processing, leverages extensive datasets to build robust pattern recognition, encoding essential responses much like human instincts. The adaptive layer, mirroring conscious thought, enables rapid adaptation and contextual reasoning. This dual-layer approach allows LLMs to perform complex tasks with foundational knowledge while adjusting to new information on the fly, much as a child learns new concepts within a developed cognitive framework. ​ 𝐃𝐢𝐬𝐭𝐢𝐧𝐜𝐭 𝐑𝐨𝐥𝐞𝐬 𝐟𝐨𝐫 𝐄𝐚𝐜𝐡 𝐋𝐋𝐌-𝐀𝐠𝐞𝐧𝐭: Each LLM-agent is assigned a specific role, such as executive (knowledge processing), legislative (behavior guardrails), and judicial (adapting to cultural norms). ​ Additionally, our validation of the LCI framework has been thoroughly tested through real-world deployment in applications across various domains, including healthcare, sales planning, investment, and debiasing news. These empirical deployments highlight the practical viability and adaptability of the LCI framework, showcasing its ability to enhance reasoning and decision-making through collaborative intelligence. ​ The eleven aphorisms presented in this free book illustrate the philosophical foundation of the LCI framework, backed by empirical studies. Through LCI, we envision a clear pathway to AGI, rooted in collaborative intelligence and flexible, context-aware adaptation. https://lnkd.in/gn74DkNj (free) https://lnkd.in/gszSvfqP ($6.98 printing fee)

9

Predictions based on Machine Learning and Statistics often rely on maximal likelihood, which can trap LLMs in biases inherited from the majority opinions of training data. This problem has been substantially addressed through conditional statistics, allowing for mitigation of biases and preventing the cancellation of minority perspectives. https://lnkd.in/gMXKqwyB

7

This is cool. The dictionary learning approach is kind of the inverse of autoencoding, which tries to learn a compressed representation of the data. With dictionary learning, you take the compressed representation in a network and expand it into a larger network so that each feature gets its own unit and each unit encodes one feature. In neuroscience parlance, you’re creating a network that actually *does* have a grandmother cell*. The activity of this expanded network is much easier to interpret. They published the methods paper in November, but here they’re applying it to their Claude 3 Sonnet model. The paper discusses the specific features encoded by the model with a particular focus on features relevant to safety. Link to the paper is below. *This is assuming grandmotherness is a feature encoded by the network, which may or may not be true, but you get the point.

14

1 Comment

🔍 Quantum AI Series: Modeling Sequences with Quantum States Picking up where I left off in my #QuantumAI series, let’s talk about Modeling Sequences with Quantum States, with insights from a fascinating paper on the topic. Classical probability distributions can be modeled using quantum states—particularly entangled states that retain information from subsystems. This approach allows quantum models to efficiently represent sequences without losing essential details. Tensor networks play a key role in optimizing the process, providing a compact, low-rank factorization of data. As I've shared previously, Singular Value Decomposition (SVD) is foundational to these models, as it helps represent the complex correlations present in sequential data. Matrix Product States (MPS) capture these rich interactions by using reduced densities to efficiently compress sequence representations while preserving their structure. This is the essence of keeping representation while building efficient lightweight models. In summary, quantum probability models allow us to go beyond classical limits in sequential tasks, contributing to information retention. Sharing the paper for a deeper dive if you'd like so: https://lnkd.in/d5aqXyyS #QuantumAI #TensorNetworks #SVD #QuantumComputing #AI

5