Andrew Gardner, Ph.D.

Thanks Werner Bogula for the really insightful study. I’m only just getting into model comparison, for now using HuggingChat and some tightly defined use cases. The difference between them is notable, interesting to see, and makes me think the future is probably going to be based around multi-LLM workflows. #genai #multillm

1

Very useful paper to read on #LLM in #healthtech, #digitalhealth, and clinical care. Thank you, Jan Beger for all your excellence and informative summaries of #AI in #healthcare papers!

10

🚨 New paper w/ interesting insights for fine-tuning and GPU poor folks 🚨 🚀 Exploring Fine-Tuning for LLMs: Full Fine-Tuning vs. #LoRA - Are They Truly Equivalent? 🤔 As LLMs grow, fine-tuning them for specific tasks becomes challenging due to high computational costs. Recently, Low-Rank Adaptation (LoRA) has become a popular alternative, promising similar performance with fewer parameters. But are LoRA-tuned models the same as fully fine-tuned ones? MIT’s latest study dives deep into this question, revealing some surprising differences! 🔍 📝Key Takeaways: 1. Spectral Differences 🧬: LoRA introduces “intruder dimensions” not found in fully fine-tuned models. These alter the model’s weight structure, suggesting that LoRA and full fine-tuning leverage different parts of the parameter space, even if they hit similar accuracy on specific tasks. 2. Generalization & Robustness 🌐: Models fine-tuned with LoRA may struggle with new or sequential tasks, forgetting previously learned knowledge more easily. Full fine-tuning, however, shows better retention and robustness in shifting environments. 3. Effective Rank & Optimal Performance ⚙️: LoRA’s rank matters! Mid-range ranks (like r=64) perform well on tasks and retain pre-trained knowledge, while lower or higher ranks can lead to greater forgetting. Smart parameter tuning (like setting α=2r) can help LoRA resemble full fine-tuning more closely. 4. Why It Matters 💡: LoRA offers incredible parameter efficiency, but its unique trade-offs might affect its use in scenarios needing high adaptability and long-term memory. With adjustments, LoRA can bridge some of these gaps, but full fine-tuning remains the go-to for complex or multi-tasking demands.

25

1 Comment

Can LLM preference tuning degrade Multi-Agent system performance? In a recent tweet, Andrej Karpathy questioned why all LLMs sound similar, prompting Sebastian Rascha to speculate about the uniformity of alignment datasets. Aligning user preferences is a last-mile step in training large language models. Regardless of the pre-training datasets used, could the preference datasets used during alignment compromise the effectiveness of LLMs? This issue could adversely affect multi-agent systems incorporating multiple LLMs in their design.

7

1 Comment

Synthetic Data for LLM applications Synthetic healthcare datasets have been around for some time. Whenever we discuss synthetic data, we often find ourselves trapped in one question: How realistic is the synthetic data? We create synthetic data because we cannot share the original data for various obvious reasons, e.g., privacy, security, etc. The challenging part is that we want the synthetic data to have all the nice characteristics of the original data. With all the challenges aside, many researchers adopted and tried to prove the value of synthetic data. This morning, I came across this well-summarized survey article, "Synthetic data in healthcare: A narrative review," by Aldren Gonzales, Guruprabha Guruswamy, and Scott Smith. According to the paper, people have used synthetic data for seven categories: 1. Simulation and Prediction Research 2. Hypothesis, Methods, and Algorithm Testing 3. Epidemiological Study/Public Health Research 4. Health IT Development and Testing 5. Education and Training 6. Public Release of Datasets 7. Linking data I have been thinking about synthetic data quite a bit these days. It's because of the Large Language Models (LLMs) like ChatGPT by OpenAI and Claude by Anthropic. I have been playing around with those LLMs. I realize these days that, even if you are an expert, it's almost impossible to predict the outputs. As we cannot logically bind the output behaviors, testing an LLM model with all possible(or available) cases is the only way. Here, I think, is the new application of synthetic data: LLM application testing. If we were to build many LLM applications, we would need a lot of synthetic healthcare data. In this case, the synthetic data do not need to be hyper-realistic. They just need to represent the "quirkiness" of the real-life use cases, so weird synthetic data should be fine. Healthcare-focused LLM applications would need to test with all sorts of available synthetic data to see if such applications produce weird outputs. We may need a mechanism to do so. I think this new use case of synthetic data will be critical in healthcare. Let's see. [1] https://lnkd.in/e8DiEH9j

64

9 Comments

Looks like fine tuning out performs RAG and Large context LLMs for data accuracy with longer contexts Paper: https://lnkd.in/gj9-3jsK Summary of "BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack" - Introduction to BABILong Benchmark: - Designed to test LLMs' reasoning abilities over extremely long documents. - Comprises 20 diverse reasoning tasks, such as fact chaining, induction, deduction, counting, and handling lists/sets. - Uses natural text from the PG19 corpus, extendable to document lengths up to 1 million tokens. - Current LLM Performance: - Popular LLMs utilize only 10-20% of the context effectively. - Performance declines with increased reasoning complexity. - Retrieval-Augmented Generation (RAG) methods achieve 60% accuracy on single-fact QA, independent of context length. - Top Performing Models: - Recurrent Memory Transformers (RMT) show the best performance, processing up to 11 million tokens. - Small models like Mamba (130M) and fine-tuned RMT (137M) achieve high accuracy on long-context tasks. - Evaluation Findings: - Current benchmarks (e.g., Longbench, L-Eval) are insufficient for LLMs with capabilities beyond 40,000 tokens. - BABILong tests models' efficiency in distinguishing relevant facts from irrelevant text over long contexts. - Popular LLMs fail to maintain high performance as context length and complexity increase. - Fine-Tuning Insights: - Fine-tuning improves model performance significantly. - Mamba and RMT demonstrate successful QA with long contexts, outperforming RAG models. - RMT's memory mechanism allows efficient processing of long sequences, with marginal quality degradation up to 11 million tokens. - Benchmark Significance: - BABILong provides a rigorous evaluation framework for LLMs' long-context reasoning abilities. - Highlights the limitations of current LLMs and the need for improved context processing mechanisms. - Key Contributions: - Introduction of BABILong, a scalable generative multi-task benchmark. - Evaluation of over 20 recent LLMs on various context lengths and tasks. - Demonstration of the limitations of current models in long-context utilization. - Setting a new record for sequence size processed by a single model with RMT.

4

5 Comments

Understanding causality is a formidable challenge for Large Language Models (LLMs) and is considered a pivotal benchmark for assessing their proximity to human cognitive abilities. Despite research efforts employing prompt-based training on causal relationships, achieving satisfactory results remains elusive. The absence of a standardized causal dataset for benchmarking exacerbates this issue. In light of this, a recent study proposes a solution in the form of CausalBench, a comprehensive benchmark designed to evaluate LLMs' understanding of causality. This benchmark addresses the limitations of existing evaluation methodologies, offering a diverse set of tasks derived from the causal research community. By encompassing three distinct causal learning tasks, CausalBench enables a thorough comparison of LLM performance against traditional causal learning algorithms. https://lnkd.in/dvV4B7FS

3

[research update] use a Transformer's multi-head attention activations to fetch RAG style, to augment multi faceted problem breakdown for LLMs in ."Multi-Head RAG: Solving Multi-Aspect Problems with LLMs": 1. Enhances LLMs by integrating document retrieval (RAG) into the LLM context. 2. Challenge: Existing RAG solutions struggle with queries needing multiple documents with varied contents due to distant embeddings. 3. Multi-Head RAG (MRAG): Introduces a novel approach using Transformer's multi-head attention activations to fetch diverse documents. 4. Activation Utilization: Different attention heads capture varied data aspects, improving complex query retrieval. 5. Improved Relevance: MRAG shows up to 20% improvement in relevance over standard RAG baselines. 6. Evaluation Methodology: Provides metrics, synthetic datasets, and real-world use cases to demonstrate effectiveness. 7. Seamless Integration: MRAG can be integrated with existing RAG frameworks and benchmarking tools like RAGAS. 8. Multi-Aspect Problem Solving: Designed specifically to handle multi-aspect queries more effectively. 9. Data Stores Compatibility: Compatible with various data store classes. 10. Potential Applications: Benefits scenarios requiring diverse information retrieval, enhancing LLM responses. Experiment The paper introduces MRAG and evaluates it using synthetic datasets and real-world data. It compares MRAG's performance against standard RAG models. The experiment focuses on the model's ability to retrieve relevant documents across various query aspects, demonstrating up to a 20% improvement in relevance over baselines. Use Case MRAG is tested on real-world scenarios like customer support, where queries often require information from multiple documents with diverse contents. The model's capability to retrieve and integrate multi-aspect data enhances the quality of responses.

81

5 Comments

#KAN | #NEURAL #NETWORKS #innovations are steadily coming ... ... possibly also helping to reduce the net result vs. training ratios. Which is dearly needed, as recent LLM fever is on unsustainable track now with quadratic punishment trap of frenzy parameter boosting. Read on what everything can be upgraded still:

2

Hallucination is Inevitable in LLMs An important new paper out that shows that even with perfect data and an idealized architecture “LLMs will always hallucinate”. I suspect that this will over time be linked to Gödel’s Incompleteness Theorem [1] and Turing’s Halting Problem [2]. Every technology has trade-offs. We love LLMs but they need to be used in the right way. [1] Good pop science depiction in Oppenheimer (Gödel was Einstein’s walking companion) or also see Rebecca Goldstein’s book Incompleteness [2] Turing is depicted in The Imitation Game and also the progenitor of the famed Turing Test.

36

In context of debate if large context #LLM models will eventually negate the need for #RAG, an article from Databricks has some good insights - • Longer contexts boost RAG accuracy... to a point • Performance often peaks at 32k-64k tokens, then declines • Top models (GPT-4, Claude 3.5) maintain consistency at scale • Others show unique failure modes (e.g., copyright concerns, summarization instead of Q&A) • Optimal context size varies by model and task • Lack of long-context training data may explain some issues Key takeaway: Long context and RAG are synergistic, but LLMs need refinement to fully leverage extended contexts. https://lnkd.in/eehdKtCm

17

Could RAFT (Retrieval-Augmented Fine-Tuning) be the next evolution of RAG architecture? I am following this with great interest. https://lnkd.in/eHDx3rHz

13

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

📝 RAG vs LC for LLMs (AI paper) 👉🏼 Paper Focus: Evaluating and comparing two main approaches to enable large language models (LLMs) to incorporate long external contexts: Long Context (LC) and Retrieval-Augmented Generation (RAG). The authors analyze recent studies on this topic, highlighting their key insights and discrepancies. 👉🏼 Results: ✍🏼 LC generally outperforms RAG in question-answering benchmarks, especially for Wikipedia-based questions. ✍🏼 Summarization-based retrieval performs comparably to LC, while chunk-based retrieval lags behind. However, RAG has advantages in dialogue-based and general question queries. Want more tips like this? ♻️ Repost and 👉 Follow Mukesh Manral🇮🇳 for daily tips on data & AI! 📋 This paper provides an in-depth analysis of results so do check it out:

25

2 Comments

💥💥💥 RieszBoost: Gradient Boosting for Riesz Regression Kaitlyn J. Lee, Alejandro Schuler Abstract Answering causal questions often involves estimating linear functionals of conditional expectations, such as the average treatment effect or the effect of a longitudinal modified treatment policy. By the Riesz representation theorem, these functionals can be expressed as the expected product of the conditional expectation of the outcome and the Riesz representer, a key component in doubly robust estimation methods. Traditionally, the Riesz representer is estimated indirectly by deriving its explicit analytical form, estimating its components, and substituting these estimates into the known form (e.g., the inverse propensity score). However, deriving or estimating the analytical form can be challenging, and substitution methods are often sensitive to practical positivity violations, leading to higher variance and wider confidence intervals. In this paper, we propose a novel gradient boosting algorithm to directly estimate the Riesz representer without requiring its explicit analytical form. This method is particularly suited for tabular data, offering a flexible, nonparametric, and computationally efficient alternative to existing methods for Riesz regression. Through simulation studies, we demonstrate that our algorithm performs on par with or better than indirect estimation techniques across a range of functionals, providing a user-friendly and robust solution for estimating causal quantities. 👉 https://lnkd.in/dCszhVaP #machinelearning

1

🧠 What’s a Semantic Cache in RAG? Here’s What We’ve Learned Semantic cache is a time and cost saving method for Retrieval Augmented Generation (RAG). Instead of always running a full vector search, a semantic cache stores embeddings of frequently asked queries and their responses. This speeds up your RAG system by skipping redundant searches for similar queries. How do you implement it? Step 1: Store history of answered questions, their embeddings and the LLM response in a semantic cache table. Step 2: When a new query comes in, compare it against the cache table using vector similarity. Step 3: If there’s a match with score over a certain threshold, serve the cached response. If not, run a fresh retrieval and add it to the cache. Now, let’s talk vector search methods: 1️⃣ Cosine Similarity: Measures angle between vectors, works well for normalized data. 2️⃣ Euclidean Distance: Measures straight-line distance but can be skewed by vector magnitude. 3️⃣ Dot Product: Our favorite—it factors in vector size and performs best for our use cases. Want to see how semantic caching powers a RAG system in real life? Check out our demo at https://lnkd.in/g2jFuHTT. 🚀

2

𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐂𝐨𝐫𝐞 𝐄𝐚𝐫𝐧𝐢𝐧𝐠𝐬 𝐌𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭 𝐰𝐢𝐭𝐡 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 This paper explores the use of LLMs to estimate core earnings, which are defined as a firm's persistent profitability from its core business activities, excluding transitory and non-recurring items. The authors evaluate LLMs using 10-K filings of U.S. public companies between 2000 and 2023, employing two prompting strategies: (i) a baseline “out of the box” approach, and (ii) a structured sequential approach. While the baseline strategy led to conceptual errors (conflating core earnings with EBITDA, etc.), the sequential strategy produced a more accurate core earnings measure. The study shows that the LLM-generated core earnings, especially from the sequential approach, outperform traditional metrics like GAAP Net Income and Compustat's OPEPS and OIADP in predicting future earnings. 𝐊𝐞𝐲 𝐂𝐨𝐧𝐜𝐥𝐮𝐬𝐢𝐨𝐧𝐬: - 𝐋𝐋𝐌𝐬 𝐟𝐨𝐫 𝐂𝐨𝐫𝐞 𝐄𝐚𝐫𝐧𝐢𝐧𝐠𝐬 𝐄𝐬𝐭𝐢𝐦𝐚𝐭𝐢𝐨𝐧: Large language models, when prompted sequentially, can effectively estimate core earnings by isolating non-recurring items from 10-K filings, providing a scalable and neutral alternative to traditional methods.    - 𝐁𝐚𝐬𝐞𝐥𝐢𝐧𝐞 𝐯𝐬. 𝐒𝐞𝐪𝐮𝐞𝐧𝐭𝐢𝐚𝐥 𝐏𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠: The baseline “out of the box” approach often led to errors, conflating core earnings with EBITDA and other financial concepts. The structured sequential prompt, however, yielded a valid and effective measure of core earnings. - 𝐈𝐦𝐩𝐫𝐨𝐯𝐞𝐝 𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐯𝐞 𝐏𝐨𝐰𝐞𝐫: The core earnings measure from the sequential LLM prompt consistently outperformed GAAP Net Income and Compustat's OPEPS and OIADP in predicting future earnings, particularly over longer time horizons. - 𝐂𝐨𝐬𝐭 𝐚𝐧𝐝 𝐓𝐢𝐦𝐞 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲: The LLM approach is remarkably cost-efficient, with API calls costing less than $1 per firm. This offers a low-cost, scalable method for analyzing increasingly complex financial disclosures. - 𝐅𝐢𝐧𝐚𝐧𝐜𝐢𝐚𝐥 𝐃𝐢𝐬𝐜𝐥𝐨𝐬𝐮𝐫𝐞 “𝐁𝐥𝐨𝐚𝐭”: As financial reports become more verbose, traditional methods struggle to extract relevant recurring earnings. LLMs have the potential to mitigate these challenges by processing large volumes of text and identifying key earnings components. - 𝐏𝐨𝐭𝐞𝐧𝐭𝐢𝐚𝐥 𝐟𝐨𝐫 𝐅𝐮𝐭𝐮𝐫𝐞 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬: The study highlights the importance of structured prompts and guidance when using LLMs for complex financial tasks. Future, more advanced LLMs could further enhance accuracy and efficiency in financial analysis. - 𝐌𝐚𝐫𝐤𝐞𝐭 𝐕𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐑𝐞𝐥𝐞𝐯𝐚𝐧𝐜𝐞: The sequential prompt LLM-based core earnings measure has a stronger association with stock prices over longer horizons compared to other measures, suggesting its relevance in capturing persistent profitability components. Link to the paper: https://lnkd.in/d--EvJXC

7

Foundational models, or large language models (LLMs), train at the scale of the entire internet to acquire the vocabulary, syntax, and narrative structure of human language. Fine-tuning involves using a smaller, domain-specific dataset to grant additional specificity to a pre-existing foundational model that has already acquired patterns and features from massive underlying training datasets. By utilizing the existing knowledge embedded in a pre-trained language model, we unlock high-performance language understanding, but fine-tuning helps to get the right answer - not just an answer that sounds right - with a minimum of hallucinations or incorrect info. In healthcare, out-of-the-box, foundational models are often too generic to sufficiently answer the precision of a healthcare query. Foundational language models are trained on a massive scale of data, which includes jokes, incorrect information, misunderstandings, and misinformation, along with the sum total of human knowledge. The problem is, if you ask a large language model to give you factually correct information when it has been trained on BOTH accurate and inaccurate information - you’re playing a bit of a dangerous game. Enter fine-tuning language models. In the old days, we used to build a single model for a single task - for example, classifying procedures into evidence-based or not as laid out by clinical practice guidelines, for patients with particular disease markers. Each disease would have its own model, which took a long time to build and didn’t scale particularly well, and we struggled with the sparsity and inherent high-dimensionality of healthcare data. By fine-tuning LLMs, however, we're able to specialize those foundational models to achieve greater domain specificity and specialization, leverage the unique predictive and planning capabilities of LLMs, and cut down on the incorrect information that such models inevitably produce, while still preserving the underlying language prowess of the LLM. If improved accuracy and specificity wasn't enough of a selling point, there's another notable upside to fine-tuning foundational models for use in a clinical context. In terms of computational efficiency, using LLMs out of the box on specific tasks can require massive compute (and spend) that amounts to computational overkill for the problem. Some LLMs charge per query, some by compute load or time - either way, as a business running LLMs in a production environment, minimizing the number of LLM calls means optimizing your compute expenditures, too. When LLMs fail at scale in healthcare, it's usually along the fault line of failure to specify adequately. Fine-tuning large language models along domain-specific lines is one way to effectively productize genAI for healthcare use cases. #genAI #LLM #ai #healthcareAI #healthtech #largelanguagemodels #generativeAI #chatGPT

61

2 Comments