Achut Manandhar

Same URL https://nhsrcommunity.com/ new look and now open source Quarto! The site is hosted and published through https://lnkd.in/ezUiryRG. Importantly, it's open to contribution so if you spot any mistakes or want to contribute content like a blog do raise and issue or a pull request. We particularly value any accessibility issues as these are an important feature of the website but one that we are still learning https://lnkd.in/eCvxTbzp

15

Conditional Computation in LLMs (MoEs): Understanding the Concept of MoEs The Mixture of Experts (MoE) works on the idea of conditional computation, allowing the model to only run through some parts (an Expert) for a given input compared to a dense model where all parameters are used. It consists of a gating mechanism that selects the experts for the given input, enabling the model to scale without increasing the required computation. Popular LLM architectures like Mixtral and Grok use the same concept. What is MoEs? The Mixture of Experts (MoE) makes a simple modification to the usual architecture by replacing all or some of the Feed Forward layers with an MoE layer. The MoE layer consists of: 1. Router: A parameter gating mechanism that selects the experts used to process the given input by producing a probability distribution over experts. 2. Experts: Standalone NN modules with independent sets of parameters trained jointly with routers. It can use more complex architecture to create hierarchical MoE modules by implementing each expert as another MoE. - Implementation in Switch Transformers : Training was found to be 7x faster than training the base T5, keeping computation and training time constant. - Gshard and MeshTransformer : These frameworks propose efficient methods to train the MoE architectures using the parallelisation of models across GPUs/TPUs. Challenges with MoEs and Solutions 1. Inconsistent Batch Size for Experts: Can be overcome with the use of "soft" constraints in training loss. 2. Load Balancing Across Experts: The issue of repeatedly using the same few experts while training can be overcome by including better balancing mechanisms. 3. Model Instability Issues: Use of router-z Loss as an auxiliary loss to encourage smaller values within the router mechanism. 4. Model Overfitting: Requires different hyperparameter settings for effective fine-tuning. It was found that models benefit from high learning rates and smaller batch sizes. Recommended Reading/References 1. Hugging Face Blog on MoE - https://lnkd.in/g36gQJTx 2. Cameron Wolfe's Substack on Conditional Computation - https://lnkd.in/gtpjMHnJ

60

1 Comment

We had this coming: MoA LLM -> Mixture of Experts (MoE) Agents -> #Mixture of Agents (MoA) Let's understand them in simple terms. #MoE: Within one large model, sub-models are created on specific parts of data to develop an expert on that part of data. E.g. One sub-model specializes in predicting health conditions for Native Hawaiians while another sub-model is specialized for Asians. MoE reduces training time, computational costs while maintaining (or improving) accuracies of #LLMs. Twitter / X's #Grok-1, Mistral AI's #Mixtral 8x7B (8 experts of 7 billion parameters each), Databricks's #DBRX are based on this architecture. Ensemble models (EM): Why am I suddenly jumping to EM? Because, in your discussions, this may come up as a concept similar to MoE at #LLM-level, just like EM is at traditional #ML-level (e.g. Gradient Boosting Random Forest). But there is one distinction - In EM, each model is trained on all the data while in MoE, each sub-model (expert) is trained on a subset of data. #MoA: While MoE combines multiple neutral networks (#experts), MoA combines outputs from multiple Agents (potentially accessing multiple LLMs internally). Advantage: More accurate response with improved quality. Disadvantage: Longer inference time; to the time order of 5x. Solution: Groq. It reduces inference time to as low as 1x! #ArtificialIntelligence #AI

20

🌟 Exciting Development in AI Evaluation! 🌟 As AI systems continue to evolve, understanding and benchmarking their performance becomes increasingly critical. The latest Medium article by the PAIR team introduces LLM Comparator, a groundbreaking tool for human-driven evaluation of Large Language Models (LLMs). This tool empowers researchers and practitioners to compare LLM outputs side by side, enabling more nuanced assessments of their capabilities. With a focus on making AI systems more interpretable and aligned with human expectations, the LLM Comparator is a step toward responsible AI development. Here’s why this matters: ✅ Facilitates human-in-the-loop evaluations for better insights. ✅ Enhances the transparency and accountability of AI systems. ✅ Encourages collaboration between developers and evaluators to refine AI performance. Whether you're a data scientist, researcher, or simply curious about LLMs, this is a tool worth exploring. Let’s continue to shape the future of AI with thoughtful evaluation techniques! 👉 Read the full article here: LLM Comparator: A Tool for Human-Driven LLM Evaluation #AI #LLMs #ResponsibleAI #HumanCenteredAI #AIResearch

28

1 Comment

This weekend I read “Zero-shot Tokenizer Transfer” by Minixhofer et al, 2024. In this paper the authors tackle the limitation of the dependence of language models (LM) to their tokenizer. In their approach, the authors propose to “mitigate” this limitation by changing the original tokenizer to an arbitrary one such that the performance of the LM does not degrade. To do this, the authors train a network that takes the tokenizer as input and predicts the embeddings for a base LM. Once trained, the authors claim that the network generalizes to new tokenizers for different LLMs. To me this paper is interesting because studies the implications of how the units of the vocabulary at hand (tokens) have to the language model. Being able to map across representations seems to be an important computing operation in artificial and biological networks. Paper: https://lnkd.in/gU-6Ja7h

4

Learn what's new with RAGs in my latest article #LLM #RAGs #MachineLearning

8

1 Comment

👉🏼 Investigating influencing factors of learning satisfaction in AI ChatGPT for research: University students perspective 🤓 Mohammed Abdullatif Almulla 👇🏻 https://lnkd.in/eYu7c93c 🔍 Focus on data insights: - Interaction learning and collaborative learning are pivotal in fostering ChatGPT adoption. - Information quality substantially influences researchers' decisions to continue using ChatGPT. - Perceived ease of use and perceived usefulness play intermediary roles in linking ChatGPT engagement to learning satisfaction. 💡 Main outcomes and implications: - User-friendly interfaces and perceived utility are crucial factors affecting overall satisfaction levels. - ChatGPT positively impacts learning motivation, enhancing student engagement and interest in learning. 📚 Field significance: - Educational practitioners can improve the implementation of AI technologies in university students by focusing on user-friendly design and collaborative learning. - Continued research is needed to comprehensively understand the interplay between AI-powered tools, learning objectives, and motivation. 🗄️: [#ChatGPT #learning satisfaction #AI adoption #interaction learning #collaborative learning]

3

👉🏼 Expansive data, extensive model: Investigating discussion topics around LLM through unsupervised machine learning in academic papers and news 🤓 Hae Sun Jung 👇🏻 https://lnkd.in/ef_MF54v 🔍 Focus on data insights: - Utilized data from Web of Science and LexisNexis to explore topic modeling methods for Large Language Models (LLMs). - Evaluated various topic modeling approaches such as LDA, NMF, CTM, and BERTopic based on diversity and coherence metrics. - BERTopic demonstrated superior performance in diversity and coherence across both platforms. 💡 Main outcomes and implications: - News articles covered a wide range of topics related to LLM applications, focusing on specialized domains. - Research papers concentrated on technical aspects of LLM technology. - Insights gained can help understand future challenges and opportunities for LLMs in different sectors. 📚 Field significance: - Enhances understanding of how LLMs are utilized in academic and news contexts. - Provides insights for enterprises leveraging LLMs for service delivery. 🗄️: #topicmodeling #unsupervisedlearning #machinelearning #datainsights

1

👉🏼 Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation 🤓 Saubhagya Joshi 👇🏻 https://lnkd.in/eVTDV-eW 🔍 Focus on data insights: - 📊 Both ChatGPT and the CDC provided mostly accurate and understandable responses to vaccination questions. - 📈 Readability scores for both platforms often exceeded recommended levels, making information less accessible. - 🌐 Spanish responses from ChatGPT tended to be direct translations of English, occasionally leading to unnatural phrasing. 💡 Main outcomes and implications: - ✅ ChatGPT shows promise as a health information resource but needs improvement in formatting and clarity to serve users effectively. - 🏥 The study highlighted the importance of providing equitable health information across languages to enhance public health. - 🔍 Default interactions with ChatGPT can affect user perceptions significantly, underlining the need for clear and accurate responses. 📚 Field significance: - 🗣️ The findings emphasize the necessity for cross-language evaluations to ensure diverse populations receive accurate health information. - 💬 As LLMs become more integrated into health communication, addressing readability and linguistic equity is crucial for health equity. - ⚖️ Improved health information access through these tools can potentially reduce barriers for individuals facing traditional healthcare obstacles. 🗄️: [#vaccination #healthinformation #ChatGPT #CDC #healthcareaccess #equity #readability #publichealth]

1

👉🏼 RDguru: A Conversational Intelligent Agent for Rare Diseases 🤓 Jian Yang 👇🏻 https://lnkd.in/eSCm_MkQ 🔍 Focus on data insights: - 📈 RDguru outperforms ChatGPT in generating authoritative descriptions, highlighting its enhanced accuracy in knowledge Q&A tasks. - 🔍 A novel multi-source fusion diagnostic model shows a significant 5.47% improvement in diagnostic recall compared to PheLR. - 🧬 The use of advanced algorithms facilitates more effective automated phenotype annotation and differential diagnosis in rare diseases. 💡 Main outcomes and implications: - 🏥 RDguru demonstrates the potential to significantly improve clinical decision-making processes in rare disease diagnostics. - 💬 The conversational agent can enhance patient interactions through evidence-based Q&A, leading to better-informed medical consultations. - ⚙️ Integration of multi-source diagnostic strategies could pave the way for the development of more robust AI tools in healthcare settings. 📚 Field significance: - 🚀 This study positions RDguru as a promising solution for addressing the challenges of diagnosing rare diseases, particularly given the limitations of traditional methods. - 🌍 The findings may inspire further research into the implementation of LLMs in clinical environments, promoting wider acceptance and use of AI technologies in healthcare. - 🔒 Ensuring accuracy and reliability in AI outputs is crucial for safeguarding patient safety and enhancing trust in AI applications within medicine. 🗄️: [#RareDiseases #AIAssistedDiagnosis #ClinicalDecisionSupport #MachineLearning #HealthTech #NaturalLanguageProcessing #DiagnosticAccuracy]

1

RAG using tabular data with function calling - my talk from @pydatalondon is now available on youtube: The talk is scattered with references to Monty Python's Quest for the Holy Grail (tabular data is the holy grail of RAG for Enterprises). https://lnkd.in/dAkSUMiE

13

👉🏼 Using Large Language Models to Address Health Literacy in mHealth: Case Report 🤓 Elliot Loughran 👇🏻 https://lnkd.in/db46KrZv 🔍 Focus on data insights: - ChatGPT effectively reduced complex language in health-related educational materials. - Content from SmartSHOTS mobile app was adapted to improve readability. - ChatGPT simplified sentence structures to align with a third-grade reading level. 💡 Main outcomes and implications: - Improved accessibility of health information for caregivers of children aged 0 to 24 months. - Enhanced adherence to vaccination schedules through clearer educational content. - Potential for ChatGPT to be a valuable editorial tool for simplifying health-related materials. 📚 Field significance: - Advancing health literacy in mHealth through AI-driven language simplification. - Enhancing communication effectiveness in healthcare education. - Promoting better understanding of medical topics among the public. 🗄️: #mHealth #HealthLiteracy #ChatGPT #DataInsights

2

1 Comment