Zongyi Liu

A substantial piece of work from Apple led by Iman Mirzadeh, keivan alizadeh vahid, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar and Hooman Shahrokhi supports the thesis that transformer-based LLMs do not reason--they perform pattern-matching. The team from  also provides a great benchmarking tool--GSM-Symbolic for assessing mathematical and formal reasoning. This supports my anecdotal findings when I evaluated o1 https://lnkd.in/ec72fecb as well as my thoughts on the incompleteness of our assessments of AGI https://lnkd.in/ehBT3HB3

16

2 Comments

𝐌𝐮𝐬𝐭-𝐫𝐞𝐚𝐝, 𝐟𝐫𝐞𝐞 𝐛𝐨𝐨𝐤 𝐭𝐡𝐚𝐭 𝐩𝐫𝐨𝐯𝐢𝐝𝐞𝐬 𝐚 𝐩𝐚𝐭𝐡𝐰𝐚𝐲 𝐟𝐫𝐨𝐦 𝐆𝐀𝐈 𝐭𝐨 𝐀𝐆𝐈! ​ Yann LeCun has expressed skepticism about the potential of LLMs to achieve Artificial General Intelligence (AGI), citing limitations in memory, planning, and grounding in real-world understanding. However, we argue that 𝗟𝗟𝗠 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝘃𝗲 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 (𝗟𝗖𝗜)—particularly with multiple multimodal LLMs working together—offers a promising architecture to address these challenges and progress toward AGI. ​ Two key architectural innovations underlie LCI: ​ 𝐒𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐔𝐧𝐜𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐚𝐧𝐝 𝐂𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐑𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐏𝐥𝐚𝐧𝐧𝐢𝐧𝐠: Inspired by the human mind’s dual-layer structure, LCI divides its architecture into foundational and adaptive layers. The foundation layer, akin to unconscious processing, leverages extensive datasets to build robust pattern recognition, encoding essential responses much like human instincts. The adaptive layer, mirroring conscious thought, enables rapid adaptation and contextual reasoning. This dual-layer approach allows LLMs to perform complex tasks with foundational knowledge while adjusting to new information on the fly, much as a child learns new concepts within a developed cognitive framework. ​ 𝐃𝐢𝐬𝐭𝐢𝐧𝐜𝐭 𝐑𝐨𝐥𝐞𝐬 𝐟𝐨𝐫 𝐄𝐚𝐜𝐡 𝐋𝐋𝐌-𝐀𝐠𝐞𝐧𝐭: Each LLM-agent is assigned a specific role, such as executive (knowledge processing), legislative (behavior guardrails), and judicial (adapting to cultural norms). ​ Additionally, our validation of the LCI framework has been thoroughly tested through real-world deployment in applications across various domains, including healthcare, sales planning, investment, and debiasing news. These empirical deployments highlight the practical viability and adaptability of the LCI framework, showcasing its ability to enhance reasoning and decision-making through collaborative intelligence. ​ The eleven aphorisms presented in this free book illustrate the philosophical foundation of the LCI framework, backed by empirical studies. Through LCI, we envision a clear pathway to AGI, rooted in collaborative intelligence and flexible, context-aware adaptation. https://lnkd.in/gn74DkNj (free) https://lnkd.in/gszSvfqP ($6.98 printing fee)

9

The paper "Attention Is All You Need" presents the Transformer model, a novel architecture for sequence transduction based entirely on attention mechanisms. https://lnkd.in/gbe9HSzE Key IDEAS: - Traditional sequence models rely heavily on recurrent networks, limiting parallelization. - The Transformer architecture replaces recurrence with self-attention, improving training times. - Self-attention allows the model to draw connections across the entire input sequence. - The Transformer achieves a BLEU score of 28.4 on the English-to-German translation task. - Processing can occur in parallel, significantly speeding up computation. - Multi-head attention enables the model to focus on different parts of the input simultaneously. - This model can generalize to other tasks beyond translation, such as constituency parsing. - The design simplifies model architecture by removing the need for complex recurrent structures. - Positional encodings are introduced to maintain the order of sequences. - Attention layers improve the model's ability to learn long-range dependencies. - The paper emphasizes the efficiency gains over state-of-the-art models. - Transformers outperform existing methods with considerably less training cost. - The model can be fine-tuned for various language tasks easily. - Experiments showed better performance with increased attention heads up to a point. - Utilization of dropout and label smoothing during training improves performance. - The model architecture allows for easier interpretation of attention distributions. - Attention heads often learn specific tasks, such as syntactic and semantic relations. - The architecture allows adapting to larger input and output sequences efficiently. - Adequate regularization contributes to superior generalization in learning from data. - Ease of implementation on hardware accelerates research and application. Key INSIGHTS: - Replacing recurrence with attention simplifies architecture and allows for efficient model training. - Self-attention is crucial for capturing dependencies across various positions in input sequences. - The Transformer demonstrates that parallel processing can drastically reduce training time and increase efficiency. - The design showcases how innovations in architecture can redefine state-of-the-art results in machine translation. - By employing a consistent embedding across input and output, the model minimizes errors in translation tasks. - Regularization methods like dropout and label smoothing enhance the robustness of predictions. - Understanding various attention heads can provide insights into the inner workings and decision-making of the model.

7

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models. It is a modular open-source framework to streamline the research, training, and evaluation of MoE algorithms. Built upon three core principles: (i) modular design (ii) efficient training (iii) comprehensive evaluation. The paper benchmarks five state-of-the-art MoE algorithms over three different LLMs and 11 datasets under the zero-shot setting. #MachineLearning #GenAI #LLM #MoE #DataScience

53

1 Comment

I'm thrilled to announce the release of two new fine-tuned large language models (LLMs) and a groundbreaking dataset for Persian NLP. 🚀 Shotor (Llama 3 8B): Fine-tuned on the Persian Alpaca Deep Clean dataset. Paya (Aya 23 8B): Also fine-tuned on the same comprehensive dataset. These models are crafted for high-quality performance in various Persian language applications, leveraging the Alpaca Deep Clean dataset which I meticulously processed. The fine-tuning was achieved using Dora for efficiency. Explore the models: Shotor: https://lnkd.in/ddge8CAh Paya: https://lnkd.in/dT55qCH4 Additionally, I’ve released the first Persian DPO dataset for LLM alignment, aiming to set a new standard in Persian language processing. Check out the datasets: Persian Alpaca Deep Clean: https://lnkd.in/d8ejpYRf Persian DPO: https://lnkd.in/dMF2sihF Your feedback and insights are invaluable as we strive to advance Persian NLP. I invite the community to test these models and datasets, and share your thoughts. Feel free to engage and share this post to spread the word! Looking forward to your feedback and collaboration. #NLP #PersianNLP #MachineLearning #AI #LanguageModels #DataScience #HuggingFace

36

6 Comments

𝐍𝐞𝐰 𝐁𝐥𝐨𝐠 𝐏𝐨𝐬𝐭 𝐀𝐥𝐞𝐫𝐭: 𝐂𝐚𝐥𝐢𝐛𝐫𝐚𝐭𝐢𝐨𝐧 𝐓𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬 𝐟𝐨𝐫 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬🚀 I'm excited to share my latest blog post where we delve into the critical aspect of AI - Calibration Techniques for Language Models. This piece highlights why meticulous calibration is essential and various ways to get calibrated response from LLM models 𝐊𝐞𝐲 𝐦𝐞𝐭𝐡𝐨𝐝𝐬: 𝐕𝐞𝐫𝐛𝐚𝐥𝐢𝐳𝐞𝐝 𝐂𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐜𝐞  - How asking models to rate their confidence can lead to more trustworthy AI. 𝐒𝐞𝐥𝐟-𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲-𝐁𝐚𝐬𝐞𝐝 𝐂𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐜𝐞 - Using consistency in model responses to gauge confidence. 𝐋𝐨𝐠𝐢𝐭-𝐁𝐚𝐬𝐞𝐝 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡𝐞𝐬 - Techniques such as Platt Scaling and Temperature Scaling to refine probabilistic outputs. 𝐒𝐮𝐫𝐫𝐨𝐠𝐚𝐭𝐞 𝐌𝐨𝐝𝐞𝐥 𝐚𝐧𝐝 𝐅𝐢𝐧𝐞-𝐓𝐮𝐧𝐢𝐧𝐠 𝐌𝐞𝐭𝐡𝐨𝐝𝐬 - Advanced strategies to enhance prediction reliability and pinpoint uncertainty. Accuracy and confidence in AI responses are crucial, especially in critical applications. Each technique offers a unique approach to better align model outputs with real-world probabilities. For a deeper insight into these techniques and detailed examples, check out the full blog post! 👀 https://lnkd.in/d9emAMEb Do share your experiences and feedback! 💡 #AI #MachineLearning #LanguageModels #ArtificialIntelligence

15

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

Shashank Sonkar and colleagues are back with another great paper and matching GitHub repo, "Pedagogical Alignment of Large Language Models". You can read the paper on arXiv here - https://lnkd.in/enZfCJgi, and browse the code here - https://lnkd.in/gyW5hSzp. They demonstrate an interesting approach to generating synthetic data to train open weights models to behave more pedagogically, as well as developing some new evaluation metrics. I think there's lots here for those of us interested in this kind of work to learn from and build on. (If synthetic data generation is new do you, check out this post from answer.ai earlier this week - https://lnkd.in/ecsG7R4b.)

38

4 Comments

Predictions based on Machine Learning and Statistics often rely on maximal likelihood, which can trap LLMs in biases inherited from the majority opinions of training data. This problem has been substantially addressed through conditional statistics, allowing for mitigation of biases and preventing the cancellation of minority perspectives. https://lnkd.in/gMXKqwyB

7

PyTorch Profiler is an interesting tool to pinpoint bottlenecks and improve efficiency. Here's how it can help: 𝗪𝗵𝗮𝘁 𝗜𝘀 𝗣𝘆𝗧𝗼𝗿𝗰𝗵 𝗣𝗿𝗼𝗳𝗶𝗹𝗲𝗿? It’s a performance analyzer for your models. You can track: ✅ CPU and GPU activity ✅ Memory usage ✅ Execution time of operations 𝗪𝗵𝘆 𝗨𝘀𝗲 𝗣𝘆𝗧𝗼𝗿𝗰𝗵 𝗣𝗿𝗼𝗳𝗶𝗹𝗲𝗿? 🔺 𝗙𝗮𝘀𝘁𝗲𝗿 𝗠𝗼𝗱𝗲𝗹 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻: Optimize performance and reduce processing time.   🔺 𝗥𝗲𝗱𝘂𝗰𝗲𝗱 𝗠𝗲𝗺𝗼𝗿𝘆 𝗨𝘀𝗮𝗴𝗲: Minimize the memory footprint of your model.   🔺 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗛𝗮𝗿𝗱𝘄𝗮𝗿𝗲 𝗨𝘁𝗶𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Make better use of available resources like CPU and GPU.   🔺 𝗜𝗱𝗲𝗮𝗹 𝗳𝗼𝗿 𝗟𝗮𝗿𝗴𝗲-𝗦𝗰𝗮𝗹𝗲 𝗠𝗼𝗱𝗲𝗹𝘀: Especially beneficial for handling complex, resource-intensive tasks.   🔺 𝗣𝗲𝗿𝗳𝗲𝗰𝘁 𝗳𝗼𝗿 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀: Crucial for deploying efficient, scalable solutions. The code example analyzes the performance of a simple linear model. It tracks CPU and GPU usage, memory consumption, and tensor shapes. By using the "record_function" decorator, specific parts of the code are labeled for easier analysis. The collected data helps identify inefficiencies, even in basic operations, so developers can make their models run more efficiently. #pytorch #llms #ai #ml #python

47

The Power of Attention Filtering for Robust Reasoning in Large Language Models A recent study published on Arxiv explored a critical challenge in large language models (LLMs) - the impact of irrelevant information on their reasoning abilities. The paper proposes a novel method called Attention Filtering Transformation (ATF) to address this issue. ATF tackles the problem in two stages: first, it identifies irrelevant information within a given prompt, and then it filters it out to improve the LLM's focus on the core elements necessary for accurate reasoning. This research has significant implications for the development of more robust and reliable LLMs. By mitigating the influence of irrelevant details, ATF paves the way for LLMs to deliver superior reasoning performance across various tasks. It's important to remember that, like any machine learning technique, the effectiveness of ATF can vary depending on the specific problem and dataset. Further research is needed to explore how ATF can be adapted and optimized for different use cases. #ai #llm #prompt * https://lnkd.in/grre_GJH

7

Is GPT-4o good at table recognition? Not quite! OCR looks fine, but tripped by spanning cell table structure! 🤩 However, our UniTable, with only 124M param trained by self-supervision, shows great understanding! 👉 UniTable paper: arxiv.org/abs/2403.04822 👉 UniTable code: https://lnkd.in/e4a7YMxJ #MachineLearning #GPT-4o #Table #Document

85

8 Comments

We are thrilled to announce that our ICML paper, titled "Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation," is now available on arXiv. This paper presents one of the first automated, interpretable, task-specific evaluation methods for Retrieval-Augmented Generation (RAG) in Q&A contexts. For a summary of our contributions, check the following 🧵 from my co-author Laurent Callot on 𝕏: https://lnkd.in/gBF5iXaR. https://lnkd.in/g6r3Xv27 #ICML #LLMEvaluation #AmazonScience #RAG

138

5 Comments

[Call for Participation: 3 ICML paper presentations and 1 KDD Tutorial presentation] We are happy to share that our group is associated with the three ICML papers in the upcoming conference and is presenting a KDD Tutorial on Reasoning and Planning with Large Language Models in Code Development. ICML papers: [1] Collage: Light-Weight Low-Precision Strategy for LLM Training: We develop the training recipe of leveraging low-precision floating point units (FPUs) effectively provided the accounted error is properly compensated at the critical locations in the training process. https://lnkd.in/gVT8TS-x. [2] Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models: we leverage (forward pass only) zero-order methods with variance reduction techniques to enhance stability and convergence for LLM training, in need of smaller memory requirement than backprop-based methods. https://lnkd.in/gnwwanV8. [3] Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation. The team proposed a new method to measure the task-specific accuracy of Retrieval-Augmented Large Language Models (RAG). https://lnkd.in/gV_paB8c KDD Tutorial: Multi-Hop Reasoning is an emergent capability of LLM. Such reasoning capability is critical for AI assisted code development. In this tutorial we explore the multifaceted impact of LLMs on code development, surveying techniques for generating high-quality codes, automatically documentation creation, code debugging, code decomposition, creating comprehensive test cases, among others.

112

3 Comments

I’m thrilled to collaborate with Josep from DataBites to share a comprehensive guide of top resources in ML. Josep Ferrer is a veteran technical writer of machine learning having published more than 50+ articles in Kdnuggets and towardsDataScience. Together, we've curated an essential list of resources to help you build a strong foundation in ML. The guide is divided into two parts. In Part 1, we share key theoretical resources: • Top-rated online courses • Must-read books. • Popular podcasts and tutorials to get you started. These high-quality resources are designed to equip you with the knowledge and confidence to tackle the challenges ahead in your ML journey. Checkout the article here: https://lnkd.in/gVDS_Uaw Stay tuned for Part 2 on Saturday, where we'll dive into practical, hands-on learning resources, some classic projects, and advanced techniques to help you apply your knowledge and build a standout portfolio. Don't miss out! Check out Part 1 now and take the first step. -- 𝘐𝘧 𝘺𝘰𝘶 𝘭𝘪𝘬𝘦𝘥 𝘵𝘩𝘦 𝘢𝘳𝘵𝘪𝘤𝘭𝘦, 𝘢𝘭𝘴𝘰 𝘤𝘰𝘯𝘴𝘪𝘥𝘦𝘳 𝘴𝘶𝘣𝘴𝘤𝘳𝘪𝘣𝘪𝘯𝘨, 𝑙𝑖𝑛𝑘 𝑏𝑒𝑙𝑜𝑤 𝑚𝑦 𝑝𝑟𝑜𝑓𝑖𝑙𝑒 𝑝𝑖𝑐𝑡𝑢𝑟𝑒 ☝️ #MachineLearning #LearningJourney #MLResources

69

20 Comments

torchtune: Easily fine-tune LLMs using PyTorch https://bit.ly/4bfQbWM

61

🚀 I'm happy to announce that our paper "Semi-supervised learning and visual transformers for product attribute extraction from e-commerce images" has been accepted at the CVPR 2024 Workshop on Fine-Grained Visual Categorization. Thanks to co-authors Manuel Lagunas, Víctor Martínez Gómez, Virginia Fernandez Arguedas, Christos Georgakis, Sofia Braun, Felipe Bertrand for the collaboration! 🙂 If you are attending CVPR later in Seattle, let's connect! Extracting product attributes from images involves classifying subtle differences between similar objects, like knowing the difference between a plaid pattern and a gingham pattern. Visual Transformers (ViT) are powerful but usually supervised, while e-commerce labels are noisy or expensive to clean. In this work, we demonstrate the benefit for e-commerce of semi-supervised techniques like Semi-ViT, a ViT model fine-tuned with unlabeled data. We demonstrate that when Semi-ViT is compared to fully supervised ResNets and ViTs, it improves precision by at least +107bps and coverage by+311bps even when trained with 25% less labeled data. #amazonscience #cvpr

128

3 Comments

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4