Kuldeep Singh Sidhu’s Post

Tables, especially when having complex layouts, contain rich semantic information. The rapid progress in natural language processing does not necessarily correspond to equivalent advancements in table parsing, which often requires joint visual and language modeling. Indeed, humans can quickly derive semantic meaning from table entries by associating them with corresponding column and/or row headers. Motivated by this observation, we propose a new heterogeneous Graph-based Table Representation Learning (GTRL) framework: https://lnkd.in/egHmV2Br

🚀 Training large-scale language models presents significant challenges due to rising computational costs and energy consumption. 🔍 Efficient optimization methods can boost AI model performance in practical scenarios. However, existing techniques have limitations, highlighting the need for more robust strategies. 🧑🔬 A group of researchers has proposed a comparative study of various optimizers to evaluate their performance and stability across different model sizes and hyperparameter configurations. This study assesses optimizers based on peak performance and stability, filling a critical gap in current research. 👉 The researchers introduce two simplified versions of Adam, named Signum and Adalayer, which retain its core benefits while isolating the effects of layerwise preconditioning. Experiments are conducted on autoregressive language models of varying parameter scales, evaluating key hyperparameters like learning rates and momentum. 🔎 Specific components of the network architecture are also analyzed to understand their impact on optimizer performance. 📊 Findings show that Adam, Adafactor, and Lion perform comparably, whereas SGD consistently underperforms. This suggests that practitioners can select from these optimizers based on practical considerations without a significant loss in performance. ✨ The research also reveals that adaptivity is crucial for certain parameters, while simpler methods like SGD can effectively train other parts of the model. #AI #MachineLearning #Optimization #LanguageModels #Research #TechInnovation https://lnkd.in/eN9QxvyP

UCLA team created Hierarchical Memory Transformers (HMT), designed to enhance long-context language processing in large language models (LLMs). Traditional transformers face limitations due to fixed context windows, which restrict their ability to handle extensive input sequences. HMT addresses this challenge by mimicking the hierarchical structure of human memory, thereby improving the model's capacity to memorize and recall information over longer contexts.   HMT employs a memory-augmented segment-level recurrence mechanism that organizes memory into sensory, short-term, and long-term categories. This organization facilitates better information selection and filtering, leading to improved performance in tasks requiring long-context understanding. The framework can be integrated into existing transformer architectures with minimal increases in parameters, ranging from 0.5% to 2%, which enhances its adaptability for future LLMs.   In various evaluations, HMT has demonstrated notable improvements compared to existing models. For instance, it achieved a 25.5% and 17.6% increase in effectiveness on the Wikitext-103 dataset when applied to the OPT and OpenLlamaV2 models, respectively. Additionally, it improved long-answer contextual reasoning by 9.81% and short-answer prediction accuracy by 1.0% in question-answering tasks using the PubMedQA dataset.   The memory recall mechanism of HMT allows for the dynamic extraction of relevant information from previous segments, which is essential for applications that require context switching. Its performance has been rigorously evaluated against benchmarks such as Wikitext-103 and PG-19, where it outperformed the Recurrent Memory Transformer (RMT) by 13% and 5.42%, respectively. Moreover, HMT demonstrates efficient memory usage, requiring less VRAM than traditional models, making it suitable for environments with limited resources.   The implications of HMT extend to the development of lifelong AI assistants capable of adapting to user behavior over time.    Arxiv: https://lnkd.in/evFenexf

what is grouped query attention? Grouped query attention refers to a mechanism in neural network architectures, particularly in natural language processing tasks like machine translation or question answering. Instead of attending to every word individually in a sequence, grouped query attention groups together similar words or tokens before attending to them collectively. This can help reduce computational complexity while still capturing relevant information effectively Here's how grouped query attention typically works: Grouping Tokens: Instead of attending to each token individually, tokens in the input sequence are grouped together based on some similarity criterion. This could be based on word embeddings, part-of-speech tags, or other linguistic features. Calculating Grouped Queries: Once tokens are grouped, grouped queries are computed for each group. These queries represent the collective attention that the group of tokens will receive. Computing Attention Scores: Attention scores are calculated between each grouped query and the keys (individual tokens) in the input sequence. This is done using a similarity measure, such as dot product or scaled dot product, followed by a softmax operation to obtain attention weights. Aggregating Attention Weights: Finally, the attention weights obtained for each grouped query are aggregated to produce the final attention distribution over the input sequences. #llama3

Cool paper that does GNN RAG! > we introduce GNN-RAG, a novel method for combining language understanding abilities of LLMs with the reasoning abilities of GNNs in a retrieval-augmented generation (RAG) style. First, a GNN reasons over a dense KG subgraph to retrieve answer candidates for a given question. Second, the shortest paths in the KG that connect question entities and answer candidates are extracted to represent KG reasoning paths. The extracted paths are verbalized and given as input for LLM reasoning with RAG. In our GNN-RAG framework, the GNN acts as a dense subgraph reasoner to extract useful graph information, while the LLM leverages its natural language processing ability for ultimate KGQA.

I’ve been seeing a lot of successful results with GraphRAG and the likes, but most of them seem to be encoding graph structure into natural language representation. That seems counterproductive to me. I am wondering if anyone here has used or has had any success with using native support for graph tokens as LLM inputs instead of using the intermediate text layer. I am guessing it would be using GNNs but most of the work seems to use GNNs only for information retrieval instead of the LLM step. I came across the Graph Neural Prompting paper which achieves something like this. It is essentially similar to LlaVa that uses cross attention with a GNN-based graph encoder instead of vision encoder. I think this is the future for graph-based approaches because it is a really elegant way of incorporating graph knowledge as a first-class LLM inputs. Theoretically this means the LLM can define what relationships to expand and explore with respect to the query by way of the weights of the message passing layers. Any thoughts or practical use cases you have come across? https://lnkd.in/dkK_Fntf

In the area of language processing, Language Models (LLM) exhibit the capability to predict subsequent words or complete missing phrases. By assessing the probability of forthcoming words and employing selection strategies like top-p or temperature settings, LLM determines the most probable word, token, or even punctuation to enhance text coherence. Delving into the intricacies of constructing an effective LLM involves understanding various components such as data acquisition, tokenizer utilization, adherence to scaling laws, and the crucial phases of training and fine-tuning (SFT) in alignment with Reinforcement Learning from Human Feedback (RLHF). The article will also touch upon the evaluation methods for the model and the generation of initial seed data. For a comprehensive guide on navigating the intricacies of developing Language Models, including insights on model evaluation and seed data generation, explore the detailed article: https://lnkd.in/gxndAXJB.

Multi-Scale Geometric Analysis of Language Model Features: From Atomic Patterns to Galaxy Structures Large Language Models (LLMs) have emerged as powerful tools in natural language processing, yet understanding their internal representations remains a significant challenge. Recent breakthroughs using sparse autoencoders have revealed interpretable “features” or concepts within the models’ activation space. While these discovered feature point clouds are now publicly accessible, comprehending their complex structural organization across different scales presents a crucial research problem. Read the full article: https://lnkd.in/ekuXxJNs Paper: https://lnkd.in/gn6Rk-ck

🚀 Solving Reasoning Challenges in LLMs with Attention Mechanisms and Why It Matters for AGI!?? In the realm of artificial intelligence, self-attention and multi-head attention mechanisms are transforming how models like GPT and BERT understand and generate language. These advanced techniques not only enhance language processing but also significantly improve reasoning capabilities, bringing us closer to achieving Artificial General Intelligence (AGI). ✨ What is Self-Attention? Self-attention allows models to weigh the importance of different words in a sentence relative to each other. This means each word can consider the entire context of the sentence simultaneously, leading to a deeper understanding of language nuances. How It Works: Input Representation: Each word is converted into a vector (embedding). Query, Key, Value Vectors: For each word, three vectors are created through linear transformations. Attention Scores: The model calculates how much focus each word should have on others by computing dot products between queries and keys. Weighted Sum: These scores are normalized using softmax and used to create a weighted sum of the value vectors, producing a contextually rich output. 🔗 Learn More About Self-Attention https://lnkd.in/dnkMHrci ✨ What is Multi-Head Attention? Building on self-attention, multi-head attention involves multiple attention layers running in parallel. This enables the model to capture various relationships and patterns within the data simultaneously, enhancing its ability to reason and generate complex concepts. How It Works: Multiple Attention Heads: The input is split into multiple sets of queries, keys, and values. Parallel Processing: Each head performs the self-attention process independently. Concatenation and Projection: The outputs from all heads are concatenated and transformed into the final output. 🔗 Explore Multi-Head Attention with Hugging Face Transformers https://lnkd.in/dgtwjdaR 🌟 Why It Matters for AGI Achieving AGI requires models that can understand, reason, and learn across a wide range of tasks with human-like flexibility. Self-attention and multi-head attention are critical in this pursuit because they: Enhance Understanding: Allow models to grasp complex relationships and dependencies in data. Boost Reasoning: Provide more coherent and logically consistent outputs, essential for intelligent decision-making. Increase Efficiency: Enable models to process information more effectively, reducing computational overhead and improving scalability. By continuing to refine these attention mechanisms, we move closer to creating AI systems that not only mimic human language understanding but also exhibit robust reasoning capabilities akin to human intelligence. 🔗 Deep Learning Book by Ian Goodfellow https://lnkd.in/djxxAwty

Want to get started in Generative AI and enjoy reading technical papers. Then here are the list of some important research articles leading to the development of the SOTA models: 1. Attention is all you need https://lnkd.in/efgn2yKY This key paper introduces Transformer architecture which are the foundation of GPTs and other LLMs 2. BERT (Bidirectional Encoder Representations from Transformers) https://lnkd.in/e-tHjguX This paper popularized the applications of Transformer models in NLP 3. T5 (Text-to-Text Transfer Transformer) https://lnkd.in/eMMvAG3P This paper presented a unified approach to NLP tasks by converting all problems into a text-to-text format 4. GPT-3 (Language Models are Few-Shot Learners) https://lnkd.in/e7tqyEiT This paper introduced GPT-3 which can perform wide variety of tasks with minimal fine tuning 5. LoRA: Low-Rank Adaptation of Large Language Models https://lnkd.in/eg4Z25qS This paper showed how to efficiently perform fine-tuning of LLMs 6. Llama 2: Open Foundation and Fine-Tuned Chat Models https://lnkd.in/ecWiiFfj This paper discusses various techniques to reduce the computational requirements of training LLMs without sacrificing performance. 7. Prompt Engineering https://lnkd.in/ePexz7Y9 This paper provides a structured approach to prompt engineering for enhancing interactions with ChatGPT and LLMs. Let me know which paper you're reading this week As always happy learning!