Adam G.

76-page survey paper on Prompting Techniques ✨ Explores structured understanding and taxonomy of 58 text-only prompting techniques, and 40 techniques for other modalities. 📌 The paper focuses on discrete prefix prompts rather than cloze prompts, because prefix prompts are widely used with modern LLM architectures like decoder-only models. It excludes soft prompts and techniques using gradient-based updates. 📌 The paper identifies 58 text-based prompting techniques broken into 6 major categories: 1) In-Context Learning (ICL) - learning from exemplars/instructions in the prompt 2) Zero-Shot - prompting without exemplars 3) Thought Generation - prompting the LLM to articulate reasoning 4) Decomposition - breaking down complex problems 5) Ensembling - using multiple prompts and aggregating outputs 6) Self-Criticism - having the LLM critique its own outputs 📌 For ICL, it discusses key design decisions like exemplar quantity, ordering, label quality, format, and similarity that critically influence output quality. It also covers ICL techniques like K-Nearest Neighbor exemplar selection. 📌 Extends the taxonomy to multilingual prompts, discussing techniques like translate-first prompting and cross-lingual ICL. It also covers multimodal prompts spanning image, audio, video, segmentation, and 3D modalities. 📌 More complex techniques like agents that access external tools, code generation, and retrieval augmented generation are also taxonomized. Evaluation techniques using LLMs are discussed. 📌 Prompting issues like security (prompt hacking), overconfidence, biases, and ambiguity are highlighted. Two case studies - benchmarking techniques on MMLU and an entrapment detection prompt engineering exercise - are presented. https://lnkd.in/d93mRWMn

3

"In this article, I’ll show how a simple change — literally adding one line of code — can transform a traditional ML model (like Random Forest, LightGBM, CatBoost, etc.) into a reliable tool for answering causal questions." Samuele Mazzanti is back with a new post on causal ML models.

88

3 Comments

Reranking Using Huggingface Transformers for Optimizing Retrieval in RAG Pipelines https://ift.tt/aN1OIbQ Understanding when reranking makes a difference Visualization of the reranking results for the user query “What is rigid motion?”. Original ranks on the left, new ranks on the right. (image create by author) In this article I will show you how you can use the Huggingface Transformers and Sentence Transformers libraries to boost you RAG pipelines using reranking models. Concretely we will do the following: Establish a baseline with a simple vanilla RAG pipeline. Integrate a simple reranking model using the Huggingface Transformers library. Evaluate in which cases the reranking model is significantly improving context quality to gain a better understanding on the benefits. For all of this, I will link to the corresponding code on Github. What is Reranking? Before we dive right into our evaluation I want to say few words on what rerankers are. Rerankers are usually applied as follows: A simple embedding-based retrieval approach is used to retrieve an initial set of candidates in the retrieval step of a RAG pipeline. A Reranker is used to reorder the results to provide a new result order that betters suits the user queries. But why should the reranker model yield something different than my already quite powerful embedding model, and why do I not leverage the semantic understanding of a reranker in an earlier stage you may ask yourself? This is quite multi-faceted but some key points are that e.g. the bge-reranker we use here is inherently processing queries and documents together in a cross-encoding approach and can thus explicitely model query-document interactions. Another major difference is that the reranking model is trained in a supervised manner on predicting relevance scores that are obtained through human annotation. What that means in practice will also be shown in the evaluation section later-on. Our Baseline For our baseline we choose the simplest possible RAG pipeline possible and focus solely on the retrieval part. Concretely, we: Choose one large PDF document. I went for my Master’s Thesis, but you can choose what ever you like. Extract the text from the PDF and split it into equal chunks of about 10 sentences each. Create embedding for our chunks and insert them in a vector database, in this case LanceDB. For details, about this part, check our the notebook on Github. After following this, a simple semantic search would be possible in two lines of code, namely: query_embedding = model.encode([query])[0] results = table.search(query_embedding).limit(INITIAL_RESULTS).to_pandas() Here query would be the query provided by the user, e.g., the question “What is shape completion about?”. Limit, in this case, is the number of results to retrieve. In a normal RAG pipeline, the retrieved results would now just be directly be provided as context to the LLM that will synthesize the answer. In many...

1

These DAnG dagitty DAGs Ever get lost in the many variables and their causal relationships in your causal analysis? Causal Directed Acyclic Graphs (DAGs) can help. A Causal DAG is a graph that provides a visual representation of the hypothesised causal relationships between variables. Each node stands for a variable, and each directed edge signifies a causal relationship from one variable to another. The acyclity in a DAG means that we assume a variable cannot cause itself, be that directly or indirectly. Why are causal DAGs so useful? -One picture is worth a thousand words: DAGs are simple, clear visual representations that are easy and intuitive for everyone to understand. -DAGs make the investigators’ assumptions explicit. Everyone can see and critique the causal model. This promotes transparency and trust in our conclusions. -DAGs can help us identify which variables should be controlled for and which ones should not be when estimating the causal effect of one variable on another. This last point is particularly useful. To accurately estimate the causal effect of one variable on another, we need to determine which variables to control for to avoid confounding and other biases. DAGs offer a clear and systematic method for making these decisions. Causal analysis without DAGs is like piecing together a puzzle in the dark; with DAGs, every piece falls into place with clarity. #CausalAI #DAGs

39

6 Comments

I’m thrilled to collaborate with Josep from DataBites to share a comprehensive guide of top resources in ML. Josep Ferrer is a veteran technical writer of machine learning having published more than 50+ articles in Kdnuggets and towardsDataScience. Together, we've curated an essential list of resources to help you build a strong foundation in ML. The guide is divided into two parts. In Part 1, we share key theoretical resources: • Top-rated online courses • Must-read books. • Popular podcasts and tutorials to get you started. These high-quality resources are designed to equip you with the knowledge and confidence to tackle the challenges ahead in your ML journey. Checkout the article here: https://lnkd.in/gVDS_Uaw Stay tuned for Part 2 on Saturday, where we'll dive into practical, hands-on learning resources, some classic projects, and advanced techniques to help you apply your knowledge and build a standout portfolio. Don't miss out! Check out Part 1 now and take the first step. -- 𝘐𝘧 𝘺𝘰𝘶 𝘭𝘪𝘬𝘦𝘥 𝘵𝘩𝘦 𝘢𝘳𝘵𝘪𝘤𝘭𝘦, 𝘢𝘭𝘴𝘰 𝘤𝘰𝘯𝘴𝘪𝘥𝘦𝘳 𝘴𝘶𝘣𝘴𝘤𝘳𝘪𝘣𝘪𝘯𝘨, 𝑙𝑖𝑛𝑘 𝑏𝑒𝑙𝑜𝑤 𝑚𝑦 𝑝𝑟𝑜𝑓𝑖𝑙𝑒 𝑝𝑖𝑐𝑡𝑢𝑟𝑒 ☝️ #MachineLearning #LearningJourney #MLResources

69

20 Comments

Our code for RAG evaluation is now publicly available at Amazon Science website: https://lnkd.in/g-HKUNZZ. This is the implementation of our evaluation mechanism performed by scoring the RAG on an automatically-generated synthetic exam composed of multiple choice questions based on the corpus of documents associated with the task. #OpenSource #OpenSourcing #AmazonScience #ICML2024 #Evaluation #LLMEvaluation

213

6 Comments

🚀 Exciting Times in Time Series Forecasting! 🚀 I recently tested TimeGPT, a foundation model with a user-friendly API, offering one-shot forecasting with minimal effort. But here's the catch—it doesn't use your entire dataset! 🤯 Curious about the details? Check out my findings on data usage and uncover the "magic numbers" for efficient forecasting in my latest blog: https://lnkd.in/gSAHZ4fv Let's dive into the future of forecasting together! 🌟 #TimeSeries #AI #Forecasting #DataScience #Innovation #timegpt #Deeplearning #TransferLearning

19

🚀 Exciting News! 🚀 I am thrilled to announce that my dataset, MMLU-tr-v0.2, has surpassed an incredible milestone with over 120K downloads on Hugging Face! 🎉 This makes it the #1 most downloaded Turkish dataset on the platform! MMLU-tr-v0.2 is a crucial subset of the 6 benchmarks used to evaluate Turkish LLMs across various benchmarks. Here's to more milestones and advancements in Turkish NLP! 🇹🇷✨ Dataset link: https://lnkd.in/d5S2zDnH #NLP #AI #MachineLearning #TurkishDataset #Benchmark #HuggingFace

59

4 Comments

Prompting BERT! Zero-shot learning ability is the hottest thing about causal LLMs. You don't need to finetune causal LLMs on each specific task. Instead, you can use prompting and get a decent performance on unseen tasks. Unfortunately, autoencoding LLMs - like our dear friend BERT 🙋‍♂️- lack this ability and you need a task-specific head for different tasks. But what if you could prompt all the BERTs in the world?! 🥁 Introducing Statement-Tuning 🥁 Now hold your horses! don't go full-LLama on it yet. Using this finetuning approach, we can get zero-shot performance from encoders by turning a problem into a yes/no problem. Binary classification all the way down! For example, a single entailment problem will be decomposed into 3 yes/no questions. This is still not super useful. But I like works that try to make a little more space for encoders in the current autoregressive era! Check the paper if interested: https://lnkd.in/ddEn7cvu

26

Read how PyTest can be used to enhance #MLOps with data validation techniques. Ensuring the integrity and quality of data is fundamental to machine learning. In his latest article, our machine learning scientist Marcello Politi shows how to effectively use PyTest for data validation, focusing on both deterministic and non-deterministic testing strategies. This approach allows for comprehensive validation of datasets, ensuring that models can handle dynamic changes in data over time. ⬇ #machinelearning #deeplearning

1

Discover how eBay created their own language model using three billion item titles in this exciting video. We'll explore the innovative techniques eBay used, including training their BERT model from scratch on eBay's massive dataset, knowledge distillation to compress the model, and fine-tuning the compressed model to learn similarity better. With a 3.5% increased purchase order rate and enhanced customer engagement, this is a must-watch for anyone interested in the power of language models. Like and subscribe for more such interesting concepts. Also, like and share over here for maximum reach. : ) Video Link: youtu.be/h51nbWr7feo YT channel Link:  youtube.com/@datatrek #datatrek #datascience #machinelearning #statistics #deeplearning #ai

9

Building a Database on S3. This is an old paper, circa 2008 - SIGMOD. But, I have been going back & forth with the hypothesis presented here. The paper explores the feasibility of utilizing Amazon S3 for general-purpose database applications. Some takeaways: - talks about s3's infinite scalability, high availability, and a pay-as-you-go pricing model, which could be an attractive option in avoiding on-prem stuff. - consistency challenges: s3 offered eventual consistency (updates will eventually become visible to all clients) - performance considerations: While S3 is cost-effective for storage, it exhibits higher latency compared to local disk storage - database kernel implementation: authors propose protocols for storing, reading, and updating objects and indexes on S3, aiming to balance scalability and availability with consistency I drilled down on the database kernel implementations (where my curiosity lies). This is what they suggest: 🌟 Client-Server Architecture: Implements a shared-disk model where clients fetch, buffer, update, and write pages from/to S3 without blocking other clients 🌟 Record Manager: Manages records within pages using CRUD operations & supporting B-tree indexing. 🌟 Page Manager: Buffers S3 pages, uses a TTL refresh protocol, and handles transactions with commit/abort functionalities. 🌟 B-tree Indexes: In terms of Access methods, it suggests implementing B-link trees on top of page manager for concurrent operations, storing nodes as S3 pages and supporting range queries. 🌟 Logging: Uses idempotent redo log records to ensure durability and consistency, with logs for inserts, deletes, updates, and secondary index mappings. The authors note that this work marks the initial step toward the ambitious vision of building comprehensive database systems on top of services like S3. But there is also the need to balance scalability, availability, and ACID guarantees to meet diverse application needs. So, how far have we come after 15+ years? Well, we do see a lot of potentials for s3. Maybe s3 by itself, cannot suffice the database kernel needs, but today s3-based data lakes have huge applications. In fact, with a #lakehouse architecture, we are actually trying to bring those database primitives and components (such as storage engine, i.e. access methods, buffer manager, lock manager & log manager) to file formats like #Parquet on S3. The difference is that now the storage formats (such as Apache Hudi, Apache Iceberg) are 'open' in nature vs proprietary DBMS-specific formats. Side notes (based on S3's latest updates): - S3 now supports 'conditional writes' using PutObject or CompleteMultipartUpload. This means multiple clients can concurrently update data in parallel across shared datasets.  - S3 increased the default bucket quota from 100 to 10,000 per AWS account. Additionally, any customer can request a quota increase up to 1 million buckets. Interesting time 🚀 #dataengineering #softwareengineering

384

26 Comments

Small Language Models Great survey on small language models (SLMs) across architectures, training datasets, and training algorithms. Analyzes 59 state-of-the-art open-source SLMs and capabilities such as reasoning, in-context learning, maths, and coding. Other discussions include on-device runtime costs, latency, memory footprint, and valuable insights. https://lnkd.in/epvkmKjx ↓ Join 85K+ AI researchers and devs so you don’t miss my weekly summary of the top AI and LLM papers: https://lnkd.in/e6ajg945

623

7 Comments

FlexAttention: Simplifying Attention in PyTorch _________________________________________________________________ FlexAttention is a new PyTorch API that combines the flexibility of PyTorch with the performance of optimized attention methods. It allows researchers to implement various attention mechanisms easily without needing to write custom kernels. Key Points: - Flexibility with Performance: -- Traditional attention methods offer high performance but limited flexibility. -- FlexAttention provides a flexible API that supports many attention variants with minimal effort. - How It Works: -- Standard Attention: Calculates attention scores using queries, keys, and values, normalizes them, and computes the weighted sum of values. -- FlexAttention: Allows for modifications to the attention scores through a flexible scoring function, enabling customization for different types of attention mechanisms. - Examples of Attention Variants: -- Relative Position Encoding: Adds positional information to attention scores based on relative positions. -- ALiBi Bias: Applies a bias to attention scores to account for position differences, enhancing model performance. -- Causal Mask: Masks future positions in the sequence to ensure the model only attends to previous positions, useful for autoregressive models. Handling Sparsity: -- BlockMask: Efficiently manages sparse attention masks, such as those used in causal attention, by creating masks that prevent unnecessary computations. Learn More: https://lnkd.in/g58pxB4A

8

While exploring various transformer-based architectures, I stumbled upon a great blog titled: "Understanding Large Language Models" by Sebastian Raschka, PhD The blog provides a concise overview of the key papers that have shaped the field of large language models (LLMs) in recent years. Sebastian mentioned the papers to read in chronological order to understand the evolution of LLM architectures and training techniques. The post starts with the original transformer paper from 2017 and covers influential works on attention mechanisms, layer normalization, and the bifurcation of LLMs into encoder-style models for predictive tasks and decoder-style models for generative tasks. It also discusses the ULMFiT approach for finetuning language models on specific tasks. The blog concludes by covering the topic of alignment, which aims to steer LLMs towards intended goals and interests. Key papers discussed include the InstructGPT approach that led to ChatGPT and a survey on parameter-efficient finetuning methods. Overall, the blog provides a well-curated reading list for researchers and practitioners looking to understand the rapid progress in LLMs over the past few years. Here's the link: https://lnkd.in/dEuq4e8p Happy Learning! #LLMs #LargeLanguageModels #GenAI #NLP #NaturalLanguageProcessing #TransformerArchitecture #ScalingLaws #ParameterEfficientFinetuning #LanguageModelAlignment #AIResearch #MachineLearning #ComputerScience

16