Chuxin Liu, PhD

A Machine Learning algorithm near to my heart: Principal Component Analysis! This dimension reduction method allows analysts identify the most important patterns in a dataset based on relative covariance. Many a nights spent on labeling faces and figuring out PCA... Special thanks to Dr. Karl Ricanek Jr. and Dr. Eric Patterson for introducing me to the method. At the end of the lecture, I discuss how PCA was used in combination to Dr. Tim Cootes' work on Active Appearance Models to generate artificial faces based on PCAs of different age demographics. https://lnkd.in/e3xf4QNS #ai #MachineLearning

1

Check out the 1st hour of Ashley Naimi's "Machine Learning for Estimating Causal Effects" on YouTube! This August 6-9 course will explore how machine learning can help minimize biases and estimate treatment effects. Subscribe to our YouTube channel for more! https://lnkd.in/emQgiZn7

6

I'm thrilled to share our new paper: Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey has been accepted by TMLR. In collaboration with Fiona Anting Tan Xi Fang Jiani ZHANG Ziqing Hu, Ph.D. Yanjun (Jane) Qi Scott Nickleach Srinivasan Sengamedu Diego Socolinsky Christos Faloutsos, we surveyed all existed methods on the LLM's application on tabular data processing, prediction, and generation. 🔗 Read full paper here: https://lnkd.in/gR2EK5pH Is there any method we are missing in the survey paper, please submit a pull request in https://lnkd.in/gXJX7Whk and we will update it in our next version. You can cite our paper using: @article{ fang2024large, title={Large Language Models ({LLM}s) on Tabular Data: Prediction, Generation, and Understanding - A Survey}, author={Xi Fang and Weijie Xu and Fiona Anting Tan and Ziqing Hu and Jiani Zhang and Yanjun Qi and Srinivasan H. Sengamedu and Christos Faloutsos}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2024}, url={https://lnkd.in/gWFBV8zk, note={} } Moving forward, I will create a series of posts about all important methods in the intersection of LLM and tabular data. Stay tuned! #AI #MachineLearning #TabularData #Research #LLM #Innovation #DataScience #ArtificialIntelligence #Tech #BigData #Analytics #DeepLearning #PredictiveAnalytics #TMLR #TableProcessing #Text2SQL #DataGeneration #SyntheticData #Prediction

187

7 Comments

Are you heading to #ICLR2024? If you're interested in causal reasoning in humans and large language models, I am happy to share my latest work. We have identified a new type of causal bias (chains vs. common causes) in humans and LLMs. By utilizing the earth mover’s distance, we show how variations in model temperature can influence this bias in LLMs. I’m looking forward to discussing our work at the Realignment Workshop (Re-Align) at #ICLR2024. ICLR

41

3 Comments

I tried to write some recent works on Advances on Large Language Models in this blog. Check out here. I gave an overview of some trending works on LLM, such as fine-tuning, prompt tuning, PEFT, and chain/tree/graph of thoughts. Some works that improve the LLM efficiency are Flash attention, Mamba, xLSTM, Deja Vu (LLM sparsity), and snapKV (KV cache approach). Also, some recent groundbreaking works like Text Diffusion, Astute RAG (Retrieval Augmented Generation), and Kolmogorov Arnold Network. I attached the paper to each blog for more details.

18

📢 New Breakthrough in Time Series Forecasting: TSMamba Model 📢 Researchers from Tsinghua University and several esteemed institutions have introduced a novel time series foundation model named TSMamba. This model leverages the Mamba architecture to tackle the complexity of time series forecasting across diverse domains. Unlike traditional models reliant on quadratic complexity architectures like Transformers, TSMamba offers a linear complexity alternative, enhancing both efficiency and scalability. Key Highlights: - Bidirectional Encoding: TSMamba incorporates both forward and backward Mamba encoders to capture time dependencies effectively. - Two-Stage Transfer Learning: To reduce the need for extensive data and computational resources, TSMamba utilizes pretrained Mamba models, allowing for effective adaptation to specific time series tasks. - Cross-Channel Attention: A compressed attention module captures dependencies across channels, enhancing predictive accuracy on multivariate datasets. Experiments show that TSMamba performs on par or even better than current state-of-the-art models, especially in scenarios with limited training data, such as zero-shot and full-shot forecasting. This approach could revolutionize forecasting across industries, from finance to healthcare, by offering more data-efficient and adaptable solutions. 📚 Read more about the research here: https://lnkd.in/d_k3sfgv #TimeSeries #Forecasting #MachineLearning

52

🚀 Enhancing Text Embeddings for Smaller Language Models 🚀 Excited to share an intriguing paper on arXiv, "Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning." This research explores how contrastive fine-tuning can enhance the performance of smaller language models like MiniCPM, Phi-2, and Gemma. The results are impressive, with MiniCPM showing an average 56.33% performance gain across various benchmarks. This method promises significant advancements in making smaller models more efficient and effective. 📄 Read more : https://lnkd.in/gMK48krE #AI #MachineLearning #NLP #TextEmbeddings #ContrastiveLearning #Innovation #Research #arXiv

6

Training LLMs to use inference-time feedback using large scale RL. Makes even the 8B Llama3.1 beat GPT-4 on CodeContests, and SOTA with the 70B. Author summary: LLMs for code should do much better if they can iterate on tests -- but they don't. Our new work (RLEF) addresses this with execution feedback at RL training time to use execution feedback at inference time. Notably, RLEF models are very sample efficient for inference. Competitive programming questions are often approached by sampling a large number of candidate programs; we can reach SOTA with just up to 3 samples. https://lnkd.in/g9_m_3Xm

18

2 Comments

Handling missing data in machine learning is a challenge, but it doesn’t have to compromise your model. In Gizem Kaya's latest article, explore how understanding the nature of missingness can help you apply the right imputation techniques.

11

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

180

2 Comments

After an unexpected hiatus, I’m pleased to announce the early release of a long overdue project: Causal Book: Design Patterns in Causal Inference. I started this project some time ago, but never had a chance to devote time to scoping it. I finally got around to it, and the first chapter is almost done.  I keep going back to it, so it might change a little more along the way. This is an accessible, interactive book for the data science / causal inference audience. Some chapters should also read well to the business audience. The book is not meant to substitute for the already great accessible books out there. The two that come to mind are The Effect and The Mixtape. Kudos to Nick and Scott for these great resources. Our goal here is to complement what’s out there by using the idea of design patterns: (1) focus on solutions to problem patterns and their code implementations in R and Python, (2) discuss the implications of different approaches to the same problem solved by modeling the same data, (3) explain some of the surprising (or seemingly surprising) challenges in applying the causal design patterns. It's a work in progress, but now that it's scoped, more is on the way. Versioning and references are up next. I will post updates along the way. Finally, why design patterns? Early in my career, I was a programmer using C# and then Java. Our most valuable resources back then were design patterns. I still have a copy of the book Head First Java Design Patterns on my bookshelf from 20 years ago. It was a lifesaver when I moved from C# to Java. This is kind of a tribute to those days. #datascience #causalinference #causalbook #datacentricity

156

4 Comments

🚀 Building Large Language Models (LLMs): Key Insights 🚀 Here’s a quick breakdown of what goes into building LLMs from what I learnt from watching the lecture from Stanford Online by Yann Dubois Architecture & Data: Large Language Models (LLMs) rely on transformer architectures, which are known for capturing complex patterns in language. High-quality, diverse data—ranging from books to web pages—enables these models to generate meaningful, accurate responses. Every bit of quality data adds to the model's overall understanding. Training & Alignment: LLMs first go through extensive pretraining, absorbing knowledge from massive datasets, which builds a general understanding of language. Post-training, or fine-tuning, aligns models with user instructions, making them responsive and more adaptable to real-world applications, from answering questions to assisting with creative tasks. Scaling & Optimization: Bigger isn’t always better—models with more parameters and data perform better but come with challenges. Efficient training, using methods like low-precision training and hardware optimization, ensures that even very large models can be trained without excessive resources, speeding up development and deployment. Human Feedback: To align with human expectations, Reinforcement Learning from Human Feedback (RLHF) is used to improve responses. This process involves human reviewers scoring the model’s outputs, which helps it learn the difference between a good and a great answer, making LLMs more intuitive and user-friendly. Link to the video: https://lnkd.in/gEWWQbCJ #AI #MachineLearning #LLMs #ArtificialIntelligence #DataScience #Tech

10

I am currently working with some of the best Machine Learning Engineers in the US who are looking for new roles. Here are some of the common themes why they are on the market: ➕ Losing confidence in the technical direction of their company ➕ Wanting to move towards a GenAI-focused role ➕ 4+ years, RSUs have vested and want a new challenge ➕ Big tech work is repetitive; they want more high-impact work ➕ Still a lot of restructures and instability ➕ Lack of long-term company vision. Short-term panic affecting work Generally, candidates are very open to startups or large companies. As long as the ML work has a high impact If you are a candidate open to new roles. Get in touch; I have clients actively interviewing this week. If you are a hiring manager, now is a good time to pick up some of the best talent. Get in touch with People In AI 🙂 📧 [email protected]

20

Understanding probability distributions is crucial for interpreting data accurately and making informed decisions. Here’s a quick guide on why they are essential in statistical analysis: 🔹 Identify Patterns: Probability distributions help us understand how data is spread across different values, which is key for predicting future events. 🔹 Risk Assessment: They are used to quantify the risk in various scenarios, enabling businesses and investors to make better risk management decisions. 🔹 Quality Control: In manufacturing, probability distributions are used to assess product quality and maintain control over the manufacturing process. 🔹 Hypothesis Testing: They are fundamental in determining the significance of data-driven insights, helping to confirm or reject initial hypotheses. Understanding these distributions can greatly enhance your analytical capabilities, whether you're a data scientist, market researcher, or student of statistics. I've created an introduction to different types of probability distributions using R programming. More information: https://lnkd.in/e98h3xB #dataanalytics #rstats #datastructure #bigdata #advancedanalytics

25

2 Comments

Understanding probability distributions is crucial for interpreting data accurately and making informed decisions. Here’s a quick guide on why they are essential in statistical analysis: 🔹 Identify Patterns: Probability distributions help us understand how data is spread across different values, which is key for predicting future events. 🔹 Risk Assessment: They are used to quantify the risk in various scenarios, enabling businesses and investors to make better risk management decisions. 🔹 Quality Control: In manufacturing, probability distributions are used to assess product quality and maintain control over the manufacturing process. 🔹 Hypothesis Testing: They are fundamental in determining the significance of data-driven insights, helping to confirm or reject initial hypotheses. Understanding these distributions can greatly enhance your analytical capabilities, whether you're a data scientist, market researcher, or student of statistics. I've created an introduction to different types of probability distributions using R programming. See this link for additional information: https://lnkd.in/e98h3xB #statistics #rstudio #dataanalytics #statisticalanalysis

10

I'm happy to finally share my latest article on Towards Data Science as part of my Machine Learning Basics Series. 😌 Writing about the K Nearest Neighbors Regressor turned out to be more involved than I initially thought. While the regressor itself is quite similar to the classifier, getting deep into nearest neighbor search algorithms like KD Trees and Ball Trees was a new challenge. I’ve spent a fair amount of time creating various visuals to illustrate these concepts clearly—hopefully helping you distinguish between the different types of trees without confusion. In this article, I aim to: - Introduce the K Nearest Neighbors Regressor and how it makes prediction - Explore the intricacies of KD Trees and Ball Trees for efficient nearest neighbor searches - Break down the concepts in a way that’s approachable, whether you're new to machine learning or looking to solidify your understanding It was a lot of tree-making, but I hope it pays off in making these ideas clearer. #MachineLearning #KNNRegressor #DataScience #KDTrees #BallTrees

21

3 Comments

About 2 months ago Li Yin didn't hold back in critiquing LLM application frameworks. Fast forward to 7/3/24, and her team has just released the alpha version of LiteRAG, with beta on the horizon(7/9/24). First Impressions? It's a nice balance between how I use instructor and DSPy. Instead of Pydantic, it uses the dataclasses, which is a standard library construct. The library also introduces DataClass as a wrapper to simplify generating schemas, adding examples, and loading JSON AND YAML. If you're familiar with Pytorch, or DSPy, then you'll feel at home composing your Components, which is a core abstraction. My favorite component as of now, Sequential, allows you to chain together your components input -> output. Definitely looking forward to seeing how this library grows! But, you don't need me to give you your opinion, Go check it out and make your own. LightRAG -> https://lnkd.in/g4cWDP88 If you want to see how I applied it to a current project, check out the video! Happy MonYay yall!

81

6 Comments

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

More opportunities

12