Juan Camilo Bustos Peláez

Hi everyone, I wanted to share with my brief write up about basic optimization algorithms used in deep learning. I start by giving the convex optimization backgrounds where these algorithms were conceived. Then following the amazing ML book by Shai Shalev-Shwartz and Shai Ben-David, I discuss the intuitive interpretation of Gradient Descent and SGD. Lastly, I do a deep dive of the absolutely amazing ICML 2013 paper by Ilya Sutskever & Hinton  ( “ On the importance of initialization and momentum in deep learning.”) which gives wonderful re interpretation of the classical momentum algo ; re -derives the Nesterov Accelerated Gradient (NAG, 1983) as a form of momentum and talks about their different experimentations regarding how these algos vs SGD vs 2nd order approximate Newton methods fare for deep learning, and how they deal with high curvature. Thus overall, this blog is supposed to be a refresher for the basics, before we dive into the newer, faster algorithms like Adam & other Adaptive algorithms in the next article. I would like to thank my Department of CSE, IIT Hyderabad professors, especially Vineeth N Balasubramanian Sir who taught us 2 courses - ML & deep learning and Saketha Nath Jagarlapudi Sir who taught us convex optimization theory and convex optimization algorithms, without whom I wouldn't have the necessary foundations to undertake this exploration. #deeplearning #optimization #gradientdescent #sgd #momentum #nesterov #nag #hinton #paperreview

14

2 commentaires

Learn why propensity-score matching is essential for causal inference workflows, and how to get started with it in Python — Ari Joury, PhD's concise guide will set you on the right path.

13

New photo, new post! My last post on interpretable time-series modelling using Gaussian processes proved to be reasonably popular, so I decided to write a follow-up. This time, we implement the same workflow, but in the probabilistic programming language Stan for full Bayesian inference. While the last post aimed to teach GPs by building them from scratch, in practice, we would usually use a tool such as Stan for fitting them. In this article, I walk through the structure of a Stan model and some of its coding conventions, show you how to fit the same model as the last post, show you how to easily produce and visualise predictions, and also provide a few examples of model diagnostics and parameter inference. I have one more post planned to conclude the series, which will explore how GPs can be extended to model non-Gaussian outcomes, such as counts (i.e., using a Poisson likelihood). Please stay tuned for that! https://lnkd.in/gRWvdV6S #statistics #timeseries #bayes #gaussianprocesses #stan #machinelearning

18

These days I’ve been in Montpellier, France, to present our last research results on combinatorial optimization problems over temporal graphs to the 31st symposium on temporal representation and reasoning (TIME2024). Check out our paper at https://lnkd.in/da66ref4 Here we discuss a new heuristic-based approach for minimum timeline cover over temporal graphs. The paper provides theoretical results to a problem largely adopted in social network analysis as interpretable network summarization framework to find important events. Particularly, we used it to analyze job postings, to find the most required skills by companies and understand how the market is evolving. We are already working on evolutions of this approach with Deep Learning. Stay tuned for next publications. #knowledgegraph #combinatorialoptimization #networksummarization

15

1 commentaire

Our latest preprint on Adaptive ML is now available on arxiv 💥 ! In this paper entitled “Evaluating the Efficacy of Instance Incremental vs. Batch Learning in Delayed Label Environments: An Empirical Study on Tabular Data Streaming for Fraud Detection”, we question the superiority of instance incremental algorithms (in terms of predictive performance/interpretability) when we take into account real conditions such as label or feedback delay, class imbalance 😑.  Our results using real world fraud data (as well as generated data widely used in academia) clearly invite researchers to rethink the design and evaluation of supervised incremental machine learning algorithms, with the aim of narrowing the gap between pure academic research and industrial applications (for example, in fraud detection). link: https://lnkd.in/eJdQYgQM code: https://lnkd.in/eWrRvHP9 Thanks to my PhD advisors and co-authors Mustapha Lebbah, Hanene Azzag, Grégoire Jaffre and Zaineb Chelly Dagdia. #FraudDetection, #DataStreams, #IncrementalLearning

44

𝐁𝐫𝐢𝐥𝐥𝐢𝐚𝐧𝐭 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐟𝐨𝐫 𝐢𝐧𝐭𝐞𝐫𝐚𝐜𝐭𝐢𝐯𝐞 𝐚𝐧𝐚𝐥𝐲𝐬𝐢𝐬 𝐚𝐧𝐝 𝐌𝐋 𝐚𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 The architecture is simply brilliant - e2b runtime - LLM middle layer- - notebook style rendering Massive applications. - fully scalable. - open source - customize - build - integrate Next up - prompt-driven iterative analytics and machine learning. Now exploring ways to incorporate similar solution into REX (rex.tigzig.com / realtime.tigzig.com - my open source analytics toolkit), to provide interactive analysis and machine learning. Check out the video demo on X (I actually saw it on X first): https://lnkd.in/gQXqMRiZ App & Code base : https://lnkd.in/eGmyXh-q

2

The Impact of Methodological Choices on Machine Learning Portfolios Studies using machine learning techniques for return forecasting have shown considerable promise. However, as in empirical asset pricing, researchers face numerous decisions around sampling methods and model estimation. This raises an important question: how do these methodological choices impact the performance of ML-driven trading strategies? Recent research by Vaibhav, Vedprakash, and Varun demonstrates that even small decisions can significantly affect overall performance. It appears that in machine learning, the old adage also holds true: the devil is in the details. https://lnkd.in/dvt--MGE

1

Check out my latest talk on "Peak Performance in Python" at this year's PyConDE Open to feedback!

19

Markov Chain Monte Carlo (MCMC) methods are powerful algorithms used to solve complex probabilistic models. MCMC methods allow us to sample from difficult-to-define probability distributions when direct computation of a distribution is impossible due to its complexity or high-dimensionality. In the insurance industry, MCMC has become increasingly popular for risk modeling and prediction. Actuaries often need to estimate the likelihood of rare, high-impact events (like natural disasters) or determine reserves under uncertainty. In this post, I introduce the Metropolis-Hastings algorithm with an implementation from scratch in Python. I also show how MCMC can be leveraged to estimate size-of-loss curves for a sample set of losses. https://lnkd.in/gT7SttAD

43

1 commentaire

How do machine learning models make decisions? In Mythili Krishnan's latest article, dive into decision trees and random forests to uncover how influential variables shape model results.

10

1 commentaire

It's fascinating to see how far AI has come in processing complex financial data! 🏦📊 Claude 3.5 Sonnet's ability to accurately extract detailed information from intricate charts is a testament to the rapid advancements in AI. This precision will undoubtedly revolutionize financial analysis and reporting. I'm curious about the level of precision Claude 3.5 Opus might achieve, given the significant improvements in Claude 3.5 Sonnet. These developments are exciting and hold great potential for enhancing our analytical capabilities in finance and beyond! 🚀🔍 #AI #MachineLearning #FinancialAnalysis #DataScience #Claude3.5 #TechInnovation #ArtificialIntelligence #FinancialReports #Innovation

3

Fine-tuning LLMs: Aligning models with specific customer needs is crucial for real-world success, yet it requires precision to ensure the model adapts effectively without compromising quality. Fine-tuning often introduces risks, such as underfit or overfit layers, which can lead to unpredictable behavior in production. Tools like WeightWatcher are essential for addressing these challenges, offering a data-free diagnostic to evaluate model quality by analyzing the spectral properties of weight matrices. Grounded in the Heavy-Tailed Self-Regularization (HT-SR) theory, WeightWatcher provides insights into layer performance and potential issues, helping developers optimize instruction-tuned models like Llama3.1 and Falcon2 for robust, production-ready use cases. #InstructionFineTuning #LLMEngineering #ModelQuality #CustomerFocusedAI

2

Let me present you the 2nd edition of "Handbook of Anomaly Detection" in response to the demand for the 1st edition. This book is special to me on its dedication page to Eileen Kuo: "To our beloved daughter, Eileen, You are the blossom that brightens us with every smile, and fills our lives with beauty, joy and love. Mom and Dad" I write with care and love, and I believe you will read with joy and smile. The second edition is an expanded edition including: - Comprehensive and in-depth coverage - Enhanced explanations and visual presentations - Supervised and unsupervised learning techniques - 200+ data science Q&A Check it out! Amazon print book: $65 https://a.co/d/c9Yeqph The print edition adopts glossy cover, color print, and the beautiful Springer font and layout for pleasing reading. Its 7.5 x 9.25 inches portal size goes with most of your books in your bookshelf. Sample eBook chapters: free https://lnkd.in/eG8HSChZ eBook on Teachable: $22.50 https://lnkd.in/eeMmwpFz This eBook is in a beautiful format for a pleasing reading experience. We have chosen the Teachable platform for the distribution of the eBook without hefty overhead and the distribution to readers around the world. The credit card transactions are managed confidentially and securely.

17

2 commentaires

Thank you for sharing this fascinating prediction and breakdown! 🌟 Your focus on multi-attribute vector indexes that combine tabular and unstructured data strikes a chord with the growing need for more adaptable and context-aware retrieval systems. 🚀 I completely agree with your assertion that tabular data dominates many practical applications like search, analytics, and retrieval systems. Leveraging structured data alongside unstructured formats (text, images, audio) for semantic search is a natural evolution and addresses a significant gap in current retrieval technologies. 🔍 Your emphasis on avoiding text-to-SQL is a refreshing perspective. Instead of adding layers of complexity, building semantic capabilities directly into multi-attribute vector spaces offers a more streamlined and intuitive solution for search and retrieval systems. 💡 The step-by-step guide for creating a vector index, particularly for e-commerce products, is spot on. Treating each table row as a node and focusing on vector similarity spaces between attributes like product descriptions, prices, and reviews is a practical and scalable approach for enhancing search relevance. 🛍️✨ The ability to customize searches by weighting semantic attributes is a game-changer. This flexibility enables businesses to optimize retrieval results based on user preferences (e.g., balancing affordability with quality), which is crucial for improving user satisfaction and conversion rates. Your use of Superlinked and MongoDB Atlas demonstrates how existing tools can be leveraged effectively to create sophisticated retrieval systems. It’s inspiring to see how these ideas can be implemented with accessible technologies. Thank you for sharing these insights! I’m excited to see how your prediction plays out in 2025 and beyond. Would love to dive deeper into the weighted attribute approach you’ve hinted at for a future discussion! 🙌 #RAG #SemanticSearch #AIInnovation #VectorIndexes #DataEngineering #FutureOfSearch

7

💡 Great insights on fine-tuning LLMs using Sparse Mixture of Experts (MoE)! This method optimizes performance by activating only specific parts of the model, which can significantly enhance efficiency and cost-effectiveness. 📊💸 The study’s focus on training efficacy, runtime metrics, GPU utilization, and cost modeling is crucial for managing cloud expenses and improving resource allocation. 🔍🚀 It’s exciting to see advancements that help us get more out of our compute resources and budget accurately. A valuable read for anyone involved in AI research and development! 🙌 #AI #MachineLearning #LLM #MixtureOfExperts #TechInnovation #CostEfficiency #AIResearch

2