Trevis Litherland

Regardless of what role you work in, there are two “musts” that everybody should be cognizant of when it comes to AI: 1. All AI/ML models produce statistical inferences, which means you can never be 100% sure the outcome will be what you expect or desire. 2. All AI/ML models are trained at a snapshot in time, meaning if an AI model works well the day after it’s trained, there’s no guarantee you’ll see continued performance down the road like a year later or so. I bring these up to highlight the importance of model validation procedures. Essentially, you want to establish a framework to make ensure you’re consistently getting the desired performance from any AI solution. And it doesn’t necessarily have to come in the form of a technical solution. It’s not ideal, but human validation can be a sufficient means of validation. Regardless of how your model validation plans manifest, you still want to ensure you have some form of plan!

26

4 Comments

We've just brought out a new update of my book, Generative AI with LangChain. It covers a lot of new information and code examples with new models: https://amzn.to/3XlSYbA. If you've bought the book already, you should be able to access the updated pdf online. If not, here's your chance to get a book about applying #genai with #Python. #llms #openai #gemini

61

1 Comment

LLMs are obliterating the Old Ways of how data scientists work. And no one's talking about it. It's NOT just code copilots. It's: - more flexible OCR - faster named entity recognition - fewer training samples to do classification - faster data labeling (with human-in-the-loop) - and even faster baselines for regression (Regression? Yes, regression! Check out "From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Example" on arvix for direct predictions on synthetic data. And "Boosting Tabular Data Predictions with Large Language Models" on Medium for an embeddings-as-features approach on a Kaggle problem's test set). LLMs are doubling data scientists' productivity by reducing time-to-value across a range of problems. With this comes new challenges: - PII leakage - prediction robustness - inference costs and speed - data leakage (e.g., from public datasets) There's an emerging playbook to handle these while delivering with speed: The Switcheroo. 1. start with LLMs on a "classic ML" problem to prove value fast without much/any feature engineering (avoid models hosted on the public cloud) 2. if predictions are performant and robust enough on an internal holdout set, deploy 3. as that generates data, train a smaller model to do the same thing 4. replace the LLM to cut ongoing costs (People have been doing this with vendors since the dawn of time. Now, we're treat the LLM as a pricey vendor that lets us get up-to-speed faster.) --- It's time to move beyond the copilot paradigm. And deliver on the promise of the sexiest job of the 21st century. It is the #datascience leader's job to think through the second-order implications of new technology developments. Your business leaders want these implications with respect to: - the Strategy (why) - the Products (what), and - the core Operating Model (how) But it's equally important to think through how it changes YOUR operating model. Meaning, how your team works, and how they can work better. The world is changing, and you must change with it. This stuff's exciting and is one of the reasons I chose to lead teams rather than staying an IC (without losing my technical skills). It is a hard transition. Soon, I'll be publishing my perspective on The Multidimensional Leader. It's a framework I wish I had 5 years ago when I started leading teams. I use it daily to balance the competing priorities of my leadership role. You'll want to do the same. Follow me so you don't miss out. Good luck, #thinksenior PS - a repost is appreciated if you think someone else will find this useful!

43

4 Comments

📊 Optimizing RAG Applications with TruLens: A Quick Guide Recently explored TruLens for evaluating and fine-tuning Large Language Models (LLMs) in Retrieval Augmented Generation (RAG) applications. Here's what I learned: Key Evaluation Metrics: ✅ Groundedness: Ensures responses are factually supported by context ✅ Answer Relevance: Measures if responses directly address queries ✅ Context Relevance: Evaluates quality of retrieved information The Process: Set up automated evaluation pipelines using TruLens Track performance across different configurations Use metrics to optimize: Chunking strategies Retrieval parameters Context window sizes Embedding models Benefits: Data-driven optimization Automated quality assessment Systematic performance tracking Clear metrics for improvement Result: More accurate, relevant, and reliable RAG applications with measurable improvements in response quality. #AI #MachineLearning #LLM #RAG #DataScience #TruLens https://lnkd.in/gHQnjREi

3

The ol' adage "a picture is worth a thousand words" is very true. Take this picture ... this is explaining CLIP - CoreLogic's Property identifier, what a SSN is to you and I. CLIP is the glue between 210M+ addresses, across 176M+ structures sitting on 157M+. Indeed we track 250,000 to 500,000 changes per quarter ... new addresses, parcels split, new development. Why is it useful? Allows our clients to focus on the insight ... traversing market position, new leads, operational efficiency all the way through to risk mitigation use cases on their book of business. So easy to CLIP your book of business. Whether handling lat/long, shape files, APNs, addresses, CoreLogic takes the pain away of mastering property & location geospatially - to allow you to focus on the insight and the business decision. Took us over 2.5 years to build and has been in the market getting used across the housing industry for over 4 years. To the product, geospatial scientists and technology geniuses who created it Anand Singh Shaunak G., Matt Kjernes Frankie Famighetti Hari Krishna Kandula Chris Tackett Manny Cuevas Patrick Dodd Devi Mateti Jay Kingsley Garret Gray Lance Lambert Kristie Vainikos Stegen Robin Wachner Amy Polvado Robin Sun Massimo Mascaro Pete Lumbert Mark Weaver Bill McGinn Heidi Russell Bill Bernabei John Sikaitis This is CoreLogic 2.0

109

7 Comments

In my exploration of Agentic AI, here’s a quick summary of the key findings and challenges I came across during my test runs with Claude 3.5 Computer Use. The study, ‘The Dawn of GUI Agent: A Preliminary Case Study,’ explores Claude’s ability to operate as a GUI agent, automating workflows and performing tasks based on natural language instructions. During my testing, I observed similar strengths and challenges highlighted in the paper, particularly around planning, action execution, and error recovery. Let’s dive into these insights and lessons learned! Key Highlights: • End-to-End Automation: Claude 3.5 Computer Use demonstrated the ability to perform complex tasks by translating natural language instructions into desktop actions, showcasing potential for automating intricate workflows. • Task Diversity: The model was evaluated across various domains and software applications, indicating its versatility in handling a wide range of tasks. • Framework Development: The study introduced an agent framework that facilitates the deployment of API-based GUI automation models, aiming to simplify implementation processes. Challenges and Lessons Learned: • Planning and Execution: The model faced difficulties in effectively planning and executing multi-step tasks, highlighting the need for improved decision-making algorithms. • Error Handling: There was a lack of robust error detection and recovery mechanisms, emphasizing the importance of developing strategies for the model to recognize and correct its mistakes. • Real-World Application: The study underscored the necessity for further research to enhance the model’s performance in complex, real-world environments, ensuring reliability and efficiency. Types of Errors Identified: 1. Action Misalignment: The AI occasionally performed actions that did not align with user instructions, such as clicking incorrect buttons or entering data into wrong fields. 2. Sequential Task Failures: In multi-step processes, the model sometimes failed to maintain the correct sequence, leading to incomplete or incorrect task execution. 3. Context Misinterpretation: The AI struggled with understanding contextual nuances, resulting in inappropriate actions or responses. Proposed Solutions: • Enhanced Contextual Understanding: Implementing advanced natural language processing techniques to improve the model’s comprehension of user instructions and contextual cues. • Adaptive Learning Mechanisms: Incorporating machine learning algorithms that allow the AI to learn from past errors and adapt its behavior accordingly. • Robust Error Recovery Protocols: Developing systems that enable the AI to detect errors in real-time and initiate corrective actions to mitigate their impact. This research marks a significant step toward integrating AI agents into everyday computer use, potentially transforming how we interact with technology. Paper Link: https://lnkd.in/eXihVDTn

14

2 Comments

Truth has a new definition, it's the answer majority of agents will converge to! Agent evaluation in unsupervised conditions will be the new frontier in assessing risk.

12

1 Comment

Common Agentic workflow patterns #llm #llms #agenticworkflow

1

In my previous post, I explained how large language models (LLMs) are evaluated on benchmarks and crowdsourced evals like LMSys Chat Arena. You can find that discussion here. https://lnkd.in/eMmMD2Yz The visualization in this post presents the evolution of rankings (ELO scores) for large language models, including proprietary and open source models on LMSys Arena (over ~1y). Featured are models from OpenAI, Anthropic, Google, with open source versions from Microsoft, LMSYS, Mistral AI, Cohere, and Meta among others. This visualization shows how the landscape is evolving rapidly. Here are some key observations: ⚡Pace of Development: While open-source models are quickly catching up (in terms of innovations), this does not imply they are nearing the quality of proprietary models like GPT4 and Gemini. 🥈Performance Nuances: It’s crucial to remember that these scores represent only certain aspects of performance, not the entirety. Therefore, proprietary models might still have significant advantages. 🥇Competition: The competition between proprietary and open-source models is intensifying. 🧾 Business Models: Open-source does not mean free—it's a different business model altogether. Among its benefits: increased reputation and community feedback. 💡Research Value: These open models serve as valuable resources for the research community, helping further the development and understanding of intelligent systems. For an explanation of the costs involved in developing these large language models, check out this post: https://lnkd.in/dEE4wEV5 #LLMs #LLM_AA_TECH #AA_TECH

7

I was researching tooling options for the "transform" part of an ELT or ETL data ingestion pipeline (with emphasis on python). Here is my summary:

2

🤔 Spurious Correlation 🤔 Correlation does not always imply Causation When people look for evidence to support their arguments, they may use spurious correlation to do so. “[Ratcliffe] informed the club’s ­approximately 1,000 ­employees that email traffic dropped by 20% when one of his companies ­experimented with work-from-home Fridays, which he cited as the reason for his diktat.” This insinuation implies that e-mail traffic means more work is getting done. So people sending less e-mails means they are doing less work. I would be highly doubtful about that, “e-mail” traffic is a sign that people are sending e-mails not necessarily being more effective. However, he wants to drive cultural change and used this as his reason. How would you measure productivity?

4

1 Comment

That says it all: “Sometimes, the most impactful research isn't about creating shiny new models. It's about saying the obvious thing out loud…. even if your work doesn't feel "novel" enough, your real-world experiences matter. Don't be afraid to state the obvious - it might just be what everyone's thinking, but no one's saying! ” 👉 It is very true, and proven in successful AI practices, that we can built excellent value using AI techniques on the same use case where others claim data quality issues, lack of resources, need for elaborate DL architectures, etc etc. 👉 In the end, experience, understanding of business processes and data, creative thinking, and vision are the key ingredients for success. #AI New York University

33

3 Comments

- SwiftKV is a novel model transformation and distillation procedure (by folks at Snowflake) designed to optimize large language model (LLM) inference. Very cool findings in the paper (as presented in the vLLM meetup on 11/13). - SingleInputKV: Prefills later layers' KV cache using earlier layer outputs, reducing computation on prompt tokens. The key point is that the first few layers of attention in LLMs are similar (nice find), allowing SwiftKV's SingleInputKV mechanism to skip computations in later layers during inference. By using a single earlier layer's output to compute the KV cache for subsequent layers, it significantly reduces both computation (by 50%) and memory usage (by 62.5%) without sacrificing model performance. - AcrossKV: Merges KV caches of neighboring layers to minimize memory usage and increase batch size for higher throughput. - For models like Llama-3.1-8B and 70B, SwiftKV reduces prefill compute by 50%, memory usage by 62.5%, and achieves up to 2× higher throughput and 60% lower time per token. https://lnkd.in/guhEBW5G

62

How I Sped Up the ChatBot Arena Ranking Calculations from 19 Minutes to 8 Seconds ⚡ Have you ever used the ChatBot Arena at https://lmarena.ai? It's a site where users can have conversations with 2 anonymized chatbots and vote for which gave the better responses. The site uses all the votes to create a leaderboard of the top AI models based directly on human preferences. In my work I'm interested in LLM Evaluation, and as a hobby I work on rating systems implementations so this was right up my alley. I did a deep dive on their codebase which is open source and found a bunch of ways to help speed up the code which fits a Bradley-Terry paired comparison model to calculate the scores. This was an extremely educational experience working on this so I decided to document the technical details with a blog post: https://lnkd.in/gY8NvtEi I was able to contribute the changes to the open source repo and now the new methods are used to produce the rankings on the site used by 10s of thousands! https://lnkd.in/gkBBtNsR #ratingsystems #pairedcomparison #chatbotarena #lmarena #lmsys #elo #numpy #scipy #vectorization #parallelization #sparsity #symmetry #dontusepandas

57

5 Comments

Yep. There are fancy ways to build a prompt behind a UI, but the only variables are the model itself and the prompt.

2

If I'm not mistaken, I just solved the Oldest Unsolved Problem in Math. Veritasium, a well-known English science communicator, stated in one of his videos that the oldest unsolved problem in mathematics is: Do any odd perfect numbers exist? A perfect number is a positive integer that equals the sum of its proper divisors (excluding itself). For example, 6 is a perfect number because its divisors are 1, 2, and 3, and 1+2+3=6. Another example is 28, whose divisors are 1, 2, 4, 7, and 14, and 1+2+4+7+14=28. Recently, the GIMPS software (Great Internet Mersenne Prime Search) discovered the largest known prime number, which has over 41 million digits (as officially announced yesterday). The search for perfect numbers has often been linked to Mersenne primes and powers of 2: Mn = 2ⁿ − 1. Here's a variation method of Mersenne primes to find perfect numbers using Mersenne primes and powers of 2: - Add powers of 2 together (starting from 1, 2, 4, etc.). - When the sum is a prime number, multiply the last power of 2 you added by that prime number. - The result is a perfect number. The objective is to establish a novel framework for identifying perfect numbers. Veritasium: https://lnkd.in/e3_Nn3kM GIMPS: https://lnkd.in/e5iKTwtf Here the SOTA: https://lnkd.in/eviHqYFQ The paper: https://lnkd.in/eYCfkT6R #odd_perfect_number

5

4 Comments

Models like OpenAI's CLIP combine text and images for search, but they have limitations, particularly with longer texts and complex relationships. This article talks about JINA's new model that improves on CLIP by integrating both text and image embeddings in one model, making it faster and more efficient for multimodal search tasks. Experiments and Findings: → Search with Both Text and Images: Combining both embeddings leads to better and more diverse search results, improving relevance and variety in results. → Image-Driven Diversification: By using images for diversification, Jina-CLIP v1 can present a broader range of results, offering users more varied options. Results Comparison: → Using both text and image embeddings (via averaging) consistently outperformed searches based solely on text or images, providing more precise and diverse retrieval across both broad and fine-grained categories. https://lnkd.in/guNRQpBU

15

Valeriy Manokhin, PhD, MBA, CQF, I attended the Pydata conference back in 2016 and your recent post on the Dempster Shaffer method caught my attention. It brought to mind the valuable insights shared by Taposh Dutta Roy and Austin Powell from KP on this topic. If you're interested, you can find more details in this video: https://lnkd.in/gmegrbTb Love your posts Valeriy Manokhin, PhD, MBA, CQF keep them coming! #pydata

7

1 Comment