Keith Toney

Whoever claims building RAG systems is easy has never built one at scale. 🤯 There is a significant gap in the literature and existing frameworks. RAG architectures are designed for POCs, not for hundreds of thousands of documents or supporting tens of thousands of users. Designing these systems requires a cross-functional team with software engineering and data science skills blended with a product mindset. Earlier this summer, I spoke at the Databricks AI Summit, highlighting guiding principles we are taking at CVS Health for building enterprise-grade RAG systems. Every company is unique, and while there is no one-size-fits-all solution, here are three principles that will help you on your journey: 👉 Spend time on discovery—It's tempting to dive in and start building immediately, but you may get lost in the desert without doing your due diligence and understanding what delivers the most ROI. Our team spent approximately a quarter on discovery. 👉 Design for modularity—The LLM space is changing rapidly, and you want to build reusable components that can easily be swapped out or upgraded when something better comes along. 👉 Humans + Machines—Technology alone can not solve the knowledge search problem. Develop processes and governance to help teams curate the data, which will be ingested into your system. These lessons are just the tip of the iceberg. Watch my full Databricks presentation and leave your comments and questions below to learn more tips and tricks. #RAGSystems #EnterpriseAI #DataEngineering #MachineLearning #AIArchitecture #AI #DataScience #LLM #SoftwareEngineering #AIAtScale #KnowledgeSearch #TechLeadership #AIFrameworks #AIInnovation #DatabricksAISummit #ProductMindset #CVSHealth

62

5 Comments

I had a great time at the Tableau Conference 2024 (TC24) in San Diego this week. I'm excited about the continuous innovation within the product and the overall 'buzz' in the space of data, analytics, and AI. For those of you who aren't as familiar with Tableau, I thought I'd summarize a few of the areas that stood out. Building a COE - Tableau Blueprint Blueprint is a methodology that organizations can use to assess their current data strategy and identify areas for improvement. Tableau has introduced a new AI tool to facilitate the assessment process and gap identification in data strategy implementation. Accelerators Ready-to-use dashboards that you can combine with your data and customize to fit your needs, to help you get to data-driven insights faster. My teams have leveraged Accelerators with fantastic results, rapidly delivering value and reducing costs while elevating our UX designs. Tableau Pulse Powerful newer product that takes Tableau Metrics to a whole new level with narrative and AI. Designed to democratize data analytics, it provides personalized insights to business users who aren't traditional data experts. The result is more intuitive and actionable insights for business users, that even integrates with apps like Microsoft Teams. Accessing On-Premises and VPC Data with Tableau Cloud Options for securely accessing external data from Tableau Cloud now include Direct Connect, Tableau Bridge, the newly launched Data Connect, and the upcoming Private Connect feature, each tailored to different security, manageability, and scaling needs. Tableau Bridge and the new Data Connect offer secure, scalable solutions for connecting to on-premise data without public internet exposure, with Data Connect providing Tableau-managed operations. It’s good to know there are several options to securely access data and maximize the value of your Tableau investment. AI - Einstein Copilot for Tableau Enables anyone to uncover valuable insights from their data and maximize their use of Tableau. Provides an AI assistant (available now in ‘beta’) that supports users throughout their data exploration and visualization journey, helping them create visualizations, generate calculations, and format dashboards through natural language commands, with a focus on increasing accessibility for users of varying expertise levels. Prioritizes data security and privacy with zero data retention, data masking, toxicity detection, and an audit trail for InfoSec teams. In my view, this is really the future of work, as it accelerates development and training, and enables SMEs to develop powerful data insights on their own. And on and on… There was a myriad of other innovations and great presentations – including in the keynote where Sophia G. introduced a great new data management cockpit. If your data, analytics, and AI capabilities aren't driving actionable insights and data-driven decisions, it might be time to explore Tableau. #Data24 #TC24 #insurance

13

If you haven't yet, make sure to read the latest article from McK on optimizing your generative AI strategy & implementation roadmap moving away from the Honeymoon & Pilot phase. Here are some key takeaways:   Eliminate the Noise: Focus on what’s truly effective. Be honest about successful pilots and reduce unnecessary experiments. Channel your efforts into solving critical business problems.   Holistic Approach: It’s about integration, not just individual components. Prioritize how the parts of a gen AI engine work together securely, rather than assessing them in isolation.   Cost Management: Models are only about 15% of the overall cost of gen AI applications. Identify hidden costs and apply the right tools to manage them efficiently.   Tool and Tech Proliferation: The abundance of infrastructures and tools can hinder scaled rollouts. Focus on the capabilities that best serve your business, leveraging cloud services while maintaining flexibility.   Build Value-Driven Teams: Success at scale requires teams with diverse skills to not only build models but also ensure they deliver value safely and securely.   Prioritize Data: Seek out the most impactful data and invest in its management for quicker scaling, rather than chasing perfect data.   Reuse Code: Reusing code can boost development speed for generative AI use cases by 30 to 50 percent.   Dive into the full article to get more insights!   #AI #GenerativeAI #BusinessStrategy #CostManagement #DataManagement #TeamBuilding

21

Simplifying a model doesn't necessarily trivialize it. A former Capital One executive once challenged my data & analytics team to make predictive models relevant by rendering statistical models relatable. The concept was to reduce the overall quantity of independent variables and describe in plain-English language (and with simplistic charts) how each impacted the dependent variable. His thesis was that models are abstractions of reality and unless explainable managers would be reluctant to use them to make business decisions (and furthermore: the smaller quantity of variables reflected the fact managers control a limited extent of factors). Statisticians rightly complained that limiting the quantity of dependent variables impaired accuracy and sensitivity. But when carefully parsed we learned the models remained directionally correct (very strongly so) with tolerable loss of fidelity. The overall result was managers confidently understood the relationships and trade-offs between models’ “levers” and the effect on forecasted performance.

15

2 Comments

Excited to try this one out! I'll be speaking at Minnesota Technology Association Tech Connect in a few weeks on Open Source Large Language Models and will discuss this and many more. Take read into why Snowflake's Arctic should be a model you explore with Streamlit. I'll give you a hint. "Arctic is on-par or better than both Llama 3 8B and Llama 2 70B on all metrics, while using less than ½ of the training compute budget." Look for covering in the upcoming Applied Artificial Intelligence Weekly News issue! https://lnkd.in/gM-wECeu

16

Everything everywhere all at once - that’s what #GenAI is and will continue to be like that given the exponential pace of innovation! All leaders within the AI space are knowingly or unknowingly driving people, process and tech transformation. It’s fun, it’s indeed interesting and a bit tough given unknown unknowns (Ajay Agrawal). Here’s some of my views on current #AI wave.

132

9 Comments

Lets be clear - LLMs are new software infrastructure. The path to creating value remains still as hard and will take as long as usual. You still need to get these things right and AI won't accelerate it much, - finding right use cases, - building great customer experience, - getting the pricing strategy right, - executing and iterating on GTM, - creating the right org with the right kind of people, and - building a smooth delivery experience Its still a 5-15yrs of hard work.. no amount of VC money will help accelerate it.

44

4 Comments

The thirst of data in all organizations is insatiable. While everyone argues over which LLM is the best, pragmatic data leaders are and should be concerned about how to serve data and semantic intelligence to these thirsty models. It’s time to talk about platform interoperability. That is the surest answer to this predicament that every CDAO finds him/her/them in. It’s a topic that does not get as much of a voice as it should.

46

6 Comments

Ever been in the middle of a negotiation and the data you built your models from changes? I have - and it was that experience (more than once) that prompted our Itemized adjusters - a very robust, flexible feature - for scenario modeling, adjustments for the unknown, and on the fly-source data corrections without delay. Let me know if you want to see how it, and we, can help you!

12

2 Comments

McKinsey released this article on GenAI - https://lnkd.in/gXxvmgTn 65% of companies now use #generativeAI regularly, but many have not seen significant impact yet. Companies need well-defined #operatingmodels to scale #genAI effectively, focusing on #datamanagement, #governance, and #compliance. Implementing #genAI in components allows for flexibility and alignment with business goals, though only a few companies have adopted this model. Q: Where are you in your #journey to #genAI adoption and what #benefits have your realized? Quinte Financial Technologies #dataanalytics #banks #creditunions #financialinstitutions

8

Statistical significance, confidence intervals, and p-values are fundamental concepts in statistical analysis, each serving distinct but interconnected roles in understanding and interpreting data. This article provides an intuitive explanation of these concepts using an example from pharmacovigilance. #statistics #dataanalaysis #learning https://lnkd.in/gvhuwPG2

40

3 Comments

The ol' adage "a picture is worth a thousand words" is very true. Take this picture ... this is explaining CLIP - CoreLogic's Property identifier, what a SSN is to you and I. CLIP is the glue between 210M+ addresses, across 176M+ structures sitting on 157M+. Indeed we track 250,000 to 500,000 changes per quarter ... new addresses, parcels split, new development. Why is it useful? Allows our clients to focus on the insight ... traversing market position, new leads, operational efficiency all the way through to risk mitigation use cases on their book of business. So easy to CLIP your book of business. Whether handling lat/long, shape files, APNs, addresses, CoreLogic takes the pain away of mastering property & location geospatially - to allow you to focus on the insight and the business decision. Took us over 2.5 years to build and has been in the market getting used across the housing industry for over 4 years. To the product, geospatial scientists and technology geniuses who created it Anand Singh Shaunak G., Matt Kjernes Frankie Famighetti Hari Krishna Kandula Chris Tackett Manny Cuevas Patrick Dodd Devi Mateti Jay Kingsley Garret Gray Lance Lambert Kristie Vainikos Stegen Robin Wachner Amy Polvado Robin Sun Massimo Mascaro Pete Lumbert Mark Weaver Bill McGinn Heidi Russell Bill Bernabei John Sikaitis This is CoreLogic 2.0

109

7 Comments

Tired of chasing elusive ROI for your data and analytics initiatives? Gartner's latest research reveals a surprising truth: focusing too narrowly on individual project ROI can actually hinder your organization's overall performance. Key takeaway: To unlock the true value of data, shift your focus from isolated metrics to the broader impact on your company's strategic goals. By blurring boundaries and aligning D&A efforts with business objectives, you can drive significant improvements. Let's discuss how your organization can avoid the "death by ROI" trap and maximize the potential of data. Read more here: https://lnkd.in/efDTcVDq #DataAnalytics #Leadership #BusinessStrategy #ROI #Gartner

5

Value-add risk insights and decision support at your fingertips. Data straight to analytics. The future is here. If you want to know more, I will be at RIMS next week. #WTW #Riskanalytics

23

1 Comment

Snowflake Agrees to Acquire Open Data Integration Platform, Datavolo. Empowering data engineers with hybrid, multimodal pipelines, and open source flexibility. Connectors incoming!

50

Why Analytics Feels Like a Pain (and How to Fix It) I was working through Medium and found this article by Tim Wilson of Facts and Feelings. It was a breakdown of analytics pains as identified by survey. Here are some high level takeaways. Let’s be honest—analytics can be a real grind. Whether you’re a data analyst or a business user, the challenges are real, recurring, and frustrating. But here’s the good news: the pain points are solvable. 🎯 Here’s what analytics teams are struggling with: 1️⃣ Unclear Goals, Undefined KPIs 69% of analytics teams feel the pain of being asked to deliver results without clear targets. Solution: Start with clear, measurable business objectives (not just data dumps). 2️⃣ Messy Data 68% struggle with data quality issues, and 56% find themselves constantly explaining inconsistencies. Solution: Streamline your data collection. More isn’t better—it’s often just more mess. 3️⃣ The Myth of Self-Service 72% of analysts say self-service tools go unused, while 91% of business users complain about data access. Solution: Self-service isn’t just about tech—it’s about training and alignment. 4️⃣ Lack of Actionable Insights 73% of business users say they don’t get actionable insights, while analysts feel stuck in reactive mode. Solution: Analysts need to shift from “answering questions” to driving business outcomes. 💡 The bigger picture: Analytics isn’t just about collecting and visualizing data—it’s about reducing friction between teams, clarifying goals, and focusing on actionable insights. Stop being “data-driven” and start being decision-driven. Build workflows, not just dashboards. Fix the systems, not just the symptoms. Check out the full article here: https://lnkd.in/gps8RNWW #Analytics #DataDriven #BusinessStrategy #ActionableInsights #DataQuality

2

One common denominator among the companies that will rise vs those that chase…is AI. Every Board of Directors and Leadership teams should be learning AI, more specifically, how to map AI to their business strategy… And in many cases, this requires new strategy or potential change in direction…for this reason, many executives who have been in their seats for some time, aren’t willing to get uncomfortable, though their organization’s growth and people depends on it… Some understand this moment and others will pay the price when they learn of its potential and opportunity late in the game. If you’re a leader and preach “growth mindset” this is your opportunity to walk the talk… Your teams and mentees are watching, and most if not all count on you to steer them in the right direction. Be a TRAILBLAZER 🔥🔥🔥 Don’t get frustrated, get educated. #ai #letsgo #winning #dreamforce #df24 #automation

1

Despite the challenges of cost pressures and catastrophic events, insurers are leveraging advancements in data and analytics, as well as innovative solutions, to effectively identify low-risk properties within the same ZIP code. My colleague, Nathan Golia, Senior Analyst on Celent's P&C Insurance team, dives into how these cutting-edge technologies are empowering insurers to make informed decisions and maintain their position in risk management. Learn more > https://lnkd.in/eVfMxMcr

9