Olof Jacobson

AI & ML is where all the fuzz is if you work in a data team. But most data teams have only just scratched the surface. dbt surveyed thousands of data professionals in their '2024 State of Analytics Engineering Report'. They found that only a third of respondents currently manage data for AI model training today. But 55% expect to soon. In this post, I've looked into the AI & ML landscape if you work in a data team, including the importance of building on top of solid foundations → State of AI in data teams → AI and ML use cases  → Data quality in AI and ML systems → The five steps to reliable data https://lnkd.in/e-6JTHQm

21

3 Comments

Over the years, I've learned a few hard lessons about scaling data teams that I wish I'd known earlier. Here are my top tips for building a team that actually delivers results: 1️⃣ Adopt a "hire for attitude over aptitude" philosophy; individuals with broad skills and enthusiasm can be more valuable than narrow technical specialists. The data space is evolving, and you need someone passionate about learning new tools 2️⃣ Proactiveness is key. Your data team doesn’t just work with data; they need to understand business and proactively engage with the business teams to discover opportunities and increase data adoption. 3️⃣ Get stakeholder buy-in early on or find a champion/sponsor before you start any data projects. Too often, I’ve seen dashboards built but rarely used. Early stakeholder buy-in ensures your work has an impact. 4️⃣ Always have a document for any data project, and more importantly, make sure everyone contributes and keeps it up to date. Trust me, it will save you countless headaches down the road. 5️⃣ Say no to ad-hoc requests. When you can’t say no, teach stakeholders how to answer those questions by themselves next team, or try to create an automated solution (those one-time requests will likely come back later). 6️⃣ If your data team is embedded within functional teams (e.g., marketing analyst, product analyst), have regular knowledge-sharing sessions and make each person an “expert” in a specific domain or tool so they know who to turn to for help. Scaling a data team is a wild ride, but with the right strategies, you can build a powerhouse that drives real business impact. What are your biggest challenges in scaling your data team? Let's swap stories and tips in the comments below! 👇 #dataleadership #datateams #scalingteams #5X

67

2 Comments

Having an ML-based optimisation model is great but a huge part of the success is nailing down what the model should optimise for - especially in domains like pricing where the goal is not that straightforward. And what if the answer is different per product, season of the year or temporary agreements with stakeholders and your current business strategy? How do you translate that into numbers that a model can digest? Our answer is a separate target variable outside of the model. Decoupling the target variable has many advantages: it means the strategy can change and you don't have to do anything to adjust the model. It also means you are forced to agree with others on what the ideal outcome should be and hand over the control what the model does to people who are the most suitable instead of the ones who built it. Read more about how we did it in Otrium here: https://lnkd.in/eR7sMWRe Thanks Nikki Bueno de Mesquita for being the best person to bounce ideas off and helping to shape this :) And Jorne, Francesca, Diana and Robert for thoroughly beta-reading the article.

48

5 Comments

Given my experience with LangChain, building PDF document chunks and generating embeddings through the Palantir Build Pipeline is straightforward. Before you learn this tool, learn the following: Chunks, Embeddings, VectorDB, Similarity Search, Ranking, Prompt, etc in LangChain. Of course, Pyhton is your AK47 Palantir Technologies

10

ONE TRICK SO POWERFUL DATA TEAMS WANT IT BANNED ... Who remembers the early days of clickbait? In this case we actually have something powerful I’d like to share. I haven’t yet spoken to a data team who doesn’t want it banned*. In our mission to make it easy to access your Webhook data we’re releasing a new hotness. Using dehook you can now write all your Webhook events directly and securely into S3 using a credential free, single permission role. Getting started with receiving Webhooks can be a pain and knowing you might lose that data while your dev team iterates is rough. This fixes that. Without a data team(!?) Ready to scale when you are ;). Have a butchers at the Loom below to see me in full tech nerd mode. https://lnkd.in/g-neZjvF More to come this week. *I haven’t spoken to any data teams about this yet. This was just released.

29

3 Comments

I've been wanting to dig into dbt's Python model capabilities for a while, and was thoroughly surprised at how easy they are to setup and run! If you're using BigQuery as your OLAP database, all you need is a dedicated storage bucket. dbt is going to run your model with PySpark under the hood, and, if you've chosen the "serverless" option in your dbt profile, it'll do so using Google Kubernetes Engine (meaning you don't need to set up any of the infrastructure yourself). I can imagine that maintaining your own Compute Engine instance in some production environments would be preferable, but the alternative approach was simple and inexpensive. Full project here: https://lnkd.in/eg6tC5vR

59

3 Comments

Wondering how to evaluate and improve your RAG response quality? Follow the Verta team this week to learn all about RAG evals and join our #ragathon later this week! #rag #evaluation

9

Fusing together our two main specialisms - AI and Segmentation - has created a compelling offer that lets users 'chat' to their segments. Having talked to lots of STRAT7 colleagues and interested clients, we have carefully built the tool to put users in control (e.g. transparency over which sources the chatbot has used in its response), limit the chance of hallucination (e.g. by splitting out numeric and qualitative questions) and repurpose the building blocks from strat7GPT (our internal tool that has been carefully built to process complex document types). The next step on our journey will be test replies from the Segment Chatbot against actual human responses, with help from colleagues at STRAT7 Audiences and STRAT7 Incite. Get in touch if you'd like to find out more or interested how this could work in your business.

10

Check out a new article from Charles Bloche who we are lucky enough to work with at DataKitchen. "Navigating the Storm: How Data Engineering Teams Can Overcome a Data Quality Crisis" https://lnkd.in/eRMSzxCh

11

2 Comments

Struggling with data chaos? If you’re managing data at scale, you know the frustration of redundancy: scattered, inconsistent data causing conflicting reports and poor decision-making. Sound familiar? 🎯 The pain: Conflicting numbers erode trust Manual fixes drain time Decisions suffer What if you could fix that? Meet Ellie—a next-gen data design and modeling tool. Ellie automates reverse engineering of source data, creating a clean, centralized repository to reduce redundancy and boost analytics efficiency. 📊 Ready to trust your data again? Book a demo here https://lnkd.in/dGhKK7Ve Let’s simplify data together.

11

1 Comment

🧠 Zombie Data Teams: A Tale of Tech Misdirection 🧠 I am extremely excited and happy to announce that one of my favourite, knowledgeable data people that always provides a pragmatic view of the Data World, Veronika Durgin the VP of Data at Saks Fifth Avenue and a recognised Data Leader will be the latest guest. To start this post. I need to address Jason Taylor's question posted on the event, "Do you feed a Data Zombie Teams Brains?" the short answer is NO. We need to be encourage data professionals to use their own brain's and be focused on what's the best things to focus on that enables business success and don't just become part of a Zombie Data Team. I can't take the credit for the Title of the show, that's from a side chat that Veronika Durgin had with Sonny Rivera. I also need to give a shout out to George Miguel for his blog on Zombie Solutuions that inspired some of the topics we will discuss on Monday's LEIT DATA Value show. 📆 Monday 2nd of December ⏰ 12pm EST | 5pm GMT 🔗 Link in the comments Here is a summary of some topics that we will cover :- 🧟 Zombie Data projects: These are projects that persist long after their initial purpose, often due to unclear objectives, organizational inertia, and a lack of critical thinking. If it's dead it's dead don't go on a death march. ⁇ Problem Definition: Clearly defining the problem is crucial to finding effective solutions. Vague or poorly defined problems can lead to misguided efforts. 🗂️ Organisational Inertia: Large organizations can be slow to change, and outdated projects can linger due to a reluctance to challenge the status quo. Easy problems have been solved. Look to find a solution to the things that people say 'It's too difficult". 🚨Difficult = Valuable 🚨 ❌ Critical Thinking: Questioning assumptions, challenging the status quo, and seeking alternative solutions are essential skills for avoiding Zombie Data activities. If delivered in the right way you can deliver critical thinking that enables the people you challenge to still be part of the journey ( and happy) ༗ Data-Driven Decision Making: Using data to inform decisions can help identify underlying issues and measure the impact of data projects. Don't follow the crowd, lead the crowd. #meandatastreets #meandatazombies

33

5 Comments

Benn Stancil latest post on dbt hits close to heart. 🥲 In parts because I remember ‘winging it’ myself whilst building an entire warehouse amount of dbt models. I had some medallions to follow, ideas for encapsulating logic, but other than it was wild west sql slinging. But also through our work with reconfigured v1 where we tried to enforce an auto-generated-yet-exposed-to-the-eyes framework for building core data models. It solved the spaghetti issue for sure, but it really didn’t address the underlying problems. Recommend the read. :-) https://lnkd.in/dmBcPhyA

10

2 Comments

If you're looking to build internal data apps, but can't yet stomach the recurring SaaS cost - especially relevant for data-centric early-stage startups... Would highly recommend taking a look at https://streamlit.io/ - a Python framework for data apps 🐍 Can deploy on GCP Cloud Run, or your preferred cloud platform, and away you go. Clean interactive front end, using Python only. For larger teams, I would definitely be looking at the likes of Count (data canvas pioneers) and Hex - but at this stage, Streamlit on Cloud Run is working very nicely 👌 #DataScience #DataEngineering #DataOps #Startups

15

You can't impose order on a complex adaptive system at the macro scale. The underlying behaviours will eventually bring it back to its strange attractor ("homeostatis") - the old order. We have to change the underlying behaviours. Macro changes start on the micro scale. More bluntly: lasting change happens bottom-up, not top-down.

21

5 Comments

I’ve said this before, but one thing we feel strongly about is offering a data pipeline that doesn’t come with a long list of exceptions. If a certain data type is supported in the source system, then we need to figure out how to replicate it over to any downstream system. Otherwise, it’s not really a complete solution. We went through every single MySQL data type that could exist to make sure we support them fully — including infrequently used spatial types like these! 👇 * GEOMETRY * POINT * LINESTRING * POLYGON * MULTIPOINT * MULTILINESTRING * MULTIPOLYGON * GEOMETRYCOLLECTION Anyone using these? I’d love to hear your use case! 🙋 #mysql #dataengineering #datareplication

16

1 Comment

What should the North Star of a data team be? It’s not data quality, literacy or insights. It’s clarity. For me, clarity is the single binding concept which ties together all that a data team does. Improving data quality? That’s because it provides better clarity to the business because the data they are looking at is accurate. Building a report? If it doesn’t clarify what the operating state of the company is then it’s failed. Building a visualisation? We should work to make sure it tells as clear a story as possible. Training the wider business how to analyse data? This helps people gain clarity by themselves. Greater business clarity is the value driver of the data team. We can’t always push the decision button but we can work damn hard to make sure people are 100% clear which one to push and why.

49

14 Comments

⚠️ Is Your Finance Team Struggling in the Dark? ⚠️ These snippets are a wake-up call for anyone still treating data as an afterthought. If you’re not leveraging data as a strategic asset, you're behind the curve, and your finance team might be stumbling into avoidable pitfalls. From reactive decisions to data overload, these challenges can severely hinder your business growth. 💡 Key Takeaways: Treat data as a strategic asset, not just numbers. Building a data-driven culture is essential for long-term success. Without reliable data, you’re setting your business up for failure. Common issues like data overload and poor quality can cripple your finance team. These are just glimpses of what we cover in our latest article. Dive deeper to ensure your finance function isn’t left in the dust. 👉 Explore the full article here: https://lnkd.in/dF4UhAfY

1

When I used to manage data teams, I often felt we were the unsung heroes of the business. We had so much potential, yet struggled to move beyond a support role. Now, at RudderStack, we’ve seen firsthand how owning the Customer 360 can be a pivotal step in a data team's evolution to strategic partners. Why is the Customer 360 so crucial? It's the foundation of data-driven decision making across the entire organization. We’ve learned that when data teams control this asset: 1. It creates a single source of truth, combining data from every touchpoint - not just marketing, but product usage, support, sales, and more. Insights are much richer when you connect all these dots. 2. Data consistency and BI reporting improves company-wide. No more conflicting definitions of "active user" or "customer lifetime value". 3. It becomes the foundation for building sophisticated use cases from better marketing attribution to app/web personalization to predictive models for LTV and churn. These are the projects that truly move the needle. Customer 360 ownership doesn't automatically make data teams strategic, but it's a powerful catalyst. We've seen data teams transform from running pipelines and writing SQL all day to driving real business growth. Let me share a few examples that still amaze me: - StatPearls' data team boosted customer retention by 1.7x - Wynn Slots increased payer revenues by 25% - A major brick-and-mortar retailer improved their ROAS by over 400% These aren't just isolated success stories. They represent what's possible when data teams take charge of the Customer 360. I've been through this shift myself, and now I'm fortunate enough to help other teams make it happen. The bottom line: When data teams own the Customer 360, they don't just manage data – they drive outsized business outcomes.

102

2 Comments

Why do we tolerate a certain level of broken data assets as normal? Think about it - dashboards that only load halfway, tables with known issues that shouldn't be used, broken reports, useless alerts... As Data Engineers, we take pride in the 'Engineer' part of our title and rightly consider ourselves a type of Software Engineer. But would a solid Software Engineer leave a known piece of code with: - Half-loaded dependencies? - Wrong results? - Functions not doing what they're intended to do? - Error-spewing code? Your data assets reflect the quality and professionalism of your work. Let's hold ourselves to higher standards. #dataengineering #dataarchitecture

17

1 Comment