Stuart Tower

BTS of the team testing the latest buyer signal at UserGems 💎 👀 This was the group chat when we turned on the Champion Referrals signal, which is now available to all UserGems customers 🥳 We use this signal to identify the current customers who used to work at our target accounts. Then, we automate a workflow for our CS and Sales teams to ask the champion for a warm intro. As you can see, it’s gotten some pretty immediate results. If you’re interested in Champion Referrals and our full Signal Library, follow UserGems 💎 to see all the signals we’re dropping.

5

If you're free tomorrow afternoon/evening check out the London Clojurians talk I'm giving about working with data in Clojure. Clojure's data ecosystem is super active and there have been tons of developments in the last few months we're excited to share: - 🤖 better integration with pure-JVM ML models - 📶 the new tablecloth columns API - 📊 the (very) beginnings of a new approach to visualizing data with a grammar-of-graphics style API - 🐍 python inter-op (not new, but something that isn't as well known) And lots more. I'll go over how to explore a few realistic datasets and gain insights using basic ML models and time series analysis, all in Clojure. Also it's free! https://lnkd.in/gkPXJwnY

12

2 Comments

It's great to revisit dbt after a few years and discover the new Python models they've introduced alongside SQL models. Recently, I tackled a messy set of SQL queries stored in various text files and in my DuckDB history. Cleaning up was essential as the data needed to be processed through a Python script for text manipulation, calls to an Elasticsearch index, and then back into DuckDB. The seamless integration of text manipulation and Elasticsearch calling into "dbt run" was a pleasant surprise. #dbt #Python #SQL #Elasticsearch

12

Following on from last week's post where I built a Python API test automation framework using Pytest I decided to see how quickly I could get a Python Performance Test Automation framework up and running against the same API using Locust (https://locust.io/). Personally I would say that it would not be unrealistic to get it up and running within an hour ignoring any organisation environmental constraints. I spent a total of about 4 hours and that included building some Python synthetic test data generation that executes on the fly when creating a booking and then stores the booking details as an instance of an object so other API calls can search for the same booking details by various criteria. There are the obvious options of JMeter, Gatling and commercial tooling such as Loadrunner but for a Python centric engineering team it's worth considering. #python #testautomation #locust #performancetesting #testing Reference: Last week's post - https://lnkd.in/eMvmDfDV

22

4 Comments

The title of this article is definitely click bait; bait I took of course. I think this article does a really nice job of looking at the underlying black box nature of LLMs and highlighting some of the inherent issues in that model. As an engineer, the concept of a "dead end" feels more like a new set of problems to solve. There are a number of ways to un dead end this technology. The case needs to be made and the conversations need to be had to get the investment. https://lnkd.in/gGPG3Cd4

3

I've written an article about how at the MoJ we've built a modern data lake using Amazon Athena, Apache Iceberg and dbt https://lnkd.in/enRN_HaJ

78

2 Comments

Why do we tolerate a certain level of broken data assets as normal? Think about it - dashboards that only load halfway, tables with known issues that shouldn't be used, broken reports, useless alerts... As Data Engineers, we take pride in the 'Engineer' part of our title and rightly consider ourselves a type of Software Engineer. But would a solid Software Engineer leave a known piece of code with: - Half-loaded dependencies? - Wrong results? - Functions not doing what they're intended to do? - Error-spewing code? Your data assets reflect the quality and professionalism of your work. Let's hold ourselves to higher standards. #dataengineering #dataarchitecture

17

1 Comment

The 2024 StackOverflow survey is out, and for the 8th year in a row, Clojure has ranked near the podium in the Top Paying Technologies category—often occupying rank 1. I don't talk much about why I choose to use a weird Lisp—mostly because nobody cares—but the results speak for themselves. If you can hold your nose long enough to learn something very different, you'll find yourself in a high-productivity environment that features: 🔥 Hot code reloading 🦆 Dynamic typing ♻️ Best-in-class garbage collection 📂 Highly performant immutable data structures 📚 Access to the entire JVM's ecosystem of libraries 🧩 Cross-platform compilation to JavaScript 🧠 An unbelievably smart and humble community Sure, you'll pay a high mental price to learn the language, but as a reward, employers will pay a high price for *you*.

72

47 Comments

Trying to hire a junior to mid-level engineer in Scotland with great benefits and remote working is hard... https://lnkd.in/eQNHVqGR Not because the role isn't attractive (you'd be working on exciting large-scale system infrastructure challenges writing code with Golang - 1,000's of Kubernetes clusters, 10,000s of cloud VMs) or because the team are all crazy (they're thought leaders in their field including multiple upstream Kubernetes contributors from 3 continents!)... you got me... maybe there's a little crazy - but only the good kind! No... my challenge is whether this AI-generated image to go with my LinkedIn post makes any sense! I never have much luck with images containing generated text. Even using prompts like "spell check any text" or as is the case with this one "do not include any text" doesn't seem to work. Not to mention, I know what a Scottish Terrier is... but that "thing" in the bottom right looks more like an angry, tartan-wearing Capybara. Comment below for your biggest concerns with this image... or if you're a junior/mid-level, talented software engineer based in Scotland - maybe just skip over the image and apply online - https://lnkd.in/eQNHVqGR

62

12 Comments

🚀 **Proud Moment: Internal Release of Ratl & Inga Agents!** 🚀 After months of hard work and collaboration, I’m excited to share a major milestone we’ve achieved at **ratl.ai**—the internal release of our intelligent agents, **Ratl** and **Inga**, marking a significant leap in API testing and automation! 🎉 Leading the development of this feature has been an incredible journey. Watching **Ratl**, our Supervisor Agent, and **Inga**, our Tester Agent, evolve into an automated solution that can orchestrate and execute complex testing workflows fills me with pride. This release is just the beginning of what’s to come as we continue innovating in the world of Product Reliability. ✨ **Key Highlights:** - Automated, comprehensive API testing with seamless orchestration. - A balance between flexibility for power users and ease-of-use for all. - Upcoming features like AI-driven load testing, no-code automation, and multi-agent collaboration. A huge thank you to my amazing team for their dedication and hard work. Together, we’re building something groundbreaking, and I couldn’t be more excited for what lies ahead! 🎥 I’m also thrilled to share a video showing Ratl and Inga in action. Stay tuned for more updates as we push towards the next stage of development. 🙌 Let’s keep pushing the boundaries of what’s possible! 👉 Join the Waitlist at https://ratl.ai to be among the first to access these groundbreaking features. #ProudLeader #Innovation #Teamwork #Automation #APITesting #AIInQA #RatlAI #FutureOfWork #DigitalTransformation

36

I’ll be there, learning from the sessions and sharing ways to progress and enable your engineers, from beginning beginners, through fluency, to mastery. Joining me?

2

At the start of my career I would often prioritise the ML algorithm I wanted to use, over the data itself. With time, experience, and mistakes - I've completely flipped the script. Here is my 8-step data preparation checklist to ensure your ML model is as robust and performant as possible: ✅ Missing values - how should they be processed? ✅ Duplicate & low-variation data - can this be removed? ✅ Incorrect & irrelevant data - how do we identify it? ✅ Categorical data - what encoding technique fits best? ✅ Outliers - could they cause issues? ✅ Feature Scaling - is this necessary? ✅ Feature Engineering & Selection - can we help the model learn? ✅ Testing & Validation - which approach makes sense? What would you add? #datascience #analytics #data #datascienceinfinity

60

4 Comments

SQL tuning? Just a flick of the wrist to make PostgreSQL fly! But beware, many brave engineers journey far and wide, dodging this ancient art like it's a spam email from a deposed prince. 📧👑 #SQLSorcery Meanwhile, sharding is like assembling IKEA furniture... without the manual. 🛠️😵 Sure, it might solve complex problems, but you're more likely to end up with a new mystery: Why do I have leftover pieces? 🤔💭 #ShardingShenanigans #PostgreSQL #PostgreSQLTips #MySQLTips #DBATips #PostgreSQLTuning #MySQLTuning #SQLisAwesome #SQLTips #SQLTuning #PerformanceTuning #PerformanceEngineering #SQLPerformance MinervaDB #ChistaDATA ChistaDATA Inc. #MinervaDB Let's not underestimate SQL; it's the unsung hero in the database saga! #DataDrama"

4

Glad to see that NGS-IQ's Excel Support has finally caught up with what I've been doing with Excel support in my now free SQL iQuery for the last 10 years. If you're an NGS-IQ user, check out their new features in this itJungle article. https://lnkd.in/geKRGm3Z

9

1 Comment

I gave a talk at the most recent London Ruby User Group (#LRUG) on how experienced programmers solve problems, and what that means about how we think and communicate as human beings: https://lnkd.in/eBzTM53t I am really proud of this talk and care deeply about this subject; I'd love if you could give it a watch! (slides available here: https://lnkd.in/eXX89qXe)

38

Introducing Git-done Aimed at Everyone but Mostly Aspiring Junior Dev's, https://lnkd.in/eGaVBmre https://lnkd.in/eWTinZhX Your New Best Mate in the Git Jungle Ah, Git, the baffling companion every developer needs but sometimes dreads. For junior developers, it can feel like trying to train a cat to fetch. That’s where Git-done comes in, your guide through the wild world of version control. Why Git-done? Because Even Heroes Need a Guide Imagine tackling Git like a quest. Without a map, you're facing merge conflicts, detached HEADs, and a few authentication dragons. Git-done hands you that map, turning your journey into a pleasant stroll instead of an epic struggle.

1

If you haven't checked out Matt Martins Substack, give it a follow and subscribe. Practical data engineering topics

25

When My MLOps Pipeline Becomes a Burnt Pancake 😂😂🥞 Had a humbling moment in tech this past month. Coming from a DevOps background, I thought extending my CI/CD practices into ML would be straightforward - like following a tried-and-true recipe. Turns out, MLOps had other plans. The initial setup looked solid: containerized environments, automated testing pipelines, infrastructure as code for AWS SageMaker - all the ingredients were there. Yet somehow, when I pushed to production, my perfectly orchestrated pipeline turned into something that resembled a burnt pancake. What I've learned? While DevOps principles are the foundation, MLOps brings its own unique challenges. Versioning isn't just about code anymore - it's about data, models, and entire training environments. Monitoring goes beyond CPU and memory metrics into model drift and prediction accuracy. The deployment automation I could handle. But when I started encountering data scientists core areas like: feature stores, model registries, and training pipelines, I realized I needed to expand my toolkit beyond traditional DevOps practices. Back at the drawing board now, diving deeper into SageMaker's model deployment patterns and rethinking my monitoring strategy. The core DevOps principles still hold true - automate everything, monitor relentlessly, iterate quickly - but the implementation looks different. For my fellow DevOps engineers venturing into ML territory - embrace the learning curve. It's very complex 🙄😰 , but that's what makes it interesting. #DevOps #MLOps #AWS #CloudComputing #TechJourney

7

3 Comments

If you come from the math/natural sciences-side of data science and haven't received any formal training in software engineering, i.e. moi, pleeease check this book out. Just from the chapter on OOP and the section on using Docker, I've learned so much. Once I read this thoroughly, I know I'll be referencing this again and again.

8