Hi everyone,

For this month’s Community Spotlight, we’re chatting with Jerry Peng, a Staff Software Engineer at Databricks and the tech lead for Spark Structured Streaming’s Real-Time Mode.

Jerry has over a decade of experience building distributed streaming systems across industry and open source, working on technologies including Apache Storm, Heron, Pulsar, and Spark. He is also a committer and PMC member of Apache Pulsar, Storm, and Heron.

In this conversation, Jerry reflects on how stream processing has evolved, the tradeoffs engineers often underestimate, what open source can teach engineering teams, and where AI can meaningfully change how distributed systems are built and operated.

Let’s get into it.

- Shubham, Swetha, & Sugandhi

📣 What’s New in DET

🎤 DET Webinar: Stop Optimizing the Wrong Part of Your Spark Job

Join us on October 16 to explore where Spark performance problems actually come from and how to focus optimization efforts on the bottlenecks that matter. (RSVP)

📍 DET Meetups in October

• Toronto meetup at Shopify Toronto Office on Thu, Oct 29 (RSVP)

• Bengaluru meetup at Koramangala, Bengaluru on Sat, Oct 31 (RSVP)

🌟 Data Engineering + AI Leadership Summit 2026

The Data Engineering + AI Leadership Summit is coming up on November 2 in San Francisco, bringing together senior data and AI practitioners and leaders for candid discussions on challenges facing modern organizations. Apply to Attend

🎟️ Next Query 2026: The Data + AI Infrastructure Summit

Next Query is a free virtual summit organized by PhoenixAI, the original creator of StarRocks, exploring the data infrastructure behind analytics and AI agents. On November 4, engineers from companies like Apple, Cisco, Microsoft, and AWS will share how they build analytics systems at scale. Sessions include:

• Apple: StarRocks as a Service: Building, Operating, and Troubleshooting at Scale

• Cisco: From Unified Real-Time Analytics to a Governed AI Data Plane

• Applovin: Scaling Analytics at AppLovin with StarRocks

👉🏼 Sign up HERE (free registration).

Spotlight: Jerry Peng

Please introduce yourself briefly to the Data Engineer Things community.

My name is Jerry Peng, and I am currently a Staff Software Engineer at Databricks, where I work extensively on Apache Spark Structured Streaming. Before joining Databricks, I was a Principal Software Engineer at Splunk, focusing on streaming and messaging systems, particularly Apache Pulsar and Apache Flink.

I have spent much of my career working on open-source stream processing systems. I am a committer and PMC member of the Apache Pulsar, Apache Storm, and Apache Heron projects. Prior to Splunk, I worked at Streamlio, where we were building a next-generation stream processing platform before the company was acquired by Splunk. Earlier in my career, I also worked at Citadel and Yahoo, where I focused on distributed systems and stream processing.

My interest in this space goes back to my graduate studies at the University of Illinois Urbana-Champaign, and much of my career since then has centered on building scalable, reliable systems for processing data in real time.

You’ve spent nearly your entire career building distributed stream processing systems. What first drew you to this space, and what has kept it interesting for so many years?

How I got started in distributed stream processing is actually a funny story. While I was in graduate school at UIUC, my advisor, Professor Indranil Gupta, told me that stream processing was going to be the “next hot area” and suggested that I do research in it. As a fairly naive and inexperienced graduate student, I agreed without really knowing what I was getting myself into. I am pretty sure I had even fallen asleep during one of the lectures where he talked about stream processing systems. But that conversation ended up being the start of my journey into the field.

What has kept me in stream processing for so many years is that I still find the problems both technically challenging and rewarding. Stream processing sits at the intersection of distributed systems and databases, and there are difficult problems to solve across many dimensions. Performance is not just about maximizing throughput; you also have to think about latency, scalability, fault tolerance, consistency, infrastructure cost, and increasingly, usability. Improving one dimension often comes at the expense of another, so building a good streaming system requires making interesting engineering tradeoffs. Even after working in this area for many years, there are still plenty of hard problems left to solve.

It has also been interesting to watch the industry evolve. When I first entered the field, stream processing was relatively specialized, and batch processing was still the dominant way most organizations approached large-scale data processing. As batch processing technologies have matured, more organizations have started asking whether there are faster and more efficient ways to process their data and make it available for use. Increasingly, they are realizing that many workloads traditionally implemented as batch pipelines can naturally be expressed as streaming or incremental computations. This shift has led to much greater interest in stream processing and, more broadly, in building data pipelines around continuous and incremental processing.

So in some sense, my advisor was right—just perhaps a little early. When he told me years ago that stream processing was going to be the “next hot thing,” I did not fully appreciate what he meant. Today, we are at a point where many more engineers and organizations are actively thinking about how to build their data systems around streaming and incremental computation, which makes it an especially exciting time to still be working in this space.

You’ve worked on Apache Storm, Heron, Pulsar, and now Spark Structured Streaming. Looking back, what ideas about stream processing have remained fundamentally true across all these systems, and which assumptions have completely changed?

One thing that has remained fundamentally true across all of these systems is the core challenge of distributed computing: how do you take a collection of commodity machines of different shapes and sizes and make them work together as a single, scalable, and fault-tolerant system? Hardware has become faster and cloud infrastructure has changed how we provision resources, but the fundamental requirements have not changed very much. A distributed stream processing system still needs to scale across many machines, tolerate failures, recover correctly, and continue processing data reliably.

Latency has also always been an important concern in stream processing, but I think the way we think about latency has evolved considerably. In the earlier days of streaming systems, lower latency was often treated as an objective by itself—the faster you could process an event, the better. Today, users are much more interested in the relationship between latency and cost. If infrastructure cost were irrelevant, almost everyone would want their data processed as quickly as possible. In practice, however, users need to make intelligent tradeoffs between latency, throughput, reliability, and infrastructure spend. The question is increasingly not simply, “How low can my latency be?” but rather, “What latency does my application actually require, and what is the most cost-effective way to achieve it?”

Another major change has been how stream processing is exposed to users. Historically, using a streaming engine often required a fairly deep understanding of streaming concepts and specialized programming models. Developers had to reason explicitly about things such as event time, windows, state, watermarks, and the lifecycle of long-running streaming applications. That created a relatively high barrier to entry.

Over time, the industry has increasingly moved toward exposing streaming capabilities through interfaces that data engineers already understand, such as SQL, tables, and materialized views. Instead of requiring every user to think of themselves as a streaming systems expert, the system can increasingly hide much of that complexity behind familiar abstractions. I think this is an important evolution: stream processing is becoming less of a specialized technology that users have to explicitly adopt and more of an underlying execution model for continuously and incrementally maintaining fresh data.

For years, organizations maintained separate systems for batch processing and low-latency streaming. Do you think Real-Time Mode represents the beginning of a unified data platform, or will there always be workloads that require specialized streaming engines?

Historically, it was common for organizations to use one set of systems for batch processing and another for low-latency stream processing. Technologies such as Apache Hadoop MapReduce and Spark were widely used for batch workloads, while specialized engines such as Apache Storm and Apache Flink were often adopted for workloads that required millisecond-level latency. I would argue, however, that Apache Spark had already gone a long way toward unifying batch and streaming even before Real-Time Mode. Spark supports both batch processing and, through Structured Streaming, incremental and streaming processing on top of the same core engine and largely the same APIs. Users can move between these processing models without having to adopt an entirely different programming model or technology stack.

💡 Editorial Note: Spark Structured Streaming’s Real-Time Mode

Real-time mode is a new Spark Structured Streaming trigger that processes events continuously as they arrive, instead of on a micro-batch schedule, cutting latency down to milliseconds for use cases like fraud detection, live personalization, and ML feature serving. For a video walkthrough, Jerry also covered Spark Structured Streaming’s Real-Time Mode in his 2026 Data Engineering Open Forum talk.

One of the strengths of Structured Streaming is the flexibility it gives users in choosing how their pipelines execute based on freshness and cost requirements. With the AvailableNow execution mode, a pipeline processes all currently available data and then shuts down. With ProcessingTime, it runs continuously and periodically processes newly arriving data. And now, with Real-Time Mode, it can continuously process data with millisecond-level latency. Together, these execution modes allow users to choose the right balance between data freshness and infrastructure cost without having to move to a different processing engine simply because their latency requirements change.

Real-Time Mode is significant because low latency was the major remaining gap in this unification. Previously, it was difficult for Structured Streaming to consistently achieve millisecond-level latency, so organizations already using Spark for batch and less latency-sensitive streaming workloads would often operate a separate engine such as Flink for their lowest-latency workloads. RTM closes that gap, allowing organizations to consolidate batch, incremental, and low-latency streaming workloads onto the same platform.

We have already seen this with customers at Databricks. Some customers that previously maintained both Spark and Flink environments have moved their low-latency workloads to Structured Streaming and deprecated their separate Flink infrastructure. Platform consolidation resonates strongly because the cost of operating multiple engines goes well beyond infrastructure: teams must learn multiple systems, maintain different tooling, understand different APIs and semantics, and develop operational expertise for each.

So I think the unification of batch and streaming was already well underway with Apache Spark; Real-Time Mode extends that unification into the low-latency domain.

Databricks serves a broad range of customers with very different workloads and latency requirements. How do you decide when a capability should become part of the core platform versus remaining a specialized solution, and what tradeoffs does that decision involve?

My view is that specialized solutions should, whenever possible, be built on top of the core platform rather than as completely separate technology stacks. That requires the core platform to have strong fundamentals—scalability, fault tolerance, reliability, and operability—so it can support a wide range of workloads.

Building a specialized system from scratch can be attractive because it allows you to optimize aggressively for a particular use case. But it also introduces significant duplicated complexity. You now have another system to deploy, monitor, upgrade, autoscale, debug, and support. More importantly, you may end up rediscovering and solving many of the same correctness and reliability problems that the core platform has already encountered and hardened itself against through years of production use.

This fragmentation also tends to create a worse user experience. Users may have to learn a different API, programming model, operational model, and debugging workflow simply because one workload has different requirements. Over time, that becomes both an operational and management burden.

Real-Time Mode is a good example of the approach I prefer. We could have built an entirely new low-latency streaming engine optimized specifically for millisecond-level processing. Instead, we deliberately chose to extend Structured Streaming itself. This allowed us to reuse Spark’s existing APIs, semantics, state management, fault tolerance, ecosystem, and operational foundations while changing the execution architecture needed to achieve much lower latency.

There will always be cases that require specialized behavior, but I think the better architectural pattern is to make that specialization a capability or execution mode of a strong core platform. A battle-tested core already embodies years of lessons around correctness, reliability, and operability. Building on that foundation lets engineers focus on what is actually unique about the new use case rather than repeatedly solving the same distributed systems problems.

The real challenge, then, is designing the core platform with clean abstractions and strong enough fundamentals that it can evolve to support new classes of workloads without becoming overly complex.

Streaming system design often involves tradeoffs such as lower latency at the cost of higher infrastructure spend, or stronger consistency at the cost of throughput and availability. What other tradeoffs do you think engineers tend to underestimate when designing streaming systems?

I think one area that has not received enough attention is usability. As I mentioned earlier, many concepts in stream processing can be difficult for users to understand and reason about. That raises an important design question: can we expose streaming and incremental processing through simpler and more familiar interfaces, such as SQL, tables, or materialized views, rather than requiring every user to become an expert in streaming concepts? A technically powerful system is much less valuable if it is too difficult for users to understand and use correctly.

Another important area is operability. Engineers who have not actually run streaming systems in production often underestimate how challenging it is to operate and support them over long periods of time. You have to handle upgrades, failures, scaling and autoscaling, workload changes, and capacity management, all while continuing to meet latency and availability SLOs. Building a system that performs well is one challenge; building one that can be operated reliably in production is another.

A third area with a lot of nuance is correctness, particularly understanding at what level a correctness guarantee actually applies. A streaming engine may provide exactly-once processing internally, but that does not automatically mean the entire pipeline is exactly-once. Sources, sinks, retries, external side effects, and application logic all contribute to the end-to-end semantics. Engineers sometimes underestimate how much correctness is ultimately a property of the entire pipeline rather than just the processing engine.

You’ve spent years building both open-source infrastructure and commercial products. What engineering habits or design principles have you learned from open source that companies should adopt more often, and vice versa?

One of the most valuable lessons I have learned from open source is the importance of a relatively egalitarian engineering culture. In a healthy open-source community, your job title, employer, or organizational seniority carries much less weight than the quality of your technical argument and the trust you have built through your contributions. A junior contributor can challenge a design proposed by a very senior engineer, and that design still has to stand on its technical merits.

This creates a healthy form of adversarial review. You have to clearly explain the problem you are solving, the assumptions you are making, the alternatives you considered, and the tradeoffs involved. You cannot simply rely on organizational authority to move a decision forward. This becomes even more valuable when contributors come from different companies, each bringing different workloads, constraints, and incentives. A design that works well for one organization may not make sense for the broader community, so ideas are tested against a much wider range of perspectives.

I think companies could benefit from adopting more of this mindset internally: creating an environment where engineers feel comfortable challenging ideas regardless of who proposed them, and where technical decisions are driven primarily by evidence, reasoning, and long-term impact rather than hierarchy.

At the same time, commercial product development has taught me lessons that open source projects can benefit from as well. One of the biggest is prioritization. There are always many technically interesting problems to solve, but engineering resources are finite. Building a commercial product forces you to ask which problems matter most to users and whether solving them will materially improve their experience.

Another lesson is the importance of usability. Open source infrastructure can sometimes be designed primarily by experts for experts. Commercial products tend to put much more pressure on simplifying workflows, providing good defaults, improving observability, and reducing the amount of specialized knowledge required to use and operate the system. A feature is not successful simply because it is technically sophisticated; users also need to be able to discover it, understand it, and operate it reliably.

Many engineering teams today are evaluating AI-assisted development. Which parts of building distributed data systems do you believe AI will meaningfully accelerate, and which areas will continue to depend primarily on engineering judgment?

One distinction I think is important is the difference between deciding what to build and figuring out how to build it.

AI is already becoming very effective at the “how.” It can accelerate implementation by generating code, writing tests, exploring unfamiliar codebases, improving documentation, and suggesting alternative implementations. I expect these capabilities to continue improving, especially for work that is time-consuming or relatively mechanical.

Another area where I think AI can have an enormous impact is production incident response. Distributed systems generate large amounts of operational data—logs, metrics, traces, alerts, configuration changes, and deployment history—and correlating all of that information during an incident can be slow and difficult. AI is well suited to quickly sift through this data, identify anomalies, narrow down likely root causes, correlate symptoms across components, and suggest potential remediations. This has the potential to significantly reduce debugging and root-cause analysis time, and ultimately shorten mean time to recovery.

Where human judgment remains especially important is deciding which problems are worth solving in the first place. That requires understanding users, real-world use cases, business priorities, system constraints, and where the technology should evolve over time. In distributed systems, there is rarely a single objectively correct design. Engineers must make tradeoffs among latency, throughput, cost, consistency, availability, usability, and operational complexity, and those decisions depend heavily on context.

AI can certainly assist with that process by exploring alternatives, challenging assumptions, and surfacing tradeoffs. But ultimately, people still need to decide which problems matter and which tradeoffs are acceptable. AI can increasingly help us determine how to build something—and even how to diagnose and remediate it when it fails—but deciding whether it is the right thing to build remains fundamentally a human decision.

So I see AI as a powerful force multiplier for engineers. It can accelerate both software development and production operations, allowing engineers to spend more time on the higher-level questions: What problem are we trying to solve? Is it worth solving? What guarantees should the system provide? And which tradeoffs are we willing to make?

Looking back at your career from Yahoo to Citadel to startups, Splunk, and Databricks, which transition taught you the most, and changed the way you think about engineering?

I think two transitions taught me complementary lessons: working at a startup taught me how to own a product end to end, while my experience at Databricks taught me what it takes to scale a product and the organization around it.

At a startup, you have to wear many hats. There may not be dedicated product managers, technical writers, support engineers, or operations teams for everything you are building, so engineers often take on parts of those roles themselves. You talk directly to users to understand their problems, help define what should be built, design and implement the solution, write documentation, operate it in production, and support customers when something goes wrong.

That experience gave me a much broader perspective on engineering. I learned to think about a product across its entire lifecycle rather than focusing only on implementation. Writing good code is just one part of building a successful product. You also have to understand whether you are solving the right problem, whether users can understand and adopt the solution, how it will behave in production, and how it will be supported over time. That sense of end-to-end ownership was one of the most valuable lessons of my startup experience.

Databricks complemented that experience by teaching me what it takes to operate at a much larger scale. As the number and diversity of customers grow, approaches that work for a small user base no longer work. I gained experience not only in scaling the technology itself, but also in scaling product adoption, engineering teams, customer support, and go-to-market efforts. I also learned how to leverage a much larger organization—across engineering, product, support, and field teams—to scale a product far beyond what any individual team could do on its own.

So the startup experience taught me how to think broadly and own a product from beginning to end, while Databricks taught me how to take that same product mindset and scale it. Together, those experiences changed how I think about engineering: it is not just about designing and implementing a system, but about the entire journey from identifying the right problem to building, operating, supporting, and ultimately scaling the solution.

Key Takeaways

• Streaming is becoming less specialized and more unified. The direction is toward using familiar abstractions like SQL and tables across batch, incremental, and low-latency workloads.

• Usability is the next frontier for streaming. A system is only as good as people’s ability to use and run it. Usability and operability are the tradeoffs engineers underestimate most: hiding streaming complexity behind familiar interfaces, and keeping systems reliable through upgrades, failures, and scaling.

• The hardest engineering problems are often about tradeoffs, not raw performance. Latency, cost, reliability, correctness, and operability all have to be balanced together.

• Strong core platforms usually beat fragmented specialized stacks. Extending a battle-tested system can reduce duplicated complexity around reliability, operations, and user experience.

• Correctness is an end-to-end property, not something a streaming engine can guarantee on its own. Sources, sinks, retries, external side effects, and application logic all shape the actual semantics of the pipeline.

Community poll

💬 How to stay connected

• LinkedIn

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hey again, dear reader,

Recently, “Your AI Slop Bores Me” meme series has been doing rounds, unmissable. And that kid on the Pepsi stack throne unamused is the one image response to shut down any AI heavy content online.

There is no doubt GenAI is a force multiplier, we all get more things done with less time and resource needs than before. But it is also silently training us to look away when something sounds “generated” and not written. And that kid on the throne is who we quietly are turning into, even if the content is worth a 10 minute read.

To address that attention hazard, we’re revamping our Data Pulse newsletter by introducing a mix of a podcast, a video and quick one-liners to give the best version of a story. Because our goal remains the same in the world of “generated AI content”: bring what’s genuinely worth your time, structured in a format that’s easier for you to consume.

So, here’s a carefully hand-picked list of what stood out to us this month in the data space and AI space. Gear up, dive in, let us know in the community polls or the comments if that inner kid on the Pepsi throne would still approve.

– Chozhan

🪴 What’s New with DET

Introducing the Data Engineering + AI Leadership Summit 2026, a new community event organized by DET. We are bringing together senior data and AI practitioners and leaders for candid discussions on challenges faced by modern organizations.

• 🗓️ When: 1:00–6:00 PM on Monday, November 2, 2026

• 📍 Where: Convene 100 Stockton, San Francisco

• 🎟️ Apply to attend HERE. Early-bird registration ends September 25.

Upcoming DET Meetup events:

• Seattle Meetup @ AWS on Thursday, Sept 24: RSVP

• Bay Area Meetup @ LinkedIn on Thursday, Sept 24: RSVP

• Toronto Meetup @ Shopify on Thursday, Oct 29: RSVP

⭐️ Reverie: The Summit for AI Builders

Reverie is a one-day summit on the data infrastructure powering frontier AI. Join us in San Francisco on November 5 as the teams behind the largest multimodal training runs share an inside look at their systems and the lessons they learned along the way.

👉🏼 Apply to attend HERE. DET members get 50% off with code DET50!

(This message is sponsored by LanceDB)

📖 Featured Read

How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference

ByteByteGo - (article, ~10 min read)

Summary: ByteByteGo sat down with the OpenAI engineers behind Codex and ChatGPT Work to unpack why “cost per successful task,” not just capability, is the metric frontier labs are now optimizing for. They break an agentic request into three layers:

• harness: assembles context, runs the tool-call loop, executes under approval policies)

• API layer: auth, tokenization, safety checks)

• inference: GPU fleet actually running the model)

and walk through what happens, layer by layer, every time an agent loops.

The techniques are the interesting part: replacing per-call HTTPS requests with a single persistent WebSocket so the harness stops re-sending the entire conversation history on every tool call; keeping prompt prefixes byte-for-byte stable so prompt caching doesn’t silently break (their example: a hash map serializing tool definitions in a different order each time, quietly making every call more expensive); “deferred tool discovery,” where instead of stuffing hundreds of tool schemas into context, the agent gets a search tool and pulls in only what it needs; and “Code Mode,” where the model writes a small script to batch several tool calls instead of paying for a full model round-trip per call.

💡 Why is this relevant for DEs?

• The harness is just orchestration with a new name. Deciding what context enters a run, executing under guardrails, appending results, looping until done, swap “agent” for “DAG” and this is a pipeline-design problem we already know.

• Deferred tool discovery maps directly onto anything you build over your own catalog or warehouse. Don’t load every table schema into an agent’s context up front. Instead index it and let the agent search, the same way you’d avoid a SELECT * on a wide table.

• Stable prompt prefixes are cache design in a new outfit. If you’ve debugged why a “identical” query wasn’t hitting your warehouse’s result cache, this will feel familiar.

• Optimize end to end, not one favorite layer is a lesson our field learns over and over, in query tuning, in pipeline cost control, and now here.

Read the full article here 👈🏼

📝 Data Bites

(The leading icon indicates the Data Bite format: 📄 article . 🎧 audio . 🎥 video)

• 📄 Stripe modeled its entire MongoDB fleet as a traversable graph and now uses pathfinding algorithms to compute and execute recovery plans automatically, cutting pager volume 30% (~200 pages/year) and eliminating 12 days of unhealthy shard states annually.

• 📄 Shopify pointed an agentic security harness at one of the largest Rails monoliths in the world and learned that the bottlenecks are partitioning, test oracles, and cross-model verification but not the models. Six weeks in: 300+ findings, an estimated $400K+ in equivalent bug bounty value.

• 🎧 Building the Context Flywheel for AI Data Agents: Prukalpa Sankar, co-founder of Atlan, describes that model intelligence alone doesn’t make an agent useful in production, what actually moves the needle is contextual intelligence. Grab your headphones and immerse yourself in this podcast episode from Data Engineering Podcast by Tobias Macey.

• 📄 Roy Daniel and Ohad Raviv make the case that Spark utilization metrics alone may not be sufficient. From "healthy" 76% memory usage masking orphaned vCores, to a 47-hour KNN pipeline traced to one line generating 13 billion throwaway records. They share four real production cases where fixing the actual bottleneck (not just adding resources) cut runtime 24–95%.

• 📄 dbt Labs breaks down why some agentic AI rollouts 10x a team (C.H. Robinson: 5,500 shipping orders/day, automated) while others become cautionary tales (Klarna) and the deciding factor is almost always whether the data can be trusted.

• 📄 Data Engineering Weekly untangles the ontology vs. semantic layer vs. knowledge graph confusion that’s been building all year, and makes the case that an “ontology for agents” is a two-loop system that has to keep running instead of something that get published once.

😂 Meme Corner

Love a good meme? We do too!

That's one of the memes my memory recalls almost everyday in the recent times. I'm guessing it's quite relatable to y'all too.

💎 Open Source Gem

Apache Ossie (Incubating)

You may know this project by its earlier name, Open Semantic Interchange (OSI). In June 2026 it entered the Apache Incubator as Apache Ossie, with its v0.1 spec already live under Apache 2.0 since January. The pitch is simple yet ambitious: a vendor-neutral, YAML-based standard for defining metrics, dimensions, relationships, and joins once so the warehouse, the BI tool, and every AI agent querying data all work from the same definition of “Monthly Active Users” instead of five slightly different ones.

The backing list has grown fast including Databricks, ThoughtSpot, Collibra, AtScale, Atlan, DataHub, and Domo notably pulling in direct competitors and tends to be the real signal that a standard is being taken seriously.

2 key aspects make this more Data Engineering relevant:

• This is the connective tissue behind the theme: whose definition of a metric does an agent trust?

• Governance is the actual product here, not the code.

👉 GitHub: https://github.com/apache/ossie

💡 DE Tip of the Month

Build a Dry-Run Mode Before You Let Agents Touch Production

More teams are preparing to or already handing agents write access to pipelines, warehouses, and orchestration tools, often before there’s a way to see the blast radius of what the agent is about to do. So, before you do so for your next agent, try this:

Rule(s) of thumb:

• Add a --dry-run / plan mode to any job an agent can trigger, that shows what would change, like the rows affected, tables touched, estimated cost, etc before it executes.

• Log the diff, not just the action. With agents taking over, “job succeeded” alerts/logs are no longer enough. So, include “what would this run add, remove, or overwrite compared to current state”.

• Separate “propose intent” from “execute.” This mirrors Stripe’s compute-a-plan-then-execute-it pattern from this month’s Data Bites, and dbt’s tiered-autonomy approach: read-only → drafting → bounded, narrow write access, with irreversible actions kept out of autonomous reach entirely.

• Treat the dry-run output as a contract. No write action should be allowed to run without first producing (and ideally, having someone or something review) its own preview.

We all know, a pipeline that fails loudly is quite annoying but succeeds confidently with a quality issue is far worse. Same way, the agent that succeeds quietly at doing the wrong thing to production data is much worse.

📊 Community Poll

Until the next one.

Chozhan, Sri &Ananda

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

For this month’s Community Spotlight, we’re talking with Himanshi Manglunia, a Senior Data Engineer at Amazon Web Services (AWS).

Himanshi has spent nearly a decade building data systems across financial services, recruiting analytics, and cloud technology. At AWS, she works on the data and MLOps foundations behind marketing attribution, forecasting, and investment optimization, helping teams turn complex signals into numbers leaders can trust and act on.

Some of you may recognize Himanshi from Data Engineering Open Forum 2026, where she shared how she built an AI agent that can make ETL changes, run and monitor pipelines, recover from failures, and open a code review. She also joined us as a speaker at the DET Seattle meetup on July 16.

In this conversation, Himanshi shares how her thinking about data engineering has evolved, from building pipelines to making sure the numbers behind important decisions are actually trustworthy. We also discuss why a green pipeline can still be wrong, how teams should handle changing historical data, and what it takes to use AI agents safely in production.

Let’s get into the conversation.

- Swetha, Shubham, & Sugandhi

🗓 DET Meetups in NYC, Seattle, and Bay Area

We have three meetup events coming up this month:

• NYC meetup at Capital One on Thu, Sep 17 (RSVP)

• Seattle meetup at AWS on Thu, Sep 24 (RSVP)

• Bay Area meetup at LinkedIn on Thu, Sep 24 (RSVP)

(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)

💡 Rows and Columns Summit - San Francisco - Sept 22

Rows & Columns Summit is a practitioner-focused data infrastructure conference taking place September 22 in San Francisco, centered on the evolving relationship between OLTP and OLAP systems. The one-day program brings together engineers, researchers, and technology leaders to discuss real-world architectures and tradeoffs around real-time analytics, CDC, open table formats, and AI-driven data systems.

Team DET will also be there to host networking sessions! Drop by and say hi!

👉🏼 RSVP (Free admission).

(This message is sponsored by ClickHouse.)

Spotlight: Himanshi Manglunia

Please introduce yourself briefly to the Data Engineer Things community and share what initially drew you to data engineering.

I am a data engineer in the Data Science and Engineering organization at AWS Marketing Tech Platform. I own the data layer behind our attribution, forecasting, and investment-optimization products. In practice, that is the data that turns short-term signals into a revenue forecast and shows leaders the return on their marketing spend. Before this, I built the data foundation for a Talent Acquisition analytics org of Amazon global. Earlier in my career, I worked with financial data to build reporting datasets that feed into dashboards.

What drew me into this was one realization: I liked being able to see how my work influenced leadership decisions. In my early days, I was in it because I liked organizing messy data with simple SQL. Now I am increasingly drawn to how big decisions get made from data, and to seeing the big picture. In most orgs, the interesting decisions are based on a handful of numbers, and the data engineers own the correctness of those numbers down to the raw event. One grain choice, one metric-definition update, one dataset freshness SLA: each ripples into every dashboard and every model downstream. The work is sometimes unglamorous and demands an eye for detail, but it pays off when you see the impact it has.

You have built data systems across financial services, recruiting analytics, and now marketing attribution at AWS. How has moving across these domains changed your understanding of what makes a data platform successful?

Today, I believe a data platform is only as successful as the trust its consumers place in the data without needing to re-transform it themselves. I did not start with that understanding. Early on, I thought success was a technical achievement: the right storage, the right compute, clean orchestration.

The hard problem turned out to be the same in every domain. In recruiting analytics, each line-of-business team had rebuilt its own pipelines. That meant data duplication and the same metric defined three different ways. In marketing, the same metric produced different totals depending on which team’s transformation you queried, because 3 different teams maintained their own logic. Different industries, identical problems: no single governed definition of the number.

So my definition of success changed. A successful platform is one where a consumer can pick up a number and use it, without asking the data engineer, “Is this the real one?” Getting there is mostly about correctness and governance, not compute. Any competent team can stand up the ETL and the pipelines. The hard part is agreeing on one definition of each number and holding every source to it. The tooling underneath is interchangeable. Agreeing on what each number means and getting every team to honor that definition is where the real work lives.

In your current role, you build data and MLOps pipelines for multi-touch attribution across online, offline, sales, campaign, and event interactions. What makes attribution particularly difficult as a data engineering problem, even before the modeling begins?

The modeling in attribution is genuinely hard, and I do not want to understate it. What I want to emphasize here is that the model sits on top of an equally hard data engineering problem, and that problem has to be solved first. The model uses whatever the data layer passes into it as input. The engineering underneath has to be right before the modeling can produce any insights.

• Identity resolution - Attribution’s whole premise is that you can join one person’s engagements to an outcome which is important to the business. Those engagements arrive across different channels like online, offline, sales, campaign, and event systems. They are logged differently in different systems (sometimes by humans), created at different times, and don’t share a common identifier. Stitching them all together and getting the identity graph right is difficult and very important.

• History does not hold still - Attribution numbers legitimately change as time progresses. Late-arriving data, compliance-related mapping deletions, and backfills, all move attribution numbers. So you are not building a pipeline that houses facts. You are building one that recomputes history and can explain why the past moved. The data modeling literature barely mentions this kind of state-management.

You mentioned that history does not hold still. Why do you think the industry still underestimates the trust problem created when historical numbers change?

Historical data is not stable, and almost nobody owns communicating when it moves. We talk about pipeline reliability, schema evolution, and cost. We barely talk about a fact that is normal in real systems: the number you reported last month is legitimately different today. Late-arriving data, backfills, and reprocessing all rewrite the past, and they are supposed to. That is correct behavior. It also quietly destroys trust. A stakeholder who saw one number in a review and a different number a month later does not even understand that “a backfill happened.” They think the data is broken. Worse, they question the decision they already made based on the older number.

The industry treats this as an edge case. In any system that recomputes history, whether that is attribution, forecasting, financial restatements, or a machine-learning feature store, it is the normal case. I built the muscle for it out of necessity: an announcement log for data-change management, an automated impact analysis that runs with every backfill and explains exactly what moved and why, before-and-after comparisons on every change, and a stack of smaller mechanisms like these. None of it is glamorous. But the trust cost of unexplained change is higher than the trust cost of an outage, and we systematically underinvest in making a moving history legible. A related problem, and just as overlooked, is metric-definition drift: the same metric ends up meaning slightly different things across dashboards and teams, until two reports disagree and no one can say which is right. Both problems look technical on the surface, a matter of pipelines and logic. What actually breaks in both is trust.

What’s a commonly accepted best practice in data engineering that you find yourself disagreeing with more as you’ve gained experience?

The practice I have come to distrust the most is success-based pipeline monitoring: alert when a job fails and treat green as done. Green DAG, therefore right. Almost every operational dashboard and alerting setup I have seen encodes that assumption, and it is wrong often enough.

I found a configuration bug once during a routine post-run deep dive. Every task in the DAG was green and the orchestration looked healthy. Underneath, a scheduled write was silently failing, so a reporting table kept serving stale numbers, and a leadership review had already gone out on them. Nothing in the “did it run” layer could have caught it, because the pipeline did run. It produced the wrong result successfully.

Execution health and data correctness are two different questions. The industry conflates them because execution health is cheap to monitor and data correctness is expensive. “It ran” tells you the plumbing worked. It tells you nothing about whether the data is correct. The more experience I get over time, the more of my attention goes to validation against an independent expectation, reconciliation against a source of truth, and honest change tracking, and I trust a group of green tasks less. A pipeline that never fails but quietly drifts is worse than one that fails loudly. At least the failing one tells you that something is wrong.

Data engineers are often asked to implement metrics or business logic after the key decisions have already been made. How can they get involved earlier and help determine whether the expected value justifies the engineering and maintenance cost?

Bring the cost side of the equation to the table, quantified, before the decision is locked. Metrics usually get designed with no one in the room who can price the implementation. The data engineer is the only person who can say “this definition costs three weeks and a permanent maintenance tax, and here is the marginal value it buys.” Surfacing that cost is not blocking the work. It supplies the piece of information that was missing, because you cannot weigh what a metric is worth against what it costs if only the value side is in the room.

I try to make the tradeoff explicit with a number rather than an opinion. Instead of arguing that something is complex or low-value, I measure it: the days of effort, the maintenance churn, the infrastructure spend, against how much of the problem it actually solves. A single number ends a debate faster than any argument, and it often changes what the team decides to build or stop doing.

The habit that earns a data engineer the seat at the design table is putting cost and value side by side, early, in the same document the business owner uses to make the decision, not flagging the problem after the decision is made.

When there is no clearly correct architecture, how do you balance reliability, speed, cost, scalability, and simplicity? How has working across different domains changed the way you make those tradeoffs?

When the output feeds a decision whose impact builds over time, like an investment recommendation for a full planning year, I weight reliability and auditability above speed, and I choose the boring, verifiable design over the clever one, every time. When the output is exploratory, I trade some rigor for iteration speed. I only pay for scalability when the growth is real, not speculative. I treat simplicity as a forcing function, not as a fifth competing axis, because complexity is where correctness becomes difficult. Consolidating several divergent copies of a dataset into one governed source, or deleting a step that no longer earns its keep, improves reliability and cost at the same time. Simplicity is often not a tradeoff. It is the thing that resolves the others.

Working across domains sharpened this. In recruiting analytics, a wrong number was easy to correct and iteration speed mattered more, so I moved faster and cleaned up afterward. In marketing attribution, the numbers feed leadership investment decisions, so the cost of a wrong number compounds across the year, and correctness and explainability dominate. The data matters more here than the code. The framework did not change. What changed is that I now calibrate it to the consequence of being wrong, and I ask about that consequence first instead of last.

💡Editorial Note:

The next few questions build on Himanshi’s talk at DEOF 2026, where she shared how she used AI agents to automate parts of the data engineering workflow. We go a little deeper into what it takes to make those systems reliable and safe in production.

Watch the full talk here:

You emphasized that the AI model is only a small part of the solution and that the real product is the context layer. What information must that context layer contain for an agent to safely modify a production pipeline?

The model is a commodity, there are a variety of them available now. Drop a capable agent into a data platform with no context, and it will write plausible code that confidently produces a wrong number.

The context layer is what makes autonomy safe, and it has to carry at least five things.

• Semantics. What each table, column, and metric actually means, plus the one canonical definition of each metric. This is why a centralized, version-controlled metric framework in a governed layer, rather than definitions scattered across dashboards, is gaining increasing importance.

• Lineage for blast radius. What is upstream, what is downstream, and what breaks if something changes. An agent must know that a change to one table will feed into three products and a leadership review before it edits anything.

• Validation expectations. What “correct” looks like, expressed as runnable checks: parity against a prior version, reconciliation targets, row-level expectations. An agent that can run the same validations a data engineer would run is an agent you can start to trust.

• Operational context. SLAs, backfill semantics, freshness cadence, and, above all, what decision the output feeds. “This number sets a budget target” and “this number is exploratory” demand different levels of caution, and validation thresholds must be set according to that.

• Guardrails. An explicit list of what the agent may not change without a human in the loop. Metric definitions and anything feeding a goal directly belong on that list.

I built a working agent that reads a requirement, makes the ETL change, triggers the run, monitors it, self-heals on failure, and opens a code review. The agent was the easy 20 percent. The 80 percent that makes it usable in production is the context and the guardrails around it.

If AI agents begin handling routine enhancements, deployments, and first-level debugging, which parts of data engineering will become less valuable, and which skills will become more important for engineers who want to have an outsized impact?

What becomes less valuable is the work already being automated:

• Routine build work. Column additions, boilerplate ETL, mechanical deployments. I took one class of routine change from about a day of work to a couple of hours, and I do not miss it.

• First-level triage and validation. The repetitive “run it, read the error, retry” loop. Anyone whose value is mostly typing that kind of code is already feeling the ground shifting.

What becomes more valuable is judgment about correctness and problem definition:

• Translating an ambiguous business ask into a data model. Agents are good at building from a specification and bad at writing one. The skill of turning “we need to make these decisions” into defined, prioritized, deliverables at the right grain only grows in value.

• Knowing what “right” means and producing that dataset continuously. When a metric swings between quarters, the useful move is to decompose the number into its parts, isolate which part broke, and prove it. Agents can accelerate that deep dive, but they still need a driver, because they have no sense of what right looks like.

• Owning the data contracts and the context layer. Someone has to define the semantics, the lineage, the validation, and the guardrails that agents run inside. That work matters more every year, as leaders increasingly want to make decisions from data: what works, what does not, and how to steer toward a target.

The job moves from writing code to specifying requirements, verifying the agent’s work, and steering it in the right direction. The engineers with outsized impact will define what “correct” means and build the guardrails that keep agents producing it.

As pipelines become more autonomous, how should teams think about accountability when an agent-generated change produces technically valid output but leads to an incorrect business decision? Which decisions do you believe should remain firmly under human control, even if they could technically be automated?

I have lived the non-agent version of this failure. A pipeline can be green, a model can retrain successfully, and every pre-defined check can pass, while the number is still wrong for the decision it feeds. Technically valid is not the same as correct enough to support a decision. An agent hits that gap faster and more confidently than a human, because it has no instinct that some parts of a metric feel off.

My view on accountability is that the human who owns the metric owns the outcome. Agents take actions, but they do not hold accountability. Every number an agent produces and puts into use still needs one specific person who is answerable for whether it is right. If you look at an agent-generated number and no name is attached, that absence is the warning sign. Automation does not remove the need for that owner. A green, hands-off pipeline can quietly feed into a number nobody watches. So autonomy does not dilute ownership. It concentrates it, because now one person answers for a system moving faster than they can check line by line.

Some decisions should remain human. Decisions that are difficult to reverse, what we call one-way doors at Amazon, need a person behind them. That includes setting or changing a goal, shifting investment from one channel to another, changing a metric definition, and deciding whether a number is trustworthy enough to publish to leadership. These are judgments about consequences, not mechanical steps. We should automate the mechanical work freely, but we should not automate the call on whether the output is safe to act on. The moment an agent’s output crosses from internal artifact to input for an irreversible decision, a human seal is necessary.

Key Takeaways

• Trust, not infrastructure, defines a successful data platform. Consumers should be able to use a number without questioning which definition or dataset is correct.

• A green pipeline does not guarantee correct data. Teams must validate outputs independently and explain when backfills, late-arriving data, or reprocessing change historical numbers.

• The model is only a small part of safe AI automation. Agents need semantics, lineage, validation rules, operational context, and explicit guardrails before they can modify production systems reliably.

• Automation does not remove human accountability. A named person must remain responsible when agent-generated outputs influence goals, investments, metric definitions, or leadership decisions.

Community poll

💬 How to stay connected

• LinkedIn

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello Everyone,

Hope everyone is doing well!

One of my favourite things about the data engineering community is how much we learn from each other. Most of us may be working with different companies, different stacks and very different scales, but the problems often sound surprisingly familiar.

A pipeline that behaves perfectly until it reaches production. A platform nobody uses quite the way we expected. A technology everyone suddenly wants to adopt. Or that one issue which takes several hours to debug and turns out to have a painfully simple explanation. 🙂

That’s also what we try to capture with Data Pulse every month not just what is new, but what we think is genuinely worth a data engineer spending some time on.

For August, we found some interesting engineering stories from Airflow, Netflix, Uber and Spotify, covering everything from pipeline reliability and data platform architecture to AI infrastructure, governance and some clever thinking around how we access data in the lake.

Grab a coffee and have a read. Hopefully a few of these are useful in something you’re building right now.

– Sri

📚 Data Pulse

How Airflow is using AI to make data engineering more resilient, not more complex

📖 Topic: Data Orchestration
🧠 Level: Intermediate

Summary: Most data engineers have dealt with the familiar 2 AM pipeline failure: open the logs, figure out whether it is a network issue, bad data, expired credentials or an upstream schema change, and then decide whether retrying will actually help.

Vikram Koka, CSO at Astronomer and an Apache Airflow PMC member, walks through three capabilities that try to make some of this operational work smarter.

LLMSchemaCompareOperator uses an LLM to compare schemas semantically rather than relying only on exact type matches. A Task State Store allows a task to retain information across retries, which is useful when Airflow is orchestrating long-running jobs in systems such as Spark or Databricks. And LLMRetryPolicy can use the team’s own runbook to classify a failure and decide whether it should retry, fail immediately or escalate.

What I like about this approach is that the AI is being applied inside a very normal data engineering problem. We are not rebuilding the pipeline around an agent; we are using AI where rigid rules start becoming difficult to maintain.

💡 Why is this relevant for DEs?

• It is a practical use of AI rather than AI for the sake of AI. Scheduling, dependencies and execution remain deterministic. The LLM is used only where interpretation can add value understanding schema differences or classifying an unfamiliar failure.

• Retry logic deserves more thought than we usually give it. retries=3 is easy to configure, but an expired credential will still be expired after the third attempt. A rate limit, on the other hand, may genuinely benefit from retrying.

• Persistent task state solves a very real operational problem. If an external Spark or Databricks job has already started, retrying the Airflow task should ideally reconnect to it rather than submit another expensive job.

• More broadly, this is a good example of AI augmenting data engineers rather than replacing the engineering around them. The team still defines the runbook and the expected behaviour; AI helps execute some of that knowledge consistently.

Simplifying Data and Product Integrations with a Data Abstraction Layer

📖 Topic: Data Platform Architecture & Data Modeling
🧠 Level: Intermediate

Summary:

Anyone who has worked with a mature data platform has probably heard some version of this:

“Use table_v2 for newer data, table_v1 for older dates, and remember that one of the metrics has a different name between the two.”

Uber built a Data Abstraction Layer (DAL) to stop pushing that complexity onto every consumer. Instead of applications querying physical tables directly, consumers request a logical table, the fields they need, a time range and filters. The DAL then figures out where that data physically lives, generates the required queries, runs them across the appropriate systems and assembles the final response.

One of the more interesting parts is the table resolution logic. Recent data might be available in a real-time OLAP table while historical data sits in a daily Hive table. Consumers still make one request; the DAL decides which datasets should serve each part of it.

Uber reports that this approach reduced the turnaround time for new advertiser reports from several weeks to under two days, and the DAL has since expanded beyond its original advertising use case.

💡 Why is this relevant for DEs?

• Consumers becoming tightly coupled to physical tables is a common source of platform fragility. Once dozens of dashboards, services and pipelines know your exact table names and schemas, even a simple migration can become painful.

• The logical-to-physical separation is the most useful idea here. Consumers describe what they need while the platform owns where and how that data is retrieved.

• The table-resolution pattern is especially interesting. Many of us have written UNION queries that combine recent real-time data with older batch data. Uber moves that complexity into platform metadata instead of repeating it in every consumer.

• This is also a good example of what useful platform abstraction looks like: not hiding everything, but hiding implementation details that consumers should never have needed to understand in the first place.

⭐️ From Sponsor: Take the Airflow AI Crash Course

Data engineering is evolving, and Orchestrate Everything on September 16 is bringing together industry leaders to explore what’s next:

• Hear from the data teams at Lyft, Wix, Ramp, and more

• Learn practical MLOps, AI inference, and context engineering patterns

• Prep for the AI Orchestration certification and take the exam for free ($150 value)

👉🏼 Register HERE.

(This message is sponsored by Astronomer.)

In-House LLM Serving at Netflix

📖 Topic: AI and Data Engineering
🧠 Level: Intermediate

Summary: Once an organization starts running enough LLM workloads, model serving begins to look less like an ML experiment and more like another distributed platform that needs APIs, deployment strategies, caching, version management and observability.

Netflix describes how it incorporated LLM serving into its existing Model Scoring Service rather than creating an entirely separate AI platform. The team uses NVIDIA Triton as part of its serving stack and selected vLLM as the primary LLM serving engine after benchmarking alternatives. An OpenAI-compatible HTTP interface sits in front of the platform, which gives application teams a familiar API and makes moving workloads between hosted and internally served models less disruptive.

The post also gets into the things that tend to appear only after going to production: model startup time, shared model storage, Triton/vLLM version compatibility, red-black versus versioned deployments, merged operational metrics, and CPU bottlenecks introduced by constrained decoding even while GPU inference itself is efficiently batched.

💡 Why is this relevant for DEs?

• The API abstraction is a pattern we already know from data platforms. Consumers talk to a stable interface while the platform team retains the freedom to change the implementation underneath. Today that implementation might be a hosted model; tomorrow it could be a self-hosted model running through vLLM.

• Self-hosting an LLM is not primarily a “download model and attach GPUs” problem. Artifact distribution, cold starts, rollout strategy, capacity, monitoring and version compatibility quickly become the bigger engineering concerns.

• The constrained-decoding example is particularly useful. Netflix found that per-request CPU-side work could become a bottleneck even when GPU inference was well optimized. It is a familiar distributed-systems lesson: after optimizing one part of the pipeline, the bottleneck simply moves somewhere else.

• As DE teams increasingly own AI infrastructure alongside batch, streaming and analytics platforms, model serving is becoming another workload whose data movement, observability and reliability we need to understand.

AI governance is now an engineering problem

📖 Topic: Data and AI Governance
🧠 Level: Beginner

Summary: AI governance often sounds like something that belongs in policy documents, review boards and compliance meetings.

This article makes a much more practical argument: once AI starts influencing production decisions, governance becomes an engineering problem.

The examples are familiar even outside AI, a production change gets approved but nobody can reconstruct exactly why it was made; an automated tool makes several changes during an incident but there is no useful decision log afterward; or a multi-step workflow fails halfway through and nobody has designed how to reverse the steps that have already completed.

The author boils this down to three questions: Who owns the outcome? Can we reconstruct what happened? Can we safely undo it?

Those questions apply just as well to AI-assisted data pipelines as they do to software delivery.

💡 Why is this relevant for DEs?

• Data engineers already work in highly governed environments. Access controls, lineage, approvals, auditability and data retention are normal parts of our platforms. AI introduces new decisions that need the same level of traceability.

• Logging the final output is not always enough. If AI recommends a schema change, transformation or production action, being able to understand what was recommended, what context influenced it and who approved it can become important during an incident.

• Rollback should be designed before automation is given more authority. A workflow that performs five actions successfully before failing at step six is very different from a transaction that simply rolls back everything.

• The larger lesson is useful beyond AI: governance works best when it is part of the engineering workflow, not something engineers are expected to remember to do after the work is complete.

Indexing the Data Lake for Online Point Queries: Random Access Parquet (RAP)

📖 Topic: Data Lake Architecture & Parquet
🧠 Level: Intermediate

Summary:

We normally think of Parquet and data lakes as being great for analytical workloads, while low-latency point lookups belong in systems such as key-value stores or serving databases. Spotify’s Random Access Parquet (RAP) work challenges that assumption.

The idea is fairly simple: maintain an external index that tells the system where a particular key lives in the data lake. Once the relevant Parquet file and location are known, the reader can use Parquet metadata to narrow things down further and fetch only the small portion of the file it actually needs from object storage.

That changes the access pattern significantly. Instead of starting a distributed query and searching through a large dataset to find one record, the system already knows where to look before it touches the file.

What makes this particularly interesting is that the underlying dataset remains regular Parquet. Analytical workloads can continue using the same files, while selected access patterns can be optimized for much faster point lookups.

💡 Why is this relevant for DEs?

• It challenges a common architecture decision. When an application needs low-latency access to data in the lake, our first instinct is often to copy that data into Redis, Cassandra, DynamoDB or another serving database. RAP shows that, for the right workload, indexing the existing data may be another option.

• Access patterns should influence architecture. Analytical scans, range queries and point lookups have very different requirements. RAP is a good example of optimizing specifically for how the data will actually be accessed instead of expecting one query pattern to handle everything efficiently.

• The Parquet fundamentals matter. File layout, row groups, metadata and minimizing unnecessary reads can have a significant impact on performance. RAP takes these ideas further by using an external index to get directly to the relevant data.

• There is a cost angle too. If an application needs one record, launching a distributed query engine and reading significantly more data than required may be unnecessary overhead. Reducing the lookup to a small number of targeted object-store reads can make the serving path much more efficient.

The broader takeaway is simple: before introducing another datastore, first ask whether the data you already have can be accessed differently.

The Technology Adoption Curve, Twenty Years On

📖 Topic: Career Development
🧠 Level: Beginner

Summary: Data engineering has no shortage of new technologies to learn. The difficult skill is figuring out which ones actually deserve our attention.

Looking back at twenty years of technology adoption, InfoQ traces how ideas such as Agile, cloud, DevOps, Kubernetes, microservices and machine learning moved from early experimentation into mainstream engineering. Some became foundational. Others were over-applied before teams eventually found more sensible boundaries around when to use them.

The article places AI engineering and agentic systems near the early-adopter end of that same curve today. The useful takeaway is less about predicting which technology will win and more about developing the judgment to separate an interesting experiment from something your production platform genuinely needs.

💡 Why is this relevant for DEs?

• We work in one of the most trend-heavy areas of engineering. Hadoop, Spark, data lakes, lakehouses, data mesh, streaming, semantic layers and now agents have all arrived with claims that they would reshape the stack. Understanding adoption maturity helps us avoid designing around hype.

• Being an early adopter and being an early majority user require different thinking. An early adopter may accept operational pain in exchange for learning or competitive advantage. A production platform team usually needs a much stronger reason.

• The underlying problem usually survives the technology. SOA became microservices; DevOps evolved into platform engineering; ML infrastructure is becoming AI infrastructure. Learning the underlying engineering problem tends to age better than memorizing whichever tool currently solves it.

• Technology judgment becomes more important as you become senior. A senior DE or architect is increasingly expected not only to know what is new, but to explain whether the organization should adopt it now, experiment with it, or simply wait.

💎 Open Source Gem

RisingWave: Event Streaming for Agentic AI

What’s new / why now:

RisingWave has been around for a while as a streaming database, but the 3.0 release line is a meaningful expansion of what the project is trying to be. RisingWave now positions itself as an event-streaming platform for agentic AI, combining CDC and event ingestion, incremental SQL computation, low-latency serving and an open Iceberg-based storage path. RisingWave 3.0 also adds capabilities around vector search and deeper Iceberg integration, while the project exposes familiar PostgreSQL interfaces and tooling for AI/agent access.

This is interesting because the “real-time context for agents” problem is starting to look very similar to a problem data engineers already know well: continuously ingest changing data, maintain derived state, and serve the latest result with low latency.

💡 Why is this useful for DEs?

• The programming model is SQL-first. Teams that do not want to build every streaming transformation as application code can maintain continuously updated results using materialized views and familiar relational concepts.

• It potentially collapses a few layers of the streaming stack. CDC/event ingestion, incremental computation and serving can happen in one system. That does not mean every Kafka + Flink architecture should suddenly be replaced, but it is worth evaluating for new workloads where the operational overhead of multiple systems is hard to justify.

• The Iceberg integration matters. Fresh operational state and durable analytical history do not have to become two completely disconnected architectures.

• Agentic AI may be the current positioning, but the underlying engineering problem is broader: keeping derived data continuously fresh and immediately queryable. That has plenty of non-AI applications as well.

👉 GitHub: https://github.com/risingwavelabs/risingwave

💡 DE Tip of the Month

Row Counts Are Not Data Quality

A pipeline loads 10 million rows yesterday and 10 million rows today. Everything looks healthy.

Except today’s customer_id is null.

Row count checks are useful, but they mainly tell us that something arrived. They don’t tell us whether the data still makes sense.

For important datasets, combine volume checks with a few checks that actually understand the data:

• Null percentage on critical columns : especially identifiers, dates and fields used in downstream joins.

• Duplicate rates on expected keys : row counts may look perfectly normal even when the same records were loaded twice.

• Min/max dates and timestamps : one of the quickest ways to detect stale data, bad filters or an upstream feed that stopped refreshing.

• Distribution changes : a sudden shift in an important category or measure may reveal a problem that schema validation will never catch.

• Referential integrity : if two datasets are expected to relate to each other, make sure that relationship still holds.

• Source-to-target reconciliation : for business-critical measures, compare what entered the pipeline with what eventually came out.

You don’t need hundreds of checks.

Five checks that actually understand the dataset can often tell you more than fifty generic checks applied everywhere.

Another lesson from production: catch the problem as close to the source as possible. Discovering bad data after it has gone through three transformations, two aggregations and seven dashboards makes troubleshooting much harder.

A successful job run only tells us that the code executed.

Data quality asks a different question: Can someone safely use what the pipeline produced?

📊 Community Poll

Until the next one.

Sri, Ananda, & Sukanya

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello, fellow Data Engineering enthusiasts!

For this issue of Community Spotlight, we chatted with Bryan Brandow, who’s a leader, mentor, data enthusiast, problem solver, and currently a Data Engineer at OpenAI.

Bryan spoke at the most recent DET Bay Area Meetup on July 23. He delivered an insightful talk on Data Engineering in the AI Era, in which he explained how maintaining data standards is crucial for building durable and maintainable systems and advocated for "harness engineering," where team standards are baked directly into the system using a structured knowledge store.

This interview explores why the fundamentals of data engineering matter more than ever in the agent era. Bryan walks through harness engineering at OpenAI, and explains why near-infinite velocity makes good design decisions the new bottleneck.

Hope you enjoy reading this as much as we did putting it together!

- Eddy, Shubham & Sugandhi

Spotlight: Bryan Brandow

For those in the DET community that may not know you yet, could you briefly introduce yourself?

My name is Bryan Brandow and I’m a Data Engineer at OpenAI. I’ve spent my entire career in Data Engineering, perhaps most notably for 11 years at Facebook/Meta. I’ve worked through the entire stack from logging, core tables, aggregate layers, and dashboards. I spent a good portion of my career specializing in data visualization with tools such as MicroStrategy and Tableau. I have strong opinions (weakly held) about all things data, but enjoy the debate and discussion.

Your first data work involved hand-building pipelines with the tooling that came bundled with SQL Server, in an era with no cloud warehouse, no orchestration frameworks as we know them now, and no version-controlled transformation layer. You're now shipping agents that do that work for you. Across all of that, what has actually stayed the same about doing this job well?

Yeah, my early days are pretty funny to think about now! My first task at my very first job was loading CSVs that got FTP’d to us from third party data providers. Data quality and the handling of exceptions (skip or fail) is probably the most consistent thing that never changes. I remember having to manually fix bad characters in the CSV and retry the job until it worked. The biggest difference is just the scale of data. The first data warehouse I worked with was only about 10gb of data, and the main ETL jobs were only a few steps. As complexity grew, the need for better tooling and platforms increased with it.

💡 Editorial Note: FTP

FTP stands for File Transfer Protocol. It's an old but simple way to move files between two computers over a network. One computer acts as the server holding the files, and you connect to it from your computer to upload or download stuff. It was hugely popular for things like publishing websites (uploading your HTML files to a web host) and sharing large files before cloud storage existed.

One caveat: plain FTP sends everything unencrypted, including your password, so, today, people use secure versions like SFTP or FTPS instead.

You've moved between management and hands-on IC work more than once, going from leading a large org to individual contributor work on harder technical problems, and later from an engineering director role into building directly again. What draws you back to the work itself, and how do you advise people thinking about that move?

I’ve always just asked how I can best help the team, whatever that role may be. I’ve never asked to be a manager, but have asked to be an IC. I personally enjoy IC more because the engineer in me loves solving problems. The feeling of satisfaction I get from fixing or enabling something for someone is really amazing. I like having direct control of the outcomes and it enables me to form strong opinions on tooling or processes that I can then turn into changes that can help everyone.

I think it’s natural for people to think that becoming a manager is taking an important step up in the career ladder, and in a lot of companies, that’s probably true. But at companies like Facebook and OpenAI, this wasn’t the case. Anyone can have influence in a company where ideas win on merit and not authority. And frankly, those are the kind of companies I want to work at anyway.

I do think everyone should try it if they have the opportunity. The experiences definitely made me a better IC because of the development of soft and organizational skills that are necessary. It also strengthened my ability to multi-task and ramp up on context quickly since as a manager, you are constantly doing that as you run from meeting to meeting.

My advice to people is to recognize that management and IC are different roles with very different skillsets. It can be hard for a first time manager to let go of the details and work through others. And it can be hard to have difficult conversations and do performance management. Being a great manager has a lot of overlap with being a great therapist. Some people enjoy it and are really good at it, but it’s not for everyone.

As Pillar Lead, you helped set Facebook's overall data engineering vision, culture, and recruiting standards alongside the Growth and Ads leads. What did you learn about what makes a DE org healthy at scale, and what translates to smaller teams?

I’m a very big proponent of standards and their importance (see my July DET Meetup talk!). I saw first hand how one person’s personal preferences would directly lead to how entire orgs of data engineers would operate in the future. I could look at Instagram, Facebook, Messenger, Growth and track all of those lineages back to a single individual that made key design decisions back when the team was small. In some ways, those differences were in support of the product or functional requirements of those business groups. But the downside was that when we needed to work across those teams, data integration was unnecessarily challenging. It’s the reason I harp on the importance of standards, even when they seem trivial. They just have such a powerful multiplicative effect over the long run.

I think a healthy organization has strong communication and debate. It’s not that any of those standards are sacred, and even though I have my own strong opinions, I don’t believe there is a correct answer. Discussion and evolution over many iterations is key. An organization that can continuously improve together, not in silos, is in a really strong position.

💡Editorial Note:

In his talk at the July 2026 DET meetup, Bryan Brandow explains why foundational data engineering standards remain critical for system longevity in the age of AI.

Watch the full talk here:

Your writing has gone from technical deep dives read by humans to standards precise enough for an agent to follow. Does public technical writing still pay off for someone starting out, now that so much content is generated?

One thing that has always helped me out tremendously is to actively monitor support channels/forums and try to reply to everything. If I don’t know the answer, I figure it out. For one, I’ve always believed in “help others the way you want to be helped”, and I greatly appreciate it when someone goes through that much effort for me. But it’s also an incredibly efficient way to learn. You’re researching the exact problems your partners are facing and building a strong reputation and relationships.

After doing this for a while on the MicroStrategy forums around 2010, I decided to start a blog so I could have some artifact to link to people when the same problems would come up. I got really into it, posting multiple technical articles per week, and eventually just put up an email address for people to directly ask me questions. I learned so much from researching and refining topics and engaging in the comments. I met so many incredible people through the community at meetups and conferences.

I do think it is still incredibly valuable. I was never worried about whether or not anyone would read it, because I was doing it for myself. It was a great personal resource for me to search and go back to. And even if you get zero traffic, it’s a fantastic artifact on a resume. My eventual hiring manager at Facebook told me that my blog was the deciding factor for them to hire me because they could easily see long form, deep dive, technical explanations and articles that would never be feasible to assess in an interview.

Early in your career you built your own object migration approach in MicroStrategy and presented it at MicroStrategy World. That idea ended up making its way into the product. Walk us through how that unfolded and what you took away from it.

I’ve always had a knack for pushing vendor software past its limits by thinking creatively. I play a lot of video games, and when I get stuck in a game, I ask myself, “What are the variables? What is my goal and what’s stopping me?” and work backwards. That same approach works very often when working with vendor software. They add so many features to meet such a wide range of needs that you can often use those features via interactions they didn’t intend, to unlock more potential.

In this particular case, when migrating objects from Dev to Prod, you would have to copy them in a particular order and piece by piece. It was laborious and error prone. It would automatically copy dependencies and I noticed that sometimes I’d get lucky and have a large nested change that did most of the work. Then I thought, “the best migration is to copy one object that takes everything with it,” and it occurred to me that there was this type of prompt that let the user pick any arbitrary object. The feature was intended to support things like “pick a filter” or “pick a metric”, but instead, I just put everything I needed to migrate in it, like a “package”. Then I could just drag that one item and it would take everything with it. It also gave the migration an artifact that could be tracked and audited to ensure only what you intended to migrate got migrated. It ended up solving a real pain point for us and I got a lot of great feedback when I shared the blog post and presented it at their annual conference. A year or so later they introduced the same feature as a first class, standalone product.

Before joining OpenAI, you were leading teams integrating LLMs into existing data workflows, back when that was still uncharted territory. What did that early work teach you, that shaped how you approach agents today?

LLMs had just burst onto the scene, and my company was understandably cautious. We worked in the financial space, so had to be very careful around new technologies, but the benefits of LLMs were obvious from the start. To unblock the project and instil confidence with our security team, we created a chat wrapper around the OpenAI API and basically re-created an internal version of ChatGPT. This gave us the ability to monitor and filter prompts to make sure an employee wasn’t sending sensitive data to a third party.

Of course today, OpenAI has excellent support for sensitive data, including Enterprise ZDR (zero data retention) and administrative reports and tools. Shortly after I left that company (to join OpenAI!), they deprecated the homegrown system and adopted the ChatGPT Enterprise tools directly.

In that project, I did learn a lot about RAG (Retrieval Augmented Generation) and how we could improve answers by giving the models context. That stuff largely happens behind the scenes now via Plugins and as the models have significantly increased their context windows and capabilities. It showed me the difference in the quality of outputs that you get when providing the model access to company information to get more relevant results.

At the DET Bay Area meetup, you argued that the core tenets of good data engineering matter more than ever in the AI era, and walked the audience through harness engineering: AGENTS.md files, data modeling contracts, and a cleanup agent called TIE-D. For readers who missed it, what is the harness, and why start an agent on cleanup work rather than net-new pipelines?

A harness is the software layer around an AI model that gives it the context, tools, rules and execution loop needed to do useful work. The model itself can reason and generate outputs, but the harness turns that capability into an agent that can operate on a real system. So when I talked about OpenAI’s blog post, “Harness engineering: leveraging Codex in an agent-first world”, I’m referring to how you build up support for the model to operate in that layer in an efficient way. I focused on how to structure context and skills for the model, but it can also include the development of tools.

As for net-new vs cleanup, I think that was more of a “low hanging fruit” situation. Where things need to be cleaned up is pretty straightforward. We know what needs to be done, and the Agent can figure out how to do it. Cleanup work is also important because in addition to your AGENTS guidance, models are also going to pull from nearby examples. So legacy approaches or non-standard implementations can propagate like weeds. But when you’re creating something new, you need to be more involved in the decisions. I wouldn’t hand over the full architecture, schema, and design choices to an Agent (yet), but I would hand over the implementation. I think Codex is in a really good place right now such that I can engage on the designs and let it implement the details.

When I do want to run longer form net-new development, right now I’m leaning heavily on ExecPlans. Here are a few posts that talk about them: Using PLANS.md for multi-hour problem solving and Modernizing your Codebase with Codex. Think of it as a materialized version of Plan mode in Codex. We can talk through all of the design decisions and then I can entrust Codex to go to work on the full end to end implementation. It can build the tables, write and test the code and validate the data. I can check in along the way, review the PRs and after they merge, it can then handle the backfills. While it’s working, I can be designing and getting the next one ready and they can work in parallel. The amount of productivity increase is really incredible! It also leaves behind a durable lot of not just what it completed, but a documentation trail for why the decisions were made.

Most teams reading this don’t have OpenAI’s headcount or infrastructure. If a small data team wanted to point an agent at their own pipelines, starting with cleanup work like you described, what is the first thing they need to write down, and what can wait?

I think it starts with instructing the model on how it should build a pipeline. Most likely, you’ve already got some kind of framework or template that you use, so document that. We had a wiki page with instructions on how to use our internal ETL framework, so I started out by having Codex turn that into an `.md` file.

Then, as you use it, you iterate on what it got wrong. Anytime it doesn’t one-shot a pipeline and you have to correct it, add that instruction into the `.md` files. You can start out with just a single AGENTS.md and as it starts to get large, break it up into the Knowledge Store volumes I described in my talk. I think it’s best to discuss these standards with your team first and give everyone the opportunity to provide input. This will become the canonical way you build your data pipelines. Once you have that, then you can build an agent like TIE-D that can apply those standards broadly as they evolve over time.

Also consider if an ExecPlan would be a better fit than a running Agent. For example, I’ve found deprecating a table and all of its downstreams to be way more efficient for an ExecPlan than for TIE-D, because the ExecPlan can scope the entire amount of work, break it into phases, and burn it down, whereas TIE-D doesn’t have that high order planning routine and chips away at lots of smaller tasks with more loops and hence more resources.

💡 Editorial Note: TIE-D

TIE-D is an autonomous Data Engineering (DE) agent utilized at OpenAI to handle maintenance tasks and cleanup.

Key features of Tidy include:

• Purpose: It automates menial but essential “cleanup” work that is often too time-consuming for engineers to do manually, preventing the accumulation of technical debt and “crust” in the system.

• Evergreen Tasks: Tidy operates on the concept of “evergreen tasks,” which are high-level goals. Instead of being pointed at every individual issue, the agent actively searches for work, such as deprecating old tables, standardizing sensors, or removing redundant data quality (DQ) checks, and handles them independently.

You can learn more about how TIE-D functions as a tireless maintenance assistant by watching Bryan’s talk at the July 2026 DET Bay Area Meetup.

You've been growing OpenAI's Data Engineering and Analytics Engineering team. Given everything agents now handle, what do you screen for in candidates, and what makes someone stand out?

We focus a lot on design and architecture. I think the fundamentals of Data Warehousing will always be a critical skill. For a while, our industry favored software engineering skills more because that was the difference in being able to produce data or not. Frameworks were commonly celebrated because of the velocity that they enabled. But now we have near infinite velocity and the bottleneck is only being able to decide what we want to build. The price of building the wrong thing is higher than ever because a bad design decision can propagate so quickly. So I think you can never go wrong with mastering good data modeling foundations.

Key Takeaways

• AI agents make writing pipeline code nearly instantaneous, shifting the core bottleneck to system design. Sound data warehousing and modeling fundamentals are more critical than ever, as poor design choices now propagate at agentic speed.

• Turning LLMs into reliable data agents requires a “harness”, which is a structured execution environment provided with explicit rules and contexts. Translating tribal knowledge into documented standards prevents model errors and eliminates long-term integration debt across teams.

• Start agents on cleanup and deprecation tasks to remove legacy patterns before they contaminate model context. For complex net-new features, engineers should direct high-level architecture while assigning end-to-end execution (coding, testing, backfilling) to structured execution plans.

• Despite shifts from manual CSV uploads to automated agents, data quality, exception handling, and creative problem-solving remain unchanged. Master these foundations, document technical insights publicly, and treat the IC and management tracks as complementary skill sets for overall growth.

Community poll

💬 How to stay connected

• Bryan’s LinkedIn

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Thanks for reading Data Engineer Things! Subscribe for free to receive new posts and support our work.

Hello Everyone,

This is Ananda again, writing from Philly. Every few years, data engineering gets a new “hard part.” First, it was scale. Then it was real-time. Now it's context and the harnesses we build around it, the scaffolding that lets an AI system, or an agent acting on its behalf, actually reason about a warehouse full of tables instead of just querying it blindly.

That raises a common question this edition keeps circling from different angles: what does it actually take to make a data system trustworthy enough for a person, a model, or an agent to rely on it?

For some teams, that means deliberately building context, curating the business knowledge an AI system needs rather than assuming raw schemas will speak for themselves. For others, it means treating schema change as a coordination problem that spans systems, not a single isolated edit, or reconciling outputs from models that interpret the same data in entirely different ways and on entirely different timelines. And for teams building on coding agents, it means designing the feedback loops that let an agent's output be trusted before it ships.

This edition spans a wide range, from large-scale data infrastructure choices to the less visible work of context, schema, and harnesses that make AI-assisted systems reliable.

– Ananda

📚 Data Pulse

The Context Layer Behind Spotify’s Data Assistant

📖 Topic: Data and Context Engineering
🧠 Level: Intermediate

Summary: Spotify built an AI data assistant (Vedder) to help teams query its 70,000+ dataset warehouse in plain English, recognizing that raw schemas alone can't capture the business context an LLM needs to write reliable SQL. The core innovation is the "cluster" model: domain-owned bundles of relevant tables, vetted question-SQL example pairs, and supplementary documentation, all curated by the data experts who actually understand that slice of the business.

💡 Why is this relevant for DEs?

• Semantic/context layer as infrastructure: This reframes the "semantic layer" problem data engineers already wrestle with (documenting tables, defining metrics, capturing knowledge) as a first-class system component, structured, versioned, health-monitored context (datasets + vetted query pairs + docs) that directly feeds a production AI system.

• Curation over automation validates governance investment: The finding that only 12.5% of auto-mined query history was usable as training examples is a strong data point for engineers arguing against "just scrape the logs" approaches to documentation/metadata. It shows human domain review remains the bottleneck worth investing in, not a step to skip.

Automated Schema Evolution: Pinterest’s Next-Gen DB Ingestion Framework

📖 Topic: Databases and Data Engineering
🧠 Level: Intermediate

Summary: Pinterest built an automated schema evolution framework for its CDC-based ingestion pipeline (Kafka, Flink, Spark, Iceberg), treating schema changes as a cross-system contract rather than an atomic operation, since a single upstream change requires coordinated updates to generated code, storage schemas, and bootstrap logic across every stage. The system deliberately restricts automation to additive, backward-compatible changes (new columns, numeric precision widening) and pushes riskier changes like type narrowing or primary key edits into manual migration paths, using both push-based (DDL-triggered) and pull-based (daily diff) detection to catch drift.

💡 Why is this relevant for DEs?

• Reusable pattern for a universally painful problem: Schema drift breaking downstream pipelines is one of the most common CDC/ELT failure modes; the staged convergence model (schema → code → data) and the “additive-only” safety boundary are directly adaptable design patterns for anyone running Debezium/Kafka/Flink-style ingestion, not just Pinterest-scale teams.

• Concrete answer to “when is eventual consistency okay?”: The SLA-based approach tolerating temporary nulls in Iceberg while Flink/Spark catch up is a well-reasoned tradeoff data engineers can cite when deciding whether their own pipelines need atomic schema changes or can accept a bounded convergence window, which has real cost/complexity implications.

• Practical techniques for ambiguous DDL: The binlog-based audit trail for resolving ambiguous create table diffs (rename vs. drop-and-add) addresses a specific, recurring headache in schema-diffing tooling and is a useful reference for data engineers building or evaluating similar catalog/DDL-tracking systems.

How Netflix is Using Multimodal AI to Power Video Search

📖 Topic: Data Engineering and Multimodal AI
🧠 Level: Advanced

Summary: Netflix built a multi-modal search system to let editorial teams find specific moments across 2,000+ hours of raw footage per season, running an ensemble of specialized AI models (character recognition, scene classification, dialogue transcription) rather than one generalist model, since each excels at its specific task but produces incompatible output formats and misaligned time intervals. The core engineering challenge was fusion: a three-stage pipeline first persists raw model outputs to Cassandra with zero transformation, then an offline job normalizes everything into one-second temporal buckets and merges overlapping annotations into unified records via upsert operations, and finally indexes those fused buckets into Elasticsearch as nested documents for querying.

💡 Why is this relevant for DEs?

• Data architecture for heterogeneous data sources: The decoupled three-stage pipeline (raw persistence → offline fusion → serving index) is a directly transferable architecture for any data engineer dealing with heterogeneous data sources on misaligned timelines or schemas, not just video, but IoT sensor streams, log correlation, or any scenario where multiple producers write disparate formats about the same entity.

• Concrete tradeoff reasoning: The explicit choice of throughput-over-freshness (offline batch fusion vs. real-time) and the upsert-with-composite-key approach to idempotent, incremental enrichment are patterns data engineers can reuse directly when designing their own late-arriving-data or multi-model-output merge jobs, plus a clear worked example of why bucket/grain size (1-second buckets) is a real design decision with volume tradeoffs (7.2M buckets from one archive).

• Data persistence justified by workload: Using Cassandra for high-throughput write-heavy ingestion and Elasticsearch for hybrid keyword+vector query serving each database used for what it’s actually good at is a clean case study for data engineers evaluating when to split storage layers by access pattern.

Reddit Data Tech Stack

📖 Topic: Data Infrastructure & Engineering
🧠 Level: Beginner

Summary: Reddit runs its massive data platform (120M+ daily active users, billions of comments, 500+ Kafka brokers processing tens of millions of messages/second) primarily on AWS, with GCP added via a commercial partnership that brought in BigQuery and Vertex AI. Kafka serves as the central event backbone, fanning data out to Flink (branded “Snooron,” built for real-time content safety rules) for stream processing and to Spark for batch reporting pipelines that feed Druid for real-time analytics; Debezium handles database change-data-capture into Kafka. Data is orchestrated with Airflow, stored raw in S3, and warehoused in BigQuery after migrating from Redshift.

💡 Why is this relevant for DEs?

• Real-world reference architecture at extreme scale: shows a concrete, battle-tested pattern (Kafka → Flink/Spark → Druid/BigQuery, with Debezium for CDC) that DEs can benchmark their own designs against, especially for systems that need real-time processing alongside batch analytics.

• Illustrates pragmatic multi-cloud trade-offs: Reddit’s AWS-for-core-infra and GCP-for-BigQuery/Vertex-AI split is a useful case study for DEs evaluating when cloud choices are technical vs. business-driven and how to run hybrid-cloud data platforms in practice.

• Concrete tool-selection rationale: explains why specific tools were chosen for specific jobs (Flink for low-latency safety rules, Spark for batch reporting, Druid for fast aggregated queries, Airflow for orchestration), giving DEs a framework for matching tools to workload characteristics rather than defaulting to one-size-fits-all.

Harness engineering for coding agent users

📖 Topic: Agentic Engineering
🧠 Level: Intermediate

Summary: The author proposes a mental model for "harness engineering" everything around a coding agent besides the model itself built on two control types: feedforward "guides" that steer the agent before it acts, and feedback "sensors" that let it self-correct after, each of which can be computational (fast, deterministic tools like linters and tests) or inferential (slower, probabilistic LLM-based judgment).

💡 Why is this relevant for DEs?

• Data pipelines already have sensors: Schema validations, data quality checks, and lineage tools map directly onto the feedback controls, meaning data engineers can bolt on harnesses to AI-assisted pipeline code fairly cheaply compared to teams without that tooling.

• “Continuous drift” is a core data engineering problem: schema drift, stale data, degrading freshness SLAs, and upstream dependency changes are exactly the kinds of ambient, out-of-lifecycle issues that need dedicated sensors. The framework provides a language for building agent-monitored drift detection into data pipelines.

• The “behavior harness” gap is evident for data transformations: verifying that a data transformation is semantically correct is hard when the tests themselves may be AI-generated, which matters a lot in data engineering, where silent logic errors in transformations can corrupt downstream data without ever throwing an error.

The Human Layer of Technical Work

📖 Topic: Career Development
🧠 Level: Beginner

Summary: “Human skills” like communication, empathy, and psychological safety matter as much as technical expertise for data professionals, especially as AI takes on more of the technical work itself, since AI can imitate connection but not genuine human presence. Human skills are as learnable and systematic as technical skills, and they can be developed through small, repeatable practices (pausing before reacting, asking a genuine question before presenting solutions) rather than treating them as innate traits. Most workplace friction technical professionals experience (defensive colleagues, meetings gone wrong, poorly received rollouts) has a human root cause, not a technical one.

💡 Why is this relevant for DEs?

• Concrete tactics for cross-functional work: DEs constantly translate technical concepts (schemas, data quality issues, root-cause explanations) for non-technical stakeholders; the advice on simplifying without undermining a colleague and on reframing “fix the dashboard” requests as source-data issues is directly applicable to that daily friction.

• Frames soft skills as a learnable, iterative system: appeals to a technical audience by treating communication and psychological safety like any other skill to practice and measure (small repeatable habits: pausing, asking questions, checking in after a tense exchange), making it easier for DEs to actually act on rather than dismiss as vague advice.

🗓 DET Bay Area Meetup on July 23

Join us at Microsoft's Mountain View office on Thursday, July 23rd, for an evening of technical talks, networking, and good food.

• When: 5:00 PM to 8:30 PM on Thursday, July 23rd

• Where: Microsoft’s Mountain View office

• 👉🏼 RSVP

(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)

💎 Open Source Gem

OLake Go: Efficient, quick, and scalable data ingestion for real-time analytics.

OLake Go is a high-performance platform that replicates data from operational datastores (PostgreSQL, MySQL, MongoDB, Oracle, Kafka, DB2, MSSQL, S3) into open lakehouse storage, writing either Apache Iceberg tables (with catalog support for AWS Glue, Hive Metastore, and REST/JDBC catalogs like Nessie, Polaris, and Unity Catalog) or Parquet files on object storage such as S3, MinIO, and GCS. It supports Full Refresh, Incremental Sync, and Change Data Capture depending on the source, with core capabilities including parallelized chunking for faster large-dataset scans, stateful/resumable syncs that recover from interruptions without full resyncs, and per-job configurable connection limits.

💡 Why is this useful for DEs?

• Faster, more reliable data pipelines: parallelized chunking speeds up large full-table scans, while stateful/resumable syncs mean a crash or network blip doesn’t force a costly resync from scratch, cutting both processing time and manual intervention.

• Cleaner data with less pipeline engineering: automatic primary-key deduplication, schema evolution, and JSON normalization (flattening nested fields into columns) handle work DEs would otherwise hand-code, while two-phase commits prevent duplicate writes or inconsistent state on the Iceberg side.

• Flexibility to fit existing lakehouse architecture: support for multiple Iceberg catalogs (Glue, Hive Metastore, REST catalogs like Nessie/Polaris/Unity) and both Iceberg and plain Parquet outputs means DEs can plug it into whatever stack they already run, rather than being locked into one catalog or storage layout.

👉 GitHub: https://github.com/datazip-inc/olake

💡 DE Tip of the Month

Read Replica vs. CDC: Choosing how to offload reads from your source database

Read replicas answer "how do I stop hammering my primary with more of the same kind of query?" CDC answers "how do I ask fundamentally different questions of this data, cheaply, with history?" They're not mutually exclusive; increasingly, teams run both from the same source, routing each consumer to whichever pattern fits its query shape.

📒 Rules of thumb

When to choose a Read Replica

• Same-engine fit: Consumers need identical SQL dialects, indexes/transactional guarantees to the source (OLTP-shaped tools, point lookups) with near-zero freshness lag.

• Low complexity, moderate scale: Minimal new tooling/schema to maintain beyond monitoring replication lag, cost-effective only while scale stays moderate, since replica compute tracks primary engine pricing.

When to choose CDC into a cheaper store

• Analytical fit, decoupled cost: Workload is analytical (aggregations, joins, trend analysis) where OLTP engines underperform, CDC lets you pay scan-optimized storage/compute rates instead of OLTP-tuned pricing.

• History and fan-out, less source pressure: CDC’s append-only change stream gives you history/time-travel and can feed multiple consumers (warehouse, search, cache, ML) from one pipeline, without adding replication load on the source as consumer count grows.

📊 Community Poll

Until the next one.

Ananda, Sukanya, & Chozhan

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

In our next community spotlight, we are chatting with Vikram Koka, Chief Strategy Officer at Astronomer and Apache Airflow PMC (Project Management Committee) member.

Vikram has been part of Airflow’s modern evolution since 2019, contributing to major milestones including Scheduler High Availability in Airflow 2.0, data-driven scheduling, dynamic tasks, setup and teardown, and the program management of Airflow 3. Many of you may also recognize him from Data Engineering Open Forum 2026, where he joined the closing panel, Frontiers of Data: The Future of Data Engineering in an AI World, alongside leaders from Netflix and OpenAI.

In this interview, we discuss Vikram’s journey across three decades of data engineering, Airflow’s evolution, why data drift remains a constant challenge, and how AI is expanding the role of data engineers from moving data to understanding context and building reliable workflows.

Let’s dive in and walk through Vikram’s journey.

- Shubham & Eddy

🗓 DET Meetups in Toronto and Seattle

We have two meetup events coming up this month:

• Seattle meetup at Downtown Seattle on Thu, July 16 (RSVP)

• Toronto meetup at Downtown Toronto on Thu, July 16 (RSVP)

(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)

Spotlight: Vikram Koka

For people who may not know your journey, how would you describe the path that brought you here?

My journey really kicked off during my master’s program at UC Santa Barbara, where I worked on the GNU Debugger (GDB), which is now distributed as part of Linux. That was my early introduction to open source quite a long time ago.

From there, I moved to Silicon Valley and got into manufacturing automation for semiconductors. My first project was in user interfaces, but my second project shifted into the data space where I was gathering data from manufacturing systems to provide process analysis. That organically evolved into building a full-fledged data warehouse pulling from manufacturing, supply chain, and financial systems to build dashboards and star schemas. I technically wrote my first data pipeline about three decades ago without ever knowing that “data pipeline” would become the actual term for it!

Since then, I’ve been through a variety of Silicon Valley companies. My last gig before Astronomer was in IoT, where I dealt with the massive explosion of data coming from embedded devices. We had to microbatch events over cellular networks purely for financial reasons, because sending data on every single event is just too expensive. That sparked my desire to get back heavily into data. When the opportunity at Astronomer presented itself in 2019, I saw the massive potential of Apache Airflow and the impending data explosion, and I leaped right in.

If you were to reflect back on your career, what did you feel was a decision that had the biggest impact on your trajectory?

Honestly, the biggest impact was my willingness to be non-linear.

We are often taught straight out of school to look at career trajectories in a very linear fashion. I was completely willing to throw that out the window. My primary focus has always been on what I enjoy most, what needs to change, and a willingness to jump across completely different roles and domains. Whether I was acting as an executive, an individual contributor, working in open source, or working in commercial software, the title didn’t matter. I just wanted to make a difference.

As a technologist, your core desire should be deeply understanding why someone wants a problem solved and having the empathy to make their life better through technology. Remaining open to non-linear shifts allowed me to focus on that.

What is the most proud you’ve been solving a problem? Is there a challenge that stands out?

I’m honestly not sure I can pick just one. I think this is true for most engineers: before you solve a problem, you look at it and think, “Oh man, this is incredibly complicated. How are we ever going to pull this off?” But the moment you crack it, you look back and say, “Huh, that actually wasn’t all that difficult after all.” I used to see this all the time with engineers when filing patent applications; once the problem is solved, they feel the solution is totally obvious.

Because of that, I don’t really dwell on past problems. I look forward. I have never been more excited about the scale of problems we can solve right now. We are standing on the shoulders of giants. With the current state of AI, computing infrastructure, and cloud evolution, we can tackle massive problems today that were completely impossible before. It’s always about the next problem for me.

At the time when you joined Apache Airflow, what did you see were the biggest limitations, and how has the project evolved?

I officially got involved in December 2019. At that time, Airflow had roughly 100,000 to 180,000 downloads a month, and the absolute biggest hurdles were reliability and enterprise scalability. Essentially, user code was running directly inside the scheduler, causing it to crash frequently because it wasn’t designed for high availability.

Recognizing this, I co-authored an Airflow Improvement Proposal (AIP) to build a highly available, active-active scheduler. We released it as part of Airflow 2 in December 2020. That completely catapulted the project’s growth because enterprises could finally deploy it reliably at massive scale. Today, Airflow sees tens of millions of downloads a month and is utilized by an estimated 80,000 to 90,000 enterprises globally.

Looking at Airflow 3, the project has taken another massive leap forward based on three pillars:

• Run tasks anywhere: You are no longer restricted to a single cluster or data center; you can dynamically run tasks across local CPUs, remote GPUs, etc.

• Run tasks in any language: While Airflow is natively Python-based, we have expanded support to execute tasks written in Java, Go, and soon TypeScript.

• Run tasks at any time: Moving past strictly batch schedules to support API triggers and message queues, making it perfectly suited to handle heavy AI training and inference workloads.

What are some lessons from Airflow’s evolution that data engineers can apply when building their own internal platforms?

The number one lesson is: Don’t get locked in. Prioritize optionality.

The biggest consistency in data engineering from my very first pipeline 30 years ago to today is data drift. Data formats change constantly, and the semantic meaning of data shifts over time, which regularly throws software engineers for a loop. Because change is inevitable, your tools and frameworks must remain highly adaptable.

What made Airflow incredibly successful is that we intentionally chose to be completely unopinionated. We refused to pick winners among databases, file systems, or cloud providers. We joke internally that we are very opinionated about being unopinionated. For instance, with AI workloads today, people are heavily reliant on commercial vendor models like OpenAI or Anthropic. But we built Airflow with the optionality to easily support self-hosted, open-source models tomorrow because security, privacy, and architecture needs will inevitably change. Build your internal platforms with that exact same flexibility.

What do you think is the role of data engineering with AI these days?

The role of data engineers has dramatically expanded, and engineers need to embrace that. The technical elements like knowing how to construct a pipeline remain fundamentally important. However, the semantic understanding of data is now paramount. This means understanding what the data actually represents from a strict business context standpoint.

In the context of AI, the objective isn’t just to dump a massive haystack of data on an LLM and assume it will find the needle. If you do that, it will hallucinate heavily and cost you a fortune.

To do that successfully, you have to deeply understand the business imperatives and context. It is a non-trivial, holistic challenge, but a massive opportunity for the field.

What did you learn by automating your own work process?

💡Editorial Note: As a PMC member of Apache Airflow, Vikram tracks many Airflow Improvement Proposals (AIPs) across Confluence specs, GitHub PRs, and the repository file tree. Every week, he was spending hours cross-referencing these sources to understand what had shipped, what was still open, and where things actually stood. So he decided to build an AI skill to do that work for him.

👉 Read Vikram’s full write-up HERE.

With that context, here is Vikram’s answer:

When I set out to automate my own daily job of tracking all the proposals, Confluence specs, and GitHub PRs happening across the open-source Airflow community, it turned into an absolute voyage of discovery.

The biggest thing I learned is that if you just give an AI agent a prompt and let it autonomously call tools to gather data, the potential for drift and hallucination skyrockets. For instance, if you ask an agent to figure out if a project is “70% complete,” it really struggles. As humans, we know one PR might be massive while another is tiny, but a stochastic model tries to guess and make assumptions, which drives you nuts.

To solve it, I had to break the process down into a 12-step structured pipeline:

• Use APIs for Data Collection: I stopped letting the agent collect the data. Instead, I used standard, deterministic APIs to pull the raw text from Confluence and GitHub.

• Isolate the Agent for Reasoning: I fed that specific data to the agent and restricted its job strictly to reasoning over what was provided.

• Layer standard code on top of AI: I introduced an evaluation model to check the reasoning, but I used a traditional, deterministic Python process to actually apply the final corrections.

By separating what an agent is actually good at (reasoning) from what deterministic code is good at (data collection and structured execution), I finally got a reliable, repeatable report.

But the most surprising lesson of all was that this pipeline approach was actually cheaper and faster. A pure agentic workflow builds up context with every sequential tool call, causing token usage to explode quadratically O(n²). By orchestrating it as a pipeline where steps are independent, token growth stays linear O(n).

Ultimately, it taught me a fundamental truth about modern orchestration: the best way to get predictable, deterministic outcomes out of AI agents is to wrap them inside a structured data pipeline.

When you’re hiring data engineers today, what are some qualities or signals that tell you someone is likely to have an outsized impact?

There are three key signals I look for:

• Technical Competence: This is the baseline requirement. You need enough technical depth to apply solid system-level thinking to complex architectures.

• Curiosity: I value intense curiosity because it drives an engineer to learn the semantic nature of data and understand its broader impact on the organization.

• Business Correlation: The best engineers can step outside of the technical box and connect their code directly to business imperatives. They don’t just build a pipeline because they were told to; they ask why someone needs this to happen and map the technical tools directly to that business outcome.

What lessons from managing commercial products translate well to open-source projects, and what doesn’t apply?

What translates perfectly: You must always ask yourself, “Why does a user or customer actually care to use my product?” Whether it’s a paid commercial enterprise system or a free open-source project, the core human element is identical. You have to listen to your users, understand their pain points, and focus intensely on solving their specific problems.

What is completely different: The dynamics of alignment and community in open source. In a commercial setting, you have organizational authority. In a project like Airflow, you are dealing with a massive community, currently around 3,800 total contributors, with 300 to 500 actively writing code every single month. These people are contributing out of pure passion on their weekends, late nights, and days off.

Managing this is a delicate balancing act. You have to encourage them, value their time, and review their PRs quickly, or they will walk away. However, you also have to maintain strict alignment with the long-term architectural goals, code maintainability, and security implications of the project. Saying “no” to a passionate volunteer because their contribution doesn’t fit the long-term roadmap is infinitely harder than managing product alignment in a commercial business.

Who should we spotlight next in the Data Engineer Things Community Newsletter, and why?

I would say Laura Pruitt from Netflix or Paul Ellwood from Open AI.

I was very impressed by the conversation we had at the Data Engineering Open Forum earlier this year. From a business impact perspective and the quality of thinking about how data engineering is evolving, I think Laura would be a great conversation for the DET community.

We discussed Data Drift as a key consistent challenge in Data Engineering since I wrote my first data pipeline three decades ago.

Paul touched on AI drift as part of the Forum panel discussion. I think that would be a great topic to have Paul expand on.

💡 Editorial Note: The discussion Vikram references came from the DEOF 2026 closing panel, Frontiers of Data: The Future of Data Engineering in an AI World. The panel covered many of the same themes from this spotlight, including data drift, AI drift, semantic context, and the evolving role of data engineers.

Watch the full panel here:

Key Takeaways

• Semantic understanding is now part of data engineering: Data engineers need to understand business context, not just move data, so they can provide AI systems with the right context.

• Optionality is a core platform principle: Airflow’s success comes from staying flexible across databases, clouds, runtimes, and model providers.

• Data drift remains the constant challenge: The interview connects long-standing data drift with newer challenges like semantic drift and AI drift.

• High-impact data engineers connect tech to business: The strongest engineers combine technical depth, curiosity, and the ability to map their work to business outcomes.

Community poll

💬 How to stay connected

• LinkedIn

• Astronomer

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello Folks,

As data platforms evolve, the hardest problems are no longer purely technical; they’re structural.

How do you maintain consistency as products expand? How do you trust AI-generated outputs without sacrificing governance? How do you measure engineering impact when traditional metrics are being reshaped by automation? And how do you keep increasingly complex systems observable and reliable?

This edition explores how leading teams are tackling these challenges—from data modeling and semantic layers to engineering metrics, observability, and platform design.

– Sukanya

📚 Data Pulse

Scaling Beyond One: How Airbnb Evolved Its Data Architecture for a Multi-Product World

📖 Topic: Data Modeling & Analytics Engineering
🧠 Level: Beginner

Summary: Airbnb began as a single product - home-sharing. As the company expanded into Experiences and Services, the original modeling approach wasn’t enough as the definitions drifted between product lines, the same concepts were modelled differently across teams, and analytics engineers spent increasing time reconciling inconsistencies rather than producing insights. This post describes how Airbnb’s data and analytics engineering teams designed a consistent, flexible data modeling framework capable of spanning multiple product verticals without collapsing into a single rigid schema. And that involves 3 key principles:

1. No hybrid data models

2. Consistent identifier naming

3. Clear namespace organization

And then the modelling guidelines as a framework for every domain team to use in analyzing their specific situation

💡 Why is this relevant for DEs?

• The single-product-to-multi-product transition is a forcing function for data modeling discipline. Teams that got away with informal modeling conventions in the early days often hit this wall when the business expands. This post is a useful reference for how to think about the transition before it becomes a crisis.

• The consistency-vs-flexibility tension is the core data modeling challenge. Airbnb’s frames it shared definitions at the core, product-specific extensions at the edges. It is a pattern that generalises well beyond their specific domain. It’s a practical answer to the question of how much standardisation is too much.

• Analytics engineers and data engineers will both find value here. The post covers both the modeling decisions (which sit closer to analytics engineering) and the platform choices that make those decisions enforceable at scale (which sit closer to data engineering). It’s one of the better cross-functional treatments of the topic.

The 8 Software Engineering Metrics AI Broke

📖 Topic: Engineering Metrics & Measurement in the AI Era
🧠 Level: Beginner

Summary: LeadDev’s June 2026 article makes a simple but uncomfortable argument: most of the metrics engineering teams rely on like deployment frequency, cycle time, PR volume, test coverage, lines of code were built on the assumption that human effort and output move roughly in proportion. AI coding tools have broken that assumption. A developer with an agent can ship multiple PRs in a day without any improvement in engineering maturity; cycle time compresses while technical debt accumulates; test coverage written by the same model that wrote the code means something different than it used to. The article walks through eight specific metrics that have been distorted or invalidated, and identifies three that still hold up with a clear explanation of why.

💡 Why is this relevant for DEs?

• Data engineering teams use these same metrics. Pipeline deployment frequency, time-to-merge for data model changes, test coverage on dbt models all of these are subject to exactly the same distortions the article describes. If your team is measuring AI productivity using pre-AI proxies, the numbers are probably misleading you.

• Goodhart’s Law has always applied; AI makes it effortless. Gaming metrics used to require deliberate effort. With AI coding tools, inflating deployment frequency or PR volume is a side effect of normal use, not a sign of bad intent. That changes how you need to think about metric design entirely.

• The three metrics that still hold up are worth identifying for your own stack. The article’s affirmative case of what to measure instead is the most actionable part and gives you a starting point for rebuilding dashboards that remain meaningful in an AI-assisted engineering environment.

How to Make Semantic Layer Work for Analytics Agents

📖 Topic: Semantic Layers & AI Analytics
🧠 Level: Intermediate

Summary: This piece explores why semantic layers are becoming foundational infrastructure for AI agents interacting with business data. Instead of letting LLMs generate raw SQL directly against warehouses (which often leads to inconsistent metrics, hallucinated joins, and governance issues), the article argues for exposing curated business logic through a semantic layer. It walks through how metrics definitions, entity relationships, dimensions, and governance rules can be standardized so AI agents operate on trusted abstractions rather than raw tables. The post also explains how semantic layers improve discoverability, consistency, and interoperability across modern analytics stacks while enabling more reliable natural-language querying.

💡 Why is this relevant for DEs?

• This is one of the clearest explanations of why semantic layers matter again in the AI era. For years, semantic layers were treated as a BI concern. Now they’re becoming the control plane between LLMs and enterprise data systems. If your company is experimenting with AI-powered analytics, this architectural shift is worth understanding early.

• The article highlights a practical reality many teams are already facing: AI-generated SQL is only as reliable as the metadata and governance underneath it. Centralized metrics definitions, entity modeling, and governed dimensions are what prevent agents from producing contradictory answers across dashboards and chat interfaces.

• The interoperability angle is especially important. As more organizations adopt tools like dbt, MetricFlow, Cube, AtScale, or Looker semantic models, the semantic layer increasingly acts as a shared contract between analytics engineers, BI tools, and AI systems. Understanding this pattern will likely become a core skill for modern data engineers and analytics engineers alike.

And this LinkedIn thread is worth a read.

The Evolution of Cassandra Data Movement at Netflix

📖 Topic: Operational Data Movement & Lakehouse Ingestion
🧠 Level: Intermediate

Summary: Apache Cassandra powers mission-critical systems at Netflix: Member, Billing, Recommendations, Subscriptions. Moving that data into Apache Iceberg for analytics has always been a core need, handled for years by an in-house connector called Casspactor. But as Netflix built higher-level abstractions on top of Cassandra - Key Value, Time Series, and Graph, the movement requirements grew more complex: transformations became necessary, schemas became non-trivial to map, and a single connector was no longer sufficient. This post walks through how Netflix evolved their Cassandra-to-Iceberg data movement architecture from a basic export connector to a more capable, abstraction-aware system that handles the full range of Cassandra-based data models and how it all sits within their broader Data Bridge platform.

💡 Why is this relevant for DEs?

• The operational-to-analytical bridge is a problem every production data team faces. Cassandra is widely used for transactional workloads; Iceberg is increasingly the standard for analytical storage. How you move data between them reliably, at scale, with schema fidelity matters a great deal. Netflix’s approach is one of the more detailed public accounts of how to do it.

• The evolution story is the valuable part. The post doesn’t present a finished system it shows how a connector built for one abstraction layer had to be redesigned as new abstractions emerged above it. That pattern of “the infrastructure you built for today’s complexity doesn’t survive tomorrow’s” is something worth understanding structurally than just technically.

Brain the Size of a Planet: Are LLMs Thonking Too Hard?

📖 Topic: LLM Reasoning Effort & Practical Model Selection
🧠 Level: Advanced

Summary: Security researcher Parsia Hakimian ran a rigorous empirical experiment across 26 combinations of Claude 4.6/4.7 and GPT-5.4/5.5 models varying reasoning effort levels (low, medium, high, xhigh) and context window sizes to test how well LLMs triage real security vulnerabilities. The results challenge a widely held assumption: higher reasoning effort is not always better, and later models don’t uniformly outperform earlier ones. GPT-5.5-medium outperformed high and xhigh on several tasks; full solve rates were nearly zero (1.9%) across all models; and a four-LLM triage council with majority voting: 86.2% unanimous agreement performed far more reliably than any single model. Total experiment cost: roughly $9,200. The methodology is detailed, reproducible, and the data is publicly available on GitHub.

💡 Why is this relevant for DEs?

• The “more reasoning = better results” assumption needs questioning in data contexts too. If you’re building LLM-powered pipeline generation, metadata enrichment, or data quality classification, the implication is important: defaulting to maximum reasoning effort increases cost and latency without a guaranteed quality improvement. The right effort level depends on the task structure, not a general rule.

• The council/ensemble approach is a practical architecture pattern. Rather than trusting a single model call, running the same task across multiple models with majority-vote reconciliation is a concrete technique for improving reliability in agentic data workflows particularly for high-stakes decisions like anomaly classification or schema change validation.

• Function-level context significantly outperformed file-level context. Giving the model exactly the code or data schema it needs rather than dumping an entire file produced dramatically better results. This maps directly to how data engineers should be structuring prompts when using LLMs to reason about pipeline logic or data contracts: smaller, precise context wins.

💎 Open Source Gem

Viaduct 1.0 — Airbnb’s Production Data Mesh, Now Open Source

Viaduct is Airbnb’s data mesh platform: a system for registering, discovering, and governing data products across a distributed, domain-owned data organisation. It handles the hard parts of data mesh: how do domains publish data products in a way that others can discover and trust? how does the platform enforce quality standards without centralising ownership? Without prescribing a fixed architecture for every team. The 1.0 release (May 2026) marks the transition from an internal Airbnb tool to a publicly maintained open-source project.

📰 What’s in 1.0

The initial open-source release includes the core data product registry, a quality enforcement layer that runs checks at registration and on a configurable schedule, a discoverability API and UI for finding data products across domains, and an access governance model that delegates ownership decisions to domain teams while maintaining platform-level audit trails. The CLI supports YAML-based product definitions (as above) as well as programmatic registration via Python and Go clients. Airbnb’s own internal deployment manages several hundred registered data products across dozens of domains.

💡 Why is this useful for DEs?

• It’s the first serious open-source reference implementation of data mesh at production scale. OpenMetadata and DataHub handle catalog and discovery well; Viaduct handles the harder problem of how data products are governed and owned across a distributed organisation, not just catalogued.

• The YAML-first product definition is approachable for small teams. You don’t need to adopt the full data mesh paradigm to get value from Viaduct’s quality and ownership primitives. Teams can start with a single domain and a handful of registered products and scale from there.

• Quality enforcement at registration time changes the dynamic. Rather than discovering data quality issues downstream when a consumer’s pipeline breaks, Viaduct’s check-at-registration model means problems surface when the producer publishes — which is the right place to catch them.

👉 GitHub: https://github.com/airbnb/viaduct

💡 DE Tip of the Month

Improving Data Observability in AI-Assisted and Agent-Driven Pipelines (Snowflake, Databricks, BigQuery)

As data platforms become more dynamic, with dbt transformations, LLM-generated queries, and agent-driven workflows, traditional pipeline monitoring is no longer enough. In Snowflake, Databricks, and BigQuery, failures often don’t show up as broken jobs but as silent data quality drift or unexplained downstream inconsistencies.

📒 Rules of thumb

• Instrument data at the query and job level, not just pipeline level.
  In Snowflake, use the information schema and QUERY_HISTORY to trace execution patterns. In Databricks, rely on system tables and job run metadata. In BigQuery, use job history and audit logs. Observability starts at execution granularity, not DAG-level success/failure.

• Track data lineage across transformations, especially in automated workflows.
  In Snowflake Dynamic Tables, dbt models, and Databricks Delta Live Tables, ensure lineage is explicitly captured so AI-generated or scheduled transformations don’t create “orphan” datasets.

• Monitor data drift, not just pipeline failures.
  In modern systems, pipelines can succeed while outputs become incorrect. Add checks for schema drift, distribution shifts, and freshness anomalies across all three platforms.

• Separate human-authored vs. agent-generated transformations.
  AI-assisted SQL and automated transformations should be tagged and tracked separately to understand where unintended changes originate.

• Centralize observability signals outside the warehouse.
  Don’t rely solely on Snowflake/Databricks/BigQuery logs. Push metadata into a unified observability layer so lineage, freshness, and quality signals can be correlated across systems.

📊 Community Poll

Until the next one.

Sukanya, Chozhan & Sri

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

In our next iteration of the community spotlight, we are chatting with Shridhar Iyer, Director of Data Engineering at Meta.

Shri has spent over a decade navigating and shaping the major shifts in data engineering. Today, he is at the forefront of transitioning traditional data systems into AI-native architectures. In this spotlight, we dive into his journey from software engineering to unifying Meta’s massive data foundations, explore how AI is rewriting the development workflow for data engineers and how to position yourself for success.

Let’s dive in and walk through Shri’s journey.

- Eddy Zulkifly & Swetha Sekhar

Spotlight: Shridhar Iyer

For people who may not know your journey, how would you describe the path that brought you here?

I started my career in software engineering working on supply chain, logistics and enterprise systems. Early on, I learned that leverage wasn’t in building the systems, but rather understanding the business deeply enough to influence it. At a retail analytics startup, I got my first taste of what it meant to scale analytics across Fortune 500 companies and found myself drawn to systematizing and unifying the chaos of different datasets and process. The work entails schematizing workflows and building tools that made fragmented processes legible. That impulse followed me to Meta.

At Meta, I started by building the analytical foundations that helped measure how people used Facebook. I curated datasets that became central to how the company understood itself and developed frameworks that standardized how growth and engagement were measured across the entire product. Later on, I moved into the Search team and worked on real-time streaming, sessionization and bridging business logic between batch and real time. Each experience exposed me to a new layer of fragmentation and each time, my instinct was to unify it.

That drive toward unification made me realize that making data accessible to more people across a complex platform required more than pipelines and dashboards. It required data to be understandable at all points of development and doing so securely.

What has changed most dramatically in data engineering during your time at Meta?

Having lived through every major shift in data engineering over the past decade, AI has been a dramatic shift in terms of data engineering work. Distributed compute, new storage formats, streaming and CI/CD changed how the work was done, but left the fundamental contract intact. A data engineer expressed logic in SQL or Python, wrapped it in pipelines, and governed it through pull requests. The interface between human and machine stayed the same.

AI breaks that contract entirely.

The nature of the work changes, from physical to conceptual, from writing logic to specifying intent and from governing code to governing knowledge. The craft is no longer in the code. It’s in the quality of the question, the rigour of the eval and the integrity of the context you feed it. Writing good evals is a hard problem. Designing agentic workflows with the right guardrails is a hard problem. Knowing what knowledge actually matters to an AI agent, and governing it well, is a hard problem. Even measuring the output from AI reliably has been harder than expected. The work is no longer about using AI as a tool but rather becoming AI-native. It’s about rebuilding how you think, design, and measure from the ground up.

Existing codebases have to be made AI-ready with appropriate tradeoffs. The development lifecycle itself is up for reinvention as spec-driven development and agentic loops are replacing what were once manual workflows. What constitutes good work is shifting from optimizing the science to encoding the art. Turning repeatable expertise into skills, expressing domain knowledge as governed context and designing for a machine that reasons differently than we do.

You’ve written about unifying semantic context across data systems. What does that look like in practice?

A while back I wrote about the intelligence pyramid: Data → Information → Knowledge → Wisdom, and more recently revisited it through an AI lens. It’s a practical blueprint for how semantic context for data can be built, evolved, and directed towards AI Agents. Each layer builds on top of the other.

Historically, the ROI for semantically modeling data for analytics alone was hard to justify at scale. AI changes this by making human workflows drastically cheaper but it requires structured knowledge to become effective.

The raw schema & data layer is where most warehouses live today. Physical tables with primitive column types that don’t convey meaning or purpose. Making this layer semantically understood required introducing richer schema annotations, a type system that encodes meaning at the column and table level, and a unified taxonomy governed at source in online systems that flows into the offline warehouse. This also required unifying the metadata and asset catalog, covering tables, columns, sub-columns, dashboards, and all asset classes, along with column-level lineage and the SQL language itself through a compiler. That effort was primarily driven by privacy and security, but it built the foundational layer of semantic understanding that everything above depends on.

The semantic layer encodes what the data means in a business context. Assets include a domain glossary, metric and dimension definitions, verified queries, and table selection guidance. This is the tribal and institutional knowledge that makes data of a domain genuinely useful but rarely governed or structured well enough to scale.

The knowledge layer encodes the analytical skills and methodologies that experts use to produce insights. The business process of growth accounting, retention, root cause analysis, opportunity sizing. Encoding them as governed primitives is what democratizes the craft. It allows non-SMEs the same analytical capability that previously required years of domain experience.

At the top, the data agent consumes all three layers to reason over enterprise data in real time, composing governed domain knowledge into trustworthy answers.

What we are formalizing is the long-term memory of the enterprise and the skills of its subject matter experts into a governed, composable system. All of this required the same design instinct at every layer: unify the representation, govern it at source, catalog it canonically, change-manage it at scale, and make it consumable, whether by a privacy enforcement system or an AI data agent.

How do you measure whether AI initiatives are truly successful in data engineering teams?

Measuring AI success in data engineering starts with recognizing that the work falls into two distinct but deeply interdependent buckets which are AI readiness and AI-native workflows.

AI readiness is about making your data systems legible to AI. Well documented schemas, semantic context, code and lineage that an agent can reason over. AI-native workflows are about reimagining the development and consumption lifecycle itself, both the inner loop of discovery, authoring, testing, review, and deployment, and the outer loop of scheduling, anomaly detection, quality enforcement, backfills, and privacy compliance, and the consumption side of visualization, reporting, analysis, and security enforcement.

The most common measure in practice has been adoption which can be measured by committed diffs, tool usage, active users. While adoption is a real signal, adoption alone masks the harder question, which is quality. Some productivity lift is nearly always present when using AI tools. The challenge is assessing if the output is trustworthy enough to act on. For example, instead of vaguely measuring "time saved," leaders must look at cycle-time reduction for specific, repeatable tickets, like tracking the average hours it takes to onboard a new data source or generate a new dbt model with an AI assistant versus without one.

That’s where evals become the more rigorous measure. Evaluating whether an AI agent produced the correct query or insight is genuinely difficult, especially for open-ended analytical tasks where there isn’t always a single correct answer. In data engineering, this means moving beyond simple code-syntax validation to behavioural evals. For instance, an inner-loop eval might test whether an AI agent can refactor a legacy SQL script into clean, optimized dbt code without altering the underlying data output. An outer-loop eval might test whether an AI agent can accurately root-cause a pipeline anomaly alert and point to the exact upstream schema change that caused it.

It also helps to think about measurement through these two distinct groups, namely builders and domain experts. Builders or engineers have taken to AI most naturally and construct the agentic workflows. Domain experts or the people who hold the institutional knowledge, are finding it harder to translate that expertise into AI leverage. The gap between these two groups is itself a measure of how much work remains.

The north star measure, autonomous agents planning, reasoning, and executing full analytical workflows with minimal human input, is still ahead of us. We are not yet there outside of the most obviously repeatable task patterns. What leadership can measure today are these metrics:

• Readiness Coverage: The percentage of production data models that possess verified, LLM-legible semantic tags and up-to-date lineage maps.

• Adoption Depth: The frequency and consistency of AI tool usage across both builders and domain experts.

• Eval-driven Quality Scores: The pass rate of AI-generated assets against rigorous regression test suites.

• Workflow Completion vs. Step Acceleration: The degree to which AI is autonomously finishing end-to-end workflows rather than just accelerating isolated steps within them.

Workflow completion, is perhaps the most honest indicator of where any team truly stands.

What is the biggest leadership shift from managing teams to operating at the Director level?

I should be honest about something upfront, I was never a traditional org manager in the strict sense. I have led teams both informally and formally as a tech lead manager, but my formal expectation has always been that of a tech lead, meaning I have always been held to an IC bar even at the director level. For the past few years I have been operating as a pure IC. So I think of this less as a shift from manager to director and more as a shift in how you operate and what you optimize for at a higher level of ambition.

The first shift is cognitive. As an IC, you are in the details. As a director, you are defining which problems are worth solving and setting the principles that guide multiple teams toward coherent solutions. The craft moves from execution to framing. Framing well requires the ability to see how the systems interact interact, where they conflict and articulating the problem concisely to a large group of people across different functions, roles, and levels. That is harder than it sounds.

The second shift is organizational, and for me it has always come back to the same instinct of unification. The biggest risk at this level is fragmentation where teams are building good things independently that don’t compose. Most of my largest initiatives have been cross-functional, spanning different orgs, roles, and levels, held together not by authority but by a shared conviction about what’s possible.

That conviction has to be earned. It starts with thinking like the exec in your reporting chain. What would a VP want to know about your product area? What does the future look like, not just for your product but for the function it serves? If you can’t answer that crisply, you are not ready to drive adoption of anything. You have to think objectively about what is best for the company, not just your team. When you frame your initiative around long-term company health rather than local goals, adoption stops being a sales pitch and starts being an obvious move.

Then there is clarity. I compensate for not being a natural presenter through writing, especially when it comes to articulating ambiguous problems. Going deep enough to produce a prototype, a proof of concept, or a clear vision is what makes the destination visible. When people can see the destination, they find their way toward it.

Pre-reads matter, but in front of a senior audience, most of the time is spent on discussion. Think through your questions deeply. What will decision makers push back on? What is the elephant in the room, and can you surface it explicitly? Influence at this level is less about presenting and more about orchestrating the right conversation. And if leadership is wrong, say so, but focus on alignment rather than proving someone wrong.

Orchestrating conversations this way builds a shared identity around the work scaling impact far beyond what any single team could achieve.

Where do you think AI genuinely helps data engineers today, and where does it create over-usage or overconfidence?

AI genuinely helps during the inner and outer development loops. In the inner loop, agentic workflows deliver the most value during coding and development, triggering autonomously on the right conditions and passing through existing guardrails like test cases, data quality checks, and human review gates. The outer loop follows the same logic, detecting what changed in the logic, handling backfills and failures, auto-healing where appropriate. Beyond these loops, everything depends on rigorous evals to validate and without them, you are only assuming effectiveness.

Overusage most commonly stems from not designing AI workflows purposefully. Almost every AI workflow is, at its core, an event-driven pipeline. When it is not designed that way, you end up reprocessing large volumes of information in unstructured formats, wasting tokens on work that could have been scoped and triggered more precisely. Another common example is not knowing which model is right for the job, reaching for a more capable and expensive model when a lighter one would suffice. Similarly, running evals on a recurring schedule to test changing context is a source of waste. The right design detects when a change happens and triggers the eval then, not continuously.

Overconfidence is the more consequential failure mode. Agents are built to find an answer, and they will, even when they should not. A silent infrastructure failure, a discovery service going down for something as straightforward as a daily active user query, will not stop an agent from finding another path and returning something that sounds authoritative. Without proper guardrails, agents take all context at face value, unable to distinguish good from bad. They can conflate similar concepts, confusing an app with a device, cross-wire descriptions, and confidently produce answers that are precisely wrong. The responsibility for governing that context and building the guardrails that prevent it sits entirely with the builder. That is not a limitation that goes away on its own. It is a design obligation.

What should the next generation of data engineers be preparing for now?

The next generation of data engineers is entering a world where the baseline job of producing data assets is getting commoditized. The differentiator is no longer who can build those artifacts but rather who can define the right spec in the first place. That is a much harder job, and it is the one worth preparing for.

In practical terms, that means owning semantics rather than scripts. What does “active user” actually mean in your business context? How do these entities relate? What are the golden structures that everything else should align behind? The ability to design a domain model that mirrors how the business actually thinks and operates will matter more than whether you can hand-craft the nth pipeline operator. Think of it as writing the markdown, a structured, precise expression of intent, that an agent then operationalizes. The quality of what the agent produces is a direct function of the quality of that spec. Treat data modelling as an act of meaning-making and not schema decoration.

The other half is evals. An agent can generate a metric, a dashboard, or a feature, but it cannot tell you on its own whether any of that is truly useful or aligned with the goals of the business. Being good at encoding goals as evals, defining what good looks like, how you would measure it, and what failure modes you are guarding against, will be one of the most important skills in an AI-saturated stack. It is the difference between the model works and the system is actually doing what we intended.

Underlying both is business fluency. If the last decade rewarded people who were very good at infrastructure and tooling, the next one will reward people who are very good at questions. Learn to think like the people you support. Understand how money flows through your company, what actually matters to your customers, how product and strategy decisions get made. Formal exposure to business thinking helps more than people expect. The goal is not to become a product manager but rather to develop the instinct to look at any dataset and identify levers, tradeoffs, and blind spots.

Near term, data engineers will be at the thick of things designing AI-ready systems and becoming AI-native themselves. Data engineers are the ones who understand the data well enough to do both. No other role can step in and own that, not a product manager, not a business analyst or an agent. The people who understand and own the data still matter, because they are also the ones responsible for ensuring that the experience of exploring data through agents is value adding for product leaders and stakeholders. Data democratization has always been the goal and AI is the most powerful lever we have ever had to achieve it, but only if the builders and domain experts collaborate together effectively.

Practically, lean into agents early. Use them to prototype quickly so you can spend more time on design. Stress-test your own understanding by having them explain or transform models you have built. Explore what-if questions at the edge of your intuition, then pull the promising ones back into rigorous evaluation. The people who will thrive are those who see agents as force multipliers on their judgment, not replacements for their labor.

If I had to compress it into a single paragraph, get very good at making meaning in messy systems. Learn the tools, but do not anchor your identity to them. Anchor it instead to business understanding, judgment, and the ability to turn fragmentation into something that both humans and machines can reason over with confidence.

Who should we spotlight next in the Data Engineer Things Community Newsletter, and why?

I would like to spotlight Tamar Phillips, my colleague at Meta, and someone whose perspective I think this community would find genuinely valuable.

Tamar has grown into senior leadership at Meta by building and leading large data engineering teams in the Integrity domain which is a good mix of cultural as it is technical. Dealing with spam, bullying and misinformation, integrity sits downstream of the entire Meta family of apps, which means data modelling is foundational. The privacy requirements are equally serious with sensitive data protection and compliance reporting to governments across multiple regulatory regimes. Getting that right, at scale sharpens your thinking in ways that are hard to replicate elsewhere.

More recently, Tamar has been driving AI readiness efforts for Integrity at Meta and leading the definition of new roles and archetypes for data engineers in an AI-native era, doing the actual legwork of figuring out what that transition looks like in practice.

Beyond the work itself, Tamar leads with empathy, domain expertise, and a clear eye on business impact and product value. He is exactly the kind of practitioner whose experience the data engineering community should hear more from.

Key Takeaways

• Own Semantics, Not Scripts: As AI commoditizes pipeline creation, a data engineer’s value shifts from hand-crafting code to defining precise business logic. Success now depends on spec-driven development which entails structuring the metadata, taxonomies, and semantic models that serve as the foundational context AI agents need to reason over data accurately.

• Measure Workflow Completion: Vanity metrics like tool usage or code generated mask the true question of quality. AI maturity can be measured by Workflow Completion (e.g., an AI agent autonomously detecting a pipeline failure, diagnosing the schema change, and opening a fixed PR) rather than mere Step Acceleration (e.g., using a copilot to write a regex string 30% faster).

• Master "Evals" to Prevent Silent Failures: Traditional data quality tests fail when AI agents generate open-ended analytical queries or insights. Data engineers must master the craft of writing behavioral evaluations and regression suites to serve as guardrails, ensuring that agents don't confidently return wrong answers or cross-wire business logic when underlying infrastructure changes.

Community poll

💬 How to stay connected

• LinkedIn

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello Folks,

Still processing. That’s the honest summary of this past week. I just got back from the Data Innovation Summit 2026 at Kistamässan in Stockholm, and I’m still untangling what I actually learned from and what I enthusiastically nodded at in a hallway.

Two moments keep coming back to me. Dael Williamson from Databricks put it simply: organisations are no longer just adopting AI tools, they’re running fleets of agents that reshape workflows in real time. Build your data foundations now, or get buried later. Then Joe Reis’s masterclass reframed data modeling as a strategic capability, not a dogma and landed two quotes I haven’t stopped thinking about:

“AI amplifies what you know to do at machine speed. It also amplifies what you don’t.”

And:

“Disciplined teams will use AI to move faster with quality. Undisciplined teams will use AI to create technical debt faster.”

That second one hit hard in our governance roundtable.

I’m Chozhan, Head of Data Platform and a passionate Data Engineer. This edition picks up right where DIS2026 left off, a real production story from Spotify on agentic dataset migrations, a fresh benchmark from dbt Labs on Semantic Layer vs. text-to-SQL, a tool-agnostic observability guide, and an honest look at what declining DE job postings are indirectly hinting us. Thanks for being here, let’s dive right in.

– Chozhan

📚 Data Pulse

Next Generation DB Ingestion at Pinterest

📖 Topic: Real-World Streaming & Lakehouse Architecture
🧠 Level: Beginner

Summary: Pinterest’s Logging Platform team replaced a patchwork of batch-oriented, full-table dump pipelines where data latency exceeded 24 hours, with a unified CDC-based framework built on Debezium, Kafka, Flink, Spark, and Iceberg. The new system delivers changes in 15 minutes to an hour, processes only changed records (saving significant compute since daily change rates are under 5% for most tables), and natively supports row-level deletes for compliance. The post walks through the architecture clearly and honestly covers the small-files problem and how they solved it with bucket joins for large-table upserts.

💡 Why is this relevant for DEs?

• This is a textbook CDC + Iceberg architecture explained plainly. If you’re building or evaluating a real-time ingestion layer, Pinterest’s dual-table design (CDC table as append-only ledger, base table as the upserted mirror) is a pattern worth understanding before you design your own.

• The small-files and bucket-join optimisations are the practical gold. Most blog posts stop at the architecture diagram. This one goes into how bucketing by primary key hash, WRITE DISTRIBUTED BY PARTITION, and a bucketed intermediate table for upserts cut compute costs by 40%+, details you can apply directly.

• The compliance angle is increasingly important. Row-level deletes for GDPR and similar requirements are a real engineering constraint, not an afterthought. Seeing how Pinterest handled it in Iceberg with Merge-on-Read is directly useful for teams facing the same challenge.

Data Observability Fundamentals for Data Engineers

📖 Topic: Data Quality & Observability
🧠 Level: Beginner

Summary: A practical, tool-agnostic guide to detecting silent pipeline failures. It introduces six concrete detection patterns (Flow Interruption, Skew, Lag, SLA Misses, Dataset Tracker, Fine-Grained Tracker) and draws a sharp distinction between observability, a flashlight that catches surprises and data contracts, a laser that prevents known failures. One of the clearest pieces on this topic I’ve read this year.

💡 Why is this relevant for DEs?

• Observability first, contracts second that sequencing matters in practice and this post explains why clearly. Most teams get the order wrong and then wonder why their contracts don’t stick.

• The six patterns are concrete enough to implement today, regardless of your stack. Most observability content stays at “monitor your freshness.” This goes further without requiring a specific vendor.

• Silent failures are qualitatively more dangerous when AI is downstream. A schema drift that once broke a dashboard now corrupts model inputs at inference time. The stakes are different and this piece explains why.

Background Coding Agents: Supercharging Dataset Migrations (Honk, Part 4)

📖 Topic: Real-World Agentic Engineering at Scale
🧠 Level: Intermediate

Summary: Spotify used their internal coding agent Honk to migrate ~1,800 downstream data pipelines off two deprecated datasets — work estimated at 10 engineering weeks manually. They generated 240 automated PRs using Backstage lineage, code search, and their Fleet Management tooling. The post is honest about where it worked (standardised dbt and BigQuery Runner pipelines) and where it didn’t (their less-consistent Scio framework).

💡 Why is this relevant for DEs?

• Context engineering - lineage, code search, test coverage is what made the agent useful, not prompt cleverness. The surrounding infrastructure determines how far an agent can actually go.

• Standardisation is now a prerequisite for automation. Honk succeeded on consistent frameworks and struggled on inconsistent ones. If your pipelines are a patchwork of bespoke patterns, agents will hit the same walls human engineers do, just faster.

• The migration use case is the most credible near-term application. Greenfield agentic pipelines are exciting in demos. Real ROI right now is in deprecations, schema migrations, and backfill rewrites; this post gives you a concrete template.

Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update

📖 Topic: Data Modeling in the AI Era
🧠 Level: Intermediate

Summary: dbt Labs re-ran their 2023 benchmark with today’s LLMs (Claude Opus 4.6, GPT-5.x and more). The gap has narrowed, models are dramatically better at text-to-SQL. But for questions within a well-modeled Semantic Layer, accuracy hits near 100%, because MetricFlow generates SQL deterministically: the LLM picks the metric and dimension; the engine handles the query. The benchmark is open-source and reproducible on your own data.

💡 Why is this relevant for DEs?

• The failure modes are what matter. Text-to-SQL fails unpredictably and silently. The Semantic Layer fails loudly (”I can’t answer that”). For production AI agents querying your data, the deterministic failure mode is far more manageable.

• This is the empirical case for data modeling in the AI era. AI doesn’t make semantic consistency less important — it makes ambiguous metric definitions actively dangerous at inference scale. If your team is debating whether to invest in MetricFlow, this benchmark is the answer.

The 2026 State of Data Engineering Survey (Interactive)

📖 Topic: Agentic AI Adoption & The State of the Profession
🧠 Level: Intermediate

Summary: Joe Reis, the same person who reframed data modeling at DIS2026, surveyed 1,101 data engineers in early 2026 and published the results as a fully interactive explorer: filterable by role, org size, industry, and region, with raw CSV download and a SQL query interface on top of the dataset. The headline findings are striking: 82% of engineers use AI tools daily, but 64% of organisations are still experimenting or using AI for tactical tasks only. The biggest bottlenecks aren’t technical, legacy systems top the list, but lack of leadership direction and poor requirements are close behind.

💡 Why is this relevant for DEs?

• The gap between individual AI usage and organisational adoption is the most actionable number here. Most engineers are already using AI heavily; most organisations haven’t built the workflows, governance, or data foundations to deploy agents at scale. That gap is where data engineers can have the most impact right now: building the infrastructure that closes it.

• The modeling pain point is validated at scale. 59% cite “pressure to move fast” as their top pain point and only 11% say modeling is going well. If you’ve felt this tension in your own team, you’re not alone and the survey gives you real data to bring into conversations about prioritisation and technical debt.

• The interactive format makes this genuinely useful beyond the headline numbers. You can cross-tab by org size, region, or industry to find findings specific to your context which is far more valuable than a static report summary. It’s worth spending 20 minutes in the explorer rather than just reading the takeaways.

How Meta Used AI to Map Tribal Knowledge in Large-Scale Data Pipelines

📖 Topic: AI-Assisted Data Platform Engineering
🧠 Level: Advanced

Summary: Meta’s engineering team built a pre-compute engine, a swarm of 50+ specialised AI agents, to systematically extract and encode the undocumented tribal knowledge buried across a large-scale data pipeline spanning four repositories, three languages, and 4,100+ files. The result: 59 concise context files covering 100% of code modules (up from 5%), 50+ non-obvious patterns documented for the first time, and ~40% fewer AI agent tool calls per task. The key design principle is “compass, not encyclopedia”: each context file is 25–35 lines, actionable, and auto-refreshed on a schedule so stale context can’t accumulate.

💡 Why is this relevant for DEs?

• Tribal knowledge is the real blocker for AI agents in data platforms. Generic coding agents fail on proprietary pipelines not because the models are weak, but because they have no map, they don’t know your naming conventions, cross-module dependencies, or the silent rules that break builds. This post is the clearest description I’ve seen of how to fix that.

• The “five questions” framework is directly applicable to your codebase today. For each module: what does it do, how do you modify it, what breaks silently, what depends on it, and what’s only in someone’s head? Running that exercise even manually will surface patterns you didn’t know were undocumented.

• Self-refreshing context is the part most teams miss. Meta automated periodic validation and re-generation of context files, because stale context causes more harm than no context. If you’re building any agent-assisted tooling on your data platform, freshness of the knowledge layer is as important as its accuracy.

💎 Open Source Gem

Apache Fluss (Incubating) — Streaming Storage for Real-Time Lakehouses

Apache Fluss is a streaming storage system, now in Apache incubation, that sits as a hot real-time layer in front of your lakehouse. The core idea: instead of choosing between a Kafka-style streaming system and a file-based lakehouse, Fluss gives you both under one table abstraction. It writes sub-second streaming data in Apache Arrow columnar format, then a built-in tiering service continuously compacts that data into standard lakehouse formats: Iceberg, Paimon, or Lance, making it queryable by Spark, StarRocks, Trino, and Flink without any extra pipeline plumbing.

📰 What’s new / why now

The 0.8 release added full Streaming Lakehouse support for Apache Iceberg (continuously tiered, with exactly-once semantics and built-in compaction) and Lance (the vector-native format for AI/ML workloads). The headline feature is Delta Joins with Flink by externalising join state into Fluss tables, Flink performs joins incrementally on data deltas, cutting CPU/memory by up to 80% and checkpoint durations from 90 seconds to 1 second in early production deployments. It’s now fully compatible with Flink 2.1.

💡 Why is this useful for DEs?

• It solves the oldest problem in streaming lakehouses. File-based formats like Iceberg have a practical lower latency bound of ~1 minute due to file commit overhead. Fluss breaks that wall by acting as the hot tier, giving you sub-second freshness without abandoning Iceberg for historical analytics.

• One table, two access patterns. Flink can union-read both the real-time Fluss data and the compacted Iceberg history in a single query, no separate pipelines, no data duplication, no metadata inconsistency between streaming and batch layers.

• Delta Joins are a genuine step forward for stateful streaming. If you’ve ever dealt with Flink state explosion on large join operations, externalising that state into Fluss is a meaningful architectural improvement, not just a configuration tweak.

👉 GitHub: https://github.com/apache/fluss

💡 DE Tip of the Month

Managing Query Costs in Cloud Data Warehouses Through Tagging and Attribution

Cloud data warehouses bill by compute consumed, which means a single expensive query from an untracked workload can inflate costs significantly before anyone notices. In many organisations, warehouse spend grows with usage but the breakdown, which pipelines, teams, or jobs are responsible, etc remains unclear until the bill arrives. Building cost attribution into your warehouse setup from the start makes spend visible, accountable, and actionable.

📒 Rules of thumb

• Apply query tags or labels at the session or job level. Most major platforms (Snowflake, BigQuery, Databricks) support attaching metadata to queries like a pipeline name, team identifier, or environment tag. Doing this consistently means your query history becomes a cost ledger you can slice by workload, not just a raw list of executions.

• Set warehouse or compute cluster size limits relative to the workload type. Transformation jobs and ad-hoc exploratory queries have very different compute requirements. Routing them to appropriately sized resources rather than a single general-purpose warehouse prevents exploratory queries from consuming resources provisioned for production pipelines.

• Schedule cost reviews as part of your regular pipeline health checks. A query that was efficient at last year’s data volumes may be expensive today. Reviewing the top consumers by compute spend on a regular cadence surfaces regressions early, before they accumulate.

• Use query result caching deliberately. Repeated identical queries common in BI dashboard refreshes can often be served from cache at zero compute cost. Verify that caching is enabled for appropriate workloads and that transformations upstream are not inadvertently invalidating it on every run.

📊 Community Poll

Until next time, cheers!

Chozhan, Sri, & Ananda

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hey everyone,

On April 16th 2026, hundreds of data engineers gathered at The Contemporary Jewish Museum in San Francisco for the third annual Data Engineering Open Forum. DEOF 2026 was a day packed with technical depth, honest conversations, and the kind of energy you only get when a community shows up for each other.

We want to share what happened, what people took away, and what it all means for where data engineering is heading.

— Team DET 💛

tl;dr: 📼 Talk recordings out on YouTube

💬 What the Community Said

“DEOF is the professional highlight of my year.”

Yaakov Bressler (LinkedIn post)

“I’ve been to quite a few conferences, and this was one of the best so far. What made it stand out? It was purely community-driven.”

Yang (Eric) Liu (LinkedIn post)

“Data engineering is changing, not disappearing. Agents rely on pipelines just as much as business users do, and everything depends on strong, reliable data foundations.”

Michelle Winters (LinkedIn post)

“At petabyte-scale, architecture isn’t just about what’s technically great, it’s about the constraints the business puts on you.”

Narasimha G. (LinkedIn post)

“The role is evolving fast, especially around owning and operating agentic workflows, building in the right guardrails, and making sure observability and alerting are part of the picture from the start.”

Jordan Lewis (LinkedIn post)

“AI doesn’t replace data engineering. What will change is the scope of the role. It’s about thinking of data as a product, how it is used, and the impact it has on the business.”

Valentin Marek (LinkedIn post)

For a truly comprehensive recap, check out Pawel Mikler’s detailed write-up. Pawel holds the unofficial title of the only attendee who crossed the Atlantic from Europe specifically for DEOF, making it three years in a row.

Read Pawel’s full recap on LinkedIn

Pictures by DuHon Photography

⚒️ Data Engineering Project: Build Real-Time IoT Analytics

This data engineering tutorial teaches you to build real-time IoT analytics using time-series datasets with relational metadata. You will learn to:

• Optimize PostgreSQL tables into time-series hypertables.

• Use compression to achieve 10x storage reduction and faster queries.

• Enable lightning-fast analytics with pre-compute aggregation.

👉 Get started with the project here: Github.

(This post is sponsored by TigerData)

💛 From the DET Crew

Our DET volunteers were embedded throughout the day, facilitating conversations, running the community booth, and soaking it all in. Here is what stood out to them.

Sanchit was struck by Dinesh Thangaraju’s talk on federated knowledge infrastructure:

“It reinforced fundamentals we’ve heard before, getting references right, avoiding multiple competing definitions for something as basic as ‘revenue’, but raised the stakes dramatically. In the AI age, a single wrong reference can cascade into serious output errors.”

His unexpected takeaway? AI adoption is rising from the bottom up, not the top down:

“I discovered use cases from talks that I hadn’t even considered, things that could be replicated at my organization immediately. We need a demo/showcase culture to surface this bottom-up innovation.”

Yaakov noticed data engineering branching into two diverging paths:

“People building coding agents, self-healing bots, context layers, to speed up DE execution. This role becomes a builder role and starts looking more like PM. Data Infrastructure, deeper technical capabilities. Building infrastructure for agents to execute on and with. This role starts evolving into a new kind of software/data builder.”

His conclusion? DE is changing fast, and the best way to keep up is to show up.

🌟 Session Highlights

Indrajit Roy (Databricks) opened with the evolution of Apache Spark Structured Streaming and the introduction of Spark Declarative Pipelines. The takeaway: real-time capabilities are no longer reserved for streaming specialists. Express the logic, and the engine handles the rest.

Jerry Wang (Airbnb) followed with a 15-year retrospective on data infrastructure, making the case for the full-stack data engineer and the death of the “chain of custody” model.

The closing keynote panel brought together Laura Pruitt (Netflix), Paul Ellwood (OpenAI), and Vikram Koka (Astronomer), moderated by Michelle Winters. They tackled the hard questions: what does the future of data engineering look like? What skills should we double down on? How do we make sure AI truly serves humanity? An absolute must-watch for every data engineer!

📼 Talk Recordings

Catch all DEOF talks in this YouTube playlist!

📙 Xinran’s Closing Words

Xinran closed the day with words that many of us will carry for a long time:

“Change is scary, but so is staying the same.”

She asked the room to take one thing home: courage. The courage to start that project you have been putting off. The courage to reach out to someone new. The courage to evolve alongside this field rather than resisting the change.

“If you were inspired by a talk, do your follow-up research and try to adopt it in your own work. If you met a new friend, connect on LinkedIn and remember to catch up again later. If you felt courageous, start on that new project that you were afraid to pick up. When we meet again, let me know how it goes.”

Xinran Waibel (LinkedIn post)

🚀 DEOF 2027 Look Ahead

Stay in the Loop

Join our Google Group for DEOF 2027 announcements, CFP updates, volunteer opportunities, and community news.

Sponsor DEOF 2027

Interested in partnering with the data engineering community? Complete our sponsorship interest form.

🙏 Thank You

DEOF 2026 would not have happened without the collective effort of so many people.

Our speakers shared their knowledge generously. From keynotes to deep dives, every session reflected real experience and genuine insight.

Our program committee designed the programming and supported speakers to deliver their best: Apoorva Bapat, Goutham Budati, Jerry Wang, Michelle Winters, Sharath Chandra, Shruthi Jaganathan, Tulika Bhatt, Will Monge, and Xinran Waibel.

Our volunteer leads kept everything running smoothly: Anna Peng, Annu Joshi, and Balachandar Paulraj, along with the entire volunteer crew who facilitated conversations and helped every attendee get the most out of the day.

Our sponsors made this community event possible. We want to give a special thank you to Team Databricks (Denny and Lisa) and Team Astronomer (Caitlin and Volker) for their support and trust from the very beginning.

Our attendees showed up, asked real questions, shared lessons honestly, and stepped out of their comfort zones. You are the reason this community exists.

🙋 Community Poll

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn.

Hello Folks,

Great to connect with you through this month’s newsletter.

I’m writing this from Coimbatore, India where summer is in full swing bright days, a bit of heat, and that constant push to slow down and pace yourself.

I’m Sri. I work in the data engineering and AI space, focusing on building scalable data platforms and Enterprise AI solutions enabling teams to make better use of data in real-world scenarios. Over time, I’ve come to appreciate that this field is more about how thoughtfully we design systems that people can rely on.

Outside of work, I enjoy reading, traveling, and watching movies small ways to recharge and stay curious. In many ways, this community feels similar: a space where we can learn from each other, share ideas, and keep growing together.

This edition brings together a set of interesting ideas and perspectives from across the data engineering world. I hope you find something here that resonates or sparks a new thought.

Happy reading, and thank you for being part of this journey.

– Sri

📚 Data Pulse

The Data Engineer’s Guide to Context Engineering

📖 Topic: Data & Context Engineering
🧠 Level: Intermediate

Summary: Modern data platforms are moving from managing raw data to managing context. The article introduces “context engineering” as a structured way to capture, enrich, and operationalize metadata so both humans and AI systems can understand data better. It focuses on building context layers like lineage, ownership, usage patterns, and semantics to improve discoverability, trust, and usability, especially in AI-driven workflows that need richer signals for accurate reasoning.

💡 Why is this relevant for DEs?

• AI-ready data platforms: As AI pipelines grow, systems need context along with raw data to interpret it correctly. Context engineering enables AI-native data platforms.

• Metadata as a core building block: Metadata like lineage, ownership, and quality signals becomes a first-class part of the system, not an afterthought.

• Improved data trust and discovery: Adding operational and semantic context helps users quickly understand what data means, where it comes from, and how reliable it is.

• Beyond static catalogs: Instead of passive catalogs, this approach enables dynamic, continuously evolving context that reflects real usage and system behavior.

Spotify: Inside the Archive: The Tech Behind Your 2025 Wrapped Highlights

📖 Topic: Data Engineering
🧠 Level: Intermediate

Summary: Spotify explains the engineering behind its 2025 Wrapped “Archive,” which goes beyond static summaries by turning listening behavior into narratives. It identifies meaningful moments like taste shifts and binge sessions and converts them into personalized stories. This required large-scale pipelines to precompute billions of user-specific insights, combined with AI-generated narratives to deliver a near real-time experience to hundreds of millions of users.

💡 Why is this relevant for DEs?

• From metrics to moments: Traditional analytics focuses on aggregates like counts and top items. This approach turns raw data into meaningful events and storytelling units.

• AI + data engineering convergence: It combines data pipelines with LLM-based narrative generation, showing how data systems are evolving for AI-driven experiences.

• Product thinking for data: Engineers move beyond pipelines to designing outputs users directly interact with, making data work more product-oriented.

• Serving at global scale: Delivering a consistent experience to hundreds of millions of users requires tight coordination across processing, storage, and serving layers.

Nextdoor’s Database Evolution: A Scaling Ladder

📖 Topic: Databases
🧠 Level: Intermediate

Summary: ByteByteGo describes Nextdoor’s database evolution as incremental scaling steps. It starts with a single Postgres instance and grows by adding connection pooling, read replicas, caching, and reconciliation as needed. Each change fixes a bottleneck but adds complexity such as replication lag and consistency challenges. The key takeaway is that scaling is continuous and every improvement involves trade-offs.

💡 Why is this relevant for DEs?

• Scaling is incremental: Systems evolve step by step as scale increases, adding new layers as bottlenecks emerge.

• Read and write separation is essential: As traffic grows, reads often become the bottleneck, requiring a primary-replica model to split read and write workloads.

• Operational complexity grows with scale: Pooling, replication, and caching add complexity, making observability, alerting, and incident management critical for debugging and reliability.

• Pragmatic architecture decisions: Each change is driven by a specific need, favoring practical, context-driven engineering over “perfect” designs.

Uber: Accelerating Deep Learning: How Uber Optimized Petastorm for High-Throughput

📖 Topic: Data Infrastructure and AI
🧠 Level: Advanced

Summary: Uber explains how it scaled its deep learning systems to support increasingly complex AI use cases. As it moved from traditional ML to deep learning and then generative AI, compute and workflow demands increased significantly. This led to investments in GPU efficiency, distributed training, and a streamlined ML lifecycle. The focus shifted toward reducing training time, improving efficiency, and standardizing processes so teams could reuse workflows instead of rebuilding them.

💡 Why is this relevant for DEs?

• Data infrastructure fuels AI: Deep learning performance depends heavily on the underlying data pipelines, making scalable, high-throughput infrastructure essential for training and inference.

• Compute-aware data design: With GPUs and distributed training, data engineering choices directly affect cost and speed, making formats, storage, and access patterns critical.

• End-to-end pipeline thinking: Data engineering extends beyond ingestion to include feature pipelines, training data preparation, and real-time serving within unified ML systems.

• Efficiency at scale: Small inefficiencies become expensive at scale, reinforcing the need to optimize data movement, reduce redundancy, and align pipelines with compute resources.

From Building Pipelines to Building Teams

📖 Topic: Data Engineering & Leadership
🧠 Level: Intermediate

Summary: This article explores the transition from hands-on data engineering to people management. It highlights that skills like solving technical problems and building pipelines do not directly translate to leadership. The focus shifts to enabling others, setting direction, and creating an environment where teams can work effectively.

💡 Why is this relevant for DEs?

• New skill set required: Technical depth alone is not enough. Communication, decision-making, and empathy become essential for managing teams and aligning stakeholders.

• Shift from systems to people: Career growth in data engineering moves from building solutions to enabling and guiding others to build them.

• Leadership before title: Effective leaders often demonstrate leadership early by mentoring, unblocking teammates, and influencing decisions before moving into formal management.

⚒️ OpenXData 2026

Block the calendar for April 29: OpenXData is a free virtual conference packed with two tracks of expert-led sessions on open data architectures, AI-native platforms, and cost and performance at scale.

The lineup includes a keynote by Vinoth Chandar (Onehouse) on data platforms for autonomous AI, Maxime Beauchemin (Preset) breaking down the anatomy of their data agent, and engineers from Uber, Walmart, Booking.com, JD.com, Zalando, Anthropic, and Snowflake sharing how they’re running open lakehouse stacks in production. If you care about Iceberg, Hudi, Spark, Polaris, or where data engineering meets AI, this one’s worth your time.

📋 Agenda and more details HERE.

👉 RSVP

💎 Open Source Gem

DuckLake v1.0: The Lakehouse Format Built on SQL Reaches Production-Readiness

DuckLake is a lakehouse format from the DuckDB team that simplifies metadata by moving it from scattered JSON and Avro files in object storage into a SQL database. Data still lives in Parquet files on object storage, similar to Iceberg or Delta Lake, but the catalog is stored in any transactional SQL database with primary key support.

👨‍💻 Quick taste

INSTALL ducklake;
LOAD ducklake;

ATTACH 'ducklake:metadata.ducklake' AS lake;
USE lake;

CREATE TABLE events (id INT, user_name VARCHAR, ts TIMESTAMP);
INSERT INTO events VALUES (1, 'vojay', NOW());

-- small writes stay in the catalog, no tiny Parquet files
FROM ducklake_list_files('lake', 'events'); -- empty
CHECKPOINT; -- flush to object storage when you're ready

📰 What/s new in v1.0

Released April 13, it is now production-ready with backward compatibility guarantees. Key updates include default data inlining for updates and deletes to avoid small-file issues in streaming writes, support for sorted tables and murmur3-based bucket partitioning for Iceberg compatibility, native GEOMETRY and VARIANT types, and deletion vectors stored as Puffin files.

💡 Why is this useful for DEs?

• One less moving piece. Your catalog is just a db you already know how to run.

• Streaming into a lakehouse actually works. Inlining means you can do small, frequent writes without drowning in compaction jobs.

• Transactions are real. ACID guarantees come from the catalog DB.

• Escape hatch included. Iceberg-compatible at the data level, so no lock-in.

👉 GitHub: https://github.com/duckdb/ducklake

💡 DE Tip of the Month

Stop Over-Partitioning Your Iceberg Tables

Partitioning can feel like free performance until it backfires. A common mistake is using a high-cardinality column like user_id or raw timestamps in an Iceberg table, which creates thousands of tiny partitions with only a few rows each. This shifts the cost to metadata, where query planning takes longer than the actual scan and effectively recreates the small-files problem through poor partition design.

📒 Rules of thumb

• Aim for partitions that hold at least ~1 GB of data. If most of yours are smaller, you’re over-partitioned.

• Partition on low-to-medium cardinality columns that actually appear in WHERE clauses, usually dates at day or month granularity, maybe region or tenant.

• Use Iceberg’s hidden partitioning transforms, not raw columns. PARTITIONED BY (days(ts)) beats partitioning on a raw timestamp, otherwise you’d get a new partition per millisecond. Transforms bucket values sensibly and Iceberg auto-prunes queries without extra derived columns.

• For high-cardinality columns you filter on, use bucket(N, col) instead. Hashes the value into N buckets, pruning without the cardinality explosion.

Quick gut check: SELECT COUNT(*), AVG(file_size_in_bytes) FROM my_table.files 👉 if average file size is in the MB range, rethink the partition spec.

📖 Iceberg Partitioning docs

📊 Community Poll

Until next time, cheers!

Sri, Ananda, & Volker

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello, everyone!

This edition of Community Spotlight features Michelle Winters, a self-proclaimed data nerd who has built petabyte-scale platform tooling for Netflix, architected diverse data systems at eBay, and founded a highly successful AI-tooling startup.

I had the great fortune of sitting down for a boba chat with Michelle to distill the architectural mental models that hold up at massive scale. We also explored the convergence of data engineering and AI, and why she believes the engineer’s role is shifting from simply moving pipelines to actively stewarding data meaning.

Read on to discover the lessons Michelle has gleaned from her expansive tech journey, the rise of the 'data artisan' who blends code with deep domain context, and the reason protecting your cognitive capacity is the ultimate career hack.

You’re in for a treat!

- Sugandhi

🎟️ Conference: Data Engineering Open Forum 2026

We are hosting the third Data Engineering Open Forum (DEOF) on April 16 in San Francisco! Here is what you will experience expect at the event:

• Sessions by speakers who are solving cutting edge problems in data engineering, for examples, Apache project creators and PMCs like Julien Le Dem, Boyang Jerry Peng, and Jack Ye.

• Intentionally-designed activities that you can sign up to engage in small-group networking (because we know it’s hard to make conversations at conference).

• Opportunities to connect with data engineering teams at top tech companies (like Airbnb, Netflix, OpenAI, and more) at their booths.

Our ultimate goal is that when you look back some day, you could confidently say “I’m so glad I went to DEOF”, because the people you met there or ideas you walked away with made a difference in your career.

👉 See the agenda HERE. RSVP today!

Spotlight: Michelle Winters

For those in the DET community who may not know you yet, could you briefly introduce yourself?

Hello, everyone—my name’s Michelle. I’m a lifelong data nerd who’s done a bit of everything in the world of data. Most of my work has focused on data engineering and warehousing, data platform and infrastructure, system intelligence, automation, efficiency, scalability, quality, and monetization. I’ve been fortunate to work at amazing companies like Netflix and eBay, and I even founded my own startup which provided collaborative data tools and became a top 10 plugin for OpenAI’s marketplace before it was acquired. Over the years, I’ve worn many hats—data engineer, architect, leader, founder—but my favorite hat is simply “data nerd.” I still love exploring new ways to structure, understand, and leverage data to create real impact. I’m especially excited about the future of data when it comes to collaboration, innovation, and solving meaningful, real-world problems.

You’ve led data engineering at GoDaddy, built platform tooling for Netflix’s 100 petabyte warehouse, co-founded a startup, and architected data systems at eBay. Each of those environments is radically different in scale, constraints, and culture. Is there a single architectural principle or mental model that has held up across all of them?

If I had to distill it down to one idea, it’s this: design for the full 100% from day one but don’t treat all 100% equally.

About 80% of what people need should be simple, obvious, and well-paved. That’s your “highway.” It’s where most users should live, and it should feel easy, reliable, and almost boring in the very best way! That’s where you invest in strong defaults, automation, and clear observability so people can move quickly with confidence.

The remaining 20% is just as important, but it’s different. That’s your off-road terrain. This is where flexibility matters. You’re giving people room to explore, to handle edge cases, and to push boundaries without breaking the system. If you over-optimize for simplicity, you box people in. If you over-optimize for flexibility, you create chaos. The real craft is in holding both at the same time.

What this forces you to do is build the right abstractions. Not just tools that work, but systems that people can understand, extend, and truly own. And ownership is the goal, because that’s what unlocks speed and innovation at scale.

The other lesson that’s stuck with me is to resist building for a single use case, even if it’s the loudest one in the room. Early on at Netflix, we intentionally designed for three very different workloads: personalization, which was deeply custom, hand-crafted scalable Java code; content analytics, which leaned heavily on SQL; and finance, which had strict accuracy and audit requirements. Those use cases pulled the system in completely different directions, and that was the point. If you can design something that works well for very different kinds of users and constraints, you’re much more likely to end up with a platform that lasts.

For folks early in their careers, I’d frame it this way: don’t just solve the problem in front of you—zoom out and ask what kind of problems this solution should be able to handle a year from now. Be intentional early. Not just about what you’re building, but about how you’re building it and what it might need to become. The earlier you start thinking that way, the better your instincts will become.

What is one architectural principle that surprised you by not holding up?

For a long time, I believed you could separate the platform from the meaning of the data—that our job was simply to move data reliably from point A to point B.

I don’t think that holds up anymore. Moving data alone doesn’t create much value, and most of this work will be automated in the near future anyway.

The real role of a data engineer is to ensure trust and preserve meaning, making sure data carries its context all the way through to feature engineering, AI models, and ultimately decisions. That means stepping back and asking harder questions: Can we trust the data? Are we capturing what actually matters? Are we keeping too much? Are we filtering or summarizing low-value data early enough?

This is where the role becomes much more impactful. You’re not just building pipelines, you’re shaping how the business understands its data. So don’t stop at “does this pipeline work?” Ask what the data means and how it creates value. That shift in thinking is what sets great data engineers apart.

The industry talks a lot about the “full-stack data engineer” as roles between data engineering, data science, and ML engineering continue to blur. Is that convergence a good thing, or are we at risk of undervaluing what people with deep, specific expertise bring to the table?

Some of the best outcomes I’ve seen come from teams where people bring different strengths and perspectives. You need both breadth and depth: generalists can connect dots across systems, but specialists provide the deep domain knowledge and technical rigor that prevent costly mistakes. There isn’t a single “full-stack” role that fits every company or every problem. What matters is understanding where the organization’s core challenges lie and building complementary teams to address them.

That’s why I think the concept of a “data artisan” is so useful. These are people who combine technical skill with deep domain expertise, whether that’s advertising, healthcare, or finance. Their knowledge adds disproportionate value because they understand the nuances of the data and the context of its use. Looking ahead, I expect we’ll see people self-select into hybrid roles that blend technical skill and domain knowledge in ways we don’t yet have names for, and those roles will be increasingly critical as data-driven products grow more complex.

What is a skill that was considered “not a data engineer’s job” a few years ago that you now think is essential?

Anything related to AI has quickly moved from “nice to have” to table stakes.

At a minimum, data engineers should understand concepts like retrieval-augmented generation (RAG) and vectorization—how data is embedded, retrieved, and used in modern AI systems. These aren’t just modeling concerns anymore; they directly shape how data needs to be stored, indexed, and served.

We’re also seeing a shift with graph technologies. Historically, that was the domain of specialists, but the ongoing responsibility of building, populating, and maintaining graphs fits naturally with data engineering. As systems become more relationship-driven, that ownership is starting to move closer to the data layer.

👉 Read our interview with the founder of PuppyGraph to learn more about graph technologies.

Data quality is evolving too. It’s no longer just binary checks or simple rules. Increasingly, it’s probabilistic and algorithmic—designed to catch real issues without overwhelming teams with false positives. That shift brings data engineering closer to disciplines like statistics and machine learning.

What all of this points to is a broader trend: the boundaries are moving. Data engineers are no longer just responsible for pipelines, they’re becoming stewards of how data is structured, interpreted, and trusted in increasingly intelligent systems.

For those earlier in their careers, this is a great moment to lean in. You don’t need to become an ML expert overnight, but building intuition around these areas, especially how data flows into and supports AI will set you apart.

There is a lot of pressure right now to embed AI into data pipelines: agentic orchestration, LLM-based data quality checks, AI-generated transformations. Where is AI genuinely changing what a data engineer does day to day, and where is the hype running ahead of reality?

A lot of what we’re seeing right now feels premature, and the reason is simple: we still have a data problem.

In most organizations, data is disjointed, inconsistent, redundant, and poorly understood. And yet, we’re expecting AI to step in and magically extract meaning from it? That’s wishful thinking. If meaning doesn’t already exist, if it’s not encoded somewhere in the system, AI isn’t going to manufacture it for us. At least, not to a level we should have confidence in.

Where I do see real value is in AI acting as a collaborator. AI agents can generate artifacts, transformations, documentation, even quality checks, but those outputs still need expert review. If that slows things down a bit, that’s okay. Speed without confidence in the data is a liability, not an advantage.

At the end of the day, the fundamentals haven’t changed. If you don’t trust your underlying data, nothing built on top of it will hold up, no matter how advanced the AI is. Any organization that wants to compete in this space needs to take that seriously. That means investing in data quality, building systems that are understandable and observable and explainable, and treating security and consumer privacy as first-class concerns, not an afterthought.

You built Noteable from the ground up as CEO, and then stepped into a Distinguished Architect role at eBay. What did running a startup change about how you read a large organisation’s data architecture?

Running a startup made me think much more about culture—both team and company—and gave me a stronger sense of ownership and connection to the business, which I think makes me a better partner overall. But the biggest shift for me was humility. As CEO, I couldn’t be the most technical person in the room anymore, and getting comfortable with that changed how I lead. I learned that when you give people clear context and direction, they often take ideas further than you could yourself. That experience reshaped how I think about both systems and teams: the goal isn’t to have all the answers, but to create an environment where talented people can do their best work. When you invest in good people, give them ownership, and get out of their way, I’ve found they’ll consistently exceed your expectations.

eBay is a marketplace with an enormous variety of data: buyer behaviour, seller inventory, pricing, fraud, logistics. How does that diversity shape architectural decisions differently from a single-product company like Netflix?

Every company has a wide variety of diverse data—it’s really a question of proportions. What matters more than the domains themselves is the company’s unique data profile: where the data comes from, what latency it requires, and what happens if it doesn’t arrive on time. That’s why I’m not a fan of one-size-fits-all architectures. The better approach is to understand where your center of gravity is and design systems that minimize unnecessary movement while supporting those core needs. At the same time, you have to build in flexibility from the start. Even if you don’t have something like IoT data today, that can change quickly, and your architecture needs to be ready to evolve with it rather than be rebuilt from scratch.

As a Distinguished Architect, you are often shaping decisions without direct authority over the teams implementing them. How do you actually get things to change at that level?

You lead with the business problem: what are we trying to solve, and what is the customer impact? Put together a long-term strategy and focus on aligning people around the outcomes it would unlock. Make it tangible, then work backward from there. The challenge is that when teams are overworked, their capacity often only reaches the current sprint or next quarter. So you start with those who still have passion and headspace, get them in the trenches building proof points, and then bring everyone else along when you can demonstrate tangible value. Change happens person-to-person—you’re not selling a technical solution, you’re selling what the organization will be able to do tomorrow that it cannot do today.

What is one tough lesson from your career that made you a better data professional?

Protect your brain! Data professionals are often pushed to deliver under sustained pressure, and it’s easy to treat mental capacity as an unlimited resource. The reality is that cognitive performance degrades long before most people notice it, and by the time the signs are obvious, the cost is already high. Don’t wait until that point—invest in yourself early. When you take care of your mental and physical well-being, you may work fewer hours, but you’re more engaged, generate better ideas, see solutions that were previously hidden, and rediscover your passion. That matters enormously when the work demands genuine innovation.

Key Takeaways

• Architecture should be designed for both simplicity and flexibility from the start. Simplicity is the 80% path; the remaining 20% is where you design for flexibility. Use contrasting use cases to stress-test your platform before it needs to scale.

• The data engineer’s role is evolving beyond moving data. Full ownership of data meaning, from ingestion through to feature engineering and AI models, is where the real value lies. AI does not fix a data quality problem, it amplifies it.

• The future of data engineering is specialization by domain, not just by technical function. Industry knowledge will become as important as technical skill, and roles that do not yet have names are already beginning to emerge.

• Influence without authority starts with business outcomes, not technical vision. Find the believers first, build proof points with them, and let the results do the convincing.

• Cognitive capacity is finite. Sustained pressure degrades mental performance before most people notice, and by the time the signs are visible, the cost is already high. Investing in yourself is foundational to a long and effective career in data.

Community poll

💬 How to stay connected

• LinkedIn

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello Folks,

Great to connect with you again for another edition! I’m writing this from Philadelphia, where the worst of winter is finally behind us and the days are slowly getting longer and warmer.

Much like the shifting seasons, data engineering itself is in the middle of a transformation. In 2026, it's no longer just about moving and storing data; it's about making data meaningful, trustworthy, and AI-ready. Whether you’re building pipelines, designing data platforms, or enabling AI systems, the common thread is clear that the future of data engineering is as much about context and reliability as it is about movement and scale.

If you’re looking for a place where these conversations are happening in person, check out the Data Engineering Open Forum in San Francisco on April 16th. The agenda is packed with sessions you'll actually want to stay for, and it's a great chance to connect with engineers navigating the same challenges you are.

- Ananda

📚 Data Pulse

Data Engineering After AI: ECL - Extract, Contextualize, Link

📖 Topic: Data & Context Engineering
🧠 Level: Beginner

Summary: As AI continues to automate data engineering tasks such as pipeline generation, transformation logic, and schema inference, the core role of data engineers is shifting from moving data to defining and managing its meaning. Traditional ETL architectures focused on data movement, and they often embedded business logic within pipelines, allowing interpretive context to drift as data passed through successive transformations. An alternative framework, ECL (Extract, Contextualize, Link), addresses this gap by emphasizing three stages: extracting reliable data from source systems, enriching it with contextual definitions, and linking entities across systems to preserve coherence.

💡 Why is this relevant for DEs?

• Shift in core responsibilities: Data engineers should focus more on designing architectures that preserve and govern the semantic meaning of data across systems.

• Need for semantic and governance infrastructure: With ECL, data engineers need to build and manage data contracts, lineage systems, and context stores that ensure data definitions remain consistent, versioned, and trustworthy as data moves through multiple transformation layers.

• Emergence of a new role: The discipline is evolving from pipeline engineering to “context architect,” where data engineers design the contextual frameworks that allow AI systems and downstream applications to interpret and use data reliably.

OpenAI: Scaling to 800 Million Users With Postgres - P99 latency

📖 Topic: Databases
🧠 Level: Intermediate

Summary: OpenAI scaled PostgreSQL to support over 800 million ChatGPT users using a single primary database with dozens of read replicas, skipping complex sharding entirely. By focusing on reducing primary writer load, optimizing queries and connections, and preventing cascading failures, they achieved low double-digit millisecond latency and 99.999% availability.

💡 Why is this relevant for DEs?

• Optimize before adding complexity: OpenAI showed that tuning PostgreSQL for query optimization, caching, and connection pooling can delay or eliminate the need for sharding and distributed systems.

• Design for your actual workload: ChatGPT is largely read-heavy, so read replicas and caching worked. Match your architecture to what your system actually does, not to generic best practices.

• Reliability is key: Tools like PgBouncer, rate limiting, and workload isolation show that modern data engineering is as much about keeping systems stable as it is about building pipelines.

Netflix: Scaling LLM Post-Training

📖 Topic: Data Engineering & AI
🧠 Level: Intermediate

Summary: Netflix built an LLM post-training framework to scale the adaptation of foundation models for production use cases such as recommendations, personalization, and search. Pre-trained models must be adapted to understand Netflix’s catalog and user behavior, but doing this at Netflix scale, with massive data pipelines and distributed GPU clusters, is a major engineering challenge. The framework, built on their ML platform Mako using PyTorch, Ray, and vLLM, enables techniques like fine-tuning, reinforcement learning, preference optimization, and knowledge distillation. The framework is organized around four pillars: data, model, compute, and workflow, providing a unified way to manage datasets, shard models, orchestrate GPUs, and run multi-stage training pipelines.

💡 Why is this relevant for DEs?

• Data pipelines are foundational to LLM training: Post-training workflows depend on well-prepared data curated datasets, proper tokenization, and efficient streaming to distributed training systems. Data engineers own the pipelines that handle all of this, from selecting and transforming domain-specific data.

• LLM systems require scalable data infrastructure: Netflix’s framework underscores the need for distributed workflows to coordinate across GPUs, storage, and orchestration layers. Data engineers are central to building the plumbing that moves, batches, and serves data reliably for both training and inference.

LinkedIn: Maintaining exabyte-scale Hadoop clusters

📖 Topic: Data Infrastructure
🧠 Level: Advanced

Summary: LinkedIn re-engineered the block placement strategy in Apache Hadoop’s HDFS to support exabyte-scale clusters storing about 5 exabytes of data and 10 billion objects while maintaining 99.99% availability. As clusters expanded to thousands of nodes, the default rack-based replication policy caused heavy overhead during maintenance because nodes had to replicate large volumes of data before going offline.

To address this, LinkedIn introduced upgrade domains, logical groupings of datanodes that distribute replicas across broader failure boundaries than racks. By updating the block placement policy without disrupting production traffic, LinkedIn eliminated large-scale replication during maintenance, reducing network congestion, speeding up upgrades, and allowing maintenance on up to 4.5% of datanodes per day while maintaining performance and reliability.

💡 Why is this relevant for DEs?

• Data reliability and availability at massive scale: Data engineers working with distributed storage systems such as Apache Hadoop must design architectures that ensure high data availability, redundancy, and fault tolerance, even when clusters contain thousands of nodes and exabytes of data.

• Operational scalability is a core data engineering challenge: As data platforms grow, routine operations such as hardware upgrades, patching, and cluster maintenance must be redesigned to avoid massive data movement or downtime, highlighting the need for an architecture that scales operationally as well as technically.

Elements of a Modern Data Strategy

📖 Topic: Data Strategy
🧠 Level: Intermediate

Summary: A modern data strategy is what separates organizations that talk about data from those that actually drive results with it. It aligns people, processes, and technology across five foundational pillars.

• Tying every data initiative directly to business outcomes through deep stakeholder engagement.

• Selecting a tech stack that scales and works as a cohesive ecosystem.

• Embedding governance for high-quality data and trust.

• Investing in talent with clear roles and continuous enablement.

• Laying out a prioritized roadmap that balances quick wins with long-term transformation.

Without this foundation, organizations end up with fragmented data, slow decision-making, shelfware investments, and an inability to capitalize on AI and automation when it matters most.

💡 Why is this relevant for DEs?

• The data stack, pipelines, storage architecture, and integration layers defined in a data strategy are largely designed and implemented by data engineers, making them central to turning strategy into working systems.

• Data engineers operationalize data governance, lineage, access controls, and reliable data pipelines, ensuring that data across the organization is trusted, consistent, and usable for analytics and AI.

⚒️ Data Engineering Open Forum 2026

The Data Engineering Open Forum (DEOF) is a community-driven conference featuring in-depth sessions that address the real challenges and innovations in data engineering today. The talks cover a wide variety of topics, including AI Agents, Multimodal data, Data lineage, Data Observability, Semantic layer, and more.

Out of a stacked lineup, the two sessions below caught my eye.

• From Manual to Magical: Building AI Agents for ETL Automation by Himanshi Manglunia, Senior Data Engineer @ AWS: This talk demonstrates a production-grade, autonomous ETL system in which AI agents (powered by Kiro and Claude) handle end-to-end pipeline changes.

• Inside OpenAI’s Internal AI Data Agent by Bonnie Xu, Staff Software Engineer @ OpenAI: This session provides a look under the hood of OpenAI’s internal data agent, a tool that lets employees turn questions into insights in minutes. Xu unpacks the agent’s core architecture, the multiple layers of context it uses to answer queries, and how the team keeps that context updated with almost zero manual intervention.

📋 Agenda and more details HERE.

👉 RSVP before March 22 to get an exclusive DET community discount (33% off).

💎 Open Source Gems

Cube Core: Semantic Layer

Cube Core is an open-source semantic layer that lets organizations define metrics, dimensions, and business logic once and reuse them across BI tools, embedded analytics, and AI agents through standard REST, GraphQL, and SQL APIs. It works with all major SQL data sources, Snowflake, Databricks, BigQuery, Postgres, and more, and includes a built-in caching engine for sub-second query performance.

💡 Why is this useful for DEs?

• With Cube Core, data engineers define metrics and business logic in one place, and every tool, BI dashboards, applications, and AI agents pull from the same definitions.

• Most organizations run multiple BI tools and data platforms. A semantic layer sits between them and provides a single, consistent way to query data, eliminating duplicate logic and unnecessary complexity.

Github: https://github.com/cube-js/cube

💡 DE Tip of the Month

Processing large time-series datasets in dbt

Microbatch incremental models in dbt efficiently process large time-series datasets by splitting transformations into small, time-bounded batches based on an event_time column.

• Atomic, idempotent batch execution: Each batch represents a self-contained unit of work that can run independently, be retried if it fails, and even execute in parallel for faster data processing.

• Simpler incremental logic: Unlike traditional incremental models that require custom SQL conditions, micro batch automatically determines which batches to run, simplifying model design.

• Flexible backfills and failure recovery: Engineers can easily reprocess historical data or retry failed batches by specifying time ranges (--event-time-start and --event-time-end) without rebuilding the entire dataset.

📊 Community Poll

Until next time, cheers!

Ananda, Sukanya & Volker

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

This series is designed for one outcome - clear lessons on how experienced builders approach data engineering problems, not just a founder story or a project overview, so that our DET community can have reusable mental model on how to think about scale, operability under real constraints.

To kick things off, we sat down with Weimo Liu, co-founder of PuppyGraph, whose career spans database research, TigerGraph, and Google’s F1 team. His perspective is a perfect stress test for the themes we care about: why graph ideas have been academically compelling for decades, why production adoption is still hard, and how “scale” in industry changes what success even means. In this interview, Weimo walks through the shift from chasing benchmark wins to optimizing for cost and operability—and shares a provocative approach to graphs: running graph queries directly on modern table formats like Apache Iceberg, instead of asking enterprises to migrate and reload everything.

Let’s dive in and walk through Weimo’s journey.

- Swetha Sekhar

🎟️ Conference: Data Engineering Open Forum 2026

We are hosting the 3rd Data Engineering Open Forum (DEOF) on April 16 in San Francisco! Here is what you will experience expect at the event:

• Sessions by speakers who are solving cutting edge problems in data engineering, for examples, Apache project creators and PMCs like Julien Le Dem, Boyang Jerry Peng, and Jack Ye.

• Intentionally-designed activities that you can sign up to engage in small-group networking (because we know it’s hard to make conversations at conference).

• Opportunities to connect with data engineering teams at top tech companies (like Netflix, Airbnb, and more) at their booths.

Our ultimate goal is that when you look back some day, you could confidently say “I’m so glad I went to DEOF”, because the people you met there or ideas you walked away with made a difference in your career.

👉 See the agenda HERE. RSVP before Early Bird price ends on March 11.

Spotlight: Weimo Liu

For those in the DET community who may not know you yet, could you briefly introduce yourself?

Hi everyone — I’m Weimo. It sounds like the “self-driving” car, Waymo. I’m the co-founder of PuppyGraph. Before starting PuppyGraph, I worked at TigerGraph and Google’s F1 team. TigerGraph is a graph database startup, and F1 is Google’s internal unified SQL query engine, serving billions of queries per day. Thanks so much for having me, it’s a real pleasure to be here.

You’ve been working in the database world since your PhD years. Tell us about your career journey — what initially drew you to databases, and what kept you in the space for so many years?

Back in college, I did very well in my data structures class, and my professor invited me to join his lab. That led me into database research, where I published papers on spatial databases.

When I applied for PhD programs, I actually looked up professors who had published heavily at SIGMOD and VLDB in recent years, and found my advisor Dr. Zhang.

After my PhD, I joined TigerGraph. The founding CTO was a close friend of Dr. Zhang, and I worked there for almost three years. Later, I joined Google — partly because it felt like a safe choice — and worked on the F1 team.

What keeps me in databases is that the field is very concrete. The problems are understandable, progress is measurable, and when something improves, you can usually prove it.

You once said that “half the database papers back then were about graphs.” What made graph workloads so captivating for researchers?

First, graphs are hard. The data is highly connected, and it’s very difficult to shard and distribute efficiently, which constantly creates room for new algorithms and system designs.

Second, motivation is very natural. Researchers don’t need to invent artificial use cases just to justify the work — real graph problems already exist everywhere, such as anti-fraud, network observability, social network analytics, and cybersecurity.

Third, graph theory in mathematics is extremely rich. Computer scientists can borrow powerful ideas from math and turn them into systems that are both interesting and practical.

After your PhD, you took your first industry role at TigerGraph. What surprised you most about the difference between academic database research and real-world database engineering?

In academia, performance is everything. You aim for benchmarks that are 10× faster than the state of the art.

In industry, speed alone doesn’t mean much. Cost, scalability, operability, these often matter far more. You can’t just spend unlimited resources to get a better benchmark number.

Also, no one really cares about “10× faster,” or even “100× faster.” If you walk around AWS re:Invent, every product claims to be 100× better than the state of the art, without even defining what that state of the art is. There’s no peer review, and nobody truly believes those numbers.

Later at Google, you worked on database systems at a much larger scale. What kinds of problems were you solving there, and how did that experience change how you thought about data systems?

F1 is a federated query engine that can query almost every data source and format inside Google. It serves billions of queries per day and handles most of Google’s OLAP workloads — from the fastest, most expensive systems to the slowest, cheapest ones.

In the early days, Google had many different OLAP systems across different organizations. Over time, more and more teams connected their data sources to F1. Eventually, most of them were either deprecated or absorbed into the F1 ecosystem.

That experience made me realize how important unified federation really is.

💡 Editorial Note: F1

F1 is Google’s globally distributed SQL database built on top of Spanner, giving full relational features (SQL, ACID, secondary indexes) at planetary scale. Spanner handles sharding, replication via Paxos, and external consistency using TrueTime (a globally synchronized clock), while F1 provides a stateless SQL layer that parses, optimizes, and executes distributed query plans. Data is organized with hierarchical, interleaved tables to co-locate related rows in the same key ranges, reducing cross-shard transactions. Writes use Spanner’s two-phase commit across replicas, and reads use consistent snapshots. In short, Spanner provides globally consistent distributed storage, and F1 turns it into a fully featured distributed SQL database.

👉 Read the full F1 paper HERE.
👉 Read the full Spanner paper HERE.

What prompted you to write a new graph query framework? What problem did you feel wasn’t being solved, or wasn’t being solved in the right way?

I had been thinking about this for a long time. At TigerGraph, many potential users showed strong interest in graph technology, but most of them couldn’t actually adopt it in production. For example, a large bank spent 18 months loading all of its data into the system — something that would be unrealistic for most enterprises. That told me something fundamental was wrong.

After joining Google’s F1 team, I realized these users weren’t really looking for a graph database, they were looking for the graph itself.

I didn’t start the project immediately because, unlike Google, the external data world wasn’t standardized. That changed when I read an a16z blog post announcing that the Apache Iceberg team had left Netflix to found Tabular. I realized the timing was finally right.

My co-founders and I actually reached out to the Apache Iceberg creators with a very simple demo: running graph queries directly on Iceberg, faster than most graph databases on the market. That surprised them too, they hadn’t optimized Iceberg for graph workloads at all. They supported us a lot, not just on engineering, but also on go-to-market.

During the development of the framework, what were some of the hardest technical problems you had to solve?

Graph systems are notoriously difficult to scale, which is one of the main reasons they have not been widely adopted by enterprises with very large datasets.

We define the operators in a graph query plan as NodeOperators and EdgeOperators, where the input and output of each operator is a collection of nodes or edges. PuppyGraph assumes that all graph queries, patterns, and algorithms can be expressed as combinations of these operators.

PuppyGraph implements a rule-based optimizer for logical query execution plans, and a hybrid rule-and-cost-based optimizer for physical execution plans. Because all inputs and outputs are collections, any individual operator can be massively parallel processed (MPP) and vectorized for evaluation.

How did you find your co-founders and early team members, and what qualities did you look for when building the team?

They’re all old friends. Our CTO was my college roommate, and our chief architect lived next door. They’re well-known competitive programming contestants, and honestly, much better programmers than I am.

Trust mattered the most. In the early days, you need to move fast and collaborate extremely smoothly. We were lucky to have already built that trust years ago, it really feels like a reunion of old teammates.

Has the graph problem space changed in response to the rise of AI? Where do you think the field is actually heading?

Yes, very much so. Initially, we didn’t think of this problem space as being closely related to AI. But more and more AI companies and AI teams started reaching out.

AI generates more data, and at the same time, it activates a lot of previously “cold” data. A human data analyst might be able to keep track of hundreds of tables, but AI systems can reason over thousands or even more simultaneously. Our view is that when you already have tables, you already have knowledge. You don’t need to build a separate knowledge graph — you can simply treat your existing data as a graph. That structure naturally provides context to LLMs.

Before we let you go… who should we interview next, and why?

Haha — I’d suggest Zhou Sun, the co-founder of Mooncake Labs. He’s sharp, opinionated, and deeply knowledgeable about databases. I think it would be a really engaging conversation.

Key Takeaways

• The central reframing: many teams don’t need “a graph database”; they need graph type queries over the data they already have, and open table formats like Apache Iceberg make that approach more feasible.

• AI shifts the bottleneck from “finding data” to “connecting data with meaningful relationships” When AI can reason across thousands of tables, the problem becomes building reliable structure/context across them.

• Weimo’s differentiator is translation: turning research-grade system design into something enterprises can run—where operability and time-to-value matter as much as performance.

Community poll

💬 How to stay connected

• LinkedIn

• PuppyGraph

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello Folks,

Great to connect with you through this month’s newsletter! I hope the resolutions you set at the start of the year are still going strong.

I'm writing this from Coimbatore, India, where the cooler winter mornings are slowly transitioning to warmer and brighter days. There's something I love about this time of year here - clear skies, fresh energy in the air, and that sense of momentum building as the season shifts.

That momentum reminds me of data engineering. The big transformations rarely happen overnight, but every new tool we explore, every system we scale, every challenge we solve it all adds up over time. It’s that steady, consistent work that fuels real progress. And through it all, we're building something meaningful: data systems people can actually rely on.

Outside of work, I enjoy reading, travelling, and watching movies - small ways to recharge and stay curious. In many ways, this community offers the same: a space to learn from each other and continue growing together.

This edition is packed with ideas and resources from across the community. I hope you find something here that sparks a new idea or gives you a fresh perspective.

Happy reading, and thank you for being on this journey with us.

- Sri

📚 Data Pulse

Netflix: Simplify Data Movement using Data Bridge

📖 Topic: Data Ingestion
🧠 Level: Intermediate

Summary: Netflix introduces Data Bridge, a unified control plane that standardizes how data is moved across its vast ecosystem of data stores. Instead of teams building case-specific/custom pipelines for every new data movement use case, Data Bridge abstracts the implementation details of how it is moved from why and what data needs to be moved. Data engineers declare their intent once, and the platform handles routing, execution, and operational concerns behind the scenes.

💡 Why is this relevant for DEs?

• Platform over Pipelines: It helps data engineers move from building siloed data movement pipelines to designing a centralized orchestration layer. This enables reuse, governance, and delivery at scale, rather than reinventing the wheel for every use case.

• Self-Service: New teams and data stores can plug into the system without reinventing ingestion or replication patterns.

• Operational Consistency: It has built-in handling for retries, monitoring, and failure management, so it ensures consistent reliability across all data transfers.

• Reduced Fragmentation: It consolidates fragmented data movement systems into one standardized approach.

Advice to Start a Blog

📖 Topic: Brand Building
🧠 Level: All levels

Summary: David Robinson argues that aspiring data scientists (note: valid for DEs as well) should start blogging as a key strategy for breaking into the field. Rather than just completing courses, candidates should publicly share analyses, tutorials, and projects on topics they find interesting. Blogging serves three critical purposes: it provides hands-on practice with real-world data analysis and communication skills; it creates a portfolio that demonstrates capabilities to potential employers better than resumes alone; and it generates feedback from the community while helping build a professional network. Robinson emphasizes that posts don’t need to be perfect! Sharing any public work is valuable, and even simple explanations of concepts you’ve mastered can resonate with audiences.

💡 Why is this relevant for DEs?

• Portfolio & Personal Brand: A blog provides concrete examples of your work that make interviews and applications more compelling than resumes alone, while establishing you as a thought leader and building a professional network that creates unexpected job opportunities.

• Practice with Purpose: Blogging forces you to work with real-world messy data and communicate findings, developing the exact skills employers need while revealing knowledge gaps and hidden strengths through community feedback.

LinkedIn: Engineering the Job Ingestion System at Scale

📖 Topic: Data Platform
🧠 Level: Intermediate

Summary: LinkedIn shares how they built a large-scale job ingestion system capable of processing millions of job postings from diverse sources reliably and efficiently. Instead of relying on fragmented ingestion workflows, LinkedIn designed a unified, scalable ingestion architecture that standardizes parsing, validation, enrichment, and indexing of job data. The system focuses on handling high-volume, heterogeneous inputs while ensuring data quality, low latency, and operational stability across downstream search and recommendation systems.

💡 Why is this relevant for DEs?

• Scalable Ingestion Architecture: Instead of building ingestion logic per partner or per feed, LinkedIn invested in a standardized ingestion framework. This represents the move from ad hoc pipelines to platform engineering. The architecture supports horizontal scaling and distributed processing, allowing millions of records to be processed per day without degrading performance or increasing operational overhead.

• Data Quality at Ingestion Layer: Schema validation, normalization, and enrichment happen early in the pipeline, reducing downstream correction cycles and improving the reliability of search and recommendation systems.

• Handling Heterogeneous Sources: In real-world systems, inputs from different sources rarely follow a single format. A strong ingestion framework abstracts this variability and converts it into a unified internal schema. For DEs, this means designing flexible parses, schema evolution strategies, and metadata-driven mappings that can scale without rewriting pipelines for every new source.

• Operational Resilience: The architecture is designed to manage failures, retries, and backpressure effectively, ensuring stability even during traffic spikes.

Uber: Engineering for Trillion-Record-Scale Data Lake Operations

📖 Topic: Data Lake
🧠 Level: Beginner

Summary: Uber shares how they adopted Apache Hudi to solve large-scale data lake challenges such as late-arriving data, incremental processing, and inconsistent batch pipelines. Hudi adds transactional semantics, upserts, and time travel capabilities on top of cloud object storage, enabling Uber to treat their data lake more like a database while retaining scalability.

Instead of rebuilding datasets from scratch, Uber uses Hudi’s incremental ingestion and record-level updates to efficiently manage continuously evolving datasets across thousands of pipelines.

💡 Why is this relevant for DEs?

• Reliable Data Lakes (Lakehouse Foundations): Hudi introduces ACID transactions, schema evolution, and rollback support to the data lakes; it helps data engineers safely handle late data, reprocessing, and failures without breaking downstream pipelines.

• Scales with Streaming and Batch: Hudi supports both streaming ingestion and batch workloads, helping teams unify real-time and batch pipelines under a single data store.

• Incremental Pipelines: It allows pipelines to process just the new or updated data since the previous run, instead of scanning terabytes of data in each run.

• Indexing and Fast Upserts: It allows data engineers to perform fast row-level updates across hundreds of partitions and billions of records.

⚒️ Workshop: Getting Started with Protobuf APIs

If schema evolution keeps biting you in production (breaking changes, inconsistent validation, tooling sprawl), this Protobuf workshop is worth your time. You will learn:

• How to adopt Protobuf to prevent breaking changes and evolve schemas safely across teams and languages

• How to use Protobuf in event streaming and data pipelines for better data quality

• Best practices for designing real-world APIs

👉 Sign up for the workshop HERE.

(This message is sponsored by Buf)

💎 Open Source Gems

skills CLI for the open agent skills ecosystem

Skills are reusable capabilities for AI agents. They provide procedural knowledge that helps agents accomplish specific tasks more effectively. skills is an open-source CLI for installing and managing skill packages for agents.

Together with skills.sh, a directory and leaderboard for skill packages, it allows you to easily install and manage skills for all common AI tools like Claude Code, Codex, Cursor, OpenClaw, Gemini CLI, and many more.

💡 Why is this relevant for DEs?

Using AI to implement data pipelines, or any type of software development, has become the new norm. However, in Data Engineering in particular, we work with highly specialized tools, different versions, and varying best practices. This often leads to significant back and forth when using AI to speed up implementation. Installing skills can help, as they target specific use cases. Tools like Claude Code automatically apply a skill when they determine it fits the use case, which can significantly improve the quality of the output.

🧑‍💻 Skills for DEs

• data-quality-frameworks: Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.

• brainstorming: Turn ideas into fully formed designs and specs.

• dbt-transformation-patterns: Production-ready patterns for dbt.

• authoring-dags: Creating and validating Airflow DAGs using best practices.

• data-storytelling: Transform raw data into compelling narratives.

And many more.

Github: https://github.com/vercel-labs/skills

💡 DE Tip of the Month

Treat data contracts as non-negotiable in modern pipelines:

As data feeds more dashboards, models, and AI systems, failures are shifting from job failures to correctness. Pipelines run successfully, but silent changes in schema, freshness, or semantics break dashboards, models, and business decisions.

Data contracts help prevent this. They define clear expectations between data producers and consumers so everyone knows what a dataset promises to deliver.

💡 Why it matters now more than ever

• AI agents, auto-generated SQL, and self-serve analytics are increasing the number of data consumers without deep context

• Faster development with a variety of tools like dbt, Spark, and Flink increases the risk of unintended schema changes.

• The cost of bad data is often higher than pipeline downtime.

💡 How to start

• Add contracts to your most critical datasets

• Enforce them with tests and freshness checks

• Make them visible in your metadata catalog

• Start small and expand gradually.

Teams that treat data contracts seriously spend less time firefighting and build stronger trust between data producers and consumers.

Let us know what you like the most in the newsletter. See you next time!

Until next time, cheers!

Sri, Sukanya & Volker

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

This interview highlights how diverse backgrounds and perspectives help people build better data tools.

I met Archie through a virtual data conference called MDS Fest where he introduced Evidence as an open-source reporting tool using only SQL and Markdown (source). What stood out to me was his non-traditional path in data moving from strategy consulting to building open-source data tools. Spending years translating raw data into business decisions gave Archie a deep perspective on how data actually gets used, and where data tools might often get in the way.

Since then, he’s been active in the open-source community shipping data tools, and he’s now leading growth at Evidence.

Hope you find the lessons from this conversation meaningful.

Let’s go!
— Eddy

Spotlight: Archie Wood (Head of Growth at Evidence)

For years, BI meant expensive, monolithic systems that stifled adaptability. But as data environments grew more complex, the demand for flexibility exploded.

This shift has driven the rise of composable BI. Instead of a “black box”, modern stacks use decoupled components to give teams total control over their metrics and visuals.

It was exactly this frustration with the “old way” that pushed Archie Wood (Head of Growth at Evidence) to move from analyzing data to building the tools behind it. His path is anything but traditional, and his story reveals a lot about where modern BI is heading.

📊 Sponsored Insight: The State of Airflow 2026 Report

The State of Apache Airflow 2026 Report, the largest data engineering survey ever, draws on insights from 5,800+ data engineers on how Airflow is actually being used today. In this report, you’ll learn:

• How the role of the data engineer is evolving

• How early adopters are leveraging Airflow 3 features

• How orchestration-first teams tend to ship AI to production faster

👉 Read the report HERE

(This message is sponsored by Astronomer.)

Could you walk us through your journey into the data engineering space?

Today I work as an open-source maintainer and lead growth for a BI tool but I got here by being deeply frustrated as a non-technical data user.

I maintain several open-source projects (DuckDB GSheets and gosql) on top of Evidence because I’ve lived the pain of data tools getting in the way of insights.

I’ve been working with data for about 8 years, though I came into data in a non-traditional way. I started in management consulting, where data was mainly used to make slides and working on Excel until it fell over at a couple hundred thousand rows. Tableau opened my eyes to what better BI tools could do, and I was hooked.

Later, I joined an e-commerce company in London called Patch Plants (UK houseplant retailer) as Strategy Manager and eventually Chief of Staff where I learned SQL and Python because I kept pestering the data team for requests. They eventually gave me developer access and let me self-serve. We used Snowflake, dbt, Looker, Metabase and a lot of Google Sheets. It was a wild couple of years. Between 2019 and 2021, Patch grew nearly 200% and delivered over a million plants to 300,000 customers. Being part of that growth while building out the BI function while navigating complex challenges like Brexit was an incredible learning experience.

In 2021, I moved to Canada and spent a bit of time working on my own data startup ideas, but nothing really stuck. Eventually, I met Adam and Sean, the founders of Evidence who were building exactly what I’d always wanted, a more flexible BI tool that gave you more control over visuals. It often felt like the BI software I used was slowing me down more than helping me and I remember thinking, “Someone has to fix this and I’d like to be part of that solution”. I traded SQL and spreadsheets for building code extensions that made it easier for data folks to tell data stories using software engineering principles.

💡 Key takeaway: Non-traditional paths into data are a superpower. Firsthand frustrations coupled with domain expertise become the fuel to build better data tools.

As an open source maintainer and Head of Growth at Evidence, how do you balance the engineering mindset with the community and adoption side of the work?

It comes in waves. Some months I’m deep in code, shipping features non-stop. Other months are more about storytelling and figuring out how to explain what we’ve built, who it’s for, and where it fits.

A big part of making that balance work is being clear about who we’re talking about when we say “users”, because we have two different audiences.

Our open-source product is something teams can self-host and deploy as a static site. The source code is fully public on GitHub, and the community around it includes contributors, maintainers, and people using it in production on their own infrastructure.

Evidence Studio is our commercial SaaS product and while it isn’t open source, we’re still working out whether we may open source parts of it over time.

When you’re balancing engineering, customer success, and an open-source community, empathy matters a lot. In open source especially, it’s easy to think “people are using it wrong,” but most of the time it means we designed something with too much friction. Evidence initially required installing Node.js and managing local dependencies, which worked fine for contributors but was a blocker for a lot of non-technical users. As a result, we’ve built a browser-based experience to make onboarding smoother and collaboration easier, while maintaining core functionality.

Keeping the project authentic to its open-source roots is a constant balance, and it’s not always straightforward. One way we’ve supported open-source development is by offering commercial support services for teams self-hosting the open-source version. Bigger enterprises often want the ability to pick up the phone and resolve issues quickly, and that support helps fund the work while keeping the open-source project healthy.

💡 Key takeaway: Balancing engineering and community comes down to empathy. Data engineers need to listen to users, understand their real needs, and remove friction wherever possible.

On a personal level, you’re balancing building tools, growing a community and parenting. What habits or systems have helped you sustain creativity and focus amid that mix?

I’m still early in parenting days so any “system” I have keeps changing. Kids force you to adapt constantly. My partner and I both work full time, and because she’s based in New York, I handle weekdays and she takes weekends. It’s intense, but it works.

I run my days in blocks: mornings and evenings are family time, the core workday is protected, and nights are for decompression or light work. Startups don’t respect schedules and if something breaks, you jump in but parenting has made me ruthless about focus. You quickly see how much time gets wasted otherwise.

💡 Key takeaway: It’s important to be intentional with time management and adaptability. Knowing when to pivot is crucial to staying productive and effective.

What’s something you’ve learned from the Evidence community that changed your thinking about how open-source BI should evolve?

BI tools are deceptively complex pieces of software. Setting one up can take weeks, and the person doing it usually has three other jobs. They might be a data engineer, director of data, or even a software developer doing BI on the side. By the time they’ve cleaned and modelled their data, users just want a tool that helps them explore and visualize it without the complex setup time.

That gap between lightweight data exploration (querying in SQL or using a notebook) and building polished dashboards is still massive. It shouldn’t be. Most people want something that feels as quick and flexible as a notebook or SQL client, but with the clarity and polish of BI.

That’s where open source really shines. It can move faster, stay flexible, and meet people where they already work. Projects like Evidence, Streamlit and Rill Data live in that middle ground between exploration and presentation. Where you can go from query to insight to something you’re proud to share, without waiting on a whole implementation cycle.

💡 Key takeaway: There’s a distinct product niche for BI tools that can offer out of the box data exploration and BI tools which provide well governed dashboards.

You mentioned other open-source projects (Streamlit, Rill) that sit in this middle ground. As a data engineer, how do these compare to Evidence, and when should one choose which tool?

Evidence is the most analytics-engineering friendly if your goal is trusted reporting as code. It’s SQL-first and Markdown-first, so dashboards behave like software artifacts: version controlled, reviewable in PRs, reproducible across environments, and easy to ship as a static site or data product. It’s especially strong when you want to pair charts with narrative and definitions (“what changed, why it changed, and what to do next”).

Streamlit is best thought of as a Python application framework, not a BI tool. It’s ideal when you need more than charts (custom workflows, business logic, forms) and UI needs to be bespoke. The tradeoff is governance isn’t built in: metric definitions, permissions, consistency, and performance patterns are on your team to implement. It’s great for prototyping, but can become “a collection of apps” without standards.

Rill is closer to a productized BI experience: fast dashboards, metrics-driven exploration, and workflows optimized for operational analytics and drill down analysis. It shines when you want a Looker-style experience without building an app. Compared to Evidence, it’s less about narrative reporting and more about interactive exploration with an opinionated structure for speed and consistency.

Rule of thumb:

• Choose Evidence for governed, version-controlled reporting with narrative.

• Choose Rill for fast metrics + interactive exploration with a BI-product feel.

• Choose Streamlit when you need a custom data application, not just dashboards.

Was there a specific GitHub issue or a discussion on a PR that provided a distinct technical learning moment you can share?

One small example that stuck with me was number formatting. A user opened a GitHub discussion pointing out that our default format:

€1,234.56 was incorrect!

In Germany, the same number should be displayed as:

1.234,56 €

That difference matters:

• Comma for decimals (,56)

• Period for thousands (1.234)

• Currency symbol at the end (€)

The user could technically “fix” it upstream in the database layer, but that would turn the number into a string and break Evidence features like charts and numeric props. This edge case required a code change to support locale-aware formatting properly.

It was a good reminder that small details can impact clarity and trust, and we only caught it because someone raised it publicly on GitHub.

💡 Key takeaway: Open source makes your blind spots visible. Real users surface real-world edge cases that make the product better for everyone.

What advice would you give to data engineers who are curious about building open source tools?

Start by contributing. If you use an open source tool and spot a bug or a feature gap, engage with the community on GitHub, Slack, or Discord. Small contributions are the best way to learn how open source really works: people building in public, improving tools they rely on, and collaborating across the world.

If you want to build your own open source tool or company, go in with equal parts excitement and realism. It’s deeply rewarding but hard to sustain. You’ll meet amazing contributors and a few demanding users, and you’ll eventually face the hardest question which is how to monetize without breaking what makes it open.

💡 Key takeaway: Start small and engage the open source community on public channels.

Community poll

💬 How to stay connected

• https://www.linkedin.com/in/archiesarrewood/

• https://github.com/archiewood

• https://evidence.dev/

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

First of all, Happy New Year, everyone!

Hope you all had a happy and restorative holiday season. My own break was a beautiful blur of firsts: spending the first Christmas and New Year with my new daughter. There is something profoundly grounding about watching a child experience the magic of a Christmas market for the first time. Between the smell of roasted nuts and the twinkling lights, I found myself doing a different kind of “data collection”capturing every giggle and wide-eyed stare.

That calm period gave me some much-needed space to reflect on 2025. In the data world, 2025 was the year we stopped talking about AI alone and started weaving it into our actual pipelines. We moved from Copilots that write SQL to AI-assisted metadata management. Personally, it taught me that while the tools change faster than a toddler’s mood, the fundamentals: reliability, cost-efficiency, and trust remain our North Star.

Let’s make 2026 the year we build less “tech debt” and more “data wealth.”

With that, we start our first newsletter of the year with a bonus section. A few of our DET editors share their New Year Resolutions with you, our amazing community. Hope you enjoy this edition and perhaps draw some inspiration for yours, if you’re in the process of creating your own.

- Chozhan

📚 Data Pulse

Templatizing Spark Declarative Pipelines with DLT-META

📖 Topic: Data Pipelines
🧠 Level: Beginner

Summary: Databricks introduces DLT-META, an open-source framework designed to solve pipeline sprawl. It allows engineers to use a metadata-driven approach to generate Delta Live Table (DLT) pipelines. Instead of writing unique code for every table, you define the pipeline logic in a JSON/YAML template, and the framework dynamically generates the Spark jobs.

💡 Why is this relevant for DEs?

• Operational Scalability: It enables us to move from “coding” individual pipelines to “architecting” frameworks, allowing a single engineer to manage thousands of tables.

• Reduced Tech Debt: By templatizing the logic, you ensure consistent data quality checks (expectations) and governance across all data assets.

• Faster Onboarding: New data sources can be integrated by simply updating a metadata file rather than deploying new code modules.

• DRY (Don’t Repeat Yourself): It enforces a “standard library” of transformations, making maintenance and debugging significantly simpler across the lakehouse.

Meta: Video Invisible Watermarking at Scale

📖 Topic: Data Engineering
🧠 Level: Intermediate

Summary: Meta uses Video Invisible Watermarking, a system designed to embed traceable metadata into video content without affecting visual quality. This enables Meta in detecting AI-generated videos, verifying who posted a video first, and identifying the source and tools used to create a video. This is a massive-scale data engineering challenge given the amount of videos being circulated across Meta platforms every minute. It requires processing billions of video uploads in real-time, ensuring the watermark survives compression, cropping, and screen recording. In this post, Meta shares how they overcame the challenges of scaling invisible watermarking, including how they built a CPU-based solution that offers comparable performance to GPUs, but with better operational efficiency.

💡 Why is this relevant for DEs?

• Complex Data Modeling: It showcases how to handle unstructured data as a first-class citizen in a pipeline.

• High-Throughput Processing: DEs can learn about the infrastructure required to run computationally expensive ML models (for watermarking) on every single write operation.

• Data Provenance and Trust: As GenAI content floods the web, building “Trust Infrastructure” like watermarking will become a core responsibility for data platforms.

• Multimodal Engineering: It bridges the gap between traditional signal processing and modern distributed data pipelines at an exabyte scale.

Lyft’s Feature Store: Architecture, Optimization, and Evolution

📖 Topic: Data Architecture
🧠 Level: Advanced

Lyft's revamped Feature Store is a mission-critical "platform of platforms" designed to centralize and scale machine learning feature management across the organization's entire rideshare stack. The implementation utilizes a "Features-as-Code" approach where engineers define features via Spark SQL for business logic and JSON for metadata. This configuration is automatically converted into production-ready Airflow DAGs that compute and publish data to both a Hive-based offline store for training and a high-performance online serving layer. For low-latency retrieval, the system employs a hybrid architecture featuring DynamoDB backed by a ValKey write-through cache, alongside OpenSearch specifically for vector embedding support. This robust setup now handles over a trillion annual operations, serving as the foundational infrastructure for everything from real-time pricing to fraud detection.

💡 Why is this relevant for DEs?

• Democratizing feature engineering: By allowing anyone with SQL and JSON knowledge to deploy production-grade pipelines.

• Separating storage and computation: To ensure high-throughput batch writes never interfere with sub-millisecond read performance.

• Solving the small file problem: Through automated background compaction and clustering within the streaming lakehouse layer.

• Enforcing organization-wide Data Contracts: Guaranteeing feature freshness, ownership, and schema stability for downstream consumers.

• Reducing operational overhead: By migrating complex orchestration from in-house tools to managed services like Astronomer (Airflow).

• Future-proofing infrastructure for AI: With native vector indexing in OpenSearch to support emerging LLM and agentic workflows.

• Standardizing multimodal data management: Unifying raw metadata, binary blobs, and embeddings into a single searchable ecosystem.

💡Blog: Building AI Agents for Data Engineering Ops

Debugging production data pipelines is a critical part of a data engineer’s job but it can be particularly difficult. The team at Corelayer explored how AI agents can assist with investigations: detecting anomalies, correlating logs and DAG runs, forming hypotheses, and producing evidence-backed root cause analyses with humans firmly in the loop. In this article, you will learn the failure modes, operational complexity, and the design principles (evals, guardrails, transparency) that matter when applying AI to real-world data engineering workflows.

👉🏼 Read the full article HERE.

(This message is sponsored by Corelayer.)

✨ New Year Resolutions from DET Editors

Volker

In our first Community Spotlight newsletter, the vote showed that the biggest blockers to write or speak about work and experiences are “stuck in drafts forever” and “too tired after coding all day.” I feel both. I tend to overthink everything I share, so ideas die in perfectionism. But data engineering moves fast, and the value of insights fades quickly. This year, I want to be braver: share earlier, share imperfectly, and don’t be afraid of having opinions.

Swetha

In 2026, I want to develop consistent technical writing in data engineering, using it to support others in the field and also make it as a learning tool. Alongside this, I want to also expand the mentoring guidance to provide aspiring data engineers, give them industry trends, interview expectations and help them achieve their goals, I have always loved contributing to the community and I look forward to taking the deliberate effort to do that this year.

Eddy

In 2026, my resolution is to create more conversations about data engineering. A lot of my growth as a data/software engineer came through meetups and having conversations with dev/product/sales/marketing folks across the tech spectrum. I’ve been fortunate to grow in a place with a vibrant data community, and that environment played a huge role in my professional and technical development. In the future, I hope to carry that same mindset forward: investing in conversations, building community, and learning in public.

Ananda

I plan to contribute to open source, as it gives me significant leverage to create impact across the data engineering community. I intend to increase my knowledge in large-scale data engineering and infrastructure optimization. A more ambitious idea is to learn a new programming language, probably Rust, to expand my skill set. I’ll share what I learnt with the community through mentoring, blogging, and speaking engagements, taking small, consistent steps that compound into decent progress.

Chozhan

Starting late 2025, we’ve been inching toward Autonomous Data Engineering, where pipelines don’t just alert us when they break, but suggest the fix or even auto heal. With that as an opportunity, I approach 2026 with the mindset of reclaiming the mental space for creative architecture & strategy that drives more business value by (re)designing every system to automate the mundane & repetitive. Also, I learned a lot in the past years from being part of communities and this year, I’d like to give back more by sharing my experiences, especially failures & lessons learned using different mediums.

So, dear community, we are curious to hear what your 2026 resolutions are. Let us know via the poll at the end 👇 or in the comments or the community chat 💬.

💎 Open Source Gems

LanceDB: The Multimodal Vector Database for the AI Era

LanceDB is an open-source, serverless-native vector database designed to simplify the management and retrieval of unstructured data for AI applications. Built on the high-performance Lance columnar data format, it offers up to 100x faster random access performance compared to traditional formats like Parquet. It is uniquely useful because it allows engineers to store raw data (images, videos), metadata, and vector embeddings together in a single, unified table. This "SQLite-like" approach means it can run embedded directly in your application or scale to massive cloud-native lakehouses without the overhead of managing a complex database cluster.

💡 Why is this relevant for DEs?

• Unifies raw media and embeddings in a single table, eliminating complex sync logic between disparate data stores.

• Eliminates Parquet’s random access bottleneck, delivering the sub-millisecond retrieval required for real-time RAG.

• Scales from local to petabyte-scale cloud storage using an embedded, zero-infrastructure serverless architecture.

GitHub: https://github.com/lancedb/lancedb

💡 DE Tip of the Month

Build Context-Aware Metadata & Smart Lineage

Most lineage systems only show which tables depend on which, but context-aware metadata explains why changes happened and what triggered them. By capturing deployment info, config changes, and upstream data shifts, teams can perform faster root-cause analysis and smarter testing. This turns metadata into an operational control plane rather than passive documentation.

💡 Key ideas & how to apply:

• Capture deployment metadata alongside data changes
  Example: tagging runs in Airflow / Dagster with context.

  run_tags = {
      "change_type": "schema_update",
      "jira_ticket": "DATA-2411",
      "release": "v2026.01",
      "affected_columns": "order_status, delivery_eta"
  }

  Store these tags in your metadata system (OpenLineage, DataHub, Marquez).

• Track schema evolution events and backfills explicitly (change_type = backfill)

Use this metadata to auto-trigger downstream tests & validations and feed lineage to impact analysis before major releases. When something breaks, you already know, what changed, who changed it, what downstream systems are affected, so the incidents resolve faster and safer.

Let us know what you like the most in the newsletter. See you next time!

Until next time, cheers!

Chozhan, Shubham & Sugandhi

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

I’m excited to launch our new Community Spotlight series! Alongside our classic Data Pulse edition, this format focuses entirely on deep-dive interviews. We have one rule for these: they must offer actionable learnings.

So, grab your favorite warm drink and meet our first guest: Ben Rogojan, aka the Seattle Data Guy! From leaving Facebook to building an independent empire, Ben breaks down how he grew a following of 100k+ and why communication often beats coding.

One thing Ben mentions really stood out to me: “You can get a lot of scale through content, but it’s hard to beat in-person relationships.”

I recently returned from AWS re:Invent in Las Vegas, and I can absolutely confirm that. It was wonderful to meet so many people there, but now it’s time to slow down 🎄.

Let’s dive in!

- Volker

Spotlight: Ben Rogojan (Seattle Data Guy)

Please introduce yourself briefly to the Data Engineer Things community and share how you’ve built your personal brand to reach over 100,000 followers.

Hello! My name is Ben Rogojan, I’ve been working in the data world for over a decade now. I started as an analyst and then found data engineering when I was looking for a role that matched the skills I enjoyed over title. While I was working full-time I had a few people ask me to help on some side projects so I started the Seattle Data Guy brand as a consulting company but I very quickly realized I needed to find a way to get myself known. So I started writing content I wish I had more of when I started.

If your journey from corporate data engineer to successful consultant and content creator was a Git repository, what would be the commit message for where you are right now? And what was the most significant “merge conflict” you had to resolve along the way?

I think the commit message would say something like “breaking out and trying new things” or “pattern interrupt phase”. I’ve been spending a lot of time trying to figure out how to shake up my thinking and habits. I’ve been consulting for a few years and it can be tempting to fall into the same habits. So I have been looking for places where I can challenge my habits.

For merge conflicts, I think one that sticks out was when I was making the decision to leave Facebook. I had spent so much time and effort getting a job there that it felt wrong. I had a consulting business that was growing but it was hard to rationalize the decision due to all the prior effort.

You made the transition from working at Facebook to running your own data consulting business. What were the most critical steps in that journey, and what would you do differently if you were starting over today?

I had been consulting off and on through most of my career. The first project I did was actually helping a client move from Access to SQL Server and now its a lot of SQL Server to cloud migration projects.

The most critical step for any consultant is figuring out how you will land clients. Some people are good at marketing, others sales motions, still others are great at networking in person. I think in terms of what I would have done differently is I would have put more effort into meeting more people in person and building relationships.

I believe that’s even more true now than it was in the past. You can get a lot of scale through content, but it’s hard to beat in person relationships.

Your LinkedIn profile states you’re “Tool-Agnostic, Outcome-Obsessed”. How did you develop this clear value statement, and how has it helped you attract the right clients for your consulting business?

I think it can be tempting to prescribe a prior solution to every client. Also, I’ve come across many clients whose data stacks have gotten decently chaotic due to the fact that a consultant decided to add their preferred stack on top of what already exists.

Instead, I aim to come into all my projects by understanding the companies business needs first, their technical talent, budget, and so forth. From there I look for the tools that meet the companies’ needs. In some cases buying a solution can prove far faster and easier on a company with limited data resources and in others the company’s goal is to make data a major part of their offering and the overhead of adding more data per customer would be too expensive for an out of the box solution.

Your newsletter and YouTube channel have both reached over 100,000 subscribers. What content strategy has been most effective for growing your audience, and how do you balance creating content with client work?

I’ve tried multiple times to create and follow a content strategy and calendar. But I always tend to drift.

So my approach is to find 2-4 themes I enjoy at the moment based on the problems I am either experiencing with recent clients, or discussing with data leaders and create content around that. I find that there are trends in problems so if you keep experiencing a similar problem, it’s likely there are plenty more people dealing with it.

If you had to start over today with zero followers across all platforms, which single platform would you focus on first, and what specific content format would you prioritize to build your audience most efficiently as a beginner?

I think writing is always the easiest place to start. Start sharing your ideas on a platform like Substack (as long as it stays friendly to email) or if you aren’t wanting to write long articles then consider a platform like LinkedIn. I don’t think there is a best place. Early on, I believe you’re focused on finding your voice. So having a large audience isn’t the goal. You’re trying to learn, figure out what type of content you enjoy, etc. Overall, good content gets noticed.

Many professionals struggle to grow their audience while maintaining focus on their core business. What’s your framework for deciding which content to create, and how do you balance creating content that attracts followers versus content that converts followers into clients?

Early on I was mostly focused on creating content that I enjoyed and felt like needed to be covered. Since then I’ve started to create a mix of both content for new data engineers as well as for possible clients.

I think this is an area I could generally improve. I don’t spend a lot of time on a content plan with clear ratios of content x and y. Instead, I write what I want to write about.

For data professionals considering consulting, what has been your most effective approach for finding and securing new clients, and how has this evolved as your brand has grown?

I’ve spoken with dozens of consultants from both technical and non-technical backgrounds and the most common way most of them land clients is through their network. I know that can seem hard as if you’re just starting out as a data engineer or analyst as you’ve likely got a small network. But there are plenty of ways to grow it.

Working, going to events, looking for chances to give whether it be sharing content, working on open source projects, etc.

I’d also add that you don’t need to find large projects first. There are plenty of projects out there where people need help automating an Excel report or some other task that you might think is small but you’ll learn a lot from (I did plenty of those projects).

What are the two most impactful actions data professionals can take today to start building their personal brand, even if they’re currently employed full-time?

We are all building our brand everyday. That doesn’t have to mean posting on LinkedIn, YouTube or Twitter.

So I’d say first, build a reputation at your job as the person that gets things done well. Be willing to take on hard problems. That’s how I started consulting as well, at a job a director who was a consultant and turning around a team learned that I was technical through the grape vine and he reached out asking if I wanted to help on a project.

Second, if you do want to do content, create content that you wished you’d had when you started.

How important are communication skills versus technical skills for data professionals today, and what are the three most impactful lessons you’ve learned?

Learning how to communicate seems to be a forever lesson for me. I am always speaking with people who are better and conveying ideas, getting buy-in, teaching and other skills that require different forms of communication.

Technical skills are always important. I really enjoy Alex Ewerlöf’s diagram of skills for making sure that’s not skipped. In terms of lessons:

1. Think about who you are communicating and tailor the message for them - It’s temping to get frustrated when you explain something you think should be simple to understand but the other party doesn’t seem to get it. I view this as my failure to understand my audience instead of their failure to understand.

2. Images work - Even an image that isn’t perfect is more likely to keep your reader engaged. It also makes it far easier to explain concepts like your internal network map of your various servers so that new employees can quickly get up to speed or if you’re trying to get buy-in for a dashboard having a mock-up makes the end state all the more real. Don’t just write when you can draw or diagram.

3. Cut out fluff - I tend to lean on the fluffy side of writing. I ramble., sometimes add sections purely for self indulgence. But when I reread it, I realize that what I’ve written adds very little to the overall piece. So cut it out.

What’s your vision for the future of independent data professionals, and what emerging opportunities should they be positioning themselves for?

I think data problems will continue to grow for a few reasons.

I believe businesses will continue to demand more and more from their data.

For some businesses that will mean more granular or complex data like images and unstructured data.

For others it’ll be integrating data sets that have never been connected.

But still for others it’ll just be answering key questions about the business. I think many people would be surprised how many businesses and organizations are early in their data journey or perhaps needing to revamp it. I like to say that many companies are in different data decades. So there will continue to be plenty of work for the next few years.

What’s one message you’d like to share with the Data Engineer Things community?

You don’t have to learn or be everything at once. Wherever your journey is now, enjoy it. You never know when it’ll shift. I didn’t know that the last day I was going to enter Facebooks offices was in March 2020. I kept assuming I’d be able to go back. Then it was gone.

And one day I am sure I’ll have my last consulting client.

I’ll put out my last YouTube video.

So enjoy being in whatever stage you’re at.

If you’re learning, learn! Dive deep, don’t worry about what other people are doing.

If you’re executing, execute to the best of your ability.

If you’re raising a family, do that. Be in that moment, because it’ll end and you’ll miss it.

Connect with Ben

• LinkedIn Ben

• LinkedIn Seattle Data Guy

• Substack

• YouTube

Community poll

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi Everyone,

Welcome to the final edition of the 2025 newsletter, a year that pushed data engineering through one of its most transformative leaps. From the rise of agentic data engineering, and multimodal distributed engines, to real-time graph architectures reshaping event processing at massive scale, one thing became clear: data engineering is evolving faster than ever, fueled by the breakneck pace of AI.

Amid all this change, one constant stood out: data engineers remain vital for organizations. They build the reliable, scalable foundations that data systems depend on, ensuring data is clean, governed, and available in real time while optimizing cost and performance as workloads grow.

Today, data engineers sit at the very center of AI readiness, designing data layers that support multimodal workloads, enforcing data contracts to keep teams aligned, and building real-time infrastructure that powers everything from fraud detection to personalized experiences.

We close the year with a powerful reminder: as our systems scale, so must our influence. Becoming a force multiplier is no longer optional for data engineers; it’s essential for driving lasting impact. 2026 will be an even more exciting and challenging year for data engineering.

Thank you for being part of this journey.

- Ananda

📚 Data Pulse

Thinking Like a Data Engineer

📖 Topic: Data Engineer
🧠 Level: Beginner

Summary: Ananth Packkildurai’s article reminds us that becoming a data engineer isn’t really about mastering technology and tools; it is about systems thinking. With the help of a few mentors, he realized that curiosity, the ability to model the world around you, the willingness to iterate, and the confidence to trust yourself matter far more than any framework or technology.

💡 Why is this relevant for DEs?
This article is valuable for data engineers because it reframes the discipline beyond tools, cloud platforms, or pipelines, and focuses on the thinking patterns that truly drive long-term success. In an industry where tools and frameworks change every year, curiosity, systems thinking, iterative design, and self-belief are timeless skills that help engineers navigate complexity and uncertainty. For anyone entering or growing in the data engineering field, these lessons offer a grounding perspective on what actually makes a data engineer effective, resilient, and innovative.

Python User-Defined Table Functions (UDTFs) in Unity Catalog

📖 Topic: Data Engineering
🧠 Level: Intermediate

Summary: Databricks introduces user-defined table functions (UDTFs) in Unity Catalog for PySpark. This feature allows users to write stateful, table-generating logic in Python and register it as a governed object in Unity Catalog (UC). Unlike a scalar UDF (which returns one value per row), a UDTF (User-Defined Table Function) returns zero, one, or many rows for each input row, making it powerful for exploding data or complex transformations. You define it as a Python class with an eval method and call it directly from SQL using the TABLE() keyword. UDTFs are useful for complex tasks like ML inference or pattern detection, but are generally not useful for simple, one-to-one transformations best handled by built-in Spark functions.

💡 Why is this relevant for DEs?
With this new UDTFs, Data Engineers can now implement complex Python logic once, register it centrally under Unity Catalog’s security model, and instantly make it available for use across all workspaces, SQL Endpoints, and pipelines.

Furthermore, it provides automated lineage capture, reduces code duplication, and enables heavy setup logic (like loading a model) to run only once per partition via the Python class structure, significantly boosting performance for stateful processing.

Netflix: Real-Time Distributed Graph (Data Ingestion and Processing)

📖 Topic: Data Architecture
🧠 Level: Advanced

Netflix has evolved from pure video streaming into a multi-vertical platform offering ads, live events, and mobile games, which requires connecting member behavior across many devices and services. Their microservices architecture created siloed data, making it challenging to understand cross-app interactions. To solve this, Netflix built Real-Time Distributed Graphs (RDG) that link events such as watching, logging in, or playing games into a unified, relationship-driven graph. Events flow from devices → API Gateway → Kafka → Apache Flink, where they are filtered, enriched, deduplicated, and transformed into graph nodes and edges before being stored through their Data Mesh at >5 million records per second. Distributed Flink jobs (one per Kafka topic) allow scalable, low-latency processing. The RDG stores a property-graph model of nodes (members, titles, devices) and edges (watching, logging in, playing).

💡 Why is this relevant for DEs?
This system illustrates the real-world challenges DEs face when dealing with large-scale, cross-device behavioral data. Netflix’s RDG shows how traditional warehouses or microservice-siloed datasets fail for real-time identity stitching, behavioral tracking, and relationship-centric analytics. The architecture highlights core DE competencies: event-driven design with Kafka, low-latency stream processing with Flink, schema governance with Avro + registry, backfill strategies with Iceberg, and graph modeling. For data engineers, the RDG is a model for designing high-throughput, low-latency, scalable systems that unify fragmented data to power personalization, fraud detection, recommendations, and cross-domain insights.

Become a better force multiplier in 4 steps

📖 Topic: Career Development
🧠 Level: Intermediate

Force multiplication means creating a large, sustained impact without being directly involved in every task. Individual Contributors (ICs) can achieve this by sharing knowledge instead of repeating answers, delegating with guidance rather than control, removing friction points that slow others down, and stepping back to connect broader patterns across teams. At its heart, force multiplication is about shifting from doing everything yourself to helping others move faster and grow. It’s the impact you create when you document what you know, coach teammates, remove friction, and help everyone stay aligned. In the end, it’s not just about taking on more work; it’s about empowering people so things keep moving forward even when you’re not in the room.

💡 Why is this relevant for DEs?
For data engineers, force multiplication is essential because modern data systems scale far beyond what any single engineer can build or maintain alone. High-impact DEs create durable value by documenting pipelines and playbooks, mentoring teammates on standards, designing reusable data frameworks, improving infrastructure bottlenecks, and aligning cross-functional teams around unified data models or platform capabilities. Whether you’re building ingestion frameworks, managing governance, introducing orchestration standards, or guiding architectural decisions, your ability to enable ML engineers, software teams, and fellow DEs, multiplies organizational velocity. Force multiplication turns a DE from a “pipeline builder” into a strategic technical leader whose influence shapes reliability, scalability, and long-term data culture.

🎤 Online Summit: What’s Ahead in 2026 for Data & Analytics

The summit focuses on preparing organizations for next-generation trends in data, analytics, and AI by showcasing evolving technologies, modern architectural best practices, and emerging AI/ML trends.

• “Streaming Intelligence: Unlocking AI’s Potential with Real-Time Context” by Jake Bengtson, VP of AI, Striim.

• “Putting GenAI to Work with Business Data: Driving Value and Managing Risk” by Kristy Hollingshead, Associate Director of Data Science, Further.

👉🏼 Sign up for the online summit HERE (free registration).

🔖 Featured Read

SIMD: The real superpower behind super fast databases

Author: Shubham Tomar

Modern analytical databases like ClickHouse, DuckDB, BigQuery, and Redshift feel incredibly fast because they’re smart at every step, how they store data, how they fetch it, and especially how they process it. Things like columnar storage, compression, and pruning help reduce the amount of data they touch, but the real boost comes from what happens inside the CPU when it starts crunching the numbers. This is where SIMD (Single Instruction, Multiple Data) becomes the hidden superpower, instead of performing operations one value at a time, SIMD allows CPUs to process multiple values simultaneously using wide vector registers (SSE, AVX, AVX-512). Because registers operate at near-zero latency compared to RAM, vectorized operations massively boost throughput for large-scale analytical workloads.

These databases utilize SIMD using vectorized execution engines, where operations are performed on batches of values rather than one row at a time. Early systems like MonetDB/X100 introduced this idea, and modern engines such as DuckDB, ClickHouse, and other columnar databases have fully embraced it. By running thousands of values through SIMD in a single instruction, these systems make far better use of the CPU’s cache, avoid unnecessary branching, and often achieve speedups of 10x or more.

For data engineers, this means queries return faster without endless tuning, pipelines scale smoothly as data grows, and compute costs stay under control because the engine does far more work per CPU cycle. Understanding how SIMD and vectorized execution work helps data engineers make more intelligent choices when selecting analytical databases during the design phase. When you know which engines fully utilize SIMD under the hood, you can pick systems that deliver faster queries, lower compute costs, and better scalability right from day one instead of discovering performance issues later and trying to fix them with tuning, caching, or hardware upgrades.

Key Features and Learnings

• SIMD enables true parallelism: one CPU instruction can operate on multiple values at once.

• CPU Registers are the real performance hotspot: operations on registers take 1 cycle vs. 100–200 cycles for RAM access.

• Massive speedups for scans & aggregates: SUM, COUNT, and predicates run across entire vectors in one instruction.

👉🏼 Read the full article HERE.

💎 Open Source Gems

Flotilla - Distributed engine - Multimodal data workloads
Flotilla is Daft’s next-generation distributed execution engine built for multimodal data workloads that involve massive PDFs, images, audio, video, embeddings, and GPU-heavy operations. Flotilla introduces a node-level, streaming, concurrent processing model powered by its Swordfish engine. Benchmarks show Flotilla achieving 2–7× faster performance than Ray Data and 4–18× faster than Spark, all while using smaller, cheaper clusters and avoiding OOM issues through bounded-memory streaming execution.

Why is this useful for data engineers?
Flotilla simplifies the challenge of building scalable multimodal pipelines that previously required complex configuration tuning, cluster resizing, and careful partition management. You can now write workflows such as PDF ingestion, image processing, video object detection, transcription, and embedding declaratively using DataFrame APIs, and Flotilla executes them efficiently across CPUs and GPUs in a distributed manner. Flotilla is faster, more reliable, and far easier to operate than traditional distributed engines, unlocking productivity gains for data engineers working on AI, LLM retrieval systems, and unstructured-data-heavy applications.

🧑‍💻 Daft on GitHub.

💡 DE Tip of the Month

Handling rapidly changing dimensions in the data warehouse:

Fast-changing dimensions should not be handled with a single SCD Type 2 approach, as highly volatile attributes can cause dimension bloat and slow performance. The best practice is to split attributes by their nature, keep slowly changing attributes in the main dimension, track the history with Type 2, and move frequently changing attributes into a new mini-dimension.

• Identify fast vs. slowly changing attributes: Attributes that seldom change should remain in the main dimension table.

• Create a junk dimension: For the fast-changing attributes, create a junk dimension table

• Create a mini-dimension table: Bridge table to link the main dimension and the junk dimension

• Update fact tables: Update the fact tables to include both the primary dimension key and the mini-dimension key

Let us know what you like the most in the newsletter. See you next time!

Cheers,

Ananda, Chozhan, Srivignesh

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.