Sakshi Batra

Databricks has open-sourced Dicer. In production, Unity Catalog kept cache hit rates above 90% after adopting it. A good example of how much impact state locality and sharding strategy can have in real systems, cutting down DB calls and improving performance. Worth reading the full article for the details: https://lnkd.in/davviaPe

1

No Unified Namespace. No scalable production. Everyone talks about AI. About MES. About analytics. But most factories are still running point-to-point integrations and fragile data pipelines. That doesn’t scale. If your data architecture can’t keep up, neither will your production. Stefan Hermann, VP Product at Cybus, explains why the Unified Namespace is the real driver of scalability and what it actually takes to turn it into industrial reality. Curious what makes a UNS truly scalable? Read the full article 👇 https://lnkd.in/dtbwiwAZ

16

5 checks I never skip in a production-grade data pipeline: 1️⃣ Null checks Never assume your source data is clean. Ever. 2️⃣ Row count validation If yesterday had 120,000 records and today has 36,000 — something broke. 3️⃣ Idempotency Rerun should not cause duplicates or drift. Your jobs must be safe to retry. 4️⃣ Simple logging Start time, end time, rows in, rows out. No fluff — just what matters when things go wrong. 5️⃣ Early failure handling Fail fast if inputs are missing or schema has changed. Don’t wait 45 minutes to crash at the last step. 🧠 Bonus: I build every pipeline as if I’ll be the one debugging it at 3AM six months from now. ✨ Fancy tools won’t save you if your fundamentals are weak. Discipline > Stack. #DataPipelines #SQL #Airflow #DBT #Snowflake #ETL #DataEngineering #ProductionReady #CleanCode #ModernDataStack #BackendEngineering #FailFast #ScalableSystems

35

9 commentaren

Quantization in Large Language Models (LLMs) As LLMs scale, intelligence is no longer the bottleneck — memory, latency, and cost are. Quantization turns out to be one of the most powerful (and misunderstood) techniques enabling real-world deployment of large models. In this blog, I break down: • Why LLMs are surprisingly quantization-friendly • INT8 vs INT4 vs NF4 — what actually works in practice • How modern architectures absorb low-bit noise • Where quantization fails and how to design around it Quantization is no longer an optimization step — it’s foundational infrastructure for production LLM systems. Link: https://lnkd.in/gqGje75G #LLMs #AIEngineering #Quantization #MachineLearning #GenAI #MLOps

9

2 commentaren

At the beginning of my career, I thought becoming a senior meant collecting as many technical skills as possible--knowing more frameworks, tools, etc. I assumed the title came from sheer volume of knowledge. Now, I realise that being a senior is mostly about mindset. It’s about how problems are approached, how knowledge is shared with others, how processes are communicated with stakeholders, and how ownership is taken--not just of one's own work, but of the team’s success. In the age of AI, skill gaps can be bridged quite easily--but no AI can teach you to have the proper mindset. Not yet, anyway.

26

Real-time isn’t always right. Batch isn’t always enough. For decades, data teams have had to choose between speed and stability: between fragile, real-time pipelines and delayed, batch processes. But every system runs on its own rhythm. Every decision has its own urgency. That’s why we’re introducing a new way to think about data movement: Right-Time Data. Right-time data means control, not compromise. It’s about moving data exactly when your business needs it — sub-second, near real-time, or scheduled — without trading off reliability or cost. At Estuary, we believe latency should be a dial you control, not a limitation you accept. Our Right-Time Data Platform unifies batch and streaming, giving enterprises dependable, exactly-once data movement at any cadence — all in one system. Read the full Right-Time Data Manifesto to see why this shift matters for the next generation of AI, analytics, and operations: 👉 https://lnkd.in/d4UZFdcE

13

I don’t think companies need analysts who “drive decisions,” I think they need analysts who stop the stupid ones. Not always. Not in every situation. But often. Most of the messes I’ve seen didn’t come from bad data, they came from overconfident timelines, half-baked assumptions and a room full of smart people nodding because they thought everyone had done their homework. Analysts see the leaps in logic and the numbers that don't quite add up, and we know no one really wants to discuss them... It's on us to ask the questions everyone is avoiding. When you bring a well-thought-out argument based on data and speak up, you'd be surprised how many people appreciate you giving voice to the things they were hesitant to say. So be bold, but bring data.

372

48 commentaren

Here’s a case study where a Fraud Model Flags the Wrong Data Scientist 🥷💰 I work in logistics. I build systems. I understand fraud. And yet, I just got classified as a “potential fraud user.” Here’s what happened (True Story): I ordered a keyboard stand on Amazon with next-day delivery (needed it as a gift for a friend). The courier reached the address. The friends relative received the package, saw my name on it and said, “No one by that name lives here.” Fair enough, wrong signal at the door. But then things got interesting. The parcel was marked Delivered. It was never delivered. It was sent back to the delivery partner. Classic RTO 3PL manipulation pattern. Now here’s the twist: This had happened once before (different context, I was out of town and the parcel was again marked delivered but wasn’t). Two operational failures = one “high risk customer.” Congratulations. I’ve now been algorithmically promoted to Fraud Tier by Amazon’s fraud detection model xD What fascinates me isn’t the loss. It’s the logic. When: • Delivery confirmation overrides ground truth • 3PL investigation = “we checked with the delivery partner” • Repeated operational failures accumulate against the customer You don’t just lose money. You lose signal integrity. False positives in fraud systems are dangerous. They quietly convert good users into lost users. The real cost isn’t RTO. It’s trust. For someone working in logistics data science, this is a reminder: Fraud prevention without fraud accountability is just automation of blame. Safe to say I’ll not be using Amazon for my deliveries anytime soon, actually considering of moving everyday purchases to Meesho. What are your thoughts? PS: For the Engineers and Data Scientists working on the problem at Amazon/Ecom-space (or if you’re just curious and want to know more), here’s an article on how you can mitigate the risk of such problems: https://lnkd.in/gPu2CWfK Now Amazon, please give me my money back 🙏 😭

13