Laura Ohrndorf

Let's Understand & Build an #AgenticRAG Pipeline in 13 Mins!

2

Andreas Scherbaum: PostgreSQL Development Conference 2024 - Review 🦸‍♂️[PostgreSQL Geeks Unite]🦸‍♀️ 🚀 Press pause on your index optimization and gather 'round! Breaking news from the PostgreSQL universe: PGCon is now PGConf.dev. Yes, you read that right. And no, you won't be catching that plane to Ottawa; get your passports ready for Vancouver, BC! 🎤 This is not a drill, my SQL wizards! New city, new vibe, new team, who absolutely nailed it this year. Imagine all your favorite extensions and modules compiled into one seamless conference experience. Shoutout to Andreas Scherbaum for the fantastic review. You had me at "PGConf.dev." Clever name, guys. Really rolls off the tongue, doesn't it? Whether you're replicating the good times or just trying to avoid deadlocks in your schedule, PGConf.dev is clearly the place to be. Pro tip: Book your flights, brush up on your CTEs, and consider this an official hint to present next year (yes, you, with the million row query and the trick up your sleeve). Let's keep those transactions committed and our spirits high. Until next time, keep your queries optimized and your connections pooled. #PostgreSQL #PGConfDev #DatabasesAreFun #TravelForTech 🔍👀 🧠 Credit: Andreas Scherbaum You can find the full article here: https://postgr.es/p/6x6

1

Peter Eisentraut: Why PostgreSQL major version upgrades are hard 🚀 **PostgreSQL Upgrades: A Tale of Trade-offs and Challenges** 🐘 Peter Eisentraut recently shed light on why PostgreSQL major version upgrades feel like a tango with complexity. Spoiler: It’s not just about incompatible formats! 😅 Turns out, while data file formats between versions are mostly compatible (thanks to pg_upgrade's magic), it's the *schema storage* that throws a wrench in the works. The system catalogs, hardcoded for efficiency, become outdated with new features. Imagine tweaking over a thousand instances of code to fix that! 🧩 Introducing conditional layouts sounds like a solution, but maintaining that? Herculean effort! Also, once upgraded, you'd still need new tools to manage the revamped catalogs. Easier transactional DDL vs. smooth upgrades: it’s a historical architecture trade-off. While the community hasn't yet demanded compatibility of every nook and cranny, like WAL formats, it’s clear: effortless upgrades require addressing these hidden intricacies. Kudos to Peter Eisentraut for delving into these geeky depths! Ready for my next round of PostgreSQL tango? 💃 Full dive into Peter’s insights: [bit.ly/PostgreSQLUpgrade] You can find the full article here: https://postgr.es/p/6IA

18

2 Kommentare

A few years back I had the opportunity to work with the Apache Arrow ecosystem. All I learned by reading pretty much most of the stuff online was that “Arrow is a language-agnostic in-memory columnar format”. I then started breaking down the jargons: ✅ Language-agnostic: for example using #ApacheArrow, Python, and R, accessing a dataset will have the exact representation in memory ✅ in-memory: With in-memory processing, we have the data directly in the system memory, thus avoiding transferring data from the disk to memory and having to wait ✅ Columnar format: keeping data associated with a column adjacent to each other, thereby limiting the amount of data that needs to be fetched by accessing the relevant columns. e.g. #parquet Arrow started as a columnar "in-memory" representation of data that every processing engine can use. Today, it is much more than that and has evolved to a full-blown software development platform for solving data problems! It is also amazing to see how the ecosystem around Arrow have grown over the years with tools such as Daft, Polars, Ray, Spark, Streamlit leveraging it underneath. If you are interested to know the origin story of Arrow & how it fits in today's data landscape, check out the blog in comments. #dataengineering #softwareengineering

104

4 Kommentare

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Kommentare

Andrew Atkinson: PostgreSQL 17: JSON_TABLE(), MERGE with RETURNING, and Updatable Views 🚀 PostgreSQL 17 is here, and it's about as flashy as your average Nordic noir detective drama. 😉 But don't be fooled—there are some gems in this new release. Andrew Atkinson's deep dive reveals the power of JSON_TABLE() for transforming JSON to table-like query results. Need to extract the author, ISBN, and publisher from a messy JSON blob? JSON_TABLE() has your back, making data retrieval as smooth as an espresso shot on a sleepy morning. Check out Andrew's runnable examples if you don't believe me. The MERGE command also got a glow-up, embracing the RETURNING clause like a long-lost friend. Why run multiple queries when you can get everything back in one neat package? It's like the Swiss Army knife of SQL operations! And let's not forget about views! They were always updatable; now they have triggers for an encore. Update a non-admin view to promote someone to admin status—no code changes needed. Pure magic. Plus, PostgreSQL 17 saves server memory like a hoarder finding extra closet space. Your database will thank you. Dig into Andrew's article for these and more. It’s worth it. 🌟 #PostgreSQL17 #JSON #SQLMagic #AndrewAtkinson You can find the full article here: https://postgr.es/p/6F4

4

𝐍𝐞𝐰 𝐀𝐈 𝐟𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐚𝐥𝐢𝐭𝐲 𝐟𝐫𝐨𝐦 𝐬𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 ❄️ No I don’t mean the new #snowflakeartic #LLM. It’s even way much cooler than that. Take a look at #classification!! Heuhhhh, classification? Yes classification!! Now in public preview in #snowflake, classification functionality that you can directly use in SQL code to train gradient boosting algorithms for binary or multi-class classification. In my opinion, companies are still not making full use of classification use cases. For example, response modeling, churn modeling, propensity to {filler}. I also think these use cases are far more present at companies than #LLM or #genai use cases. The data for these use cases are often already in data warehouses like snowflake. I.e. customer details, customer transactions, canceled subscriptions, etc… The new snowflake classification functionality allows you to train a #MLmodel directly on data in snowflake without the need for separate python environments. Moreover, the model that is trained is just a snowflake object that can be used to score / predict new data. #snowflake_is_cool See for detail and examples 👉

29

4 Kommentare

Now you can enjoy Postgres Europe 2024 too Exciting news for PostgreSQL aficionados! The PostgreSQL Conference Europe 2024 hosted in Athens has wrapped up, and now you can catch the action with sixty recorded talks. If your binge-watching isn't complete without some Postgres content, dive in! For a solid start, check out Claire Giordano's session on What’s in a Postgres major release and Lætitia Avrot’s take on Untangling the Web of PostgreSQL Permissions. These talks were true highlights! In the world of updates, pgvector has just released version 0.8.0. It brings enhanced performance and introduces iterative index scans. If indexing is your jam, this one's for you. For those stepping into the Postgres arena, Redgate's learning resources hub is a treasure trove for getting up to speed. Lastly, a gentle reminder: no one is above writing analytical SQL queries. Start small, build up, and avoid chaos. For the geeky details, visit the provided links. So grab your popcorn and let’s geek out on Postgres together! Source: https://lnkd.in/eeNFMkCZ

2

A total weird thing. Spent hours debugging a query in #Postgres today. The issue? Searching for a word with a German #umlaut ("ü") couldn't find the articles anymore, at least on my machine, despite them existing. I have never faced such issue any more for a long time (although Umlauts caused much trouble very often in older times). Turns out, two 'ü's can look identical but differ in their underlying #Unicode. Ironically, #MacOS underlines the correct one as 'wrong' and types the incorrect one from a German keyboard. Our database (#Postgres) had the correct 'ü', imported from news agencies, but searching with my #Macbook and a german keyboard typed the wrong one. Now I'm going to implement Unicode #normalization in our database for a better search. Finally I understand why airlines and banks still have so many issues with my last name, even in 2024. #UnicodeProblems #DeveloperLife #utf8

12

10 Kommentare

There's a common myth in the cloud consulting business landscape. In the quest to offer end-to-end solutions, many companies attempt to juggle more balls than they can. They often forget they are not circus performers - if you think you are, consider this a friendly reminder that you’re (probably) not. Cloud Consulting boutiques love to brag about their in-house capabilities to manage the entire customer journey from 0-100. On paper, this sounds amazing - and many customers are falling for these flattery terms: “We are the only cloud partner you need” “Comprehensive control and seamless integration” “All your needs handled under one roof” I totally disagree with that approach. I know my data stuff well but e.g. have no idea how to set up a proper landing zone like Manuel, and no idea how to make CI/CD a joy like Johannes. And now, I also know the cost guru of the cloud - if a shiver runs down your spine every time you look at your AWS bill, you know what to do ➡ Cristian Cristian Cristian So, instead of trying to be a jack of all trades: Focus on what you do best and partner for the rest. Did we forget how niche specialisation made the “Mittelstand” the backbone of our economy?

10

5 Kommentare

ATTENTION 🚨 This post is for Airflow advanced users only How to optimize the Scheduler to parse your DAGs faster 👇 Is your scheduler getting slow in parsing your DAGs? 1️⃣ How to diagnose the issue? Imagine you have 200 DAGs to parse. Set logging.dag_processor_log_target=stdout With that, you will know how long it takes to parse every single DAG. Let's say it takes ~300ms/DAG file. By default, the scheduler parses DAGs every 30s with two processes. Which means, 300ms * 200 = 60s > 30s parsing loop. You end up with: ❌ The scheduler falling behind ❌ Overlap between parsing loops ❌ Cpu resource to the roof 2️⃣ How to fix this issue? Here are some solutions: ✅ scheduler.min_file_process_interval=120 (increase the parsing interval) 📋 DAG updates will take longer to appear ✅ scheduler.parsing_processes=4 (more processes) 📋 You need more CPU Cores ✅ scheduler.standalone_dag_processor=True (DAG processor separated) ✅ avoid fetching variables outside of tasks (top-level dag code) ✅ set secrets.use_cache=True ✅ make your imports in your tasks, especially if one task needs an import That's it. Now you know how to make your Scheduler fast again 😎 Anything I missed? Enjoy ❤️ P.S: Like and share to help your teammates #airflow #apacheairflow #dataengineering #dataengineer

327

18 Kommentare

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Kommentare

𝐓𝐡𝐞 𝐑𝐀𝐆 𝐃𝐢𝐥𝐞𝐦𝐦𝐚: 𝐆𝐫𝐞𝐚𝐭 𝐏𝐨𝐰𝐞𝐫, 𝐁𝐮𝐭 𝐚𝐭 𝐖𝐡𝐚𝐭 𝐂𝐨𝐬𝐭? 🤔 Retrieval-Augmented Generation (RAG) has unlocked incredible possibilities for language models, but it's not without its challenges. We're talking retrieval latency, errors, and complex system architectures. These issues can significantly impact performance and scalability. So, what's the solution? Enter 𝐂𝐚𝐜𝐡𝐞-𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 (𝐂𝐀𝐆)! 💡 CAG offers a refreshing approach by leveraging the power of extended context windows. We preload relevant resources into the model's context and cache runtime parameters, eliminating the need for real-time retrieval. The benefits are clear: 1. 𝐅𝐚𝐬𝐭𝐞𝐫 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞: No more retrieval bottlenecks slowing you down. 2. 𝐌𝐨𝐫𝐞 𝐀𝐜𝐜𝐮𝐫𝐚𝐭𝐞 𝐑𝐞𝐬𝐮𝐥𝐭𝐬: Minimize retrieval errors and maintain context relevance. 3. 𝐒𝐢𝐦𝐩𝐥𝐞𝐫 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞: A streamlined, efficient design that's easier to manage. Of course, CAG has its limitations, particularly with massive datasets and extremely long contexts. But for many use cases, it offers a compelling alternative to RAG. Read more about it here :- https://lnkd.in/dhGbbu-f What are your thoughts on CAG vs. RAG? Share your insights in the comments! #AI #LLM #RAG #CAG #MachineLearning #Innovation #DeepLearning #NLP #ArtificialIntelligence

8

Tomas Vondra: The state of the Postgres community 🚀 Back from Swiss PGDay 2024! About a month ago, @TomasVondra delivered a keynote on the state of the Postgres community with some eye-opening charts. 🎤 If you missed it, don't worry. I’m here to share the highlights. Spoiler alert: we’re doing great, but we’re not perfect. Tomas pointed out what's working, what's not, and what will most likely keep us up at night. Think of it as a Postgres report card. 📝 Some of the trends were intriguing – even surprising! For instance, some parts of the community are thriving like never before. Kudos to all the contributors and tireless developers. You rock! 💪 But, it’s not all rainbows and butterflies. We’ve got challenges ahead that need more TLC. So, gear up. This is a marathon, not a sprint. Ready for a deep dive? Here’s the full talk: [Swiss PGDay 2024](https://www.pgday.ch/2024/). Big shoutout to Tomas Vondra for his insights and stellar presentation. 👏 #PostgreSQL #DatabaseGeek #SwissPGDay2024 #CommunityInsights #TechTalk You can find the full article here: https://postgr.es/p/6Bf

2

Breaking down Apache XTable's Architecture. Apache XTable (Incubating) is an omni-directional translation layer on top of open table formats such as Apache Hudi, Apache Iceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Blog in comments for a detailed read. #dataengineering #softwareengineering

73

3 Kommentare

Apache XTable's Architecture. XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, Apache Iceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure 🌟 Check out the Github repo: https://lnkd.in/geVGqRvE 📙 Get Started: https://lnkd.in/d3wsE_ux #dataengineering #softwareengineering

27

1 Kommentar