Vincent Claes

🔥 Struggling to monitor thousands of Spark applications? It’s a common challenge! The diversity of Spark users often leads to inefficiencies, soaring costs, and endless frustration. In our latest blog “𝐌𝐨𝐧𝐢𝐭𝐨𝐫𝐢𝐧𝐠 𝐭𝐡𝐨𝐮𝐬𝐚𝐧𝐝𝐬 𝐨𝐟 𝐒𝐩𝐚𝐫𝐤 𝐚𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 𝐰𝐢𝐭𝐡𝐨𝐮𝐭 𝐥𝐨𝐬𝐢𝐧𝐠 𝐲𝐨𝐮𝐫 𝐜𝐨𝐨𝐥” Lead Data Engineer Niels Claeys shares expertise to help you take control. Here's what you’ll learn: 🚀 Tools to analyze Spark applications (and when to use them). 🔧 How Dataflint streamlines issue detection. 💸 Pro tips to save resources while scaling Spark efficiently. Don’t let Spark inefficiencies burn your budget. Read the blog now and take charge of your Spark landscape! 👉 Read the full article here: https://hubs.li/Q02_n6Hd0

15

What's with all the smiling faces at our latest sprint review? ⁣ ⁣ Well, this sprint, we optimized the Spark composer. Now, the code that Konekti generates runs faster and more efficient, directly on your own tech stack. ⁣  ⁣ And we also discovered that Niek, our Spark code maestro, has the same stylish curls as Mozart. Coincidence? 👀👱‍♀️⁣  ⁣ 𝐖𝐡𝐚𝐭 𝐚𝐫𝐞 𝐲𝐨𝐮 𝐛𝐢𝐠𝐠𝐞𝐬𝐭 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞𝐬 𝐰𝐢𝐭𝐡 𝐡𝐚𝐧𝐝𝐥𝐢𝐧𝐠 𝐥𝐚𝐫𝐠𝐞 𝐝𝐚𝐭𝐚 𝐯𝐨𝐥𝐮𝐦𝐞𝐬 𝐢𝐧 𝐩𝐫𝐨𝐜𝐞𝐬𝐬 𝐦𝐢𝐧𝐢𝐧𝐠? #Spark #datatransformation #processmining with Bram van Bebber, Durairaja V., Ryan Houben, Maarten van Asseldonk, Niek van Son, Lotte Vugs

36

𝐃𝐨 𝐲𝐨𝐮 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐮𝐧𝐥𝐨𝐜𝐤 𝐭𝐡𝐞 𝐟𝐮𝐥𝐥 𝐩𝐨𝐰𝐞𝐫 𝐨𝐟 Databricks 𝐟𝐨𝐫 𝐲𝐨𝐮𝐫 𝐝𝐚𝐭𝐚? Then check out our new “Databricks in a Week” workshop! In this 5 day workshop we will (optionally together with you): ✅ rapidly prototype and validate your data-driven product or solution ✅ showcase the Databricks technology hands-on with your data in your Databricks account ✅ demonstrate feasibility and value to your stakeholders Sounds good? Then head on over to our website and get in touch with our team! 👉 https://lnkd.in/dxHyFgfK What is your biggest challenge with Databricks at the moment? Let us know in the comments and let’s discuss! #databricks #data #DataIntelligence #DatabricksInAWeek

36

1 Comment

A few years back I had the opportunity to work with the Apache Arrow ecosystem. All I learned by reading pretty much most of the stuff online was that “Arrow is a language-agnostic in-memory columnar format”. I then started breaking down the jargons: ✅ Language-agnostic: for example using #ApacheArrow, Python, and R, accessing a dataset will have the exact representation in memory ✅ in-memory: With in-memory processing, we have the data directly in the system memory, thus avoiding transferring data from the disk to memory and having to wait ✅ Columnar format: keeping data associated with a column adjacent to each other, thereby limiting the amount of data that needs to be fetched by accessing the relevant columns. e.g. #parquet Arrow started as a columnar "in-memory" representation of data that every processing engine can use. Today, it is much more than that and has evolved to a full-blown software development platform for solving data problems! It is also amazing to see how the ecosystem around Arrow have grown over the years with tools such as Daft, Polars, Ray, Spark, Streamlit leveraging it underneath. If you are interested to know the origin story of Arrow & how it fits in today's data landscape, check out the blog in comments. #dataengineering #softwareengineering

104

4 Comments

Your contribution for the evening 🌖: sodadata/soda-core🥤 Co-founded by Maarten Masschelein and Tom Baeyens, Soda is a data quality platform built for the modern data stack. Here’s why you should contribute to this repo: 1️⃣ With a growing repo now at ~1,700 stars, contributing to the soda-core repo allows you to refine your skills in data quality management, gain hands-on experience with the Soda Checks Language (SodaCL) and learn how to implement practical data quality tests across various data sources. You also have the opportunity to connect with other data professionals, expanding your network and opening doors to working opportunities. 2️⃣ Maintainers ensure a response rate of 100%, with typically a median PR merge time of around 3 hours. This guarantees that your contributions receive the attention they deserve. 🩶 3️⃣ The repo is more accessible. At Quira, we rank Developers by their DevRank (more on this here: https://lnkd.in/eNtrs97F). We found that the median developer working on the repo belongs to the 90th percentile of GitHub developers. For reference, many of the open-source companies we reported on have a median contributor level upwards of the 95th percentile. This means that you have a real opportunity to make an impact. ⚡️ If you want to help improve a pivotal tool for many organisations in ensuring data integrity, you can check more of their open-source stats on Quira (https://lnkd.in/egxYTC4N) or access the GitHub page: https://lnkd.in/e2r3iazu 📖

6

Have you ever wondered what it means to be a Data Engineer and Cloud Architect? 🤔 Our Coder, Marc Vea, tells you about his journey to become a Data Engineer: his studies, experience and what he enjoys the most on the day to day. 👏 Do you want to know more about our roles and positions? Let us know in the comments! 💬👇 #SEATCODE #Coders #Careers #Jobs #Engineer

38

The 2024 State of Postgres is now open for voting until the end of September! Each year, Timescale colleagues publish an insightful report on the state of our beloved database. Curious about how #Postgres is being used? Take part in the survey—it only takes 15 minutes! #StateofPostgres #survey #opensource #database

7

Today's Tip - API for GraphQL in Microsoft Fabric Learn how to share data currently stored in a Microsoft Fabric warehouse using a GraphQL API along with the steps to set this up. Author: Koen Verbeeck >>> https://lnkd.in/e2gQ-zk6 #sql #sqlserver #microsoftsqlserver #mssql #mssqlserver #microsoftfabric #graphqlapi

2

New Tool Alert 🚨 Introducing ZIO Apache Parquet 🔥 Yesterday, Mykhailo Nedokushev from Ziverge posted in the Scala Reddit channel about his ZIO-powered wrapper for Apache Parquet's Java implementation 🎁 The tool leverages ZIO Schema to automatically derive codecs and provide type-safe filter predicates. Operate your parquet files easily using a top-notch ZIO-powered ecosystem without running a Spark cluster ⚡️ Why is this useful? 👉 No Spark required - you don't need to run a Spark cluster to read/write Parquet files. 👉 ZIO native - utilizes various ZIO features to offer a FP-oriented way of working with the Parquet API. 👉 ZIO Schema - the backbone that powers all the cool features of this library such as type-safe filter predicates and codecs derivation. Want to find out more? Head over to the GitHub: https://lnkd.in/dSmFZ-AJ

37

3 Comments

next post on my series about #AngularQuery Below is a first simple example on how to use injectQuery to fetch data. The best? You get respective states like loading for free! No more cumbersome #RxJs tap()ing needed #Angular

61

Now you can enjoy Postgres Europe 2024 too Exciting news for PostgreSQL aficionados! The PostgreSQL Conference Europe 2024 hosted in Athens has wrapped up, and now you can catch the action with sixty recorded talks. If your binge-watching isn't complete without some Postgres content, dive in! For a solid start, check out Claire Giordano's session on What’s in a Postgres major release and Lætitia Avrot’s take on Untangling the Web of PostgreSQL Permissions. These talks were true highlights! In the world of updates, pgvector has just released version 0.8.0. It brings enhanced performance and introduces iterative index scans. If indexing is your jam, this one's for you. For those stepping into the Postgres arena, Redgate's learning resources hub is a treasure trove for getting up to speed. Lastly, a gentle reminder: no one is above writing analytical SQL queries. Start small, build up, and avoid chaos. For the geeky details, visit the provided links. So grab your popcorn and let’s geek out on Postgres together! Source: https://lnkd.in/eeNFMkCZ

2

Kafka is about to get a lot cheaper 🔥 Just one week after my calculator post, where I shared the idea of a Produce to Same-AZ Leader KIP - Ivan Yurchenko has published KIP-1123: Rack-aware partitioning for the Kafka Producer. 🎅 What a christmas gift! 🎁 It basically adds two configs to the producer client which will allow it to preferentially produce to a partition in the same AZ when there is no need to pick a specific partition. Producers pick a specific partition if you specifically pick one through the API, or if your record uses a key (so that the ordering guarantee per key is kept) I think this is an amazing development for the community which will bring Kafka one step closer to matching the cost of the no-replication solutions like WarpStream. The yearly cost of a 100 MiB/s Kafka workload on AWS with 3x consumer fanout and Tiered Storage enabled is roughly $348,292 a year. 38% of that cost is consume. It can be removed completely with KIP-392 to have consumers fetch from the local replica. 👌 When we remove it, we get to $215,942 a year. 🤯 20% of that cost is produce. If none of your consumers depend on ordering, you can remove ALL of that with KIP-1123. It would bring the cost down to $171,826! Depending on the compression ratio, that's even cheaper than WarpStream. So you will have one more knob to bring down your Kafka pricing to be ultra-competitive! Go Kafka! 💪 Thanks Ivan Yurchenko 🙇‍♂️

182

10 Comments

How to create data for your built-in database! 👇 In this video Tristan Van Poucke explains how you can easily create data within your data formats. 😎 ** NoCode-X simplifies data management with flexible data formats and intuitive creation tools. Learn how to: ➡ Define Data Formats: Create custom data formats like 'Soccer Team' with attributes such as team name and number of players. ➡ Add Properties and Constraints: Specify data types, set character limits, and add descriptions to ensure data accuracy. ➡ Generate Data: Populate your database with new data entries directly from the back office or via API, pages, or files. Watch the full video to see how easy it is to manage your data with NoCode-X! 🚀 #nocode #technology #dataformats

5

1 Comment

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Comments

🥇 𝐀𝐜𝐡𝐢𝐞𝐯𝐞 𝐮𝐧𝐢𝐟𝐢𝐞𝐝 𝐝𝐚𝐭𝐚 𝐚𝐜𝐜𝐞𝐬𝐬 𝐚𝐧𝐝 𝐞𝐧𝐡𝐚𝐧𝐜𝐞𝐝 𝐝𝐞𝐜𝐢𝐬𝐢𝐨𝐧-𝐦𝐚𝐤𝐢𝐧𝐠 🥇 Our new article is available here 👇🏻 https://lnkd.in/e7b_368V #Data #Database #VirtualDatabase #BlackTiger

7

❓ Question: Will test-driven data engineering become as standard as test-driven development? 💬 Answer: Where we're heading is what I think is exactly the same as with software engineering... Test-driven development was a radical new thing back then. But then it turns out, you can much more reliably release software. And this is exactly the same here. If you don't inject data testing, data observability throughout your data stack, then how are you going to trust the data that you put into your machine learning model? This is something that people are realizing, but we're still figuring out the best practices, the dos, the don'ts. We've come a long way, but there's still a way to go before this is as common and as normal as in the test-driven development software engineering space. 👆 There's this and more in the Open Source Data podcast hosted by Sam Ramji, when he sat down with Tom to discuss the evolution of workflow engines, data contracts, and more including: - What open source data means to Tom - Why data quality needs a language approach - Tom’s advice for engineers who want to leverage data observability tools 🎧 Sit back, listen, and enjoy: https://lnkd.in/ev_e4RGD #opensource #dataengineering #softwareengineering #dataquality #testdrivendevelopment #dataobservability #datacontracts

11