Prathy Kamasani

People, if your data pipelines are as precarious as this then spend some time tackling those, they will come back and haunt you. #dataengineer #datawarehouse #datapipeline #dataleaders

17

5 Comments

Do you currently hold this certification? 🎓 ✅ Are you proficient in using Databricks for data engineering and machine learning workflows? ✅ Can you analyse large datasets using Apache Spark? ✅ Are you familiar with with tools like pandas and Spark DataFrames? ✅ Have you built and trained machine learning models? If yes, and you would be interested in representing Databricks in a contracting capacity, I'd love to hear from you. Drop me a DM or email for more info - [email protected] #databricks #machinelearning #datascience #contract

10

🚀 Elevate your data skills with SQL! 🚀 Already in the data profession but want to deepen your SQL knowledge? Understanding how to write SQL queries can be a game-changer for your career, no matter your role. On September 12th at Midday (UK time), we’ll cover the fundamentals of SQL, from SELECT statements to JOINs. This session is perfect for those looking to enhance their database management skills. 👉 https://lnkd.in/eG2FZDnA Take the next step in your data career. Don't miss out! #DataEngineering #SQL #ProfessionalDevelopment #CareerGrowth

11

𝗠𝘆 𝗣𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝗻 𝗬𝗼𝘂𝗧𝘂𝗯𝗲 That's right, Power BI & Fabric Manchester UG have given me a platform, so I'm going to shout about it. If you're interested in: - Incremental Refresh - The XMLA Endpoint - TMSL Scripts - Power BI's Advanced Partitioning capabilities Or, if you just want to find out how me and my thick Bristolian accent got on the first time I spoke in front of a crowd 𝗜 𝘀𝘂𝗴𝗴𝗲𝘀𝘁 𝘆𝗼𝘂 𝗳𝗼𝗹𝗹𝗼𝘄 𝘁𝗵𝗲 𝗹𝗶𝗻𝗸 𝗯𝗲𝗹𝗼𝘄. https://lnkd.in/eRTcxpkc * 𝘉𝘰𝘯𝘶𝘴 𝘱𝘰𝘪𝘯𝘵𝘴 𝘢𝘸𝘢𝘳𝘥𝘦𝘥 𝘵𝘰 𝘢𝘯𝘺𝘰𝘯𝘦 𝘵𝘩𝘢𝘵 𝘤𝘰𝘶𝘯𝘵𝘴 𝘵𝘩𝘦 𝘯𝘶𝘮𝘣𝘦𝘳 𝘰𝘧 𝘵𝘪𝘮𝘦𝘴 𝘐 𝘴𝘢𝘺 𝘱𝘢𝘳𝘵𝘪𝘵𝘪𝘰𝘯 𝘰𝘳 𝘱𝘢𝘳𝘵𝘪𝘵𝘪𝘰𝘯𝘪𝘯𝘨. ** 𝘛𝘩𝘢𝘵'𝘴 𝘯𝘰𝘵 𝘢 𝘮𝘰𝘶𝘴𝘦 𝘤𝘶𝘳𝘴𝘰𝘳 𝘮𝘰𝘷𝘪𝘯𝘨 𝘢𝘳𝘰𝘶𝘯𝘥 𝘵𝘩𝘦 𝘴𝘤𝘳𝘦𝘦𝘯, 𝘪𝘵'𝘴 𝘢 𝘥𝘪𝘨𝘪𝘵𝘢𝘭 𝘧𝘭𝘺.

17

9 Comments

An important point from this piece I'd like to call out here: "𝘛𝘩𝘦 𝘣𝘪𝘨𝘨𝘦𝘴𝘵 𝘭𝘪𝘮𝘪𝘵𝘢𝘵𝘪𝘰𝘯 𝘰𝘯 𝘦𝘹𝘵𝘦𝘳𝘯𝘢𝘭 𝘥𝘢𝘵𝘢 𝘮𝘰𝘯𝘦𝘵𝘪𝘴𝘢𝘵𝘪𝘰𝘯 𝘪𝘴 𝘨𝘦𝘵𝘵𝘪𝘯𝘨 𝘳𝘦𝘴𝘵𝘳𝘪𝘤𝘵𝘦𝘥 𝘵𝘰 𝘚𝘯𝘰𝘸𝘧𝘭𝘢𝘬𝘦 𝘢𝘤𝘤𝘰𝘶𝘯𝘵𝘴 𝘢𝘯𝘥 𝘵𝘩𝘦 𝘚𝘯𝘰𝘸𝘧𝘭𝘢𝘬𝘦 𝘦𝘤𝘰𝘴𝘺𝘴𝘵𝘦𝘮 𝘰𝘯 𝘣𝘰𝘵𝘩 𝘵𝘩𝘦 𝘱𝘳𝘰𝘷𝘪𝘥𝘦𝘳’𝘴 𝘢𝘯𝘥 𝘤𝘰𝘯𝘴𝘶𝘮𝘦𝘳’𝘴 𝘱𝘢𝘳𝘵, 𝘢𝘭𝘰𝘯𝘨𝘴𝘪𝘥𝘦 𝘪𝘯𝘤𝘶𝘳𝘳𝘪𝘯𝘨 𝘩𝘦𝘢𝘷𝘺 𝘤𝘰𝘴𝘵𝘴 𝘰𝘯 𝘤𝘰𝘮𝘱𝘶𝘵𝘦 & 𝘴𝘵𝘰𝘳𝘢𝘨𝘦 𝘥𝘶𝘦 𝘵𝘰 𝘮𝘰𝘴𝘵𝘭𝘺 𝘶𝘯𝘥𝘪𝘳𝘦𝘤𝘵𝘦𝘥 𝘥𝘢𝘵𝘢 𝘦𝘧𝘧𝘰𝘳𝘵𝘴." When we as data organisations aim to build direct revenue streams out of data efforts by creating both internal and external monetisation opportunities, high cost of data access, availability, and management right at the start is a heavy downer. On a side note, really enjoyed Samir and Saurabh's chat on The Data Strategy Show and it has been one of the inspirations for storytelling around driving value with Snowflakes of the world. This is another collab with Dr. Balaji Muthusubramanian, second in the series of leveraging your existing toolset and data stack to prime for a data product ecosystem and the value it brings! In this piece, we explore how to combine true product approach & familiar stacks like Snowflake for an enhanced experience of existing tools. Data Products are not here to replace tool choices (the tactics); they are here to give a direction to data - consistently point North to the data consumer. It's important to note that data products or self-service layers are not meant to clash with the trio of people, process, and tech; instead, they are enablers of this trident 🔱 In this piece you'll find how to: ♜ Make data access easier for data consumers by eliminating tool lock-ins ♝ Increase data-sharing options and flexibility for data producers ♞ Create THE experience with product-driven discoverability & propelled usage ♚ Enable data ops teams to show a much cheaper end-of-month billing with explicit reduction in compute costs ♛ Build a cheap layer for data monetisation on top of Snowflakes of the world Access the full read here: https://lnkd.in/dXxNPhrM This is part of an ongoing series where we talk about leveraging existing stacks instead of a more aggressive rip-and-replace strategy. Keep watching Modern Data 101 for upcoming pieces on some of your other favourite/familiar stacks! #datastrategy #snowflake #dataproducts

50

3 Comments

Undisciplined use of Kafka leads to poor quality data I was at Confluent’s Data In Motion event in London yesterday and this quote by Tim Berglund in they keynote caught my attention: "Undisciplined use of Kafka leads to poor quality data." That’s 100% true - not just for Kafka but for any data movement/integration tool. So, how do we use Kafka with discipline? We: 1️⃣ Design schemas that meet our users requirements 2️⃣ Implement change management on those schemas 3️⃣ Publish the in the schema registry for serialisation and deserialisation The producers of the data can then publish data that matches the schema, and the consumer can be confident that the schema won’t change without going through an agreed change management process. These can all be implemented with data contracts.

121

15 Comments

I've read some debate around whether or not to skip this badge, but I'd definitely recommend to do the Databricks Certified Data Engineer Associate exam, it reinforces and demonstrates core product knowledge. Looking forward to self-prepping the Professional badge over the coming weeks! #databricks #dataengineer #dataarchitect #databrickscertified

23

5 Comments

With a focus on data monetisation, every CDO and CDAO is being asked to deliver value – often through the delivery of data products. But what exactly is a data product? This question has been doing the rounds on LinkedIn – clearly there is some confusion. In this session, Gary will explore the concept of data products and propose a workflow for delivering trusted data products that add value. Join DAMA Southern Africa and Gary Allemann for our Monthly User Group Discussion to learn more! RSVP link in the comments below. #dataproducts #datamanagement #datamonetisation #damasa

16

5 Comments

It was a good end of the week, wrapping up the DBT Fundamentals course. It is a fascinating piece of tech, allowing you to perform ELT, embed tests for your data models and automated documentation all within 1 tool. Kudos to the dbt Labs training team for designing and explaining the whole content in the easiest possible way. #dbt #dbtlearn #analyticsengineering

38

4 Comments

Do you want a better understanding of Databricks Lakehouse Monitoring, how to implement it, and if it is a good fit for you? Join Anna-Maria Wykes at the next Manchester Databricks User Group Thursday 15th August - 18.00 - 20.00 Sign up in the comments below. #meetup #databricks #lakehouse #monitoring

5

3 Comments

I’m probably going to let rip on the next person to lecture me on ETL vs ELT. The whole #data industry appears to think that there’s a clear distinction between them… and “The Industry” is just wrong, no matter what the vendors pay us to say. More than 90% of the systems I’ve worked on for a couple of decades across hundreds have customers have been too complex to let a single shonky ETL tool do all the heavy lifting. Most data warehouses (call it a lake, and we might properly fall out) are ELTL. One of these days, us #data folk will realise that there’s no “one true way” of doing things… if that’s the message we want to spread, then AI would have taken our jobs by now. Now that we’re all clear that people who don’t use git/dbt/airflow etc don’t give two hoots about which order we put our letters, can we just focus on delivering some value, and quickly, please? Rant over.

191

69 Comments

New Enterprise Big Data Engineer official guide is OUT!!! Don't missed it for the world if you are serious about BIG DATA! https://lnkd.in/gn2hnfHw #EnterpriseBigDataFramework #BigData #BigDataEngineer #Data #DataEngineer #EBDF #FreeGuide #Free #EnterpriseData

8

Heading to BigData London. I’ll be presenting on Day 2 at 2:40 PM on the Role of Metadata in Business, Governance, and AI! 👇 Abstract: Step into the dynamic world of data governance, business operations, and artificial intelligence (AI), where the unsung hero, metadata, takes center stage. Just like the perfect sandwich relies on clear definitions of its ingredients, this talk unveils the indispensable role of metadata in defining and organizing data. George will share captivating real-life stories and examples on how clarity in definitions and metadata not only streamlines operations but also empowers decision-makers with invaluable insights. Explore the backbone of AI advancement through essential data management tools: the Business Glossary, Data Dictionary, Data Catalog, and Machine Learning Metadata Store. Let's embark on a journey where unified interpretations pave the way for accuracy, efficiency, and success in the data-driven era. Hope to see you there! #data #datagovernance #AI

108

36 Comments

Check out this interview I did with Gary Short! Gary is a Data Science Consultant that has done some really interesting projects! In this video he discusses an innovative project he did, and also gives advice about setting up a Data Science team. Thank you Gary! It was a really enjoyable chat! Enjoy!

5

4 Comments

Tired of creeping costs paying for JSON to look like a table EVERY Query? Get the best price performance with the TPC-DS record holder #dbsql Check out the latest #databricks features that can transform your traditional analytics. #Innover can quickly get you on your next analytic value explosion with a production ready Databrick SQL lakehouse in as little as two weeks!! https://lnkd.in/gjb6Hxeg

1

📣 New Blog Alert! 📣 Looking to understand and manage costs within Databricks? Grace O’Halloran’s "How to Effectively Manage Databricks Costs" explores tools like the Cost and Usage Dashboard, Power BI templates, and system tables to help you track and optimise spend effectively. Read the full guide 👉 https://hubs.la/Q02ZNWkY0 #Databricks #BigData #DataEngineering

42

1 Comment

Yesterday the #AzureMaps team posted a blog announcing their updated Route Range API, which can create HD isochrones representing how far can be travelled by car or HGV within a given time. I was keen to see how this could be used alongside #IconMapPro in #PowerBI, and also whether I could incorporate it as part of a report load in #PowerQuery without the need to use a python notebook. I was able to create a report that represents all of the West Sussex Fire & Rescue Service stations within West Sussex County Council and show how far can be travelled within 10 minutes from each. I added labels to quickly identify them, and a Power BI slicer to be able to zoom in to a specific station. I've published a blog post to show step by step how this was created, and shared a PowerQuery function you can re-use to use the API in your own reports. You can preview the report, or download to examine it fully. Check the comments for the link.

129

15 Comments