Ganeshprabu kathiresan

CLASSES in SCALA are pretty much like CLASSES in C++ & Other OBJECT ORIENTED PROGRAMMING LANGUAGE. #SCALA #BIGDATA #CPP #C++ Karthik K. object CLASS { class Point{ private var _x = 0 private var _y = 0 private val bound = 100 def x: Int ={ return _x} 👈 //GETTER for private var _x def y: Int ={ return _y} 👈 //GETTER for private var _y def coord( x:Int, y:Int)={ 👈 //SETTER for private vars _x & _y this._x = x 👆👇 /"This" keyword store address of the object calling the function this._y = y println(this._x,this._y) } } def main(args: Array[String]):Unit = { val a = new Point 👈 //Object creation of Point in Heap Memory a.coord(20,30) print(a) } } OUTPUT:

4

3 Comments

Today's Topic - Why is Redis so Fast? Redis consistently ranks among the fastest databases available. Several factors contribute to its remarkable speed and performance - 1. 𝐈𝐧-𝐌𝐞𝐦𝐨𝐫𝐲 𝐃𝐚𝐭𝐚 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 Lightning-Fast Access - Redis primarily operates in-memory. - This means data resides in RAM, eliminating the latency of disk reads and writes. - The image clearly demonstrates the significant speed difference between RAM and storage mediums like SSDs or HDDs. Data Persistence Options - While primarily in-memory, Redis offers options for persistence (RDB snapshots, AOF logging) to safeguard data durability. 2. 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞𝐝 𝐃𝐚𝐭𝐚 𝐒𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐬 Efficient Implementation - Redis data structures (strings, lists, hashes, sets, sorted sets) are implemented with highly optimized algorithms and data structures, tailored for specific use cases. - The image showcases how each Redis structure maps to a well-chosen underlying implementation (e.g., ziplists for small lists, skip lists for sorted sets). Minimal Overhead - These specialized structures minimize memory overhead and computational complexity, leading to faster operations. 3. 𝐒𝐢𝐧𝐠𝐥𝐞-𝐓𝐡𝐫𝐞𝐚𝐝𝐞𝐝 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 No Context Switching - Redis uses a single thread for command processing, eliminating the overhead of context switching between multiple threads. Event Loop - The event loop model allows Redis to handle multiple client connections concurrently without the need for multithreading, avoiding the contention and synchronization issues that can arise. 4. 𝐈/𝐎 𝐌𝐮𝐥𝐭𝐢𝐩𝐥𝐞𝐱𝐢𝐧𝐠 (𝐑𝐞𝐝𝐢𝐬 6.0 𝐎𝐧𝐰𝐚𝐫𝐝𝐬) Enhanced Scalability - I/O multiplexing (using the io_uring interface on Linux) further improves Redis' ability to handle many concurrent connections efficiently. - The image depicts how this mechanism enables a single thread to manage multiple client connections simultaneously. Networking Performance - Redis leverages various networking optimizations (e.g., TCP backlog, SO_REUSEPORT) to enhance network throughput. Minimalistic Design - Redis focuses on core features and avoids unnecessary bloat, resulting in a lean and efficient codebase. Connect/Follow - Pallavi Ghanshani

4

Spark Streaming!! Interviewer: "Today, let's explore Spark Streaming, an integral component of Apache Spark for processing real-time streaming data. Are you familiar with Spark Streaming?" Candidate: "Yes, I've worked with Spark Streaming. It provides a scalable and fault-tolerant framework for processing real-time data streams." Interviewer: "Excellent! Spark Streaming enables developers to build robust and scalable stream processing applications. Could you elaborate on the key features and benefits of Spark Streaming?" Candidate: "Certainly! Spark Streaming offers several advantages. Firstly, it provides high throughput and low latency processing of real-time data streams. Secondly, it integrates seamlessly with the core Spark API, enabling users to leverage Spark's rich ecosystem of libraries and tools. Additionally, Spark Streaming offers fault tolerance through lineage and checkpointing, ensuring that data processing is resilient to failures. Lastly, it supports various data sources and sinks, allowing users to ingest data from sources like Kafka, Flume, or HDFS, and output processed data to databases, file systems, or dashboards." Interviewer: "Well summarized! Now, let's delve into the architecture of Spark Streaming. How does Spark Streaming process and analyze real-time data streams?" Candidate: "Spark Streaming follows a micro-batch processing model, where real-time data streams are divided into small batches of data, which are then processed using Spark's batch processing engine. The architecture consists of a Receiver that ingests data from input sources, a StreamingContext that manages the execution of streaming jobs, and a Discretized Stream (DStream) abstraction that represents a continuous stream of data. Each batch of data is processed using RDDs (Resilient Distributed Datasets), enabling parallel and fault-tolerant stream processing." Interviewer: "Spot on! Spark Streaming's micro-batch processing model provides fault tolerance and scalability, making it suitable for handling large-scale streaming data. Now, let's discuss the use cases of Spark Streaming. Can you provide examples of scenarios where Spark Streaming excels?" Candidate: "Certainly! Spark Streaming is well-suited for various real-time analytics and monitoring applications. For example, in e-commerce, it can be used to analyze user activity and recommend products in real-time. In finance, it can process streaming market data to detect anomalies or fraud in transactions. Additionally, in telecommunications, it can analyze network logs to identify network issues or optimize network performance in real-time." Interviewer: "Impressive examples! Spark Streaming's versatility makes it applicable across different industries for real-time data processing and analysis. Thank you for sharing your insights on Spark Streaming!" Candidate: "You're welcome! #bigdata #spark #sparkstreaming #pyspark #python

7

**𝐋𝐞𝐯𝐞𝐥 𝐔𝐩 𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐒𝐤𝐢𝐥𝐥𝐬 𝐰𝐢𝐭𝐡 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠!** A huge shoutout to my friends, Vishal Kaushal , a Snowflake Squad member, and Sachin Mittal, a 4x Snowflake Data Superhero, who are starting Snowflake Training: Zero to Hero course starting May 18th! . **Why Snowflake? ** Snowflake is more than just a data warehouse. It's a powerful, cloud-native data platform that empowers data engineers to build next-generation data solutions. Here's why learning Snowflake can be a game-changer for your career. By mastering Snowflake, you'll equip yourself with the skills to design and implement robust, scalable data solutions that are in high demand across industries. This positions you as a valuable asset in the ever-evolving world of data engineering. This comprehensive training, will equip you with the knowledge and skills to become a Snowflake master. Here's what you'll gain: Solid Foundation: Grasp the core concepts and functionalities of Snowflake. Hands-on Learning: Get practical experience through real-world exercises and projects. Unlimited doubt support Real time Projects : 2 End to End Real time projects , 1 using Matillion as well. Ready to take your data engineering skills to the next level? Don't miss out on this amazing opportunity! Sign up for the Snowflake Training: Zero to Hero course starting May 18th. 👉🏻 Enroll Here - https://lnkd.in/gcfxzNx4 📅 Live classes starts from 18th May 2024 Discount Coupon : LIVE20 PS : This is not a paid post. #Snowflake #DataEngineering #CloudDataWarehouse #Training #Upskilling #DataSuperhero

121

4 Comments

Snowflake Optimization - Summary of the week 1. Remove failing workloads Failing workloads are just using credits and not performing any real work. To identify them, use this: https://lnkd.in/gP8bgV-a 2. Identify unused storage Identify and remove unused storage. Here's a blog to identify: https://lnkd.in/gMTrHxmM 3. Set query timeouts Set appropriate query timeouts based on the size of the warehouse and the business use case. 4. Split or merge files before ingestion The appropriate file size for ingestion into Snowflake is 100-250MB. Split or merge files before ingesting to ensure the file size is correct. 5. Use deferred merge If the frequency of ingestion is high, use the concept of deferred merge. Here's a blog: https://lnkd.in/gk4jgz2g

20

3 Comments

Reposting for Visibility: I recently shared details about our next meet-up in the LinkedIn group. In case you missed it there, I'm sharing it again here! #AzureSQL #AzSQLBangUG

7

How the Sort Merge Join Happens in the Apache Spark , Know the SMJ internals Sort-Merge Join (SMJ): In Apache Spark, Sort-Merge Join (SMJ) is the default join strategy for combining datasets based on a shared join key.  It excels at handling large datasets that might not entirely reside in memory on a single worker node. its execution phases: 1. Shuffle Phase: Spark partitions both input DataFrames according to the join key. This ensures records with matching join keys end up on the same worker node. Each partition is shuffled and sent across the network to the appropriate worker nodes based on the join key's hash value. 2. Sort Phase: Within each worker node, the shuffled partitions are sorted independently and in parallel using a stable sorting algorithm (like Timsort) to maintain the order of join keys. 3. Merge Phase: Worker nodes iterate through their sorted partitions simultaneously, performing a merge join operation. They compare the join keys of each DataFrame, and when a match is found, they combine the corresponding rows into a new joined row. This process continues until all matching join keys from both DataFrames have been processed. Example: Imagine you have two DataFrames: customers: (customer_id, name, city) orders: (order_id, customer_id, amount) You want to join them on the customer_id to get customer details along with their orders. Execution Steps: Shuffle: Both customers and orders are partitioned and shuffled based on customer_id. Sort: On each worker node, the shuffled partitions for customers and orders are sorted independently by customer_id. Merge: Worker nodes iterate through their sorted partitions, comparing customer_id values. When a match is found, a new row is created with data from both DataFrames (e.g., (customer_id, name, city, order_id, amount)). Key Points: SMJ works well for datasets that don't fit entirely in memory or where the join key has good cardinality (distinct values). It requires shuffling data, which can be network-intensive. Spark can automatically choose a more efficient join strategy like broadcast join if one DataFrame is significantly smaller than the other. Additional Considerations: You can configure the number of shuffle partitions using spark.sql.shuffle.partitions. SMJ can be optimized by using in-memory sorting if data fits in memory (spark.sql.sortMergeJoin.forceInMemory=true). For very large datasets, Spark might spill intermediate results to disk during shuffling and sorting, potentially impacting performance. TrendyTech #Spark, #BigData, #DataEngineering, #SoftwareEngineering

36

Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE in Databricks 13.3 LTS: Changes to Implicit Casting in Databricks 13.3 LTS Databricks Runtime 13.3 LTS, powered by Apache Spark 3.4.1, introduced several enhancements and improvements. One notable change relates to implicit casting when storing rows in a table using the MERGE INTO and UPDATE operations. Previous Behavior Before this update, when storing values in a table, Databricks would store them as NULL by default if they couldn’t be safely cast to the target table schema. This behavior could lead to unexpected results, especially when dealing with data types that might overflow. New Behavior With Databricks 13.3 LTS, the platform now follows the configuration parameter spark.sql.storeAssignmentPolicy for implicit casting during storage. The default value for this parameter is ANSI, which throws an error when storing values that overflow. In other words, if a value cannot be safely cast to the type specified in the target table schema, Databricks will raise an error instead of storing it as NULL. Example Usage Consider the following example using the MERGE INTO statement: SQL -- Assume we have a target table named 'my_table' and a source table named 'my_source'. -- We want to update rows in 'my_table' based on matching keys from 'my_source'. MERGE INTO my_table AS t USING my_source AS s ON t.key = s.key WHEN MATCHED THEN  UPDATE SET * WHEN NOT MATCHED THEN  INSERT * WHEN NOT MATCHED BY SOURCE THEN  DELETE;  In this above example, MY_Table table column Key is "Varchar", and my_source table column Key is "Int" ,databricks willnot be able to saefly cast it.   Databricks will raise an error during the MERGE INTO operation. Benefits By enforcing safe casts during data storage, Databricks ensures data consistency and helps prevent unexpected behavior. Developers can rely on more predictable results when working with different data types. Things I Learned: 1. Before any LTS version upgrade, even though no code change is involved, notebooks can fail in production if they encounter above scenario 2. Go through the Release Notes especially "Breaking changes" section to identify if any components will fail For more details, you can refer to the official Databricks documentation on Databricks Runtime 13.3 LTS. https://lnkd.in/eTmjp6Ke #databricks #Data_Architect #Databricks_SQL

3

1 Comment

I have published new article on Medium. In this article, you'll get fundamental overview on how to solution design architecture for data engineering pipeline leveraging Apache Airflow and Kafka (open source) to read real-time weather updates and analyze its event streams on Twitter to understand public reactions and engagement trends 📈 #DataEngineering #Airflow #Kafka #Opensource #LearningandGrowing #ShareandLearn

24

Q. Why Snowflake & Databricks moves ahead and did acquisitions in upstream & downstream, "Data Ingestion" & "AI Modeling/AI Ops" ? 💭 Embracing the evolution of the warehouse/lakehouse where storage, compute, and knowledge are independent yet scalable, adapting dynamically to the volume or changes in data. Q. Can the prime focus on scalability, elasticity, and performance lead to deep vendor lock-in, limiting options in their data maturity journey? Even with tight vendor lock-in, organizations must still push for rapid innovation and accelerate AI product development. But could this focus become a cost barrier that hampers the ability to see real ROI? #warehouse #lakehouse #modernwarehouse

21

Latest blog on how to connect Oracle Autonomous Data Warehouse (ADW) to Apache Superset using the python-oracledb thick client on an Oracle Cloud Infrastructure (OCI) compute instance. Connection between Oracle ADW and Apache Superset can be quite the maze, given the scattered pieces of information out there. This step-by-step guide brings all the necessary details together in one place, making it much easier for you to set up and start exploring your data without much hassle. Thanks to @christopher jones (https://github.com/cjbj). His insights have been helpful for this. https://lnkd.in/g6h5Fi5u #Superset #PythonOracledb #OracleCloud

10

Hot off the press just minutes ago, Confluent just announced Freight clusters in #KafkaSummit Bangalore. They’re a new innovative cluster type that lowers prices by up to 90% (!) 🔥 What is it and how does it work? It’s all about latency. Some workloads like: 🔷telemetry 🔷metrics 🔷logs don’t care whether the latency is 50ms, 500ms or 2s. But the same workloads typically have a TON of data. And when you need to replicate that data 3x and pipe it through the network… the network costs can eat you alive 💀 So what did #CFLT do? They decided to write the data straight to S3! Instead of having Kafka replicate it across brokers, they just write it directly to S3. S3 itself does the replication under the hood (for much cheaper) 💸 The catch? S3 is much slower. But as we mentioned, those workloads don’t care. Similar to a cargo ship, you’re fine letting the cargo (data) take longer to ship as long as the price is good. For the critical use cases - you have planes. ✈️ For everything else - you have Freight. 🛳️

600

36 Comments

🌟 Exciting News for Snowflake Enthusiasts in Hyderabad! 🌟 The #Snowflake Hyderabad User Group is back with another amazing in-person meetup, hosted by Snowflake. Join us on December 07, 2024, from 9:00 AM to 1:00 PM, for a hands-on workshop led by Satish T, Cloud Data Architect at Infocepts Pvt Ltd and Snowflake Data SuperHero. In this workshop, you’ll learn how to: 🔍 Translate multilingual customer reviews 📊 Categorize and analyze unstructured review data at scale ⭐ Extract ratings and sentiments from reviews 💡 Identify customer issues and take action with LLM assistance 💻 What You’ll Build: We’ll analyze Tasty Bytes customer reviews using Snowflake Cortex within Snowflake Notebooks to discover: ✅ How likely customers are to recommend Tasty Bytes ✅ Sentiments around food, pricing, and other aspects ✅ Key issues customers highlight and how to address them 🎯 Prerequisites: Familiarity with Python and DataFrame API Experience with Snowflake and Snowpark A Snowflake account with access to Cortex LLM functions/models and Snowflake Notebooks. Don’t miss this opportunity to level up your skills with Snowflake Cortex. #Snowflake #LLM #dataengineer #Cortex #dbt #genai

15

In my view, the question that is concerning is - what and why of use case selection at the organisation level vs starting with a Big Bang approach in favour of few tools that drags the organisation into managing cost and scope creeps Success of the #dataquality foundation using Gen AI and its applicability to handful use cases is the key. Structured Documentation aligned with newer roles within CDO organisation like knowledge creators in favour of strong actionable metadata layer is an important area that we might see unfolding. Who should be a Knowledge creator in a large organisation? Should this be traditional Product organisation or a revamp that brings product, business and technology together who form Knowledge COEs targeted towards evolutionary and foundational knowledge base for key use cases that Gen AI, or AI models can leverage? Reference: https://lnkd.in/geFTx98j

2

Databricks announces General Availability of Liquid Clustering in AWS, Azure & GCP India Regions today. Liquid Clustering is a new data management technique that adapts to your data over time & completely changes the paradigm of how traditional partitioning & indexing were thought of. Key Benefits of Liquid Clustering - Improves write time by up to 7x compared to Hive-style partitioning and Z-order Improves query performance by up to 12x Databricks offers the only lakehouse with an open data format that offers row-level concurrency Check it out today !! #DeltaLake #LiquidClustering

45

Snowflake and AWS??? Understanding how Snowflake integrates with AWS is significantly important for data engineers. 🔗 Key Integration Points: 𝐒𝐞𝐚𝐦𝐥𝐞𝐬𝐬 𝐃𝐚𝐭𝐚 𝐒𝐭𝐨𝐫𝐚𝐠𝐞: Snowflake operates natively on AWS, utilizing Amazon S3 for scalable and secure data storage. This integration ensures high availability and durability of our datasets while allowing for efficient data retrieval. 𝐄𝐥𝐚𝐬𝐭𝐢𝐜 𝐂𝐨𝐦𝐩𝐮𝐭𝐞 𝐑𝐞𝐬𝐨𝐮𝐫𝐜𝐞𝐬: With Snowflake's decoupled architecture, we can scale compute resources independently from storage. This means we can allocate compute power on-demand, optimizing costs while ensuring performance for various workloads—whether it’s running complex queries or handling large data transformations. 𝐃𝐚𝐭𝐚 𝐒𝐡𝐚𝐫𝐢𝐧𝐠 𝐚𝐧𝐝 𝐂𝐨𝐥𝐥𝐚𝐛𝐨𝐫𝐚𝐭𝐢𝐨𝐧: Snowflake’s capabilities allow for easy data sharing across teams and organizations. By integrating with AWS services, we can facilitate real-time collaboration on datasets stored in Snowflake without the need for cumbersome data movement. 🌟 𝐖𝐡𝐲 𝐓𝐡𝐢𝐬 𝐌𝐚𝐭𝐭𝐞𝐫𝐬: For data engineers and architects, this integration simplifies the architecture by eliminating silos and enabling a unified approach to data management. 💡 Attached is a handy doc by Shwetank Singh #DataEngineering #AWS #Snowflake #CloudComputing #DataIntegration #BigData P.S ✅Version 3 of 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐛𝐚𝐬𝐞𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 𝐑𝐞𝐚𝐥-world AWS Data Engineering ( RADE™) program 𝐰𝐢𝐭𝐡 80+ 𝐚𝐦𝐚𝐳𝐢𝐧𝐠 𝐩𝐞𝐨𝐩𝐥𝐞 completed on Oct 5th. Onboarding for the Next one has begun. ✅𝐈𝐟 𝐲𝐨𝐮 𝐚𝐫𝐞 𝐚𝐜𝐭𝐢𝐨𝐧-𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝 and want to get in the next batch, fill the form below to be get invited for the webinar for RADE™ program V4 Click https://lnkd.in/ervbN-zt 𝐔𝐒𝐏: 𝐑𝐞𝐚𝐥-𝐰𝐨𝐫𝐥𝐝 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐛𝐚𝐬𝐞𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 program ! #aws #dataengineering

241

23 Comments

**Databricks is like Mutual funds, Using Databricks is subject to market risk.** 🎙 Fundamental of migrating towards databricks & cloud is to reduce the cost and create scalable applications. 2 years back databricks clusters start time and termination is quicker (considered the permutations, library and users etc..), now it is taking more time and if we go with server less its will cost. 😉 No doubt on databricks R&D its improved and using serverless clusters will enhance the power, but question is why the all-purpose clusters start time has latency ? – is any timeout implanted to make it lazy to use forceful of new features? it my doubt only. SQL Warehouse a year back start time is instant, now it is taking more time then they claim. Databricks features are increasing and each feature with cluster usage has different cost and it’s appreciated but why existing and old features are taking more time is a question. Gradually features are increasing, if we use those features automatically DBU’s cost will be increased and old billing setup clusters startup and other issues impacting the speed/performance, felt somewhere we are missing the fundamental of cloud migration became true commercial. Note : Executors servers and CPU’s are provided by the respective cloud above lines I am not talking about the performance of the execution, its about only the cluster/sql warehouse starting time. Databricks Amazon Web Services (AWS) Microsoft Azure Google Cloud

22

3 Comments

DATA ENGINEERING (DE) FOR DOWNSTREAM USECASES 🙂 Basically, what I did was feed FEW RAW JSON FILES Residing in AWS S3 to SnowFlake & Perform ELT VIA DBT (Data Build Tool) in VS Code where I made use of JINJA Script to make Transformations on the Data Residing in SnowFlake in a series of steps ensuring Modularity of CODE abiding by Medallion Architecture- Schematic Data for Downstream Users (BI / AI / ML / DS) By me DE! 😎