shubham srivastava

Today's Topic - Why is Redis so Fast? Redis consistently ranks among the fastest databases available. Several factors contribute to its remarkable speed and performance - 1. 𝐈𝐧-𝐌𝐞𝐦𝐨𝐫𝐲 𝐃𝐚𝐭𝐚 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 Lightning-Fast Access - Redis primarily operates in-memory. - This means data resides in RAM, eliminating the latency of disk reads and writes. - The image clearly demonstrates the significant speed difference between RAM and storage mediums like SSDs or HDDs. Data Persistence Options - While primarily in-memory, Redis offers options for persistence (RDB snapshots, AOF logging) to safeguard data durability. 2. 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞𝐝 𝐃𝐚𝐭𝐚 𝐒𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐬 Efficient Implementation - Redis data structures (strings, lists, hashes, sets, sorted sets) are implemented with highly optimized algorithms and data structures, tailored for specific use cases. - The image showcases how each Redis structure maps to a well-chosen underlying implementation (e.g., ziplists for small lists, skip lists for sorted sets). Minimal Overhead - These specialized structures minimize memory overhead and computational complexity, leading to faster operations. 3. 𝐒𝐢𝐧𝐠𝐥𝐞-𝐓𝐡𝐫𝐞𝐚𝐝𝐞𝐝 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 No Context Switching - Redis uses a single thread for command processing, eliminating the overhead of context switching between multiple threads. Event Loop - The event loop model allows Redis to handle multiple client connections concurrently without the need for multithreading, avoiding the contention and synchronization issues that can arise. 4. 𝐈/𝐎 𝐌𝐮𝐥𝐭𝐢𝐩𝐥𝐞𝐱𝐢𝐧𝐠 (𝐑𝐞𝐝𝐢𝐬 6.0 𝐎𝐧𝐰𝐚𝐫𝐝𝐬) Enhanced Scalability - I/O multiplexing (using the io_uring interface on Linux) further improves Redis' ability to handle many concurrent connections efficiently. - The image depicts how this mechanism enables a single thread to manage multiple client connections simultaneously. Networking Performance - Redis leverages various networking optimizations (e.g., TCP backlog, SO_REUSEPORT) to enhance network throughput. Minimalistic Design - Redis focuses on core features and avoids unnecessary bloat, resulting in a lean and efficient codebase. Connect/Follow - Pallavi Ghanshani

4

How Quora Scaled Writes with MySQL ? 1. Initial Setup: They began with a non-sharding setup, scaling reads using a leader-follower configuration + caching. 2. Increasing Write Throughput: They adopted two main approaches: Vertical Sharding: - They moved high-traffic tables to different partition groups, so multiple partitions ran different sets of tables using a master-follower setup. - They used Zookeeper to store information about partitions, their tables, and master and replica instances. - This method worked well for small tables but faced challenges with large tables, joins, and transactional operations. Horizontal Sharding: - This approach involves splitting a logical table into multiple physical tables, known as shards. - They used range-based sharding as they make frequent use of range queries. eg: If a shard of table T holds values >= x and < y, they named it T_shard_x_y. - They used the tool pt-online-schema-change to modify the schema, which requires 2x space as it creates a copy of the original table. Although it is an "online" schema change tool, it initially acquires a short-duration X lock on the entire table to create triggers. This X lock sometimes impacts production traffic. However, if the table has 10 shards, at any given time only about 10% of the table would be locked. This article provides an in-depth explanation and discusses the challenges involved. https://lnkd.in/gWcz26mY.

5

Traffic split is not the same as A/B testing! Almost all #MLOps tools that provide #machinelearning model serving (like AWS Sagemaker, Azure ML, Databricks, and Seldon) understand a simple traffic split under A/B testing. If the routing to a certain version of the model is done based on the traffic %, the same customer may get a different version of the model served to him during the experiment. This may result in inaccurate experiment results and a bad customer experience. Most mature companies have experimentation frameworks in place (built in-house or a 3rd party tool), where an experiment can be registered, and the treatment group gets assigned to the customers. It should be fast enough to comply with latency requirements for your project. If you do not have such a tool, you should consider using a hash function to split group assignment in the routing interface and log all the information for future analysis. #datascience

184

16 Comments

Hey folks, diving into Kubeflow Pipelines with Google Cloud VertexAI? Let's talk architecture! Whether you're into the open-source vibe or prefer native GCP services, there's a pattern for you. Check out the supported GCP integrations and let's elevate those workflows together!

15

1 Comment

𝐓𝐢𝐞𝐫𝐞𝐝 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 𝐢𝐧 𝐊𝐚𝐟𝐤𝐚 Kafka’s architecture is built on the concept of distributed, partitioned, and replicated logs, designed for high-throughput, low-latency streaming. It’s great at handling concurrent producers and consumers, but with the growing demand for real-time data and PubSub systems, the tight coupling of storage and compute can lead to challenges around scalability and maintenance. Pinterest's implemented an alternate architecture in this regard. In a nutshell, they’ve decoupled storage and compute by leveraging cheaper remote storage solutions like AWS S3, a concept they call Tiered Storage. By offloading data storage to S3, Pinterest ensures Kafka brokers focus on what they do best—streaming data—without the overhead of managing and serving stored data to consumers. This shift allows them to maximize the compute power of brokers while making the system more scalable and cost-effective. Huge shoutout to the Pinterest engineering team for the deep dive on this—definitely a must-read for anyone building real-time streaming applications #kafka #streaming #realtimedata #engineering #aws #s3 #decoupled #architecture #cloudcomputing #scalability #data #dataengineering Confluent

5

DATA ENGINEERING (DE) FOR DOWNSTREAM USECASES 🙂 Basically, what I did was feed FEW RAW JSON FILES Residing in AWS S3 to SnowFlake & Perform ELT VIA DBT (Data Build Tool) in VS Code where I made use of JINJA Script to make Transformations on the Data Residing in SnowFlake in a series of steps ensuring Modularity of CODE abiding by Medallion Architecture- Schematic Data for Downstream Users (BI / AI / ML / DS) By me DE! 😎

CLASSES in SCALA are pretty much like CLASSES in C++ & Other OBJECT ORIENTED PROGRAMMING LANGUAGE. #SCALA #BIGDATA #CPP #C++ Karthik K. object CLASS { class Point{ private var _x = 0 private var _y = 0 private val bound = 100 def x: Int ={ return _x} 👈 //GETTER for private var _x def y: Int ={ return _y} 👈 //GETTER for private var _y def coord( x:Int, y:Int)={ 👈 //SETTER for private vars _x & _y this._x = x 👆👇 /"This" keyword store address of the object calling the function this._y = y println(this._x,this._y) } } def main(args: Array[String]):Unit = { val a = new Point 👈 //Object creation of Point in Heap Memory a.coord(20,30) print(a) } } OUTPUT:

4

3 Comments

**𝐋𝐞𝐯𝐞𝐥 𝐔𝐩 𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐒𝐤𝐢𝐥𝐥𝐬 𝐰𝐢𝐭𝐡 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠!** A huge shoutout to my friends, Vishal Kaushal , a Snowflake Squad member, and Sachin Mittal, a 4x Snowflake Data Superhero, who are starting Snowflake Training: Zero to Hero course starting May 18th! . **Why Snowflake? ** Snowflake is more than just a data warehouse. It's a powerful, cloud-native data platform that empowers data engineers to build next-generation data solutions. Here's why learning Snowflake can be a game-changer for your career. By mastering Snowflake, you'll equip yourself with the skills to design and implement robust, scalable data solutions that are in high demand across industries. This positions you as a valuable asset in the ever-evolving world of data engineering. This comprehensive training, will equip you with the knowledge and skills to become a Snowflake master. Here's what you'll gain: Solid Foundation: Grasp the core concepts and functionalities of Snowflake. Hands-on Learning: Get practical experience through real-world exercises and projects. Unlimited doubt support Real time Projects : 2 End to End Real time projects , 1 using Matillion as well. Ready to take your data engineering skills to the next level? Don't miss out on this amazing opportunity! Sign up for the Snowflake Training: Zero to Hero course starting May 18th. 👉🏻 Enroll Here - https://lnkd.in/gcfxzNx4 📅 Live classes starts from 18th May 2024 Discount Coupon : LIVE20 PS : This is not a paid post. #Snowflake #DataEngineering #CloudDataWarehouse #Training #Upskilling #DataSuperhero

121

4 Comments

𝑻𝒊𝒑𝒔 𝒇𝒐𝒓 𝒏𝒆𝒘 𝑫𝒂𝒕𝒂 𝑺𝒄𝒊𝒆𝒏𝒄𝒆 𝑴𝒂𝒏𝒂𝒈𝒆𝒓𝒔 I became a manager in data science early in my career. It has been a journey fraught with failures and some successes. Here are some of my learnings along the way - 📈 𝑹𝒐𝒍𝒆, 𝒓𝒆𝒔𝒑𝒐𝒏𝒔𝒊𝒃𝒊𝒍𝒊𝒕𝒚 𝒂𝒏𝒅 𝒑𝒓𝒊𝒐𝒓𝒊𝒕𝒊𝒆𝒔 Early on, my vision was small which narrowed the value I was able to create. I had two focuses - 1) ship out high quality work, 2) coach & promote my team. After >10 years in leadership roles, here is how I NOW think of the responsibilities of a data science manager in the hierarchy of importance - 🔹 Build authentic relationships with stakeholders and team members 🔹 Identify and align on highest impact work the team can enable and unlock 🔹 Set up systems, processes, and people to execute on these projects 🔹 Manage projects and priorities 🔹 Coach your direct reports on execution Depending on the seniority of the role, the distribution across these may vary but the core buckets remain the same. 🏆 𝑩𝒆 𝒊𝒏 𝒕𝒉𝒆 𝑨𝒓𝒆𝒏𝒂 Don’t be carried away by the business of meetings that make you feel important but don’t create real value. Spend at least 1/3rd of your time towards hands-on work. Don’t get rusty on the technical chops. Be an SME in several areas in the workflow - tech stack/tools, data models, methodologies, algorithms and specific tables. This will build trust with your reports while reducing dependency, increasing team efficiency and productivity. 🔄 𝑻𝒂𝒊𝒍𝒐𝒓 𝑷𝒓𝒐𝒄𝒆𝒔𝒔𝒆𝒔 & 𝑷𝒓𝒐𝒅𝒖𝒄𝒕𝒔 Data science lacks standard processes, unlike software engineering, due to its evolving role, inherent variability in methods depending on project goals. Creating processes and systems that will help with the ground reality of the ecosystem and culture at your company, is important. 🎲 𝑼𝒏𝒅𝒆𝒓-𝑷𝒓𝒐𝒎𝒊𝒔𝒆 𝒂𝒏𝒅 𝑶𝒗𝒆𝒓-𝑫𝒆𝒍𝒊𝒗𝒆𝒓 In the unpredictable world of predictive modeling 😉, where factors such as data anomalies, algorithm biases, and unexpected market shifts can disrupt the accuracy of forecasts, it pays to temper expectations and try to exceed them. #datascience #analytics #businessanalytics #leadership #techleadership #careers #techmanagers

28

1 Comment

Spark Streaming!! Interviewer: "Today, let's explore Spark Streaming, an integral component of Apache Spark for processing real-time streaming data. Are you familiar with Spark Streaming?" Candidate: "Yes, I've worked with Spark Streaming. It provides a scalable and fault-tolerant framework for processing real-time data streams." Interviewer: "Excellent! Spark Streaming enables developers to build robust and scalable stream processing applications. Could you elaborate on the key features and benefits of Spark Streaming?" Candidate: "Certainly! Spark Streaming offers several advantages. Firstly, it provides high throughput and low latency processing of real-time data streams. Secondly, it integrates seamlessly with the core Spark API, enabling users to leverage Spark's rich ecosystem of libraries and tools. Additionally, Spark Streaming offers fault tolerance through lineage and checkpointing, ensuring that data processing is resilient to failures. Lastly, it supports various data sources and sinks, allowing users to ingest data from sources like Kafka, Flume, or HDFS, and output processed data to databases, file systems, or dashboards." Interviewer: "Well summarized! Now, let's delve into the architecture of Spark Streaming. How does Spark Streaming process and analyze real-time data streams?" Candidate: "Spark Streaming follows a micro-batch processing model, where real-time data streams are divided into small batches of data, which are then processed using Spark's batch processing engine. The architecture consists of a Receiver that ingests data from input sources, a StreamingContext that manages the execution of streaming jobs, and a Discretized Stream (DStream) abstraction that represents a continuous stream of data. Each batch of data is processed using RDDs (Resilient Distributed Datasets), enabling parallel and fault-tolerant stream processing." Interviewer: "Spot on! Spark Streaming's micro-batch processing model provides fault tolerance and scalability, making it suitable for handling large-scale streaming data. Now, let's discuss the use cases of Spark Streaming. Can you provide examples of scenarios where Spark Streaming excels?" Candidate: "Certainly! Spark Streaming is well-suited for various real-time analytics and monitoring applications. For example, in e-commerce, it can be used to analyze user activity and recommend products in real-time. In finance, it can process streaming market data to detect anomalies or fraud in transactions. Additionally, in telecommunications, it can analyze network logs to identify network issues or optimize network performance in real-time." Interviewer: "Impressive examples! Spark Streaming's versatility makes it applicable across different industries for real-time data processing and analysis. Thank you for sharing your insights on Spark Streaming!" Candidate: "You're welcome! #bigdata #spark #sparkstreaming #pyspark #python

7

Dealing with private #python packages #Databricks can be painful. Let's say we have package mlops-test stored in a private PyPi repository (we assume basic authentication). The following is required to access this package within your Databricks job: ➡ Secret scope (in our case, "mlops") and secret "pypi_token". The secret scope can be KeyVault-backed (recommended) or Databricks-backed. ➡ Init script to specify extra-index-url. ➡ Unity Catalog Volume (in our example, /Volumes/mlops_test/mlops_volumes/init_scripts) where we upload the init script. ➡ Because init scripts can not access secret scopes directly, we need to create an environment variable. See an example of the Databricks Asset Bundle definition, and read our new article for more details: https://lnkd.in/eBEakGg5 We have not seen it documented anywhere, and hope it will save you from your struggles! #mlops

288

43 Comments

🚀 Native Execution Engine for Fabric Runtime 1.2 is now available in public preview! - Boost query execution performance with technologies like a columnar format and vectorized processing. - Validation by the TPC-DS 1TB benchmark showcases a remarkable 4x speed enhancement over OSS Spark. - The engine translates SparkSQL code into optimized C++ code, thanks to Microsoft's query optimizations in Apache Spark on Fabric. - Seamless integration with Apache Spark™ APIs means no code alterations needed, ensuring no concerns about vendor lock-in. - Compatible with PySpark, Scala, R, or Spark SQL, working effortlessly with Parquet and Delta formats for efficient data processing. - Accelerated processing times result in significant operational cost savings for users. Explore more about this update here: https://lnkd.in/g7vGzYfA #DataScience #DataEngineering #ApacheSpark #FabricRuntime #TechInnovation

4

For the data nerds who like to make data topics clear and accessible for non-data nerds, I really like this Kafka fundamentals by Sahn Lam. Well done. #data #kafka #API #datanerds

3

Revised my skills on apache kafka build streaming pipeline. pre requisite: go to compute engine>> create vm instances 3 to input 1 for ouput kafka data. 1. ingest the data to kafka, console producer in kafka. 2. process data using java application wordcounts. 3.inspect output data in kafka, console consumer in kafka.

1

#Parquet stores data in columnar format on disk. #Spark reads it and stores it in row-oriented format in memory for further processing. There is an obvious overhead of conversion from columnar to row-oriented. Apart from #Photon, #Apache #Arrow also stores data in memory in columnar format. "I do know that the Photon engine replaces Spark's row-orientation with a columnar representation." #databricks #bigdata #dataanalytics

2

3 days ago I raved about the team adding Fine Grain Access Control (FGAC) in EMR Spark, making it the most secure Apache Spark engine in the market. Here is the VLDB paper to back that claim up: https://lnkd.in/g7GMRNWh Spark is known for its extensive API, which makes it a very powerful tool but notoriously porous and hard to secure. The most common way customer secure it is by using Apache Ranger, but the technology does not meet modern security standards which renders it unsuitable for scenarios like cross user/account sharing. The approach taken by more sophisticated DBMS engines is to sandbox UDF and other user code artifacts in process-based containers. Although the approach works, we believe it is not well suited for large scale analytics processing: it suffers from performance problems in data marshaling, it doesn't solve problems with SQL-side channel attack on complex policies and in general, process-based isolation is not the gold-standard compared to full VM isolation. Our approach leverages Spark's use of exchange materialization between operators as a natural security boundary and uses the optimizer to rearrange operators to evaluate the constraints on both performance and security. The paper describes some of this in more detail as well as the challenges implementing it in Spark. where customers are free to write SQL, DataFrames and low-level RDDs. It has been amazing ideating this project with the team and supporting them make to make it a reality. Andrei Paduroiu, Sungheun Wi and Yan Yan will be presenting on VLDB if you are interested to learn more.

166

4 Comments

Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE in Databricks 13.3 LTS: Changes to Implicit Casting in Databricks 13.3 LTS Databricks Runtime 13.3 LTS, powered by Apache Spark 3.4.1, introduced several enhancements and improvements. One notable change relates to implicit casting when storing rows in a table using the MERGE INTO and UPDATE operations. Previous Behavior Before this update, when storing values in a table, Databricks would store them as NULL by default if they couldn’t be safely cast to the target table schema. This behavior could lead to unexpected results, especially when dealing with data types that might overflow. New Behavior With Databricks 13.3 LTS, the platform now follows the configuration parameter spark.sql.storeAssignmentPolicy for implicit casting during storage. The default value for this parameter is ANSI, which throws an error when storing values that overflow. In other words, if a value cannot be safely cast to the type specified in the target table schema, Databricks will raise an error instead of storing it as NULL. Example Usage Consider the following example using the MERGE INTO statement: SQL -- Assume we have a target table named 'my_table' and a source table named 'my_source'. -- We want to update rows in 'my_table' based on matching keys from 'my_source'. MERGE INTO my_table AS t USING my_source AS s ON t.key = s.key WHEN MATCHED THEN  UPDATE SET * WHEN NOT MATCHED THEN  INSERT * WHEN NOT MATCHED BY SOURCE THEN  DELETE;  In this above example, MY_Table table column Key is "Varchar", and my_source table column Key is "Int" ,databricks willnot be able to saefly cast it.   Databricks will raise an error during the MERGE INTO operation. Benefits By enforcing safe casts during data storage, Databricks ensures data consistency and helps prevent unexpected behavior. Developers can rely on more predictable results when working with different data types. Things I Learned: 1. Before any LTS version upgrade, even though no code change is involved, notebooks can fail in production if they encounter above scenario 2. Go through the Release Notes especially "Breaking changes" section to identify if any components will fail For more details, you can refer to the official Databricks documentation on Databricks Runtime 13.3 LTS. https://lnkd.in/eTmjp6Ke #databricks #Data_Architect #Databricks_SQL

3

1 Comment