Ganeshprabu kathiresan

Hadoop Vs Spark Let's explore a few scenarios!! Scenario 1: Interviewer: We've discussed how Spark replaced MapReduce in many data processing workflows. Can you explain a scenario where Spark's in-memory processing model provides a significant performance advantage over Hadoop MapReduce? Candidate: Certainly. Consider a scenario where we need to perform iterative machine learning algorithms, such as training a large-scale model using gradient descent. In Hadoop MapReduce, each iteration involves reading the data from disk, processing it, and writing intermediate results back to disk, incurring significant overhead due to disk I/O operations. In contrast, Spark keeps the data in memory between iterations, eliminating the need for costly disk reads and writes. This allows Spark to perform iterative algorithms much faster than MapReduce, making it well-suited for machine learning tasks. Interviewer: That's a clear illustration of Spark's performance advantage in iterative workloads. Thank you for explaining. Scenario 2: Interviewer: Let's discuss the versatility of Spark compared to Hadoop MapReduce. Can you provide an example of a scenario where Spark's support for multiple processing modes enables users to address diverse analytical use cases? Candidate: Absolutely. Consider a scenario where an organization needs to build a real-time analytics dashboard to monitor key performance metrics in their business operations. In this scenario, Spark's support for streaming data processing with Spark Streaming allows the organization to ingest, process, and analyze streaming data in real-time. By integrating streaming analytics with batch processing and machine learning using the same Spark framework, the organization can build an end-to-end data pipeline to address both real-time and historical analytics use cases seamlessly. Interviewer: That's a compelling example of how Spark's versatility enables organizations to address a wide range of analytical use cases. Thank you for sharing. Scenario 3: Interviewer: Let's discuss Spark's programming model and its impact on developer productivity. Can you provide an example of how Spark's high-level APIs simplify the development of complex data processing workflows? Candidate: Certainly. Consider a scenario where we need to perform complex data transformations and aggregations on a large dataset. With Spark's DataFrame API, developers can express these transformations using high-level operations like filter, groupBy, and agg, which abstract away the underlying details of distributed processing. This allows developers to write concise and expressive code, reducing the complexity of data processing workflows and accelerating development. Additionally, Spark's support for multiple programming languages such as Scala, Python, and Java enables developers to work with their preferred language! Interviewer: That's an excellent demonstration. #bigdata #dataengineering

26

**𝐋𝐞𝐯𝐞𝐥 𝐔𝐩 𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐒𝐤𝐢𝐥𝐥𝐬 𝐰𝐢𝐭𝐡 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠!** A huge shoutout to my friends, Vishal Kaushal , a Snowflake Squad member, and Sachin Mittal, a 4x Snowflake Data Superhero, who are starting Snowflake Training: Zero to Hero course starting May 18th! . **Why Snowflake? ** Snowflake is more than just a data warehouse. It's a powerful, cloud-native data platform that empowers data engineers to build next-generation data solutions. Here's why learning Snowflake can be a game-changer for your career. By mastering Snowflake, you'll equip yourself with the skills to design and implement robust, scalable data solutions that are in high demand across industries. This positions you as a valuable asset in the ever-evolving world of data engineering. This comprehensive training, will equip you with the knowledge and skills to become a Snowflake master. Here's what you'll gain: Solid Foundation: Grasp the core concepts and functionalities of Snowflake. Hands-on Learning: Get practical experience through real-world exercises and projects. Unlimited doubt support Real time Projects : 2 End to End Real time projects , 1 using Matillion as well. Ready to take your data engineering skills to the next level? Don't miss out on this amazing opportunity! Sign up for the Snowflake Training: Zero to Hero course starting May 18th. 👉🏻 Enroll Here - https://lnkd.in/gcfxzNx4 📅 Live classes starts from 18th May 2024 Discount Coupon : LIVE20 PS : This is not a paid post. #Snowflake #DataEngineering #CloudDataWarehouse #Training #Upskilling #DataSuperhero

121

4 Comments

An under appreciated reason why data processing companies like Confluent, Databricks, Splunk etc get really big is how data scales. Data usage in a company scales O(EQS), if we use Big-O notation, E=number of employees using the data, Q=number of queries being run , S=size of data queried. The reason customer ACVs can get absolutely massive is that if you’ve built an incredible product you will be able to process more data if the customer does even one of the 3: - store more data - add more employees who want to work with the data - have more computations being done on the data Most of the time, companies that are growing have all 3 increasing simultaneously. Ofcourse, because data tools are sticky this growth also persists and even when employee growth slows down, often more data and uses of the data have emerged If you’re building any tools, or even side projects that involve data processing, I’d love to learn more and talk to you!

50

Apache Spark Vs Hadoop MapReduce! Interviewer: Let's delve into the comparison between Apache Spark and Hadoop MapReduce. Can you elaborate on their core differences? Candidate: Certainly. Apache Spark is characterized by its in-memory processing, enabling faster computation by keeping data in memory across multiple processing steps. This contrasts with Hadoop MapReduce, which follows a disk-based processing model, reading data from and writing data to disk after each Map and Reduce phase. This fundamental difference in processing models greatly influences their performance and suitability for various types of workloads. Interviewer: That's insightful. How about their ecosystem support? Candidate: While Hadoop MapReduce benefits from its longstanding presence in the ecosystem, Spark has rapidly gained popularity and built a comprehensive ecosystem of its own. Spark seamlessly integrates with various data sources and storage systems, and its high-level APIs for SQL, streaming, machine learning, and graph processing simplify application development. Additionally, Spark can run both standalone and on existing Hadoop clusters, offering flexibility and compatibility with existing infrastructure. Interviewer: Good points. Now, let's talk about fault tolerance. How do Spark and MapReduce handle failures in distributed environments? Candidate: Both frameworks employ fault tolerance mechanisms, but they differ in their approaches. In MapReduce, fault tolerance is achieved through data replication and re-execution of failed tasks. Intermediate data is persisted to disk after each phase, allowing tasks to be rerun on other nodes in case of failure. On the other hand, Spark leverages lineage and resilient distributed datasets (RDDs) to achieve fault tolerance. RDDs track the lineage of each partition, enabling lost partitions to be recomputed from the original data source. Because Spark primarily operates in-memory, it can recover from failures more quickly compared to MapReduce. Interviewer: That's a comprehensive explanation. Lastly, in what scenarios would you recommend using Apache Spark over Hadoop MapReduce, and vice versa? Candidate: I would recommend using Apache Spark for applications that require real-time processing, iterative algorithms, or interactive analytics. Its in-memory processing capabilities and high-level APIs make it well-suited for these use cases. Conversely, Hadoop MapReduce may be more suitable for batch processing tasks that involve large-scale data processing and do not require real-time or iterative computation. It's essential to consider factors such as performance requirements, processing models, and ecosystem compatibility when choosing between the two frameworks. #spark #hadoop #bigdata

11

In an ideal world, we should not have the concept of warehouses in Snowflake. I don’t think I have ever met a data engineer who wants to delve deep into warehouse internals when they are already juggling tasks from different business stakeholders. Ideally, warehouses in Snowflake or DBUs in Databricks are just units of compute. One should be able to define what needs to be done and the underlying data warehouse should be capable enough to figure out the size of machine and the parallelism it needs to complete my task within a particular timeframe and minimum cost. No one cares about the configuration of the warehouse if the task succeeds within the business SLA and it’s priced fairly. Though to be honest, accurately predicting the memory and runtime requirements of an SQL before actually executing is a very tough technical problem to solve. Snowflake has always been famous for ‘it just works’ and my gut says that it’ll be the first to solve this problem (if it isn't solved by academia), but for now, it remains unsolved and the users need to take care of this! 

49

13 Comments

Ever wondered, why synchronized block doesn’t work as expected with Scala’s Futures and reason for facing: • race conditions • inconsistent results Read my below blog to know more -

8

Q. Why Snowflake & Databricks moves ahead and did acquisitions in upstream & downstream, "Data Ingestion" & "AI Modeling/AI Ops" ? 💭 Embracing the evolution of the warehouse/lakehouse where storage, compute, and knowledge are independent yet scalable, adapting dynamically to the volume or changes in data. Q. Can the prime focus on scalability, elasticity, and performance lead to deep vendor lock-in, limiting options in their data maturity journey? Even with tight vendor lock-in, organizations must still push for rapid innovation and accelerate AI product development. But could this focus become a cost barrier that hampers the ability to see real ROI? #warehouse #lakehouse #modernwarehouse

21

Can you believe it's already been two weeks since the the first-ever #KafkaSummit Bangalore? ICYMI we have the full rundown of all the key moments from the premiere event for the #ApacheKafka community. We’ve got: 🔑 Keynote recaps (Big product announcements!) 💡 Session highlights (Flink, GenAI, and of course Kafka are top of mind) 🚀 Women in Technology spotlight ➕ And MORE Find it on our blog!

33

1 Comment

Today's Learning ✌️ While going through one PBIX file. I saw that data was more than 5 Million Rows in Fact Table where Import Mode was used as Connectivity Mode. Snowflake Schema was used. But PBIX file was only of 25 MB. How Come? 🤔🤔🤔 Reason: ⬆️ 1) Data was Fetched of only last 6 months not all ✅ 2) Unwanted Columns where removed from Table ✅ 3) Grouping was done ✅ It looks very simple but Very Important observation real time work perspective 👍 Hope this helps 🙏 #powerbi #powerbideveloper

57

11 Comments

Enter the ultimate face-off: Apache Spark vs. Apache Hadoop. These two powerful tools serve distinct purposes.                Know more ......          👇👇👇👇👇👇👇👇👇👇👇👇👇 https://lnkd.in/gYAFCM7b #Apachespark #ApacheHadoop #DataProcessing #RealTimeAnalytics #BatchProcessing #BigDataTools #DataAnalytics #CloudComputing #DataStorageSolutions #MachineLearning #DataArchitecture

2

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Comments

Here's a comprehensive explanation of Adaptive Query Execution (AQE) in Spark with Scala implementation : Overview Adaptive Query Execution (AQE) is a runtime optimization technique in Spark SQL that dynamically reoptimizes and adjusts query plans based on runtime statistics. It's enabled by default since Spark 3.2.0. Enabling AQE ———————— // Enable AQE spark.conf.set("spark.sql.adaptive.enabled", true) // Enable partition coalescing spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", true) Key Features ————————— Dynamic Partition Coalescing // Set initial large number of partitions spark.conf.set("spark.sql.adaptive.coalescePartitions.initialPartitionNum", 200) // Set target partition size (default 64MB) spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", 67108864) Join Strategy Selection Spark supports different join hints that can be used with AQE: // Example using broadcast hint df1.join(broadcast(df2), Seq("join_column")) // Example using merge hint df1.join(df2.hint("merge"), Seq("join_column")) // Example using shuffle hash join df1.join(df2.hint("shuffle_hash"), Seq("join_column")) Performance Benefits AQE provides several performance improvements: 1. Dynamic Partition Optimization: Automatically coalesces small partitions into larger ones 2. Smart Join Strategy: Switches between broadcast and sort-merge joins based on runtime statistics 3. Skew Handling: Automatically detects and handles data skew in joins Implementation Details The AQE framework works in stages: 1. Initial Execution: Starts with leaf stages that don't depend on other stages 2. Statistics Collection: Gathers runtime statistics after stage completion 3. Plan Reoptimization: Updates query plan based on collected statistics 4. Adaptive Execution: Executes optimized plan for remaining stages // Example showing query with AQE optimization val query = spark.sql(""" SELECT /*+ BROADCAST(small_table) */ big_table.key, small_table.value FROM big_table JOIN small_table ON big_table.key = small_table.key """) // Explain the adaptive plan query.explain(true) Monitoring and Debugging You can monitor AQE behavior through[7]: // View adaptive plan details df.explain("adaptive") // Check if final plan is reached // Look for AdaptiveSparkPlan node with isFinalPlan = true Best Practices 1. Set appropriate initial partition numbers for large datasets 2. Monitor partition sizes using Spark UI 3. Use appropriate data types to optimize memory usage 4. Enable statistics collection for better optimization decisions AQE is particularly effective for complex queries with multiple joins and aggregations, offering significant performance improvements without manual tuning. #spark #scala

6

Interview Discussion - Driver machine fails in Databricks! Interviewer: What happens if the driver machine fails in Databricks? Candidate: If the driver machine fails in Databricks, it can have significant implications depending on the stage of the Spark job execution. Interviewer: Can you elaborate on the impact of a driver machine failure during different stages of Spark job execution? Candidate: Certainly. During the initialization phase, if the driver machine fails before the SparkContext is fully initialized, the job will fail to start, and any resources allocated for the job will be released. However, if the driver machine fails during the execution phase, while tasks are running on worker nodes, the job will typically fail, and any partially completed tasks will need to be rerun upon job restart. Interviewer: What steps can be taken to mitigate the impact of a driver machine failure? Candidate: To mitigate the impact of a driver machine failure, it's essential to design Spark jobs with fault tolerance in mind. This includes enabling checkpointing and persisting intermediate results to resilient storage systems like HDFS or cloud storage. Additionally, leveraging features like speculative execution and job retries can help ensure job completion even in the event of driver machine failures. Interviewer: How does Databricks handle driver machine failures in terms of fault tolerance and job recovery? Candidate: Databricks provides built-in fault tolerance mechanisms to handle driver machine failures gracefully. For example, it automatically restarts the driver process on a different node if the original driver machine fails. Additionally, Databricks integrates with cloud storage services like AWS S3 and Azure Blob Storage, allowing it to recover job state and intermediate results from resilient storage in case of failures. Interviewer: Can you discuss the impact of driver machine failures on interactive notebooks in Databricks? Candidate: In interactive notebooks, a driver machine failure can disrupt the user's session and result in loss of unsaved work. However, Databricks provides session recovery capabilities that automatically restore the notebook session and code state to the last saved checkpoint in case of a driver machine failure. This ensures minimal disruption and allows users to resume their work seamlessly. Interviewer: How would you proactively monitor and mitigate the risk of driver machine failures in Databricks? Candidate: Proactive monitoring involves tracking key performance metrics such as driver CPU and memory utilization, job execution times, and resource availability. Implementing automated alerting mechanisms based on predefined thresholds can help detect and address potential issues before they escalate into failures. Additionally, implementing redundancy and load balancing strategies for critical components like the driver process can further mitigate the risk of failures.

11

Interview Discussion - Driver machine fails in Databricks! Interviewer: What happens if the driver machine fails in Databricks? Candidate: If the driver machine fails in Databricks, it can have significant implications depending on the stage of the Spark job execution. Interviewer: Can you elaborate on the impact of a driver machine failure during different stages of Spark job execution? Candidate: Certainly. During the initialization phase, if the driver machine fails before the SparkContext is fully initialized, the job will fail to start, and any resources allocated for the job will be released. However, if the driver machine fails during the execution phase, while tasks are running on worker nodes, the job will typically fail, and any partially completed tasks will need to be rerun upon job restart. Interviewer: What steps can be taken to mitigate the impact of a driver machine failure? Candidate: To mitigate the impact of a driver machine failure, it's essential to design Spark jobs with fault tolerance in mind. This includes enabling checkpointing and persisting intermediate results to resilient storage systems like HDFS or cloud storage. Additionally, leveraging features like speculative execution and job retries can help ensure job completion even in the event of driver machine failures. Interviewer: How does Databricks handle driver machine failures in terms of fault tolerance and job recovery? Candidate: Databricks provides built-in fault tolerance mechanisms to handle driver machine failures gracefully. For example, it automatically restarts the driver process on a different node if the original driver machine fails. Additionally, Databricks integrates with cloud storage services like AWS S3 and Azure Blob Storage, allowing it to recover job state and intermediate results from resilient storage in case of failures. Interviewer: Can you discuss the impact of driver machine failures on interactive notebooks in Databricks? Candidate: In interactive notebooks, a driver machine failure can disrupt the user's session and result in loss of unsaved work. However, Databricks provides session recovery capabilities that automatically restore the notebook session and code state to the last saved checkpoint in case of a driver machine failure. This ensures minimal disruption and allows users to resume their work seamlessly. Interviewer: How would you proactively monitor and mitigate the risk of driver machine failures in Databricks? Candidate: Proactive monitoring involves tracking key performance metrics such as driver CPU and memory utilization, job execution times, and resource availability. Implementing automated alerting mechanisms based on predefined thresholds can help detect and address potential issues before they escalate into failures. Additionally, implementing redundancy and load balancing strategies for critical components like the driver process can further mitigate the risk of failures.

68

Interview Discussion - Driver machine fails in Databricks! Interviewer: What happens if the driver machine fails in Databricks? Candidate: If the driver machine fails in Databricks, it can have significant implications depending on the stage of the Spark job execution. Interviewer: Can you elaborate on the impact of a driver machine failure during different stages of Spark job execution? Candidate: Certainly. During the initialization phase, if the driver machine fails before the SparkContext is fully initialized, the job will fail to start, and any resources allocated for the job will be released. However, if the driver machine fails during the execution phase, while tasks are running on worker nodes, the job will typically fail, and any partially completed tasks will need to be rerun upon job restart. Interviewer: What steps can be taken to mitigate the impact of a driver machine failure? Candidate: To mitigate the impact of a driver machine failure, it's essential to design Spark jobs with fault tolerance in mind. This includes enabling checkpointing and persisting intermediate results to resilient storage systems like HDFS or cloud storage. Additionally, leveraging features like speculative execution and job retries can help ensure job completion even in the event of driver machine failures. Interviewer: How does Databricks handle driver machine failures in terms of fault tolerance and job recovery? Candidate: Databricks provides built-in fault tolerance mechanisms to handle driver machine failures gracefully. For example, it automatically restarts the driver process on a different node if the original driver machine fails. Additionally, Databricks integrates with cloud storage services like AWS S3 and Azure Blob Storage, allowing it to recover job state and intermediate results from resilient storage in case of failures. Interviewer: Can you discuss the impact of driver machine failures on interactive notebooks in Databricks? Candidate: In interactive notebooks, a driver machine failure can disrupt the user's session and result in loss of unsaved work. However, Databricks provides session recovery capabilities that automatically restore the notebook session and code state to the last saved checkpoint in case of a driver machine failure. This ensures minimal disruption and allows users to resume their work seamlessly. Interviewer: How would you proactively monitor and mitigate the risk of driver machine failures in Databricks? Candidate: Proactive monitoring involves tracking key performance metrics such as driver CPU and memory utilization, job execution times, and resource availability. Implementing automated alerting mechanisms based on predefined thresholds can help detect and address potential issues before they escalate into failures. Additionally, implementing redundancy and load balancing strategies for critical components like the driver process can further mitigate the risk of failures.

21

1 Comment

Interview Discussion - Driver machine fails in Databricks! Interviewer: What happens if the driver machine fails in Databricks? Candidate: If the driver machine fails in Databricks, it can have significant implications depending on the stage of the Spark job execution. Interviewer: Can you elaborate on the impact of a driver machine failure during different stages of Spark job execution? Candidate: Certainly. During the initialization phase, if the driver machine fails before the SparkContext is fully initialized, the job will fail to start, and any resources allocated for the job will be released. However, if the driver machine fails during the execution phase, while tasks are running on worker nodes, the job will typically fail, and any partially completed tasks will need to be rerun upon job restart. Interviewer: What steps can be taken to mitigate the impact of a driver machine failure? Candidate: To mitigate the impact of a driver machine failure, it's essential to design Spark jobs with fault tolerance in mind. This includes enabling checkpointing and persisting intermediate results to resilient storage systems like HDFS or cloud storage. Additionally, leveraging features like speculative execution and job retries can help ensure job completion even in the event of driver machine failures. Interviewer: How does Databricks handle driver machine failures in terms of fault tolerance and job recovery? Candidate: Databricks provides built-in fault tolerance mechanisms to handle driver machine failures gracefully. For example, it automatically restarts the driver process on a different node if the original driver machine fails. Additionally, Databricks integrates with cloud storage services like AWS S3 and Azure Blob Storage, allowing it to recover job state and intermediate results from resilient storage in case of failures. Interviewer: Can you discuss the impact of driver machine failures on interactive notebooks in Databricks? Candidate: In interactive notebooks, a driver machine failure can disrupt the user's session and result in loss of unsaved work. However, Databricks provides session recovery capabilities that automatically restore the notebook session and code state to the last saved checkpoint in case of a driver machine failure. This ensures minimal disruption and allows users to resume their work seamlessly. Interviewer: How would you proactively monitor and mitigate the risk of driver machine failures in Databricks? Candidate: Proactive monitoring involves tracking key performance metrics such as driver CPU and memory utilization, job execution times, and resource availability. Implementing automated alerting mechanisms based on predefined thresholds can help detect and address potential issues before they escalate into failures. Additionally, implementing redundancy and load balancing strategies for critical components like the driver process can further mitigate the risk of failures.

21

1 Comment

Benefits of Bucketing in Pyspark! 1.By bucketizing the tables, you minimize data shuffling during join operations, leading to improved query performance. 2.Bucketing can be particularly effective when the join column has high cardinality and the tables are large. Bucketing in Apache Spark is a technique used to improve the performance of certain types of queries, particularly those involving join operations, by organizing data into a fixed number of buckets based on a specific column's hash value. Here's an example scenario where bucketing can be beneficial: Suppose you have two large tables: transactions and clients. The transactions table contains transactional data with columns like transaction_id, client_id, amount, etc. The clients table contains information about clients, including client_id, name, email, etc. When to Use Bucketing: In this scenario, bucketing can be useful. You can bucketize both tables on the client_id column. This ensures that rows with the same client_id are co-located in the same bucket across both tables. When you perform a join operation, Spark can quickly match rows with the same client_id without the need for extensive shuffling. Example Code: Here's how you can bucketize the transactions and clients tables using PySpark: #pyspark #bigdata #dataengineer

22

Someone asked today why my focus shifted from Kafka/Confluent to Spark/Databricks? Here is the answer. I have my own senses of looking into the future of data technologies. I worked with the Oracle database for years, shifted to Hadoop, then saw huge potential in streaming and got into Kafka and finally, I saw this. Steve Jobs launched the iPhone as a unified platform for offering three capabilities. 1. Mobile Phone (Killed Nokia type of phones) 2. Internet access on the go (Killed BlackBerry phones) 3. Media and entertainment on the go (Killed music players and also video players) A unified platform has its own power when designed properly with innovation and clear vision of the future. Let's see what Databricks bring to the table. A unified platform for the following. 1. Big data (Killed Hadoop) 2. Data warehouse (Killed Teradata and exadata and will be killing many others) 3. ML/AI (Killing ML API, if not completely then taking majority of enterprises away) The key to the success of Databricks Unified platform will be based on the openness, adaptability, integration and ecosystem. And Databricks knows that very well. Look at what they are doing. 1. Open source the delta lake. Make it interoperable with iceberg and hudi. 2. Open source Unity catalog and make it adaptable as a single metadata across systems. 3. Develop and open source AI models and offer an end to end platform for pre training, fine train, automate and serve the entire pipeline and API. They are betting on open source and build your own principal and I believe it will outperform proprietory solutions. Note: I am not employed with Databricks and views above are my personal senses. Enjoying the post? Please share with your network and follow Prashant Kumar Pandey for more.

661

41 Comments