Douglas Moore

Worth reading: How the Apache Arrow Format Accelerates Query Result Transfer by Ian Cook One of the hidden problems of intersystem communication is transcription of data formats. Ian does a wonderful job of explaining the "why" of Apache Arrow as well as the "how" of its advantages. Read on...

8

Let's transform raw page view data from Apache Kafka into aggregated insights directly loaded into Databricks Delta tables. 🧱 We explore real-time data aggregation, demonstrating how to downsample page view data across a 10-second time window using just a few SQL queries. By processing data before it reaches Databricks, you can dramatically reduce storage and compute costs while minimizing latency. It's all here 👇 https://lnkd.in/g7QwbSUD

3

Attention Snowflake users ! Wouldn't it be great if every time anything anomalous happened in your infrastructure it got flagged? Well in Orchestra that's pretty easy to set-up. Check it out below! https://lnkd.in/eP8bj6MU #dataengineering #anomalydetection #snowflake

7

Why wait for slow queries when you can speed things up? On the Apache Arrow blog, Ian Cook, David Li, and Matt Topol, break down how Arrow's format accelerates query result transfers by eliminating inefficiencies. Key takeaways: ✅ Columnar format reduces costly transpositions ✅ Zero-copy architecture minimizes overhead ✅ Universal language support ensures seamless integration across platforms Read the full post to learn how Arrow boosts query performance and discover real-world speedups from tools like Theseus, DuckDB, and Snowflake. 🔗 Read more: https://lnkd.in/eBbjz5uZ #DataAnalytics #ApacheArrow #BigData #OpenSource

12

Did you miss this awesome demo Hugo Lu did with Youssef Mrini over at Databricks 🏘 In Orchestra you can trigger and monitor Databricks Workbooks, Workflows, data quality tests, and lots more. Our users also leverage Orchestra's overarching monitoring and alerting capabilities to link upstream jobs like data ingestion (e.g. ADF) through Databricks into downstream jobs like Reverse ELT with Hightouch or Power BI dataset refreshes. Check it out here 🔗 https://lnkd.in/dU2grpum #dataengineering #databricks

7

1 Comment

Back in March, #Databricks launched #DBRX—a super-advanced state-of-the-art open language model that's truly next-level. This new model marks a major step forward for #OpenSource #LLMs. What’s really cool is that DBRX outperforms nearly all established open source models and even rivals #ClosedSource giants like #GPT4, #Gemini, and #Mistral across various benchmarks. Want to know what makes #DatabricksDBRX so awesome? We've got you covered! Check out the full article below to learn everything you need to know about it. https://lnkd.in/g3w845G7

2

Senior Consultant and machine learning expert, Kenny Shaevel, breaks down 3 ways Databricks is helping clients implement ML best practices through MLflow. - Practical use case identification: AutoML helps to quickly evaluate the data you already have available to see how well it would work for a machine learning app. - Repeatability of complex models: Feature Store allows you to “register” features you’ve engineered to use in multiple models as needed. - Governance: Model Registry provides governance and traceability to ensure everyone knows how the models were trained and meant to be used Watch Kenny’s 2-min video for the details! #Databricks #MachineLearning #MLFlow #DataScience #AutoML #FeatureEngineering #ModelRegistry #ExpertInsights

27

1 Comment

The latest update for #SnowflakeDB includes "Celebrating Innovation and Excellence: Announcing Snowflake's Data Drivers" and "Introducing Snowflake Notebooks, an End-to-End Interactive Environment for Data & AI Teams". #analytics #BusinessIntelligence https://lnkd.in/dH8h5ftY

4

Why should Machine Learning teams use Feature Stores within Databricks? 🤔 This question and more were answered by Ben and Sepideh at Databricks when they spoke to Gavi in our most recent Advancing AI video: https://hubs.ly/Q02vyj4R0 #MachineLearning #ML #AI #FeatureStores #Databricks

19

1 Comment

Join PyTorch at Snowflake's #DataCloudDevDay on June 6 in San Francisco to learn how to accelerate GenAI performance with Native PyTorch. Register: https://hubs.la/Q02xQStt0

18

🚀 Databricks Unveils Groundbreaking Innovations 🚀 Databricks has announced the release of the DBRX model, setting a new standard for open large language models (LLMs) with up to 2x faster inference than competitors like Llama2-70B. This model aims to democratize LLM training for organizations, making it more accessible and cost-effective. Key Highlights: Unity Catalog: Now open-sourced, enhancing data governance across cloud platforms. Features include centralized access management, Role-Based Access Control (RBAC), and data lineage tracking. Databricks AI/BI: An intelligent business intelligence solution featuring AI-powered dashboards and a conversational interface, enabling self-service data analysis. Mosaic AI: A comprehensive platform for building and managing AI applications with unified tooling, centralized model management, and cost-effective custom LLM training. Data Intelligence Platform: Transforms data management by combining data lakes and warehouses, utilizing Delta Lake for real-time processing, and facilitating secure data sharing. These innovations reinforce Databricks' commitment to enhancing data management and AI capabilities for organizations of all sizes. #Databricks #AI #MachineLearning #DataGovernance #OpenSource #Innovation #Logimonk

4

"The new Columnar capabilities enable rapid real-time intelligence for data analysis and quick insights, along with further opportunities for traditional ML and AI" ⚡ Check out this write up on Capella columnar from private preview user Will Thayer, MBA Technologist 📝 https://bit.ly/3JGo8UR

24

1 Comment

🚨 Is your data engineering workflow holding you back? 🚨 MotherDuck and Dagster Labs are teaming up to discuss the secrets of modern data engineering. We'll cover: - Crafting an end-to-end data platform with Bluesky data, MotherDuck, and Dagster - Proactively solving problems with "shift-left" testing to catch issues early - Practical automation strategies to streamline your workflow and boost productivity 📅 January 14th ⏰ 9:00 AM PT ☑️ Registration link in the comments

13

1 Comment

When a physician-led healthcare organization found that their utilization of Databricks was not functioning at optimal efficiency, they decided to transition to Azure Synapse. Here's how we assisted with this urgent project: https://lnkd.in/gD_trzDt #Databricks #AzureSynapse #datalake #datamgmt

11

🚀 KAWA Analytics is the easiest way to build enterprise-grade data apps on Snowflake From risk management to dynamic pricing, ESG to market abuse surveillance, KAWA AI Co-builder makes it effortless. 🎥 Watch our product manager Narjisse Bouri demonstrate how to build a market abuse monitoring system in no time.

5

1 Comment

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43