Sadik Bakiu

Tomorrow is the day, the Databricks Roundtable is taking place in Cologne.🚀 Join us for an exclusive hands-on session where Philipp Schwartenbeck will present a multi-agent system and Felix Xaver Krisch will demonstrate the complete lifecycle of a data product directly in Databricks! In the live demo, we will take you through the entire journey of a data project - starting with data exploration, followed by incremental data processing, workflow scheduling and scaling. We'll also show you how to implement CI/CD pipelines and monitoring tools to ensure seamless project delivery in Databricks. In the second live demo, we'll dive into the world of LLM-based multi-agent systems. See how autonomous collaboration between agents can solve complex tasks in real time, helping organisations improve the efficiency of their data processing and decision making! Find all the event details and register now... there are only 3 places left❗ 🔗https://lnkd.in/g_EyDuHp Looking forward to seeing you tomorrow! #databricks #dataengineering #AI #multiagent #LLM #cloudmigration #datagovernance #realtimedata #dataplatform #changemanagement

6

🤝 Data Exchange Strategies between Airflow DAGs 🖋️ Author: Shivananda D 🔗 Read the article: https://lnkd.in/e_HaSKz7 ------------------------------------------- ✅ Follow Data Engineer Things for more insights and updates. 💬 Hit the 'Like' button if you enjoyed the article. ------------------------------------------- #dataengineering #airflow #orchestration #data

25

Reasons to monitor dbt in Orchestra 🕊️ Ease of use Orchestra is the easiest way to set-up dbt core orchestration and monitoring. The Data Catalog even gets updated without any additional integration 💸 Cost Orchestra provides an execution environment for dbt core with lightweight pricing, which can save on existing dbt compute costs. We also support monitoring dbt core projects in AWS EC2, Virtual Machines, etc. so you can run dbt core your way without compromising on visibility 🔀 End to End Orchestration There is sometimes more to life than synchronously pushing down SQL Queries to your data warehouse. Orchestra provides a feature-complete, end-to-end orchestration framework so you can trigger and monitor any job from a single place What other problems do you want us to solve? See it in action below. https://lnkd.in/eTzuG7Ty #dataengineering #analytics #dbt

2

𝗗𝗮𝘁𝗮 + 𝗔𝗜 𝗪𝗼𝗿𝗹𝗱 𝗧𝗼𝘂𝗿 𝟮𝟬𝟮𝟰 𝗠𝘂𝗻𝗶𝗰𝗵 - 𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀, 𝗮 𝗳𝗮𝗻𝘁𝗮𝘀𝘁𝗶𝗰 𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻, 𝗶𝗻𝘀𝗽𝗶𝗿𝗶𝗻𝗴 𝗽𝗲𝗼𝗽𝗹𝗲, 𝗮𝗻𝗱 𝗮𝗻 𝗶𝗻𝗳𝗲𝗰𝘁𝗶𝗼𝘂𝘀 𝘀𝗽𝗶𝗿𝗶𝘁 … This is how our colleagues Frank Liebrand and Sebastian Moritz summed up yesterday’s event by Databricks. The message was clear: Databricks Data Intelligence Platform is the backbone of innovative AI use cases across various industries, enabling companies to seamlessly integrate and process even petabytes of data. It was inspiring to hear firsthand success stories from leading companies like 𝗖𝗮𝗿𝗶𝗮𝗱, 𝗙𝗿𝗮𝗽𝗼𝗿𝘁, 𝗔𝗹𝗹𝗶𝗮𝗻𝘇 𝗮𝗻𝗱 𝗕𝗮𝘆𝗲𝗿 𝗮𝗻𝗱 𝗺𝘂𝗰𝗵 𝗺𝗼𝗿𝗲. Each showcased their unique journey in harnessing Databricks to drive their data and AI initiatives. AI/BI was the big topic of the day and Databricks itself presented their journey on how they use AI/BI to gain faster and deeper insights from their own data. Beyond the informative presentations, the event provided a valuable opportunity to exchange experiences with different companies and discuss current challenges. As a partner of Databricks, we are inspired by the platform's versatility and its impact on empowering a wide range of customers in their data transformation journey. A big thank you to the Databricks team especially Felix Völker & Inga Roos for hosting such an engaging and insightful event. We’re already looking forward to next year!   𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗲𝗱 𝗶𝗻 𝗹𝗲𝘃𝗲𝗿𝗮𝗴𝗶𝗻𝗴 𝗮 𝗰𝗹𝗼𝘂𝗱 𝗱𝗮𝘁𝗮 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗮𝗻𝗱 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗻𝗴 𝗶𝘁 𝗶𝗻𝘁𝗼 𝘆𝗼𝘂𝗿 𝗲𝘅𝗶𝘀𝘁𝗶𝗻𝗴 𝗦𝗔𝗣 𝘀𝘁𝗮𝗰𝗸, 𝗳𝗲𝗲𝗹 𝗳𝗿𝗲𝗲 𝘁𝗼 𝗿𝗲𝗮𝗰𝗵 𝗼𝘂𝘁 𝘁𝗼 𝘂𝘀!   #dataplatform #AI #databricks #bixConsulting

42

1 Kommentar

What if your data tools let you focus entirely on what matters — your use case? That’s exactly what Felix Krueger, senior data scientist at DZ Bank AG, experiences with Dataiku. ✨ The benefit: With no setup hassles or distractions, Felix simply logs into Dataiku, and everything he needs is ready. This simplicity allows him to focus on his use cases without extra technical overhead. 🔄 The transformation: Felix’s team at DZ Bank moved from isolated reporting solutions to a unified system within Dataiku. Now, they generate reports and dashboards across departments and even perform additional analysis, extending their previous workflows. ⭐ Favorite feature? The prepare recipe which, as he says, "just makes things so easy." 💡 Advice for companies starting with AI: Felix recommends having a use case to start but thinking ahead. “Start slow, but build an architecture that can scale over time” to prepare for future needs.

19

1 Kommentar

Real-time data processing is crucial for modern businesses, and Apache Flink's powerful stream processing capabilities set it apart 🐿️ In our latest blog, Maciej Maciejko explores strategies to mitigate data skew in Flink SQL, ensuring efficient and balanced processing even with large-scale data. Learn how to handle NULL values deterministically and use pseudo-broadcast joins to boost performance and maintain dependable real-time analytics 👇 https://lnkd.in/dGFXJAki #getindata #datascience #flinksql #realtimeanalytics #streaming

8

📢 Exciting Read for the Apache Airflow Community! We highly recommend checking out this insightful article by Giorgos Myrianthous, which delves into deploying dbt on GCP. This is a must-read for anyone in the data engineering space, especially those integrating modern data stacks with Apache Airflow. 🌐🚀 Giorgos does an excellent job breaking down the process and providing actionable steps that can help streamline your data workflows. Whether you're new to dbt or looking to optimize your GCP deployments, this article has something valuable for you. 🔗 Read the full article here: https://lnkd.in/gq_tgc5n #ApacheAirflow #DataEngineering #GCP #dbt #DataOps

73

📷 New on YouTube: In his 2023 #SAGconf session, Matthias Niehoff delves into the latest developments in modern data architectures, including Data Mesh, Data Lakehouse, and the integration of analytics with operational systems, offering insights into their practical implementation. 👀 Watch now: https://lnkd.in/eHs3EkZa #SoftwareArchitecture #SoftwareArchitectureGathering #SoftwareArchitectureConference #DataArchitecture #DataMesh #DataLakehouse #ELTPattern #CloudDataWarehouse #ModernDataStack #DataStrategy #SoftwareEngineering

8

Decoding Databricks - how, why, and where?⚙️ Aditya K. and Arman Yazdani from our Berlin Data Engineering team presented challenging and interesting parts of their practical experience with Databricks at our latest Meetup. We invite you to watch the full keynote on our YouTube channel: https://lnkd.in/dhhc4RDu What can you learn from our experts: - Which challenges did they face while analyzing and processing e-vehicle data? - Dive into the topics of Medallion Architecture and Data Quality - And simply the most important question – why Databricks at all? #databricks #keynote #dataarchitecture #dataengineering

13

Meet us at Data + AI Databricks Summit on June 13, 2024, where Veysel Kocaman, PhD, our Head of Data Science will be presenting: ‘State-of-the-Art Retrieval Augmented Generation at Scale in Spark NLP’ Current #RAG #LLM systems struggle to efficiently scale the number or size of processed documents or handle the complex pre- and post-processing pipelines needed when going from POC to production. This session shows how the open-source #SparkNLP library addresses these issues -         Natively scale pre-processing pipelines to handle multimodal inputs, document segmentation, semantic sentence, paragraph splitting, and data normalization challenges. -         Calculate state-of-the-art text embeddings, which are then loaded into a vector database, several times faster than Hugging Face on a single machine or an order of magnitude faster & cheaper on a commodity cluster. -         Provide post-processing modules such as reranking, post-filtering, expansion, augmentation, or keyword extraction without requiring other libraries. -         Use the native integration with #LangChain and 3HayStack when these libraries are being used This is a session for data scientists building production-grade LLM systems. Book a meeting with our Team: https://lnkd.in/dD_RBEtr #Databricks #LLMs #HealthcareLLMs #nocode #GenerativeAI #HealthcareAI #MedicalLLMs #nlp #naturallanguageprocessing #DataScience #DataEngineering #DataAnalysis #DataInsights #TextProcessing #DeepLearning #DataScience

7

A few months ago, we shared our collaboration with an online higher education institution. Our Senior MLOps Engineer, Bart Grasza, Data Scientist, Kamil Abram, and Senior Data Engineer, Mateusz Kijewski developed an MLOps platform that provides a foundational template. 🙌 This platform integrates MLOps best practices, automates the setup and update processes, and allows for seamless workflow adaptation across different environments. You can now see this platform in action, presented at the Databricks Data Summit: https://lnkd.in/dUbJrbVi 

23

Tune into a new webinar with Tamara Janina Fingerlin to master Airflow and data engineering best practices! Learn to: 💡 Write smarter DAGs 💡 Master advanced scheduling 💡 Boost readability & maintenance Save your spot now! > https://shorturl.at/1IFpq Sponsor: Astronomer #Airflow #DataEngineering #Scheduling #Readability

2

Apache XTable provides users with the ability to translate metadata from one #lakehouse table format to another omni-directionally. What exactly happens after the XTable "Sync" Process is run? The sync process provides users with the following: ✅ Syncs the data files along with their column-level statistics and partition metadata information ✅ All the schema-level updates in the source table are reflected on to the target format metadata ✅ Metadata maintenance for the target table format. - If the target format is Apache Hudi, unreferenced files will be marked as ‘cleaned’ to control metadata table size - If the target format is Apache Iceberg, snapshots will be expired after a configured amount of time - If the target format is Delta Lake, the transaction log will be retained for a configured amount of time ⭐️ Want to try out XTable? - Here is a link to the getting started page: https://lnkd.in/gHMBQeqV #dataengineering #softwareengineering

49

1 Kommentar

🚀 Stepping into a Data Engineering Career from a Non-Technical Background As someone who transitioned into a Data Engineering career from a different field, I recently had the pleasure of watching the 🎥 "Becoming a Data Engineer from a Non-Technical Background" talk, organized by Veri Bilimi Okulu, featuring my former colleagues Ömer Gencay Ünlü and moderated by Erkan SIRIN. The session was an invaluable guide for anyone looking to embark on this path. ✨ The insights shared were both inspiring and practical, offering a clear roadmap for those who might feel uncertain about making the leap into data engineering. I thoroughly enjoyed the discussion and found it to be a great resource for anyone considering a similar career shift. If you're thinking about transitioning into the world of data engineering, I highly recommend checking out this talk—it could be the spark you need to start your journey. 💡 #DataEngineering #CareerChange #VeriBilimiOkulu

27

🌟 Exciting news for those that work with Snowflake: They announced a partnership last week with Mistral AI, a french AI startup known for their open-source philosophy. 🤝 The partnership: Snowflake has invested in Mistral AI and has integrated their Language Models into the Snowflake Cloud. This empowers their customers to gain access to 'Mistral Large', 'Mistral 7B' and 'Mixtral 8x7B'. 🤔 Why this is relevant: This collaboration will revolutionise how companies deploy AI and is poised to set a new standard in accessibility and innovation by offering tools that excel in logical reasoning and multi-language fluency, all while maintaining data security standards. Mistral AI & Snowflake #AI #DataCloud #Innovation #Partnership #Snowflake #MistralAI #cloudllms

4

🚀 𝐄𝐱𝐜𝐢𝐭𝐢𝐧𝐠 𝐍𝐞𝐰𝐬! 🚀 Our CTO Sebastian Stremming will be speaking at a Snowflake event on the intriguing topic: "Snowflake Unistore and Massive Parallel Cloud Processes - Perfect Match or Trap?" 🌐   With Snowflake Unistore's unique feature of combining option of transaction-based data processing (OLTP) andanalyzing and querying the data (OLAP), the session will explore the potential of implementing massively parallel workloads. A proof of concept in #AWS Lambda will be presented to evaluate its advantages, disadvantages, and product maturity. Don't miss it! 📅✨   Intrigued? Can't wait? Excited? Please, register here: https://lnkd.in/dw2qNB_P #AI #SnowflakeEvent #CloudComputing #TechInnovation

14

6 Kommentare

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43

Building Text-to-SQL from Scratch using DAGs 🛠️🧱 Most existing pre-packaged text-to-SQL modules are not suitable for production - inevitably you’ll have to go in and customize the prompts, and define custom orchestration. An underrated feature of LlamaIndex for advanced LLM developers is query pipelines: the ability to define very custom LLM pipelines as DAGs, letting you mix both prepackaged modules you don’t want to write (e.g. retrievers, SQL integration) as well as custom functions. Here Aruna Withanage shows you how to compose an advanced text-to-SQL pipeline:  1. Index the table schemas via a vector database 2. Retrieve relevant table schemas given a user query 3. Given schemas, convert query to a valid SQL statement 4. Retrieve relevant rows from the SQL database 5. Define a custom response synthesis prompt https://lnkd.in/gkhTJqjt

576

21 Kommentare