Introduction to

Databricks
Lakehouse
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Analytics Practitioner
 The Data Warehouse
Data Warehouse
Pros

Great for structured data

Highly performant

Easy to keep data clean

Cons

Very expensive

Cannot support modern applications

Not built for Machine Learning

1 https://www.databricks.com/blog/2021/05/19/evolution-to-the-data-lakehouse.html

DATABRICKS CONCEPTS
 The Data Lake
Data Lake
Pros

Support for all use cases

Very flexible

Cost effective

Cons

Data can become messy

Historically not very performant

1 https://www.databricks.com/blog/2021/05/19/evolution-to-the-data-lakehouse.html

DATABRICKS CONCEPTS
 Birth of the Lakehouse

1 https://www.databricks.com/blog/2021/05/19/evolution-to-the-data-lakehouse.html

DATABRICKS CONCEPTS
 Birth of the Lakehouse

1 https://www.databricks.com/blog/2021/05/19/evolution-to-the-data-lakehouse.html

DATABRICKS CONCEPTS
 The Databricks Lakehouse
The Databricks Lakehouse Platform

Single platform for all data workloads

Built on open source technology

Collaborative environment

Simplified architecture

1 https://www.databricks.com/blog/2021/05/19/evolution-to-the-data-lakehouse.html

DATABRICKS CONCEPTS
Databricks Architecture Benefits
Unification Multi-Cloud

Every use case from AI to BI Bring powerful platform to your data

Benefits of data warehouse and data lake No lock-in to a specific cloud platform

DATABRICKS CONCEPTS
Databricks Development Benefits
Collaborative Open-Source

Every data persona Underpinned by Apache Spark

Ability to work in same platform in real- Support for most popular languages
time (Python, R, Scala, SQL)

DATABRICKS CONCEPTS
Let's practice!
D ATA B R I C K S C O N C E P T S
 Core features of the
Databricks
Lakehouse Platform
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Practitioner
Apache Spark
Apache Spark is an open-source data processing framework and is the engine underneath
Databricks.

DataCamp Courses

Introduction to Pyspark

Big Data Fundamentals with Pyspark

Cleaning Data with Pyspark

Machine Learning with Pyspark

Introduction to Spark SQL in Python

DATABRICKS CONCEPTS
 Benefits of Spark
Key Benefits:

1. Extensible, flexible open-source framework

2. Large developer community

3. High performing

4. Databricks optimizations

1 https://spark.apache.org/docs/latest/cluster-overview.html

DATABRICKS CONCEPTS
Cloud computing basics

DATABRICKS CONCEPTS
Databricks Compute
Clusters

Collection of computational resources

All workloads, any use case

All-purpose vs. Jobs

SQL Warehouses

SQL only

BI use cases

Photon

DATABRICKS CONCEPTS
Cloud data storage

DATABRICKS CONCEPTS
 Delta
Delta is an open-source data storage file
format, and provides:

ACID transactions

Unified batch and streaming

Schema evolution

Table history

Time-travel

1 delta.io

DATABRICKS CONCEPTS
Unity Catalog
Unity Catalog is an open data governance
strategy that controls access to all data
assets in the Databricks Lakehouse platform.

SQL GRANT , REVOKE statements to control

access

Simple interface for governance

DATABRICKS CONCEPTS
Databricks UI
Designed for easier access to capabilities
based on your data workload.

All users have access to data and compute

SQL users get a familiar interface for

queries and reports

Data engineers leverage Delta Live Tables

Machine Learning workloads use models,

features, and more

DATABRICKS CONCEPTS
Let's review!
D ATA B R I C K S C O N C E P T S
 Administering a
Databricks
workspace
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Practitioner
Account Admin
Key Responsibilities:

Creating and managing workspaces

Enabling Unity Catalog

Managing identities

Managing the account subscription

DATABRICKS CONCEPTS
Account Console

https://accounts.cloud.databricks.com/

DATABRICKS CONCEPTS
Account Console - Workspaces

https://accounts.cloud.databricks.com/

DATABRICKS CONCEPTS
Account Console - Data

https://accounts.cloud.databricks.com/

DATABRICKS CONCEPTS
Account Console - Users & Groups

https://accounts.cloud.databricks.com/

DATABRICKS CONCEPTS
Account Console - Settings

https://accounts.cloud.databricks.com/

DATABRICKS CONCEPTS
Workspace Admin
Key Responsibilities:

Managing identities in your workspace

Creating and managing compute resources

Managing workspace features and settings

DATABRICKS CONCEPTS
Data Plane
Contains all of the customer's assets needed for computation with Databricks.

Data is stored in the customer's cloud environment

Clusters / SQL Warehouses run in customer's cloud tenant.

DATABRICKS CONCEPTS
Control Plane
The portion of the platform that is managed and hosted by Databricks.

Orchestrates various background tasks in Databricks

Sends requests to Data Plane to create clusters, run jobs, etc.

DATABRICKS CONCEPTS
 Databricks Platform Architecture
Each cloud will have the same general
options to create a workspace:

Cloud Service Provider marketplace

Account Console

Using the Accounts API with Databricks

Programmatic deployment (e.g., Terraform)

1 https://docs.databricks.com/getting-started/overview.html

DATABRICKS CONCEPTS
Let's review!
D ATA B R I C K S C O N C E P T S
 Setting up a
Databricks
workspace example
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Practitioner
Let's practice!
D ATA B R I C K S C O N C E P T S