Data Engineers

Guide to
Apache Spark
and Delta Lake
DATA ENGINEER S GUIDE T O
A PAC HE S PA R K A ND D ELTA L A K E

Table of Contents
Apache Spark™ has seen immense growth over the past several years, including
Chapter 1: A Gentle Introduction to its compatibility with Delta Lake.
Apache Spark 3

Chapter 2: A Tour of Spark’s Toolset 24

Delta Lake is an open-source storage layer that sits on top of your existing
Chapter 3: Working with
Different Types of Data 42 data lake file storage, such as AWS S3, Azure Data Lake Storage, or HDFS. Delta
Chapter 4: Delta Lake Quickstart 84 Lake brings reliability, performance, and lifecycle management to data lakes.
Databricks is proud to share excerpts from the Delta Lake Quickstart and the book,
Spark: The Definitive Guide.
DATA ENGINEER S GUIDE T O 3
A PAC HE S PA R K A ND D ELTA L A K E

CHAPTER 1: A Gentle Introduction to Spark Now that we took our history lesson on Apache Spark, it’s time to start using it and applying it! This chapter
will present a gentle introduction to Spark — we will walk through the core architecture of a cluster, Spark
Application, and Spark’s Structured APIs using DataFrames and SQL. Along the way we will touch on
Spark’s core terminology and concepts so that you are empowered start using Spark right away. Let’s get
started with some basic background terminology and concepts.
DATA ENGINEER S GUIDE T O 4
A PAC HE S PA R K A ND D ELTA L A K E

Spark’s Basic Architecture Spark Applications

Typically when you think of a “computer” you think about one machine sitting on your Spark Applications consist of a driver process and a set of executor processes.
desk at home or at work. This machine works perfectly well for watching movies The driver process runs your main() function, sits on a node in the cluster, and is
or working with spreadsheet software. However, as many users likely experience responsible for three things: maintaining information about the Spark Application;
at some point, there are some things that your computer is not powerful enough to responding to a user’s program or input; and analyzing, distributing, and scheduling
perform. One particularly challenging area is data processing. Single machines do work across the executors (defined momentarily). The driver process is absolutely
not have enough power and resources to perform computations on huge amounts essential — it’s the heart of a Spark Application and maintains all relevant information
of information (or the user may not have time to wait for the computation to finish). during the lifetime of the application.
A cluster, or group of machines, pools the resources of many machines together
allowing us to use all the cumulative resources as if they were one. Now a group of The executors are responsible for actually executing the work that the driver assigns
machines alone is not powerful, you need a framework to coordinate work across them. This means, each executor is responsible for only two things: executing
them. Spark is a tool for just that, managing and coordinating the execution of tasks code assigned to it by the driver and reporting the state of the computation, on that
on data across a cluster of computers. executor, back to the driver node.

The cluster of machines that Spark will leverage to execute tasks will be managed by
a cluster manager like Spark’s Standalone cluster manager, YARN, or Mesos. We then
submit Spark Applications to these cluster managers which will grant resources to
our application so that we can complete our work.
DATA ENGINEER S GUIDE T O 5
A PAC HE S PA R K A ND D ELTA L A K E

The cluster manager controls physical machines and allocates resources to Spark Applications. This can be one
DRIVER PROCESS EXECUTORS of several core cluster managers: Spark’s standalone cluster manager, YARN, or Mesos. This means that there
can be multiple Spark Applications running on a cluster at the same time. We will talk more in depth about cluster
managers in Part IV: Production Applications of this book.

Spark Session
In the previous illustration we see on the left, our driver and on the right the four executors on the right. In this
diagram, we removed the concept of cluster nodes. The user can specify how many executors should fall on each
node through configurations.
User Code

N O T E | Spark, in addition to its cluster mode, also has a local mode. The driver and executors are simply processes,
this means that they can live on the same machine or different machines. In local mode, these both run (as threads)
on your individual computer instead of a cluster. We wrote this book with local mode in mind, so everything should be
CLUSTER MANAGER
runnable on a single machine.

As a short review of Spark Applications, the key points to understand at this point are that:
• Spark has some cluster manager that maintains an understanding of the resources available.
• The driver process is responsible for executing our driver program’s commands across the executors in order
to complete our task.

Now while our executors, for the most part, will always be running Spark code. Our driver can be “driven” from
a number of different languages through Spark’s Language APIs.
DATA ENGINEER S GUIDE T O 6
A PAC HE S PA R K A ND D ELTA L A K E

Spark’s Language APIs

Spark’s language APIs allow you to run Spark code from other languages. For the most part, Spark presents some core “concepts” in every language and these concepts
are translated into Spark code that runs on the cluster of machines. If you use the Structured APIs (Part II of this book), you can expect all languages to have the same
performance characteristics.

N O T E | This is a bit more nuanced than we are letting on at this point but for now, it’s the right amount of information for new users. In Part II of this book, we’ll dive into the details
of how this actually works.

SCALA
Spark is primarily written in Scala, making it Spark’s “default” language. This book will include Scala code examples wherever relevant.

JAVA
Even though Spark is written in Scala, Spark’s authors have been careful to ensure that you can write Spark code in Java. This book will focus primarily on Scala but will
provide Java examples where relevant.

PYTHON
Python supports nearly all constructs that Scala supports. This book will include Python code examples whenever we include Scala code examples and a Python API exists.

SQL
Spark supports ANSI SQL 2003 standard. This makes it easy for analysts and non-programmers to leverage the big data powers of Spark. This book will include SQL
code examples wherever relevant

R
Spark has two commonly used R libraries, one as a part of Spark core (SparkR) and another as an R community driven package (sparklyr). We will cover these two
different integrations in Part VII: Ecosystem.
 DATA ENGINEER S GUIDE T O 7
A PAC HE S PA R K A ND D ELTA L A K E

Each language API will maintain the same core concepts that we described above. There is a SparkSession available
Python Process to the user, the SparkSession will be the entrance point to running Spark code. When using Spark from a Python or
JVM JVM R, the user never writes explicit JVM instructions, but instead writes Python and R code that Spark will translate into
TO EXECUTORS

code that Spark can then run on the executor JVMs.

Spark Session R Process Spark’s APIs

While Spark is available from a variety of languages, what Spark makes available in those languages is worth
mentioning. Spark has two fundamental sets of APIs: the low level “Unstructured” APIs and the higher level
Structured APIs. We discuss both in this book but these introductory chapters will focus primarily on the higher
level APIs.


Here’s a simple illustration of this relationship.

Starting Spark
Thus far we covered the basic concepts of Spark Applications. This has all been conceptual in nature. When we
actually go about writing our Spark Application, we are going to need a way to send user commands and data to the
Spark Application. We do that with a SparkSession.

N O T E | To do this we will start Spark’s local mode, just like we did in the previous chapter. This means running
./bin/spark-shell to access the Scala console to start an interactive session. You can also start Python console
with ./bin/pyspark. This starts an interactive Spark Application. There is also a process for submitting standalone
applications to Spark called spark-submit where you can submit a precompiled application to Spark. We’ll show you
how to do that in the next chapter.

When we start Spark in this interactive mode, we implicitly create a SparkSession which manages the Spark
Application. When we start it through a job submission, we must go about creating it or accessing it.
DATA ENGINEER S GUIDE T O 8
A PAC HE S PA R K A ND D ELTA L A K E

The SparkSession
As discussed in the beginning of this chapter, we control our Spark Application through a driver process. This driver process manifests itself to the user as an object called
the SparkSession. The SparkSession instance is the way Spark executes user-defined manipulations across the cluster. There is a one to one correspondence between a
SparkSession and a Spark Application. In Scala and Python the variable is available as spark when you start up the console. Let’s go ahead and look at the SparkSession in
both Scala and/or Python.

spark

In Scala, you should see something like:

res0: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@27159a24

In Python you’ll see something like:

<pyspark.sql.session.SparkSession at 0x7efda4c1ccd0>

Let’s now perform the simple task of creating a range of numbers. This range of numbers is just like a named column in a spreadsheet.

%scala
val myRange = spark.range(1000).toDF(“number”)

%python
myRange = spark.range(1000).toDF(“number”)

You just ran your first Spark code! We created a DataFrame with one column containing 1000 rows with values from 0 to 999. This range of number represents a distributed
collection. When run on a cluster, each part of this range of numbers exists on a different executor. This is a Spark DataFrame.
DATA ENGINEER S GUIDE T O 9
A PAC HE S PA R K A ND D ELTA L A K E

DataFrames
A DataFrame is the most common Structured API and simply represents a table of data with rows and columns.
The list of columns and the types in those columns the schema. A simple analogy would be a spreadsheet with
Spreadsheet on a named columns. The fundamental difference is that while a spreadsheet sits on one computer in one specific
single machine
location, a Spark DataFrame can span thousands of computers. The reason for putting the data on more than one
computer should be intuitive: either the data is too large to fit on one machine or it would simply take too long to
perform that computation on one machine.

The DataFrame concept is not unique to Spark. R and Python both have similar concepts. However, Python/R
DataFrames (with some exceptions) exist on one machine rather than multiple machines. This limits what you can
Table or DataFrame partitioned
across servers in data center do with a given DataFrame in python and R to the resources that exist on that specific machine. However, since
Spark has language interfaces for both Python and R, it’s quite easy to convert to Pandas (Python) DataFrames to
Spark DataFrames and R DataFrames to Spark DataFrames (in R).

N O T E | Spark has several core abstractions: Datasets, DataFrames, SQL Tables, and Resilient Distributed Datasets
(RDDs). These abstractions all represent distributed collections of data however they have different interfaces for
working with that data. The easiest and most efficient are DataFrames, which are available in all languages. We
cover Datasets at the end of Part II and RDDs in Part III of this book. The following concepts apply to all of the core
abstractions.
DATA ENGINEER S GUIDE T O 10
A PAC HE S PA R K A ND D ELTA L A K E

Partitions
In order to allow every executor to perform work in parallel, Spark breaks up the data into chunks, called partitions. A partition is a collection of rows that sit on one physical
machine in our cluster. A DataFrame’s partitions represent how the data is physically distributed across your cluster of machines during execution. If you have one partition,
Spark will only have a parallelism of one even if you have thousands of executors. If you have many partitions, but only one executor Spark will still only have a parallelism of one
because there is only one computation resource.

An important thing to note, is that with DataFrames, we do not (for the most part) manipulate partitions manually (on an individual basis). We simply specify high-level
transformations of data in the physical partitions and Spark determines how this work will actually execute on the cluster. Lower level APIs do exist (via the Resilient Distributed
Datasets interface) and we cover those in Part III of this book.

Transformations
In Spark, the core data structures are immutable meaning they cannot be changed once created. This might seem like a strange concept at first, if you cannot change it, how are
you supposed to use it? In order to “change” a DataFrame you will have to instruct Spark how you would like to modify the DataFrame you have into the one that you want. These
instructions are called transformations. Let’s perform a simple transformation to find all even numbers in our current DataFrame.

%scala
val divisBy2 = myRange.where(“number % 2 = 0”)

%python
divisBy2 = myRange.where(“number % 2 = 0”)

You will notice that these return no output, that’s because we only specified an abstract transformation and Spark will not act on transformations until we call an action, dis-
cussed shortly. Transformations are the core of how you will be expressing your business logic using Spark. There are two types of transformations, those that specify narrow
dependencies and those that specify wide dependencies.
DATA ENGINEER S GUIDE T O 11
A PAC HE S PA R K A ND D ELTA L A K E

Transformations consisting of narrow dependencies (we’ll call them narrow transformations) are those where each
input partition will contribute to only one output partition. In the preceding code snippet, our where statement
N A R R O W T R A N S F O R M AT I O N S
1 to 1 specifies a narrow dependency, where only one partition contributes to at most one output partition.

A wide dependency (or wide transformation) style transformation will have input partitions contributing to many
output partitions. You will often hear this referred to as a shuffle where Spark will exchange partitions across the
cluster. With narrow transformations, Spark will automatically perform an operation called pipelining on narrow
dependencies, this means that if we specify multiple filters on DataFrames they’ll all be performed in-memory.
The same cannot be said for shuffles. When we perform a shuffle, Spark will write the results to disk. You’ll see
lots of talks about shuffle optimization across the web because it’s an important topic but for now all you need to
W I D E T R A N S F O R M AT I O N S understand are that there are two kinds of transformations.
(SHUFFLES)
1 to 1
We now see how transformations are simply ways of specifying different series of data manipulation. This leads
us to a topic called lazy evaluation.

Lazy Evaluation
Lazy evaluation means that Spark will wait until the very last moment to execute the graph of computation
instructions. In Spark, instead of modifying the data immediately when we express some operation, we build up
a plan of transformations that we would like to apply to our source data. Spark, by waiting until the last minute to
execute the code, will compile this plan from your raw, DataFrame transformations, to an efficient physical plan that
will run as efficiently as possible across the cluster. This provides immense benefits to the end user because Spark
can optimize the entire data flow from end to end. An example of this is something called “predicate pushdown”
on DataFrames. If we build a large Spark job but specify a filter at the end that only requires us to fetch one row
from our source data, the most efficient way to execute this is to access the single record that we need. Spark will
actually optimize this for us by pushing the filter down automatically.
DATA ENGINEER S GUIDE T O 12
A PAC HE S PA R K A ND D ELTA L A K E

Actions Spark UI
Transformations allow us to build up our logical transformation plan. To trigger the During Spark’s execution of the previous code block, users can monitor the progress
computation, we run an action. An action instructs Spark to compute a result from of their job through the Spark UI. The Spark UI is available on port 4040 of the driver
a series of transformations. The simplest action is count which gives us the total node. If you are running in local mode this will just be the http://localhost:4040.
number of records in the DataFrame. The Spark UI maintains information on the state of our Spark jobs, environment, and
cluster state. It’s very useful, especially for tuning and debugging. In this case, we can
divisBy2.count() see one Spark job with two stages and nine tasks were executed.

We now see a result! There are 500 numbers divisible by two from 0 to 999 (big
surprise!). Now count is not the only action. There are three kinds of actions:
• actions to view data in the console;
• actions to collect data to native objects in the respective language;
• and actions to write to output data sources.

In specifying our action, we started a Spark job that runs our filter transformation
(a narrow transformation), then an aggregation (a wide transformation) that performs
the counts on a per partition basis, then a collect with brings our result to a native
This chapter avoids the details of Spark jobs and the Spark UI, we cover the Spark UI in
object in the respective language. We can see all of this by inspecting the Spark UI,
detail in Part IV: Production Applications. At this point you should understand that a
a tool included in Spark that allows us to monitor the Spark jobs running on a cluster.
Spark job represents a set of transformations triggered by an individual action and we
can monitor that from the Spark UI.
DATA ENGINEER S GUIDE T O 13
A PAC HE S PA R K A ND D ELTA L A K E

An End to End Example

In the previous example, we created a DataFrame of a range of numbers; not exactly groundbreaking big data. In this section we will reinforce everything we learned previously
in this chapter with a worked example and explaining step by step what is happening under the hood. We’ll be using some flight data available here from the United States
Bureau of Transportation statistics.

Inside of the CSV folder linked above, you’ll see that we have a number of files. You will also notice a number of other folders with different file formats that we will discuss in
Part II: Reading and Writing Data. We will focus on the CSV files.

Each file has a number of rows inside of it. Now these files are CSV files, meaning that they’re a semi-structured data format with a row in the file representing a row in our
future DataFrame.

$ head /mnt/defg/flight-data/csv/2015-summary.csv
DEST_COUNTRY_NAME,ORIGIN_COUNTRY_NAME,count
United States,Romania,15
United States,Croatia,1
United States,Ireland,344

Spark includes the ability to read and write from a large number of data sources. In order to read this data in, we will use a DataFrameReader that is associated with our
SparkSession. In doing so, we will specify the file format as well as any options we want to specify. In our case, we want to do something called schema inference, we want Spark
to take the best guess at what the schema of our DataFrame should be. The reason for this is that CSV files are not completely structured data formats. We also want to specify
that the first row is the header in the file, we’ll specify that as an option too.
DATA ENGINEER S GUIDE T O 14
A PAC HE S PA R K A ND D ELTA L A K E

To get this information Spark will read in a little bit of the data and then attempt to CSV FILE D ATA F R A M E
parse the types in those rows according to the types available in Spark. You’ll see that
this works just fine. We also have the option of strictly specifying a schema when we
Array(Row(...),Row(...))
read in data (which we recommend in production scenarios).

Read Take (N)

%scala
val flightData2015 = spark
.read
.option(“inferSchema”, “true”) Each of these DataFrames (in Scala and Python) each have a set of columns with an
.option(“header”, “true”) unspecified number of rows. The reason the number of rows is “unspecified” is because
.csv(“/mnt/defg/flight-data/csv/2015-summary.csv”) reading data is a transformation, and is therefore a lazy operation. Spark only peeked
at a couple of rows of data to try to guess what types each column should be.
%python
flightData2015 = spark\ If we perform the take action on the DataFrame, we will be able to see the same
.read\ results that we saw before when we used the command line.
.option(“inferSchema”, “true”)\
.option(“header”, “true”)\ flightData2015.take(3)
.csv(“/mnt/defg/flight-data/csv/2015-summary.csv”) Array([United States,Romania,15], [United States,Croatia...
DATA ENGINEER S GUIDE T O 15
A PAC HE S PA R K A ND D ELTA L A K E

Let’s specify some more transformations! Now we will sort our data according to the Nothing happens to the data when we call sort because it’s just a transformation.
count column which is an integer type. However, we can see that Spark is building up a plan for how it will execute this across
the cluster by looking at the explain plan. We can call explain on any DataFrame
N O T E | Remember, the sort does not modify the DataFrame. We use the sort is a object to see the DataFrame’s lineage (or how Spark will execute this query).
transformation that returns a new DataFrame by transforming the previous DataFrame.
Let’s illustrate what’s happening when we call take on that resulting DataFrame. flightData2015.sort(“count”).explain()

Congratulations, you’ve just read your first explain plan! Explain plans are a bit arcane,
but with a bit of practice it becomes second nature. Explain plans can be read from
CSV FILE D ATA F R A M E D ATA F R A M E
top to bottom, the top being the end result and the bottom being the source(s) of data.
In our case, just take a look at the first keywords. You will see “sort”, “exchange”, and
Array(...) “FileScan”. That’s because the sort of our data is actually a wide transformation be-
cause rows will have to be compared with one another. Don’t worry too much about
Read Sort take(3) understanding everything about explain plans at this point, they can just be helpful
(Narrow) (Wide) (Wide) tools for debugging and improving your knowledge as you progress with Spark.
DATA ENGINEER S GUIDE T O 16
A PAC HE S PA R K A ND D ELTA L A K E

Now, just like we did before, we can specify an action in order to kick off this plan. CSV FILE D ATA F R A M E D ATA F R A M E
However before doing that, we’re going to set a configuration. By default, when we
perform a shuffle Spark will output two hundred shuffle partitions. We will set this
Array(...)
value to five in order to reduce the number of the output partitions from the shuffle
from two hundred to five.
Read Sort take(3)

(Narrow) (Wide) (Wide)

spark.conf.set(“spark.sql.shuffle.partitions”, “5”)
flightData2015.sort(“count”).take(2)
1 PA R T I T I O N 5 PA R T I T I O N S
... Array([United States,Singapore,1], [Moldova,United States,1])

This operation is illustrated in the following image. You’ll notice that in addition to the
logical transformations, we include the physical partition count as well.

The logical plan of transformations that we build up defines a lineage for the DataFrame so that at any given point in time Spark knows how to recompute any partition by per-
forming all of the operations it had before on the same input data. This sits at the heart of Spark’s programming model, functional programming where the same inputs always
result in the same outputs when the transformations on that data stay constant.

We do not manipulate the physical data, but rather configure physical execution characteristics through things like the shuffle partitions parameter we set above. We got five
output partitions because that’s what we changed the shuffle partition value to. You can change this to help control the physical execution characteristics of your Spark jobs.
Go ahead and experiment with different values and see the number of partitions yourself. In experimenting with different values, you should see drastically different run times.
Remeber that you can monitor the job progress by navigating to the Spark UI on port 4040 to see the physical and logical execution characteristics of our jobs.
DATA ENGINEER S GUIDE T O 17
A PAC HE S PA R K A ND D ELTA L A K E

DataFrames and SQL

We worked through a simple example in the previous example, let’s now work through Now we can query our data in SQL. To execute a SQL query, we’ll use the spark.sql
a more complex example and follow along in both DataFrames and SQL. Spark the function (remember spark is our SparkSession variable?) that conveniently, returns a
same transformations, regardless of the language, in the exact same way. You can new DataFrame. While this may seem a bit circular in logic – that a SQL query against
express your business logic in SQL or DataFrames (either in R, Python, Scala, or a DataFrame returns another DataFrame, it’s actually quite powerful. As a user, you
Java) and Spark will compile that logic down to an underlying plan (that we see in the can specify transformations in the manner most convenient to you at any given
explain plan) before actually executing your code. Spark SQL allows you as a user to point in time and not have to trade any efficiency to do so! To understand that this is
register any DataFrame as a table or view (a temporary table) and query it using pure happening, let’s take a look at two explain plans.
SQL. There is no performance difference between writing SQL queries or writing
DataFrame code, they both “compile” to the same underlying plan that we specify in
DataFrame code.

Any DataFrame can be made into a table or view with one simple method call.

%scala
flightData2015.createOrReplaceTempView(“flight_data_2015”)

%python
flightData2015.createOrReplaceTempView(“flight_data_2015”)
DATA ENGINEER S GUIDE T O 18
A PAC HE S PA R K A ND D ELTA L A K E

%scala == Physical Plan ==

val sqlWay = spark.sql(“”” *HashAggregate(keys=[DEST_COUNTRY_NAME#182], functions=[count(1)])
SELECT DEST_COUNTRY_NAME, count(1) +- Exchange hashpartitioning(DEST_COUNTRY_NAME#182, 5)
FROM flight_data_2015 +- *HashAggregate(keys=[DEST_COUNTRY_NAME#182], functions=[partial_
GROUP BY DEST_COUNTRY_NAME count(1)])
“””) +- *FileScan csv [DEST_COUNTRY_NAME#182] ...
val dataFrameWay = flightData2015 == Physical Plan ==
.groupBy(‘DEST_COUNTRY_NAME) *HashAggregate(keys=[DEST_COUNTRY_NAME#182], functions=[count(1)])
.count() +- Exchange hashpartitioning(DEST_COUNTRY_NAME#182, 5)
sqlWay.explain +- *HashAggregate(keys=[DEST_COUNTRY_NAME#182], functions=[partial_
dataFrameWay.explain count(1)])
+- *FileScan csv [DEST_COUNTRY_NAME#182] ...
%python
sqlWay = spark.sql(“””
SELECT DEST_COUNTRY_NAME, count(1)
FROM flight_data_2015
GROUP BY DEST_COUNTRY_NAME
“””)
dataFrameWay = flightData2015\
.groupBy(“DEST_COUNTRY_NAME”)\
.count()
sqlWay.explain()
dataFrameWay.explain()
DATA ENGINEER S GUIDE T O 19
A PAC HE S PA R K A ND D ELTA L A K E

We can see that these plans compile to the exact same underlying plan! Great, that’s a simple example. Let’s perform something a bit more complicated
and find out the top five destination countries in the data? This is our first multi-
To reinforce the tools available to us, let’s pull out some interesting statistics from transformation query so we’ll take it step by step. We will start with a fairly
our data. One thing to understand is that DataFrames (and SQL) in Spark already straightforward SQL aggregation.
have a huge number of manipulations available. There are hundreds of functions
that you can leverage and import to help you resolve your big data problems faster. %scala
We will use the max function, to find out what the maximum number of flights to val maxSql = spark.sql(“””
and from any given location are. This just scans each value in relevant column the SELECT DEST_COUNTRY_NAME, sum(count) as destination_total
DataFrame and sees if it’s bigger than the previous values that have been seen. This is FROM flight_data_2015
a transformation, as we are effectively filtering down to one row. Let’s see what that GROUP BY DEST_COUNTRY_NAME
looks like. ORDER BY sum(count) DESC
LIMIT 5
spark.sql(“SELECT max(count) from flight_data_2015”).take(1) “””)
maxSql.collect()
%scala
import org.apache.spark.sql.functions.max %python
flightData2015.select(max(“count”)).take(1) maxSql = spark.sql(“””
SELECT DEST_COUNTRY_NAME, sum(count) as destination_total
%python FROM flight_data_2015
from pyspark.sql.functions import max GROUP BY DEST_COUNTRY_NAME
flightData2015.select(max(“count”)).take(1) ORDER BY sum(count) DESC
LIMIT 5
“””)
maxSql.collect()
DATA ENGINEER S GUIDE T O 20
A PAC HE S PA R K A ND D ELTA L A K E

Now let’s move to the DataFrame syntax that is semantically similar but slightly
different in implementation and ordering. But, as we mentioned, the underlying plans
for both of them are the same. Let’s execute the queries and see their results as a
sanity check.

%scala %python
import org.apache.spark.sql.functions.desc from pyspark.sql.functions import desc
flightData2015 flightData2015\
.groupBy(“DEST_COUNTRY_NAME”) .groupBy(“DEST_COUNTRY_NAME”)\
.sum(“count”) .sum(“count”)\
.withColumnRenamed(“sum(count)”, “destination_total”) .withColumnRenamed(“sum(count)”, “destination_total”)\
.sort(desc(“destination_total”)) .sort(desc(“destination_total”))\
.limit(5) .limit(5)\
.collect() .collect()
DATA ENGINEER S GUIDE T O 21
A PAC HE S PA R K A ND D ELTA L A K E

Now there are 7 steps that take us all the way back to the source data. You can see The first step is to read in the data. We defined the DataFrame previously but, as a
this in the explain plan on those DataFrames. Illustrated below are the set of steps reminder, Spark does not actually read it in until an action is called on that DataFrame
that we perform in “code”. The true execution plan (the one visible in explain) will differ or one derived from the original DataFrame.
from what we have below because of optimizations in physical execution, however
the illustration is as good of a starting point as any. This execution plan is a directed The second step is our grouping, technically when we call groupBy we end up with a
acyclic graph (DAG) of transformations, each resulting in a new immutable DataFrame, RelationalGroupedDataset which is a fancy name for a DataFrame that has a group-
on which we call an action to generate a result. ing specified but needs the user to specify an aggregation before it can be queried
further. We can see this by trying to perform an action on it (which will not work). We
basically specified that we’re going to be grouping by a key (or set of keys) and that
O N E O P E R AT I O N now we’re going to perform an aggregation over each one of those keys.

Read GroupBy Sum Therefore the third step is to specify the aggregation. Let’s use the sum aggregation
method. This takes as input a column expression or simply, a column name. The result
CSV FILE D ATA F R A M E G R O U P E D D ATA S E T D ATA F R A M E
of the sum method call is a new dataFrame. You’ll see that it has a new schema but
that it does know the type of each column. It’s important to reinforce (again!) that no
computation has been performed. This is simply another transformation that we’ve
expressed and Spark is simply able to trace the type information we have supplied.
Rename
D ATA F R A M E D ATA F R A M E D A T A F R A M E Column

Array(...)

Collect Limit Sort

DATA ENGINEER S GUIDE T O 22
A PAC HE S PA R K A ND D ELTA L A K E

The fourth step is a simple renaming, we use the withColumnRenamed method that takes two arguments, the original column name and the new column name. Of course, this
doesn’t perform computation — this is just another transformation!

The fifth step sorts the data such that if we were to take results off of the top of the DataFrame, they would be the largest values found in the destination_total column.

You likely noticed that we had to import a function to do this, the desc function. You might also notice that desc does not return a string but a Column. In general, many
DataFrame methods will accept Strings (as column names) or Column types or expressions. Columns and expressions are actually the exact same thing.

Penultimately, we’ll specify a limit. This just specifies that we only want five values. This is just like a filter except that it filters by position instead of by value. It’s safe to say that
it basically just specifies a DataFrame of a certain size.

The last step is our action! Now we actually begin the process of collecting the results of our DataFrame above and Spark will give us back a list or array in the language that
we’re executing. Now to reinforce all of this, let’s look at the explain plan for the above query.

%scala %python
flightData2015 flightData2015\
.groupBy(“DEST_COUNTRY_NAME”) .groupBy(“DEST_COUNTRY_NAME”)\
.sum(“count”) .sum(“count”)\
.withColumnRenamed(“sum(count)”, “destination_total”) .withColumnRenamed(“sum(count)”, “destination_total”)\
.sort(desc(“destination_total”)) .sort(desc(“destination_total”))\
.limit(5) .limit(5)\
.explain() .explain()
DATA ENGINEER S GUIDE T O 23
A PAC HE S PA R K A ND D ELTA L A K E

== Physical Plan ==
TakeOrderedAndProject(limit=5, orderBy=[destination_total#16194L DESC], output=[DEST_COUNTRY_NAME#7323,...
+- *HashAggregate(keys=[DEST_COUNTRY_NAME#7323], functions=[sum(count#7325L)])
+- Exchange hashpartitioning(DEST_COUNTRY_NAME#7323, 5)
+- *HashAggregate(keys=[DEST_COUNTRY_NAME#7323], functions=[partial
sum(count#7325L)])
+- InMemoryTableScan [DEST_COUNTRY_NAME#7323, count#7325L]
+- InMemoryRelation [DEST_COUNTRY_NAME#7323, ORIGIN_COUNTRY_NAME#7324, count#7325L]...
+- *Scan csv [DEST_COUNTRY_NAME#7578,ORIGIN_COUNTRY_NAME#7579,count#7580L]...

While this explain plan doesn’t match our exact “conceptual plan” all of the pieces are there. You can see the limit statement as well as the orderBy (in the first line). You can also
see how our aggregation happens in two phases, in the partial_sum calls. This is because summing a list of numbers is commutative and Spark can perform the sum, parti-
tion by partition. Of course we can see how we read in the DataFrame as well.

Naturally, we don’t always have to collect the data. We can also write it out to any data source that Spark supports. For instance, let’s say that we wanted to store the information
in a database like PostgreSQL or write them out to another file.
DATA ENGINEER S GUIDE T O 24
A PAC HE S PA R K A ND D ELTA L A K E

CHAPTER 2: A Tour of Spark’s Toolset In the previous chapter we introduced Spark’s core concepts, like transformations and actions, in the
context of Spark’s Structured APIs. These simple conceptual building blocks are the foundation of Apache
Spark’s vast ecosystem of tools and libraries. Spark is composed of the simple primitives, the lower level
APIs and the Structured APIs, then a series of “standard libraries” included in Spark.

Developers use these tools for a variety of different tasks, from graph analysis and machine learning to
Advanced analytics Ecosystem
Structured
ML graph + streaming and integrations with a host of libraries and databases. This chapter will present a whirlwind
streaming
Deep learning Packages
tour of much of what Spark has to offer. Each section in this chapter are elaborated upon by other parts of
this book, this chapter is simply here to show you what’s possible.
STRUCTURED APIS

Datasets DataFrames SQL This chapter will cover:

• Production applications with spark-submit
• Datasets: structured and type safe APIs
LOW LEVEL APIS
• Structured Streaming
Distributed variables RDDs
• Machine learning and advanced analytics
• Spark’s lower level APIs
• SparkR
• Spark’s package ecosystem

The entire book covers these topics in depth, the goal of this chapter is simply to provide a whirlwind tour
of Spark. Once you’ve gotten the tour, you’ll be able to jump to many different parts of the book to find
answers to your questions about particular topics. This chapter aims for breadth, instead of depth. Let’s
get started!
DATA ENGINEER S GUIDE T O 25
A PAC HE S PA R K A ND D ELTA L A K E

Production Applications
Spark makes it easy to make simple to reason about and simple to evolve big data ./bin/spark-submit \
programs. Spark also makes it easy to turn in your interactive exploration into --class org.apache.spark.examples.SparkPi \
production applications with a tool called spark-submit that is included in the core --master local \
of Spark. spark-submit does one thing, it allows you to submit your applications to a ./examples/jars/spark-examples_2.11-2.2.0.jar 10
currently managed cluster to run. When you submit this, the application will run until
the application exists or errors. You can do this with all of Spark’s support cluster What this will do is calculate the digits of pi to a certain level of estimation. What
managers including Standalone, Mesos, and YARN. we’ve done here is specified that we want to run it on our local machine, specified
which class and which jar we would like to run as well as any command line
In the process of doing so, you have a number of knobs that you can turn and control arguments to that particular class.
to specify the resources this application has as well, how it should be run, and the
parameters for your specific application. We can do this in Python with the following command line arguments.

You can write these production applications in any of Spark’s supported languages ./bin/spark-submit \
and then submit those applications for execution. The simplest example is one that --master local \
you can do on your local machine by running the following command line snippet on ./examples/src/main/python/pi.py 10
your local machine in the directory into which you downloaded Spark.
By swapping out the path to the file and the cluster configurations, we can write and
run production applications. Now Spark provides a lot more than just DataFrames
that we can run as production applications. The rest of this chapter will walk through
several different APIs that we can leverage to run all sorts of production applications.
DATA ENGINEER S GUIDE T O 26
A PAC HE S PA R K A ND D ELTA L A K E

Datasets: Type-Safe Structured APIs

The next topic we’ll cover is a type-safe version of Spark’s structured API for Java and The awesome thing about Datasets is that we can use them only when we need or
Scala, called Datasets. This API is not available in Python and R, because those are want to. For instance, in the follow example I’ll define my own object and manipulate it
dynamically typed languages, but it is a powerful tool for writing large applications in via arbitrary map and filter functions. Once we’ve performed our manipulations, Spark
Scala and Java. can automatically turn it back into a DataFrame and we can manipulate it further
using the hundreds of functions that Spark includes. This makes it easy to drop down
Recall that DataFrames, which we saw earlier, are a distributed collection of objects to lower level, perform type-safe coding when necessary, and move higher up to SQL
of type Row, which can hold various types of tabular data. The Dataset API allows for more rapid analysis. We cover this material extensively in the next part of this
users to assign a Java class to the records inside a DataFrame, and manipulate it book, but here is a small example showing how we can use both type-safe functions
as a collection of typed objects, similar to a Java ArrayList or Scala Seq. The APIs and DataFrame-like SQL expressions to quickly write business logic.
available on Datasets are type-safe, meaning that you cannot accidentally view the
objects in a Dataset as being of another class than the class you put in initially. This
makes Datasets especially attractive for writing large applications where multiple
software engineers must interact through well-defined interfaces.

The Dataset class is parametrized with the type of object contained inside:
Dataset<T> in Java and Dataset[T] in Scala. As of Spark 2.0, the types T supported
are all classes following the JavaBean pattern in Java, and case classes in Scala.
These types are restricted because Spark needs to be able to automatically analyze
the type T and create an appropriate schema for the tabular data inside your Dataset.
DATA ENGINEER S GUIDE T O 27
A PAC HE S PA R K A ND D ELTA L A K E

%scala
// A Scala case class (similar to a struct) that will automatically
// be mapped into a structured data table in Spark
case class Flight(DEST_COUNTRY_NAME: String, ORIGIN_COUNTRY_NAME: String, count: BigInt)
val flightsDF = spark.read.parquet(“/mnt/defg/flight-data/parquet/2010-summary.parquet/”)
val flights = flightsDF.as[Flight]

One final advantage is that when you call collect or take on a Dataset, we’re going to collect to objects of the proper type in your Dataset, not DataFrame Rows. This makes it
easy to get type safety and safely perform manipulation in a distributed and a local manner without code changes.

%scala
flights
.filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME != “Canada”)
.take(5)
DATA ENGINEER S GUIDE T O 28
A PAC HE S PA R K A ND D ELTA L A K E

Structured Streaming
Structured Streaming is a high-level API for stream processing that became Let’s walk through a simple example of how easy it is to get started with Structured
production-ready in Spark 2.2. Structured Streaming allows you to take the same Streaming. For this we will use a retail dataset. One that has specific dates and times
operations that you perform in batch mode using Spark’s structured APIs, and run for us to be able to use. We will use the “by-day” set of files where one file represents
them in a streaming fashion. This can reduce latency and allow for incremental one day of data.
processing. The best thing about Structured Streaming is that it allows you to rapidly
and quickly get value out of streaming systems with virtually no code changes. It We put it in this format to simulate data being produced in a consistent and regular
also makes it easy to reason about because you can write your batch job as a way to manner by a different process. Now this is retail data so imagine that these are
prototype it and then you can convert it to streaming job. The way all of this works is being produced by retail stores and sent to a location where they will be read by our
by incrementally processing that data. Structured Streaming job.

It’s worth sharing a sample of the data so you can reference what the data looks like.

InvoiceNo,StockCode,Description,Quantity,InvoiceDate,UnitPrice,CustomerID,Country
536365,85123A,WHITE HANGING HEART T-LIGHT HOLDER,6,2010-12-01 08:26:00,2.55,17850.0,United Kingdom
536365,71053,WHITE METAL LANTERN,6,2010-12-01 08:26:00,3.39,17850.0,United Kingdom
536365,84406B,CREAM CUPID HEARTS COAT HANGER,8,2010-12-01 08:26:00,2.75,17850.0,United Kingdom
DATA ENGINEER S GUIDE T O 29
A PAC HE S PA R K A ND D ELTA L A K E

Now in order to ground this, let’s first analyze the data as a static dataset and create Now since we’re working with time series data it’s worth mentioning how we might
a DataFrame to do so. We’ll also create a schema from this static dataset. There are go along grouping and aggregating our data. In this example we’ll take a look at the
ways of using schema inference with streaming that we will touch on in the Part V of largest sale hours where a given customer (identified by CustomerId) makes a large
this book. purchase. For example, let’s add a total cost column and see on what days a customer
spent the most.
%scala
val staticDataFrame = spark.read.format(“csv”) The window function will include all data from each day in the aggregation. It’s
.option(“header”, “true”) simply a window over the time series column in our data. This is a helpful tool for
.option(“inferSchema”, “true”) manipulating date and timestamps because we can specify our requirements in a
.load(“/mnt/defg/retail-data/by-day/*.csv”) more human form (via intervals) and Spark will group all of them together for us.
staticDataFrame.createOrReplaceTempView(“retail_data”)
val staticSchema = staticDataFrame.schema

%python
staticDataFrame = spark.read.format(“csv”)\
.option(“header”, “true”)\
.option(“inferSchema”, “true”)\
.load(“/mnt/defg/retail-data/by-day/*.csv”)
staticDataFrame.createOrReplaceTempView(“retail_data”)
staticSchema = staticDataFrame.schema
DATA ENGINEER S GUIDE T O 30
A PAC HE S PA R K A ND D ELTA L A K E

%scala It’s worth mentioning that we can also run this as SQL code, just as we saw in the
import org.apache.spark.sql.functions.{window, column, desc, col} previous chapter.
staticDataFrame
.selectExpr( Here’s a sample of the output that you’ll see.
“CustomerId”,
“(UnitPrice * Quantity) as total_cost”, +----------+--------------------+------------------+
“InvoiceDate”) |CustomerId| window| sum(total_cost)|
+----------+--------------------+------------------+
.groupBy(
| 17450.0|[2011-09-20 00:00...| 71601.44|
col(“CustomerId”), window(col(“InvoiceDate”), “1 day”)) | null|[2011-11-14 00:00...| 55316.08|
.sum(“total_cost”) | null|[2011-11-07 00:00...| 42939.17|
.show(5) | null|[2011-03-29 00:00...| 33521.39999999998|
| null|[2011-12-08 00:00...|31975.590000000007|
+----------+--------------------+------------------+
%python
from pyspark.sql.functions import window, column, desc, col
The null values represent the fact that we don’t have a customerId for some
staticDataFrame\
transactions.
.selectExpr(
“CustomerId”,
That’s the static DataFrame version, there shouldn’t be any big surprises in there
“(UnitPrice * Quantity) as total_cost” ,
if you’re familiar with the syntax. Now we’ve seen how that works, let’s take a
“InvoiceDate” )\
look at the streaming code! You’ll notice that very little actually changes about
.groupBy(
our code. The biggest change is that we used readStream instead of read,
col(“CustomerId”), window(col(“InvoiceDate”), “1 day”))\
additionally you’ll notice maxFilesPerTrigger option which simply specifies the
.sum(“total_cost”)\
number of files we should read in at once. This is to make our demonstration more
.show(5)
“streaming” and in a production scenario this would be omitted.
DATA ENGINEER S GUIDE T O 31
A PAC HE S PA R K A ND D ELTA L A K E

Now since you’re likely running this in local mode, it’s a good practice to set the Now we can see the DataFrame is streaming.
number of shuffle partitions to something that’s going to be a better fit for local mode.
This configuration simple specifies the number of partitions that should be created streamingDataFrame.isStreaming // returns true
after a shuffle, by default the value is two hundred but since there aren’t many
executors on this machine it’s worth reducing this to five. We did this same operation Let’s set up the same business logic as the previous DataFrame manipulation, we’ll
in the previous chapter, so if you don’t remember why this is important feel free to flip perform a summation in the process.
back to the previous chapter to review.
%scala
%scala val purchaseByCustomerPerHour = streamingDataFrame
val streamingDataFrame = spark.readStream .selectExpr(
.schema(staticSchema) “CustomerId”,
.option(“maxFilesPerTrigger”, 1) “(UnitPrice * Quantity) as total_cost”,
.format(“csv”) “InvoiceDate”)
.option(“header”, “true”) .groupBy(
.load(“d/mnt/defg/retail-data/by-day/*.csv”) $”CustomerId”, window($”InvoiceDate”, “1 day”))
.sum(“total_cost”)
%python
streamingDataFrame = spark.readStream\ %python
.schema(staticSchema)\ purchaseByCustomerPerHour = streamingDataFrame\
.option(“maxFilesPerTrigger”, 1)\ .selectExpr(
.format(“csv”)\ “CustomerId”,
.option(“header”, “true”)\ “(UnitPrice * Quantity) as total_cost” ,
.load(“/mnt/defg/retail-data/by-day/*.csv”) “InvoiceDate” )\
.groupBy(
col(“CustomerId”), window(col(“InvoiceDate”), “1 day”))\
.sum(“total_cost”)
DATA ENGINEER S GUIDE T O 32
A PAC HE S PA R K A ND D ELTA L A K E

This is still a lazy operation, so we will need to call a streaming action to start the execution of this data flow.

N O T E | Before kicking off the stream, we will set a small optimization that will allow this to run better on a single machine. This simply limits the number of output partitions after a
shuffle, a concept we discussed in the last chapter. We discuss this in Part VI of the book.

spark.conf.set(“spark.sql.shuffle.partitions”, “5”)

Streaming actions are a bit different from our conventional static action because we’re going to be populating data somewhere instead of just calling something like count
(which doesn’t make any sense on a stream anyways). The action we will use will out to an in-memory table that we will update after each trigger. In this case, each trigger is
based on an individual file (the read option that we set). Spark will mutate the data in the in-memory table such that we will always have the highest value as specified in our
aggregation above.

%scala
purchaseByCustomerPerHour.writeStream
.format(“memory”) // memory = store in-memory table
.queryName(“customer_purchases”) // counts = name of the in-memory table
.outputMode(“complete”) // complete = all the counts should be in the table
.start()

%python
purchaseByCustomerPerHour.writeStream\
.format(“memory”)\
.queryName(“customer_purchases”)\
.outputMode(“complete”)\
.start()
DATA ENGINEER S GUIDE T O 33
A PAC HE S PA R K A ND D ELTA L A K E

Once we start the stream, we can run queries against the stream to debug what our You’ll notice that as we read in more data - the composition of our table changes! With
result will look like if we were to write this out to a production sink. each file the results may or may not be changing based on the data. Naturally since
we’re grouping customers we hope to see an increase in the top customer purchase
%scala amounts over time (and do for a period of time!). Another option you can use is to just
spark.sql(“”” simply write the results out to the console.
SELECT *
FROM customer_purchases purchaseByCustomerPerHour.writeStream
ORDER BY `sum(total_cost)` DESC .format(“console”)
“””) .queryName(“customer_purchases_2”)
.show(5) .outputMode(“complete”)
.start()
%python
spark.sql(“”” Neither of these streaming methods should be used in production but they do make
SELECT * for convenient demonstration of Structured Streaming’s power. Notice how this win-
FROM customer_purchases dow is built on event time as well, not the time at which the data Spark processes the
ORDER BY `sum(total_cost)` DESC data. This was one of the shortcoming of Spark Streaming that Structured Streaming
“””)\ as resolved. We cover Structured Streaming in depth in Part V of this book.
.show(5)
DATA ENGINEER S GUIDE T O 34
A PAC HE S PA R K A ND D ELTA L A K E

Machine Learning and Advanced Analytics

Another popular aspect of Spark is its ability to perform large scale machine learning Spark includes a number of preprocessing methods out of the box. To demonstrate
with a built-in library of machine learning algorithms called MLlib. MLlib allows for these methods, we will start with some raw data, build up transformations before
preprocessing, munging, training of models, and making predictions at scale on getting the data into the right format at which point we can actually train our model
data. You can even use models trained in MLlib to make predictions in Strucutred and then serve predictions.
Streaming. Spark provides a sophisticated machine learning API for performing a
variety of machine learning tasks, from classification to regression, clustering to deep staticDataFrame.printSchema()
learning. To demonstrate this functionality, we will perform some basic clustering on
our data using a common algorithm called K-Means. root
|-- InvoiceNo: string (nullable = true)
|-- StockCode: string (nullable = true)
WHAT IS K-MEANS? K-means is a clustering algorithm where “K” centers are
|-- Description: string (nullable = true)
randomly assigned within the data. The points closest to that point are then
|-- Quantity: integer (nullable = true)
“assigned” to a particular cluster. Then a new center for this cluster is computed
|-- InvoiceDate: timestamp (nullable = true)
(called a centroid). We then label the points closest to that centroid, to the
|-- UnitPrice: double (nullable = true)
centroid’s class, and shift the centroid to the new center of that cluster of points.
|-- CustomerID: double (nullable = true)
We repeat this process for a finite set of iterations or until convergence (where
|-- Country: string (nullable = true)
our centroid and clusters stop changing.
DATA ENGINEER S GUIDE T O 35
A PAC HE S PA R K A ND D ELTA L A K E

Machine learning algorithms in MLlib require data to be represented as numerical Now we are also going to need to split our data into training and test sets. In this
values. Our current data is represented by a variety of different types including instance we are going to do this manually by the data that a certain purchase
timestamps, integers, and strings. Therefore we need to transform this data into occurred however we could also leverage MLlib’s transformation APIs to create a
some numerical representation. In this instance, we will use several DataFrame training and test set via train validation splits or cross validation. These topics are
transformations to manipulate our date data. covered extensively in Part VI of this book.

%scala %scala
import org.apache.spark.sql.functions.date_format val trainDataFrame = preppedDataFrame
val preppedDataFrame = staticDataFrame .where(“InvoiceDate < ‘2011-07-01’”)
.na.fill(0) val testDataFrame = preppedDataFrame
.withColumn(“day_of_week”, date_format($“InvoiceDate”, “EEEE”)) .where(“InvoiceDate >= ‘2011-07-01’”)
.coalesce(5)
%python
%python trainDataFrame = preppedDataFrame\
from pyspark.sql.functions import date_format, col .where(“InvoiceDate < ‘2011-07-01’”)
preppedDataFrame = staticDataFrame\ testDataFrame = preppedDataFrame\
.na.fill(0)\ .where(“InvoiceDate >= ‘2011-07-01’”)
.withColumn(“day_of_week”, date_format(col(“InvoiceDate”), “EEEE”))\
.coalesce(5)
DATA ENGINEER S GUIDE T O 36
A PAC HE S PA R K A ND D ELTA L A K E

Now that we prepared our data, let’s split it into a training and test set. Since this is a This will turn our days of weeks into corresponding numerical values. For example,
time-series set of data, we will split by an arbitrary date in the dataset. While this may Spark may represent Saturday as 6 and Monday as 1. However with this numbering
not be the optimal split for our training and test, for the intents and purposes of this scheme, we are implicitly stating that Saturday is greater than Monday (by
example it will work just fine. We’ll see that this splits our dataset roughly in half. pure numerical values). This is obviously incorrect. Therefore we need to use a
OneHotEncoder to encode each of these values as their own column. These boolean
trainDataFrame.count() flags state whether that day of week is the relevant day of the week.

trainDataFrame.count() %scala
import org.apache.spark.ml.feature.OneHotEncoder
Now these transformations are DataFrame transformations, covered extensively in val encoder = new OneHotEncoder()
part two of this book. Spark’s MLlib also provides a number of transformations that .setInputCol(“day_of_week_index”)
allow us to automate some of our general transformations. One such transformer is a .setOutputCol(“day_of_week_encoded”)
StringIndexer.
%python
%scala from pyspark.ml.feature import OneHotEncoder
import org.apache.spark.ml.feature.StringIndexer encoder = OneHotEncoder()\
val indexer = new StringIndexer() .setInputCol(“day_of_week_index”)\
.setInputCol(“day_of_week”) .setOutputCol(“day_of_week_encoded”)
.setOutputCol(“day_of_week_index”)

%python
from pyspark.ml.feature import StringIndexer
indexer = StringIndexer()\
.setInputCol(“day_of_week”)\
.setOutputCol(“day_of_week_index”)
DATA ENGINEER S GUIDE T O 37
A PAC HE S PA R K A ND D ELTA L A K E

Each of these will result in a set of columns that we will “assemble” into a vector. All We can see that we have 3 key features, the price, the quantity, and the day of week.
machine learning algorithms in Spark take as input a Vector type, which must be a Now we’ll set this up into a pipeline so any future data we need to transform can go
set of numerical values. through the exact same process.

%scala %scala
import org.apache.spark.ml.feature.VectorAssembler import org.apache.spark.ml.Pipeline
val vectorAssembler = new VectorAssembler() val transformationPipeline = new Pipeline()
.setInputCols(Array(“UnitPrice”, “Quantity”, “day_of_week_encoded”)) .setStages(Array(indexer, encoder, vectorAssembler))
.setOutputCol(“features”)
%python
%python from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler transformationPipeline = Pipeline()\
vectorAssembler = VectorAssembler()\ .setStages([indexer, encoder, vectorAssembler])
.setInputCols([“UnitPrice”, “Quantity”, “day_of_week_encoded”])\
.setOutputCol(“features”)
DATA ENGINEER S GUIDE T O 38
A PAC HE S PA R K A ND D ELTA L A K E

Now preparing for training is a two step process. We first need to fit our transformers This will put a copy of this intermediately transformed dataset into memory, allowing
to this dataset. We cover this in depth, but basically our StringIndexer needs to us to repeatedly access it at much lower cost than running the entire pipeline again.
know how many unique values there are to be index. Once those exist, encoding is If you’re curious to see how much of a difference this makes, skip this line and run
easy but Spark must look at all the distinct values in the column to be indexed in order the training without caching the data. Then try it after caching, you’ll see the results
to store those values later on. are significant.

%scala transformedTraining.cache()
val fittedPipeline = transformationPipeline.fit(trainDataFrame)
Now we have a training set, now it’s time to train the model. First we’ll import the
%python relevant model that we’d like to use and instantiate it.
fittedPipeline = transformationPipeline.fit(trainDataFrame)
%scala
Once we fit the training data, we are now create to take that fitted pipeline and use it import org.apache.spark.ml.clustering.KMeans
to transform all of our data in a consistent and repeatable way. val kmeans = new KMeans()
.setK(20)
%scala .setSeed(1L)
val transformedTraining = fittedPipeline.transform(trainDataFrame)
%python
%python from pyspark.ml.clustering import KMeans
transformedTraining = fittedPipeline.transform(trainDataFrame) kmeans = KMeans()\
.setK(20)\
At this point, it’s worth mentioning that we could have included our model training in .setSeed(1L)
our pipeline. We chose not to in order to demonstrate a use case for caching the data.
At this point, we’re going to perform some hyperparameter tuning on the model, since
we do not want to repeat the exact same transformations over and over again, we’ll
leverage an optimization we discuss in Part IV of this book, caching.
DATA ENGINEER S GUIDE T O 39
A PAC HE S PA R K A ND D ELTA L A K E

In Spark, training machine learning models is a two phase process. First we initialize We can see the resulting cost at this point. Which is quite high, that’s likely because
an untrained model, then we train it. There are always two types for every algorithm we didn’t necessary scale our data or transform.
in MLlib’s DataFrame API. They following the naming pattern of Algorithm, for the
untrained version, and AlgorithmModel for the trained version. In our case, this is kmModel.computeCost(transformedTraining)
KMeans and then KMeansModel.
%scala
Predictors in MLlib’s DataFrame API share roughly the same interface that we saw val transformedTest = fittedPipeline.transform(testDataFrame)
above with our preprocessing transformers like the StringIndexer. This should
come as no surprise because it makes training an entire pipeline (which includes the %python
model) simple. In our case we want to do things a bit more step by step, so we chose transformedTest = fittedPipeline.transform(testDataFrame)
to not do this at this point.
kmModel.computeCost(transformedTest)
%scala
val kmModel = kmeans.fit(transformedTraining) Naturally we could continue to improve this model, layering more preprocessing as
well as performing hyperparameter tuning to ensure that we’re getting a good model.
%python We leave that discussion for Part VI of this book.
kmModel = kmeans.fit(transformedTraining)
DATA ENGINEER S GUIDE T O 40
A PAC HE S PA R K A ND D ELTA L A K E

Lower Level APIs

Spark includes a number of lower level primitives to allow for arbitrary Java and RDDs are available in Scala as well as Python. However, they’re not equivalent. This
Python object manipulation via Resilient Distributed Datasets (RDDs). Virtually differs from the DataFrame API (where the execution characteristics are the same)
everything in Spark is built on top of RDDs. As we will cover in the next chapter, due to some underlying implementation details. We cover lower level APIs, including
DataFrame operations are built on top of RDDs and compile down to these lower level RDDs in Part IV of this book. As end users, you shouldn’t need to use RDDs much in
tools for convenient and extremely efficient distributed execution. There are some order to perform many tasks unless you’re maintaining older Spark code. There are
things that you might use RDDs for, especially when you’re reading or manipulating basically no instances in modern Spark where you should be using RDDs instead
raw data, but for the most part you should stick to the Structured APIs. RDDs are of the structured APIs beyond manipulating some very raw unprocessed and
lower level that DataFrames because they reveal physical execution characteristics unstructured data.
(like partitions) to end users.

One thing you might use RDDs for is to parallelize raw data you have stored in memory
on the driver machine. For instance let’s parallelize some simple numbers and create
a DataFrame after we do so. We can then convert that to a DataFrame to use it with
other DataFrames.

%scala
spark.sparkContext.parallelize(Seq(1, 2, 3)).toDF()

%python
from pyspark.sql import Row
spark.sparkContext.parallelize([Row(1), Row(2), Row(3)]).toDF()
DATA ENGINEER S GUIDE T O 41
A PAC HE S PA R K A ND D ELTA L A K E

SparkR
SparkR is a tool for running R on Spark. It follows the same principles as all of Spark’s %r
other language bindings. To use SparkR, we simply import it into our environment library(magrittr)
and run our code. It’s all very similar to the Python API except that it follows R’s sparkDF %>%
syntax instead of Python. For the most part, almost everything available in Python is orderBy(desc(sparkDF$count)) %>%
available in SparkR. groupBy(“ORIGIN_COUNTRY_NAME”) %>%
count() %>%
%r limit(10) %>%
library(SparkR) collect()
sparkDF <- read.df(“/mnt/defg/flight-data/csv/2015-summary.csv”,
source = “csv”, header=”true”, inferSchema = “true”) We cover SparkR more in the Ecosystem Part of this book along with short discussion
take(sparkDF, 5) of PySpark specifics (PySpark is covered heavily through this book), and the new
sparklyr package.
%r
collect(orderBy(sparkDF, “count”), 20) Spark’s Ecosystem and Packages
One of the best parts about Spark is the ecosystem of packages and tools that the
R users can also leverage other R libraries like the pipe operator in magrittr in order to community has created. Some of these tools even move into the core Spark project
make Spark transformations a bit more R like. This can make it easy to use with other as they mature and become widely used. The list of packages is rather large at over
libraries like ggplot for more sophisticated plotting. 300 at the time of this writing and more are added frequently. The largest index of
Spark Packages can be found at spark-packages.org, where any user can publish to
this package repository. There are also various other projects and packages that can
be found through the web, for example on GitHub.
DATA ENGINEER S GUIDE T O 42
A PAC HE S PA R K A ND D ELTA L A K E

CHAPTER 3: Working with Different In the previous chapter, we covered basic DataFrame concepts and abstractions. This chapter will cover building
Types of Data expressions, which are the bread and butter of Spark’s structured operations.

This chapter will cover working with a variety of different kinds of data including:
• Booleans
• Numbers
• Strings
• Dates and Timestamps
• Handling Null
• Complex Types
• User Defined Functions
DATA ENGINEER S GUIDE T O 43
A PAC HE S PA R K A ND D ELTA L A K E

Where to Look for APIs

Before we get started, it’s worth explaining where you as a user should start looking org.apache.spark.sql.functions contains a variety of functions for a variety of
for transformations. Spark is a growing project and any book (including this one) is a different data types. Often you’ll see the entire package imported because they are
snapshot in time. Therefore it is our priority to educate you as a user as to where you used so often. These are available here.
should look for functions in order to transform your data. The key places to look for
transformations are: Now this may feel a bit overwhelming but have no fear, the majority of these functions
are ones that you will find in SQL and analytics systems. All of these tools exist to
DataFrame (Dataset) Methods. This is actually a bit of a trick because a DataFrame achieve one purpose, to transform rows of data in one format or structure to another.
is just a Dataset of Row types so you’ll actually end up looking at the Dataset This may create more rows or reduce the number of rows available. To get started,
methods. These are available here. let’s read in the DataFrame that we’ll be using for this analysis.

Dataset sub-modules like DataFrameStatFunctions and %scala

DataFrameNaFunctions have more methods that solve specific sets of problems. val df = spark.read.format(“csv”)
For example, DataFrameStatFunctions holds a variety of statistically related .option(“header”, “true”)
functions while DataFrameNaFunctions refers to functions that are relevant when .option(“inferSchema”, “true”)
working with null data. .load(“/mnt/defg/retail-data/by-day/2010-12-01.csv”)
• Null Functions available here. df.printSchema()
• Stat Functions vailable here. df.createOrReplaceTempView(“dfTable”)

Column Methods. These were introduced for the most part in the previous chapter %python
are hold a variety of general column related methods like alias or contains. These df = spark.read.format(“csv”)\
are available here. .option(“header”, “true”)\
.option(“inferSchema”, “true”)\
.load(“/mnt/defg/retail-data/by-day/2010-12-01.csv”)
df.printSchema()
df.createOrReplaceTempView(“dfTable”)
DATA ENGINEER S GUIDE T O 44
A PAC HE S PA R K A ND D ELTA L A K E

Here’s the result of the schema and a small sample of the data.

root
|-- InvoiceNo: string (nullable = true)
|-- StockCode: string (nullable = true)
|-- Description: string (nullable = true)
|-- Quantity: integer (nullable = true)
|-- InvoiceDate: timestamp (nullable = true)
|-- UnitPrice: double (nullable = true)
|-- CustomerID: double (nullable = true)
|-- Country: string (nullable = true)

+---------+---------+--------------------+--------+-------------------+---------+----------+--------------+
|InvoiceNo|StockCode| Description|Quantity| InvoiceDate|UnitPrice|CustomerID| Country|
+---------+---------+--------------------+--------+-------------------+---------+----------+--------------+
| 536365| 85123A|WHITE HANGING HEA...| 6|2010-12-01 08:26:00| 2.55| 17850.0|United Kingdom|
| 536365| 71053| WHITE METAL LANTERN| 6|2010-12-01 08:26:00| 3.39| 17850.0|United Kingdom|
| 536365| 84406B|CREAM CUPID HEART...| 8|2010-12-01 08:26:00| 2.75| 17850.0|United Kingdom|
| 536365| 84029G|KNITTED UNION FLA...| 6|2010-12-01 08:26:00| 3.39| 17850.0|United Kingdom|
...
| 536367| 21754|HOME BUILDING BLO...| 3|2010-12-01 08:34:00| 5.95| 13047.0|United Kingdom|
| 536367| 21755|LOVE BUILDING BLO...| 3|2010-12-01 08:34:00| 5.95| 13047.0|United Kingdom|
| 536367| 21777|RECIPE BOX WITH M...| 4|2010-12-01 08:34:00| 7.95| 13047.0|United Kingdom|
+---------+---------+--------------------+--------+-------------------+---------+----------+--------------+
DATA ENGINEER S GUIDE T O 45
A PAC HE S PA R K A ND D ELTA L A K E

Converting to Spark Types Working with Booleans

One thing you’ll see us do throughout this chapter is convert native types into Spark Booleans are foundational when it comes to data analysis because they are the
types. We do this with our first function, the lit function. The lit with take a type in foundation for all filtering. Boolean statements consist of four elements: and, or, true
a native language and convert it into the Spark representation. Here’s how we can and false. We use these simple structures to build logical statements that evaluate
convert a couple of different kinds of Scala and Python values into their respective to either true or false. These statements are often used as conditional requirements
Spark types. where a row of data must either pass this test (evaluate to true) or else it will be
filtered out.
%scala
import org.apache.spark.sql.functions.lit Let’s use our retail dataset to explore working with booleans. We can specify equality
df.select(lit(5), lit(“five”), lit(5.0)) as well as less or greater than.

%python %scala
from pyspark.sql.functions import lit import org.apache.spark.sql.functions.col
df.select(lit(5), lit(“five”), lit(5.0)) df.where(col(“InvoiceNo”).equalTo(536365))
.select(“InvoiceNo”, “Description”)
There’s no equivalent function necessary in SQL, so we can just use the values directly. .show(5, false)

%sql N O T E | Scala has some particular semantics around the use of == and ===. In Spark, if
SELECT 5, “five”, 5.0 you wish to filter by equality you should use === (equal) or =!= (not equal). You can also
use not function and the equalTo method.

%scala
import org.apache.spark.sql.functions.col
df.where(col(“InvoiceNo”) === 536365)
.select(“InvoiceNo”, “Description”)
.show(5, false)
DATA ENGINEER S GUIDE T O 46
A PAC HE S PA R K A ND D ELTA L A K E

Python keeps a more conventional notation. Now we mentioned that we can specify boolean expressions with multiple parts
when we use and or or. In Spark you should always chain together and filters as a
%python sequential filter.
from pyspark.sql.functions import col
df.where(col(“InvoiceNo”) != 536365)\ The reason for this is that even if boolean expressions are expressed serially (one after
.select(“InvoiceNo”, “Description”)\ the other) Spark will flatten all of these filters into one statement and perform the filter
.show(5, False) at the same time, creating the and statement for us. While you may specify your
statements explicitly using and if you like, it’s often easier to reason about and to read if
+---------+-----------------------------+ you specify them serially. or statements need to be specified in the same statement.
|InvoiceNo|Description |
+---------+-----------------------------+
%scala
|536366 |HAND WARMER UNION JACK |
... val priceFilter = col(“UnitPrice”) > 600
|536367 |POPPY’S PLAYHOUSE KITCHEN | val descripFilter = col(“Description”).contains(“POSTAGE”
+---------+-----------------------------+

df.where(col(“StockCode”).isin(“DOT”))
Another option, and probably the cleanest, is to specify the predicate as an .where(priceFilter.or(descripFilter))
expression in a string. This is valid for Python or Scala. Note that this also gives us .show()
access to another way of expressing “does not equal”.
%python
df.where(“InvoiceNo = 536365”) from pyspark.sql.functions import instr
.show(5, false) priceFilter = col(“UnitPrice”) > 600
df.where(“InvoiceNo <> 536365”) descripFilter = instr(df.Description, “POSTAGE”) >= 1
.show(5, false) df.where(df.StockCode.isin(“DOT”))\
.where(priceFilter | descripFilter)\
.show()
DATA ENGINEER S GUIDE T O 47
A PAC HE S PA R K A ND D ELTA L A K E

%sql
SELECT
*
FROM dfTable
WHERE
StockCode in (“DOT”) AND
(UnitPrice > 600 OR
instr(Description, “POSTAGE”) >= 1)

+---------+---------+--------------+--------+-------------------+---------+----------+--------------+
|InvoiceNo|StockCode| Description|Quantity| InvoiceDate|UnitPrice|CustomerID| Country|
+---------+---------+--------------+--------+-------------------+---------+----------+--------------+
| 536544| DOT|DOTCOM POSTAGE| 1|2010-12-01 14:32:00| 569.77| null|United Kingdom|
| 536592| DOT|DOTCOM POSTAGE| 1|2010-12-01 17:06:00| 607.49| null|United Kingdom|
+---------+---------+--------------+--------+-------------------+---------+----------+--------------+
DATA ENGINEER S GUIDE T O 48
A PAC HE S PA R K A ND D ELTA L A K E

Boolean expressions are not just reserved to filters. In order to filter a DataFrame we
can also just specify a boolean column.

%scala %sql
val DOTCodeFilter = col(“StockCode”) === “DOT” SELECT
val priceFilter = col(“UnitPrice”) > 600 UnitPrice,
val descripFilter = col(“Description”).contains(“POSTAGE”) (StockCode = ‘DOT’ AND
df.withColumn(“isExpensive”, (UnitPrice > 600 OR
DOTCodeFilter.and(priceFilter.or(descripFilter))) instr(Description, “POSTAGE”) >= 1)) as isExpensive
.where(“isExpensive”) FROM dfTable
.select(“unitPrice”, “isExpensive”) WHERE
.show(5) (StockCode = ‘DOT’ AND
(UnitPrice > 600 OR
%python instr(Description, “POSTAGE”) >= 1))
from pyspark.sql.functions import instr
DOTCodeFilter = col(“StockCode”) == “DOT” Notice how we did not have to specify our filter as an expression and how we could
priceFilter = col(“UnitPrice”) > 600 use a column name without any extra work.
descripFilter = instr(col(“Description”), “POSTAGE”) >= 1
df.withColumn(“isExpensive”,
DOTCodeFilter & (priceFilter | descripFilter))\
.where(“isExpensive”)\
.select(“unitPrice”, “isExpensive”)\
.show(5)
DATA ENGINEER S GUIDE T O 49
A PAC HE S PA R K A ND D ELTA L A K E

If you’re coming from a SQL background all of these statements should seem quite Here’s our state definition.
familiar. Indeed, all of them can be expressed as a where clause. In fact, it’s often
easier to just express filters as SQL statements than using the programmatic %python
DataFrame interface and Spark SQL allows us to do this without paying any from pyspark.sql.functions import expr
performance penalty. For example, the two following statements are equivalent. df.withColumn(“isExpensive”, expr(“NOT UnitPrice <= 250”))\
.where(“isExpensive”)\
%scala .select(“Description”, “UnitPrice”)
import org.apache.spark.sql.functions.{expr, not, col} .show(5)
df.withColumn(“isExpensive”, not(col(“UnitPrice”).leq(250)))
.filter(“isExpensive”) W A R N I N G | One “gotcha” that can come up is working with null data when creating
.select(“Description”, “UnitPrice”) boolean expressions. If there is a null in your data, you’re going to have to treat things a
.show(5) bit differently. Here’s how we can ensure that we perform a null safe equivalence test.
df.withColumn(“isExpensive”, expr(“NOT UnitPrice <= 250”))
.filter(“isExpensive”) df.where(col(“Description”).eqNullSafe(“hello”)).show()
.select(“Description”, “UnitPrice”)
.show(5) While not currently available (Spark 2.2), IS [NOT] DISTINCT FROM will be coming in
Spark 2.3 to do the same thing in SQL.
DATA ENGINEER S GUIDE T O 50
A PAC HE S PA R K A ND D ELTA L A K E

+----------+------------------+
Converting to Spark Types
|CustomerId| realQuantity|
When working with big data, the second most common task you will do after filtering +----------+------------------+
things is counting things. For the most part, we simply need to express our computation | 17850.0|239.08999999999997|
and that should be valid assuming we’re working with numerical data types. | 17850.0| 418.7156|
+----------+------------------+

To fabricate a contrived example, let’s imagine that we found out that we misrecorded
You’ll notice that we were able to multiply our columns together because they were
the quantity in our retail dataset and true quantity is equal to (the current quantity *
both numerical. Naturally we can add and subtract as necessary as well. In fact we
the unit price) ˆ 2 + 5. This will introduce our first numerical function as well the pow
can do all of this a SQL expression as well.
function that raises a column to the expressed power.

%scala
%scala
df.selectExpr(
import org.apache.spark.sql.functions.{expr, pow}
“CustomerId”,
val fabricatedQuantity = pow(col(“Quantity”) * col(“UnitPrice”), 2) + 5
“(POWER((Quantity * UnitPrice), 2.0) + 5) as realQuantity”)
df.select(
.show(2)
expr(“CustomerId”),
fabricatedQuantity.alias(“realQuantity”))
%python
.show(2)
df.selectExpr(
“CustomerId”,
%python
“(POWER((Quantity * UnitPrice), 2.0) + 5) as realQuantity” )
from pyspark.sql.functions import expr, pow
.show(2)
fabricatedQuantity = pow(col(“Quantity”) * col(“UnitPrice”), 2) + 5
df.select(
%sql
expr(“CustomerId”),
SELECT
fabricatedQuantity.alias(“realQuantity”))\
customerId,
.show(2)
(POWER((Quantity * UnitPrice), 2.0) + 5) as realQuantity
FROM dfTable
DATA ENGINEER S GUIDE T O 51
A PAC HE S PA R K A ND D ELTA L A K E

Another common numerical task is rounding. Now if you’d like to just round to a whole
number, often times you can cast it to an integer and that will work just fine. However %python
Spark also has more detailed functions for performing this explicitly and to a certain from pyspark.sql.functions import lit, round, bround
level of precision. In this case we will round to one decimal place. df.select(
round(lit(“2.5”)),
%scala bround(lit(“2.5”)))\
import org.apache.spark.sql.functions.{round, bround} .show(2)
df.select(
round(col(“UnitPrice”), 1).alias(“rounded”), %sql
col(“UnitPrice”)) SELECT
.show(5) round(2.5),
bround(2.5)
By default, the round function will round up if you’re exactly in between two numbers.
You can round down with the bround. +-------------+--------------+
|round(2.5, 0)|bround(2.5, 0)|
+-------------+--------------+
%scala
| 3.0| 2.0|
import org.apache.spark.sql.functions.lit | 3.0| 2.0|
df.select( +-------------+--------------+
round(lit(“2.5”)),
bround(lit(“2.5”)))
.show(2)
DATA ENGINEER S GUIDE T O 52
A PAC HE S PA R K A ND D ELTA L A K E

Another numerical task is to compute the correlation of two columns. For example, A common task is to compute summary statistics for a column or set of columns.
we can see the Pearson Correlation Coefficient for two columns to see if cheaper We can use the describe method to achieve exactly this. This will take all numeric
things are typically bought in greater quantities. We can do this through a function as columns and calculate the count, mean, standard deviation, min, and max. This should
well as through the DataFrame statistic methods. be used primarily for viewing in the console as the schema may change in the future.

%scala %scala
import org.apache.spark.sql.functions.{corr} df.describe().show()
df.stat.corr(“Quantity”, “UnitPrice”)
df.select(corr(“Quantity”, “UnitPrice”)).show() %python
df.describe().show()
%python
from pyspark.sql.functions import corr +-------+------------------+------------------+------------------+
df.stat.corr(“Quantity”, “UnitPrice”) |Summary| Quantity| UnitPrice| CustomerID|
+-------+------------------+------------------+------------------+
df.select(corr(“Quantity”, “UnitPrice”)).show()
| count| 3108| 3108| 1968|
| mean| 8.627413127413128| 4.151946589446603|15661.388719512195|
%sql | stddev|26.371821677029203|15.638659854603892|1854.4496996893627|
SELECT | min| -24| 0.0| 12431.0|
| max| 600| 607.49| 18229.0|
corr(Quantity, UnitPrice)
+-------+------------------+------------------+------------------+
FROM
dfTable

+-------------------------+
|corr(Quantity, UnitPrice)|
+-------------------------+
| -0.04112314436835551|
+-------------------------+
DATA ENGINEER S GUIDE T O 53
A PAC HE S PA R K A ND D ELTA L A K E

If you need these exact numbers you can also perform this as an aggregation yourself We can also use this to see a cross tabulation or frequent item pairs (Be careful, this
by importing the functions and applying them to the columns that you need. output will be large and is omitted for this reason).

%scala %scala
import org.apache.spark.sql.functions.{count, mean, stddev_pop, min, max} df.stat.crosstab(“StockCode”, “Quantity”).show()

%python %python
from pyspark.sql.functions import count, mean, stddev_pop, min, max df.stat.crosstab(“StockCode”, “Quantity”).show()

There are a number of statistical functions available in the StatFunctions Package. %scala
These are DataFrame methods that allow you to calculate a variety of different things. df.stat.freqItems(Seq(“StockCode”, “Quantity”)).show()
For instance, we can calculate either exact or approximate quantiles of our data using
the approxQuantile method. %python
df.stat.freqItems([“StockCode”, “Quantity”]).show()
%scala
val colName = “UnitPrice”
val quantileProbs = Array(0.5)
val relError = 0.05
df.stat.approxQuantile(“UnitPrice”, quantileProbs, relError) // 2.51

%python
colName = “UnitPrice”
quantileProbs = [0.5]
relError = 0.05
df.stat.approxQuantile(“UnitPrice”, quantileProbs, relError) # 2.51
DATA ENGINEER S GUIDE T O 54
A PAC HE S PA R K A ND D ELTA L A K E

Working with Strings

As a last note, we can also add a unique id to each row using the monotonically_ String manipulation shows up in nearly every data flow and its worth explaining
increasing_id function. This function will generate a unique value for each row, what you can do with strings. You may be manipulating log files performing regular
starting with 0. expression extraction or substitution, or checking for simple string existence, or
simply making all strings upper or lower case.
%scala
import org.apache.spark.sql.functions.monotonically_increasing_id We will start with the last task as it’s one of the simplest. The initcap function will
df.select(monotonically_increasing_id()).show(2) capitalize every word in a given string when that word is separated from another via
whitespace.
%python
from pyspark.sql.functions import monotonically_increasing_id %scala
df.select(monotonically_increasing_id()).show(2) import org.apache.spark.sql.functions.{initcap}
df.select(initcap(col(“Description”))).show(2, false)
There are functions added every release and so by the time you’re reading this, it may
already not include everything. For instance, there are some random data generation %python
tools (rand() randn()) that allow you to randomly generate data however there are from pyspark.sql.functions import initcap
potential determinism issues when doing so. Discussions of these challenges can df.select(initcap(col(“Description”))).show()
be found on the Spark mailing list. There are also a number of more advanced tasks
like bloom filtering and sketching algorithms available in the stat functions that we %sql
mentioned (and linked to) at the beginning of this chapter. Be sure to search the API SELECT
documentation for more information and functions. initcap(Description)
FROM
dfTable
DATA ENGINEER S GUIDE T O 55
A PAC HE S PA R K A ND D ELTA L A K E

+----------------------------------+ %sql
|initcap(Description) | SELECT
+----------------------------------+
Description,
|White Hanging Heart T-light Holder|
|White Metal Lantern | lower(Description),
+----------------------------------+ Upper(lower(Description))
FROM
As mentioned above, we can also quite simply lower case and upper case strings dfTable
as well.
+--------------------+--------------------+-------------------------+
%scala | Description| lower(Description)|upper(lower(Description))|
+--------------------+--------------------+-------------------------+
import org.apache.spark.sql.functions.{lower, upper}
|WHITE HANGING HEA...|white hanging hea...| WHITE HANGING HEA...|
df.select( | WHITE METAL LANTERN| white metal lantern| WHITE METAL LANTERN|
col(“Description”), +--------------------+--------------------+-------------------------+
lower(col(“Description”)),
upper(lower(col(“Description”))))
.show(2)

%python
from pyspark.sql.functions import lower, upper
df.select(
col(“Description”),
lower(col(“Description”)),
upper(lower(col(“Description”))))\
.show(2)
DATA ENGINEER S GUIDE T O 56
A PAC HE S PA R K A ND D ELTA L A K E

Another trivial task is adding or removing whitespace around a string. We can do this
with lpad, ltrim, rpad and rtrim, trim.

%scala %sql
import org.apache.spark.sql.functions.{lit, ltrim, rtrim, rpad, lpad, SELECT
trim} ltrim(‘ HELLLOOOO ‘),
df.select( rtrim(‘ HELLLOOOO ‘),
ltrim(lit(“ HELLO “)).as(“ltrim”), trim(‘ HELLLOOOO ‘),
rtrim(lit(“ HELLO “)).as(“rtrim”), lpad(‘HELLOOOO ‘, 3, ‘ ‘),
trim(lit(“ HELLO “)).as(“trim”), rpad(‘HELLOOOO ‘, 10, ‘ ‘)
lpad(lit(“HELLO”), 3, “ “).as(“lp”), FROM
rpad(lit(“HELLO”), 10, “ “).as(“rp”)) dfTable
.show(2)
+---------+---------+-----+---+----------+
%python | ltrim| rtrim| trim| lp| rp|
+---------+---------+-----+---+----------+
from pyspark.sql.functions import lit, ltrim, rtrim, rpad, lpad, trim
|HELLO | HELLO|HELLO| HE|HELLO |
df.select( |HELLO | HELLO|HELLO| HE|HELLO |
ltrim(lit(“ HELLO “)).alias(“ltrim”), +---------+---------+-----+---+----------+
rtrim(lit(“ HELLO “)).alias(“rtrim”),
trim(lit(“ HELLO “)).alias(“trim”), You’ll notice that if lpad or rpad takes a number less than the length of the string, it
lpad(lit(“HELLO”), 3, “ “).alias(“lp”), will always remove values from the right side of the string.
rpad(lit(“HELLO”), 10, “ “).alias(“rp”))\
.show(2)
DATA ENGINEER S GUIDE T O 57
A PAC HE S PA R K A ND D ELTA L A K E

Regular Expressions
Probably one of the most frequently performed tasks is searching for the existence of %python
one string on another or replacing all mentions of a string with another value. This is from pyspark.sql.functions import regexp_replace
often done with a tool called “Regular Expressions” that exist in many programming regex_string = “BLACK|WHITE|RED|GREEN|BLUE”
languages. Regular expressions give the user an ability to specify a set of rules to use df.select(
to either extract values from a string or replace them with some other values. regexp_replace(col(“Description”), regex_string, “COLOR”)
.alias(“color_cleaned”),
Spark leverages the complete power of Java Regular Expressions. The Java RegEx col(“Description”))\
syntax departs slightly from other programming languages so it is worth reviewing .show(2)
before putting anything into production. There are two key functions in Spark that
you’ll need to perform regular expression tasks: regexp_extract and regexp_ %sql
replace. These functions extract values and replace values respectively. SELECT
regexp_replace(Description, ‘BLACK|WHITE|RED|GREEN|BLUE’, ‘COLOR’)
Let’s explore how to use the regexp_replace function to replace substitute colors as color_cleaned,
names in our description column. Description
FROM
%scala dfTable
import org.apache.spark.sql.functions.regexp_replace
val simpleColors = Seq(“black”, “white”, “red”, “green”, “blue”) +--------------------+--------------------+
val regexString = simpleColors.map(_.toUpperCase).mkString(“|”) | color_cleaned| Description|
+--------------------+--------------------+
// the | signifies `OR` in regular expression syntax
|COLOR HANGING HEA...|WHITE HANGING HEA...|
df.select( | COLOR METAL LANTERN| WHITE METAL LANTERN|
regexp_replace(col(“Description”), regexString, “COLOR”) +--------------------+--------------------+
.alias(“color_cleaned”),
col(“Description”))
.show(2)
DATA ENGINEER S GUIDE T O 58
A PAC HE S PA R K A ND D ELTA L A K E

Another task may be to replace given characters with other characters. Building
this as regular expression could be tedious so Spark also provides the translate
function to replace these values. This is done at the character level and will replace all
instances of a character with the indexed character in the replacement string.

%scala %sql
import org.apache.spark.sql.functions.translate SELECT
df.select( translate(Description, ‘LEET’, ‘1337’),
translate(col(“Description”), “LEET”, “1337”), Description
col(“Description”)) FROM
.show(2) dfTable

%python +----------------------------------+--------------------+
from pyspark.sql.functions import translate |translate(Description, LEET, 1337)| Description|
+----------------------------------+--------------------+
df.select(
| WHI73 HANGING H3A...|WHITE HANGING HEA...|
translate(col(“Description”), “LEET”, “1337”), | WHI73 M37A1 1AN73RN| WHITE METAL LANTERN|
col(“Description”))\ +----------------------------------+--------------------+
.show(2)
DATA ENGINEER S GUIDE T O 59
A PAC HE S PA R K A ND D ELTA L A K E

We can also perform something similar like pulling out the first mentioned color.

%scala %sql
import org.apache.spark.sql.functions.regexp_extract SELECT
val regexString = simpleColors regexp_extract(Description, ‘(BLACK|WHITE|RED|GREEN|BLUE)’, 1),
.map(_.toUpperCase) Description
FROM
.mkString(“(“, “|”, “)”) dfTable
// the | signifies OR in regular expression syntax
df.select( +-------------+--------------------+
regexp_extract(col(“Description”), regexString, 1) |color_cleaned| Description|
+-------------+--------------------+
.alias(“color_cleaned”),
| WHITE|WHITE HANGING HEA...|
col(“Description”)) | WHITE| WHITE METAL LANTERN|
.show(2) +-------------+--------------------+

%python
from pyspark.sql.functions import regexp_extract
extract_str = “(BLACK|WHITE|RED|GREEN|BLUE)”
df.select(
regexp_extract(col(“Description”), extract_str, 1)
.alias(“color_cleaned”),
col(“Description”))\
.show(2)
DATA ENGINEER S GUIDE T O 60
A PAC HE S PA R K A ND D ELTA L A K E

Sometimes, rather than extracting values, we simply want to check for existence.
We can do this with the contains method on each column. This will return a boolean
declaring whether it can find that string in the column’s string.

%scala %sql
val containsBlack = col(“Description”).contains(“BLACK”) SELECT
val containsWhite = col(“DESCRIPTION”).contains(“WHITE”) Description
df.withColumn(“hasSimpleColor”, containsBlack.or(containsWhite)) FROM
.filter(“hasSimpleColor”) dfTable
.select(“Description”) WHERE
.show(3, false) instr(Description, ‘BLACK’) >= 1 OR
instr(Description, ‘WHITE’) >= 1
In Python we can use the instr function.
+----------------------------------+
%python |Description |
+----------------------------------+
from pyspark.sql.functions import instr
|WHITE HANGING HEART T-LIGHT HOLDER|
containsBlack = instr(col(“Description”), “BLACK”) >= 1 |WHITE METAL LANTERN |
containsWhite = instr(col(“Description”), “WHITE”) >= 1 |RED WOOLLY HOTTIE WHITE HEART. |
df.withColumn(“hasSimpleColor”, containsBlack | containsWhite)\ +----------------------------------+
only showing top 3 rows
.filter(“hasSimpleColor”)\
.select(“Description”)\
This is trivial with just two values but gets much more complicated with more values.
.show(3, False)
DATA ENGINEER S GUIDE T O 61
A PAC HE S PA R K A ND D ELTA L A K E

Let’s work through this in a more dynamic way and take advantage of Spark’s ability We can also do this quite easily in Python. In this case we’re going to use a different
to accept a dynamic number of arguments. When we convert a list of values into a set function locate that returns the integer location (1 based location). We then convert
of arguments and pass them into a function, we use a language feature called varargs. that to a boolean before using it as a the same basic feature.
This feature allows us to effectively unravel an array of arbitrary length and pass it
as arguments to a function. This, coupled with select allows us to create arbitrary %python
numbers of columns dynamically. from pyspark.sql.functions import expr, locate
simpleColors = [“black”, “white”, “red”, “green”, “blue”]
%scala def color_locator(column, color_string):
val simpleColors = Seq(“black”, “white”, “red”, “green”, “blue”) “””This function creates a column declaring whether or not a given
val selectedColumns = simpleColors.map(color => { pySpark column contains the UPPERCASED color. Returns a new column
col(“Description”) type that can be used in a select statement.“””
.contains(color.toUpperCase) return locate(color_string.upper(), column)\
.alias(s”is_$color”) .cast(“boolean”)\
}):+expr(“*”) // could also append this value .alias(“is_” + c)
df selectedColumns = [color_locator(df.Description, c) for c in simple-
.select(selectedColumns:_*) Colors]
.where(col(“is_white”).or(col(“is_red”))) selectedColumns.append(expr(“*”)) # has to a be Column type
.select(“Description”) df\
.show(3, false) .select(*selectedColumns)\
.where(expr(“is_white OR is_red”))\
+----------------------------------+ .select(“Description”)\
|Description |
.show(3, False)
+----------------------------------+
|WHITE HANGING HEART T-LIGHT HOLDER|
|WHITE METAL LANTERN | This simple feature is often one that can help you programmatically generate columns
|RED WOOLLY HOTTIE WHITE HEART. | or boolean filters in a way that is simple to reason about and extend. We could extend
+----------------------------------+
this to calculating the smallest common denominator for a given input value or
whether or not a number is a prime.
DATA ENGINEER S GUIDE T O 62
A PAC HE S PA R K A ND D ELTA L A K E

Working with Dates and Timestamps

Dates and times are a constant challenge in programming languages and databases. df.printSchema()
It’s always necessary to keep track of timezones and make sure that formats are
correct and valid. Spark does its best to keep things simple by focusing explicitly on root
two kinds of time related information. There are dates, which focus exclusively on |-- InvoiceNo: string (nullable = true)
calendar dates, and timestamps that include both date and time information. Spark, |-- StockCode: string (nullable = true)
as we saw with our current dataset, will make a best effort to correctly identify column |-- Description: string (nullable = true)
types, including dates and timestamps when we enable inferSchema. We can see |-- Quantity: integer (nullable = true)
that this worked quite well with our current dataset because it was able to identify |-- InvoiceDate: timestamp (nullable = true)
and read our date format without us having to provide some specification for it. |-- UnitPrice: double (nullable = true)
|-- CustomerID: double (nullable = true)
Now as we hinted at above, working with dates and timestamps closely relates to |-- Country: string (nullable = true)
working with strings because we often store our timestamps or dates as strings and
convert them into date types at runtime. This is less common when working with While Spark will do this on a best effort basis, sometimes there will be no getting
databases and structured data but much more common when we are working with around working with strangely formatted dates and times. Now the key to reasoning
text and csv files. We will experiment with that shortly. about the transformations that you are going to need to apply is to ensure that you
know exactly what type and format you have at each given step of the way. Another
W A R N I N G | There are a lot of caveats, unfortunately, when working with dates and common gotcha is that Spark’s TimestampType only supports second-level precision,
timestamps, especially when it comes to timezone handling. In 2.1 and before, Spark this means that if you’re going to be working with milliseconds or microseconds,
will parse according to the machine’s timezone if timezones are not explicitly specified you’re going to have to work around this problem by potentially operating on them as
in the value that you are parsing. You can set a session local timezone if necessary by longs. Any more precision when coercing to a TimestampType will be removed.
setting spark.conf.sessionLocalTimeZone in the SQL configurations. This should
be set according to the Java TimeZone format.
DATA ENGINEER S GUIDE T O 63
A PAC HE S PA R K A ND D ELTA L A K E

Spark can be a bit particular about what format you have at any given point in time. Now that we have a simple DataFrame to work with, let’s add and subtract 5 days
It’s important to be explicit when parsing or converting to make sure there are from today. These functions take a column and then the number of days to either add
no issues in doing so. At the end of the day, Spark is working with Java dates and or subtract as the arguments.
timestamps and therefore conforms to those standards. Let’s start with the basics
and get the current date and the current timestamps. %scala
import org.apache.spark.sql.functions.{date_add, date_sub}
%scala dateDF
import org.apache.spark.sql.functions.{current_date, current_timestamp} .select(
val dateDF = spark.range(10) date_sub(col(“today”), 5),
.withColumn(“today”, current_date()) date_add(col(“today”), 5))
.withColumn(“now”, current_timestamp()) .show(1)
dateDF.createOrReplaceTempView(“dateTable”)
%python
%python from pyspark.sql.functions import date_add, date_sub
from pyspark.sql.functions import current_date, current_timestamp dateDF\
dateDF = spark.range(10)\ .select(
.withColumn(“today”, current_date())\ date_sub(col(“today”), 5),
.withColumn(“now”, current_timestamp()) date_add(col(“today”), 5))\
dateDF.createOrReplaceTempView(“dateTable”) .show(1)

dateDF.printSchema() %sql
SELECT
root date_sub(today, 5),
|-- id: long (nullable = false) date_add(today, 5)
|-- today: date (nullable = false) FROM
|-- now: timestamp (nullable = false) dateTable
DATA ENGINEER S GUIDE T O 64
A PAC HE S PA R K A ND D ELTA L A K E

+------------------+------------------+
|date_sub(today, 5)|date_add(today, 5)|
+------------------+------------------+
| 2017-06-12| 2017-06-22|
+------------------+------------------+

Another common task is to take a look at the difference between two dates. We can
do this with the datediff function that will return the number of days in between
two dates. Most often we just care about the days although since months can have a
strange number of days there also exists a function months_between that gives you
the number of months between two dates.

%scala %python
import org.apache.spark.sql.functions.{datediff, months_between, to_date} from pyspark.sql.functions import datediff, months_between, to_date
dateDF dateDF\
.withColumn(“week_ago”, date_sub(col(“today”), 7)) .withColumn(“week_ago”, date_sub(col(“today”), 7))\
.select(datediff(col(“week_ago”), col(“today”))) .select(datediff(col(“week_ago”), col(“today”)))\
.show(1) .show(1)
dateDF dateDF\
.select( .select(
to_date(lit(“2016-01-01”)).alias(“start”), to_date(lit(“2016-01-01”)).alias(“start”),
to_date(lit(“2017-05-22”)).alias(“end”)) to_date(lit(“2017-05-22”)).alias(“end”))\
.select(months_between(col(“start”), col(“end”))) .select(months_between(col(“start”), col(“end”)))\
.show(1) .show(1)
DATA ENGINEER S GUIDE T O 65
A PAC HE S PA R K A ND D ELTA L A K E

%sql %scala
SELECT import org.apache.spark.sql.functions.{to_date, lit}
to_date(‘2016-01-01’), spark.range(5).withColumn(“date”, lit(“2017-01-01”))
months_between(‘2016-01-01’, ‘2017-01-01’), .select(to_date(col(“date”)))
datediff(‘2016-01-01’, ‘2017-01-01’) .show(1)
FROM
dateTable %python
from pyspark.sql.functions import to_date, lit
+-------------------------+ spark.range(5).withColumn(“date”, lit(“2017-01-01”))\
|datediff(week_ago, today)| .select(to_date(col(“date”)))\
+-------------------------+
.show(1)
| -7|
+-------------------------+
+-------------------------+
|months_between(start,end)|
+-------------------------+
| -16.67741935|
+-------------------------+

You’ll notice that I introduced a new function above, the to_date function. The
to_date function allows you to convert a string to a date, optionally with a specified
format. We specify our format in the Java simpleDateFormat which will be important
to reference if you use this function.
DATA ENGINEER S GUIDE T O 66
A PAC HE S PA R K A ND D ELTA L A K E

W A R N I N G | Spark will not throw an error if it cannot parse the date, it’ll just return null. We will use two functions to fix this, to_date and to_timestamp. The former
This can be a bit tricky in larger pipelines because you may be expecting your data in optionally expects a format while the latter requires one.
one format and getting it in another. To illustrate, let’s take a look at the date format that
has switched from year-month-day to year-day-month. Spark will fail to parse this date import org.apache.spark.sql.functions.{unix_timestamp, from_unixtime}
and silently return null instead. val dateFormat = “yyyy-dd-MM”
val cleanDateDF = spark.range(1)
dateDF.select(to_date(lit(“2016-20-12”)),to_date(lit(“2017-12-11”))). .select(
show(1) to_date(lit(“2017-12-11”), dateFormat)
.alias(“date”),
+-------------------+-------------------+ to_date(lit(“2017-20-12”), dateFormat)
|to_date(2016-20-12)|to_date(2017-12-11)| .alias(“date2”))
+-------------------+-------------------+
cleanDateDF.createOrReplaceTempView(“dateTable2”)
| null| 2017-12-11|
+-------------------+-------------------+
%python
from pyspark.sql.functions import unix_timestamp, from_unixtime
We find this to be an especially tricky situation for bugs because some dates may
dateFormat = “yyyy-dd-MM”
match the correct format while others do not. See how above, the second date is
cleanDateDF = spark.range(1)\
show to be December 11th instead of the correct day, November 12th? Spark doesn’t
.select(
throw an error because it cannot know whether the days are mixed up or if that
to_date(unix_timestamp(lit(“2017-12-11”), dateFormat).cast(“time-
specific row is incorrect.
stamp”))\
.alias(“date”),
Let’s fix this pipeline, step by step and come up with a robust way to avoid these
to_date(unix_timestamp(lit(“2017-20-12”), dateFormat).cast(“time-
issues entirely. The first step is to remember that we need to specify our date format
stamp”))\
according to the Java SimpleDateFormat standard as documented here.
.alias(“date2”))
cleanDateDF.createOrReplaceTempView(“dateTable2”)
DATA ENGINEER S GUIDE T O 67
A PAC HE S PA R K A ND D ELTA L A K E

+----------+----------+ Now let’s use an example of to_timestamp which always requires a format to be
| date| date2|
specified.
+----------+----------+
|2017-11-12|2017-12-20|
+----------+----------+ %scala
import org.apache.spark.sql.functions.to_timestamp
%sql cleanDateDF
SELECT .select(
to_date(date, ‘yyyy-dd-MM’), to_timestamp(col(“date”), dateFormat))
to_date(date2, ‘yyyy-dd-MM’), .show()
to_date(date)
FROM %python
dateTable2 from pyspark.sql.functions import to_timestamp
cleanDateDF\
.select(
to_timestamp(col(“date”), dateFormat))\
.show()

+----------------------------------+
|to_timestamp(`date`, ‘yyyy-dd-MM’)|
+----------------------------------+
| 2017-11-12 00:00:00|
+----------------------------------+
DATA ENGINEER S GUIDE T O 68
A PAC HE S PA R K A ND D ELTA L A K E

We can check all of this from SQL. One minor point is that we can also set this as a string which Spark parses to a literal.

%sql cleanDateDF.filter(col(“date2”) > “’2017-12-12’”).show()

SELECT
to_timestamp(date, ‘yyyy-dd-MM’), W A R N I N G | Implicit type casting is an easy way to shoot yourself in the foot,
to_timestamp(date2, ‘yyyy-dd-MM’) especially when dealing with null values or dates in different timezones or formats. We
FROM recommend that you parse them explicitly instead of relying on implicit changes.
dateTable2

Casting between dates and timestamps is simple in all languages, in SQL we would do
it in the following way.

%sql
SELECT cast(to_date(“2017-01-01”, “yyyy-dd-MM”) as timestamp)

Once we’ve gotten our date or timestamp into the correct format and type, comparing
between them is actually quite easy. We just need to be sure to either use a date/
timestamp type or specify our string according to the right format of yyyy-MM-dd if
we’re comparing a date.

cleanDateDF.filter(col(“date2”) > lit(“2017-12-12”)).show()

DATA ENGINEER S GUIDE T O 69
A PAC HE S PA R K A ND D ELTA L A K E

Working with Nulls in Data

As a best practice, you should always use nulls to represent missing or empty data Coalesce
in your DataFrames. Spark can optimize working with null values more than it can Spark includes a function to allow you to select the first null value from a set of
if you use empty strings or other values. The primary way of interacting with null columns by using the coalesce function. In this case there are no null values, so it
values, at DataFrame scale, is to use the .na subpackage on a DataFrame. There are simply returns the first column.
also several functions for performing operations and explicitly specifying how Spark
should handle null values. See the previous chapter where we discuss ordering and %scala
the section on boolean expressions previously in this chapter. import org.apache.spark.sql.functions.coalesce
df.select(coalesce(col(“Description”), col(“CustomerId”))).show()
W A R N I N G | Nulls are a challenge part of all programming and Spark is no exception.
We recommend being explicit is always better than being implicit when handling null %python
values. For instance, in this part of the book we saw how we can define columns as from pyspark.sql.functions import coalesce
having null types. However, this comes with a catch. When we declare a column as not df.select(coalesce(col(“Description”), col(“CustomerId”))).show()
having a null time, that is not actually enforced. To reiterate, when you define a schema
where all columns are declared to not have null values - Spark will not enforce that and
will happily let null values into that column. The nullable signal is somply to help Spark
SQL optimize for handling that column. If you have null values in columns that should not
have null values, you can get an incorrect result or see strange exceptions that can be
hard to debug.

There are two things you can do with null values. You can explicitly drop nulls or you
can fill them with a value (globally or on a per column basis). Let’s experiment with
each of these now.
DATA ENGINEER S GUIDE T O 70
A PAC HE S PA R K A ND D ELTA L A K E

NullIf, Ifnull, nvl, and nvl2 Drop

There are several SQL functions that allow us to achieve similar things. ifnull allows The simplest is probably drop, which simply removes rows that contain nulls. The
you to select the second value if the first is null, and defaults to the first. nullif default is to drop any row where any value is null.
allows you to return null if the two values are equal or else return the second if they
are not. nvl will return the second value if the first is null, but defaults to the first. df.na.drop()
Lastly, nvl2 will return the second value is the first is not null, otherwise it will return df.na.drop(“any”)
last specified value (else_value below).
In SQL we have to do this column by column.
%sql
SELECT %sql
ifnull(null, ‘return_value’), SELECT
nullif(‘value’, ‘value’), *
nvl(null, ‘return_value’), FROM
nvl2(‘not_null’, ‘return_value’, “else_value”) dfTable
FROM dfTable WHERE
LIMIT 1 Description IS NOT NULL

+------------+----+------------+------------+ Passing in “any” as an argument will drop a row if any of the values are null. Passing in
| a| b| c| d| “all” will only drop the row if all values are null or NaN for that row.
+------------+----+------------+------------+
|return_value|null|return_value|return_value|
+------------+----+------------+------------+ df.na.drop(“all”)

Naturally, we can use these in select expressions on DataFrames as well.

DATA ENGINEER S GUIDE T O 71
A PAC HE S PA R K A ND D ELTA L A K E

We can also apply this to certain sets of columns by passing in an array of columns. We can also do with with a Scala Map where the key is the column name and the value
is the value we would like to use to fill null values.
%scala
df.na.drop(“all”, Seq(“StockCode”, “InvoiceNo”)) %scala
val fillColValues = Map(
%python “StockCode” -> 5,
df.na.drop(“all”, subset=[“StockCode”, “InvoiceNo”]) “Description” -> “No Value”
)
Fill df.na.fill(fillColValues)
Fill allows you to fill one or more columns with a set of values. This can be done by
specifying a map, specific value and a set of columns. %python
fill_cols_vals = {
For example to fill all null values in String columns I might specify. “StockCode”: 5,
“Description” : “No Value”
df.na.fill(“All Null values become this string”) }
df.na.fill(fill_cols_vals)
We could do the same for integer columns with df.na.fill(5:Integer) or for
Doubles df.na.fill(5:Double). In order to specify columns, we just pass in an array
of column names like we did above.

%scala
df.na.fill(5, Seq(“StockCode”, “InvoiceNo”))

%python
df.na.fill(“all”, subset=[“StockCode”, “InvoiceNo”])
DATA ENGINEER S GUIDE T O 72
A PAC HE S PA R K A ND D ELTA L A K E

Working with Complex Types

Replace Complex types can help you organize and structure your data in ways that make more
In addition to replacing null values like we did with drop and fill, there are more sense for the problem you are hoping to solve. There are three kinds of complex types,
flexible options that we can use with more than just null values. Probably the most structs, arrays, and maps.
common use case is to replace all values in a certain column according to their current
value. The only requirement is that this value be the same type as the original value. Structs
You can think of structs as DataFrames within DataFrames. A worked example will
%scala illustrate this more clearly. We can create a struct by wrapping a set of columns in
df.na.replace(“Description”, Map(“” -> “UNKNOWN”)) parenthesis in a query.

%python df.selectExpr(“(Description, InvoiceNo) as complex”, “*”)

df.na.replace([“”], [“UNKNOWN”], “Description”)
df.selectExpr(“struct(Description, InvoiceNo) as complex”, “*”)

Ordering %scala
As discussed in the previous chapter, you can use asc_nulls_first, desc_nulls_ import org.apache.spark.sql.functions.struct
first, asc_nulls_last, or desc_nulls_last to specify where we would like our val complexDF = df
null values to appear in an ordered DataFrame. .select(struct(“Description”, “InvoiceNo”).alias(“complex”))
complexDF.createOrReplaceTempView(“complexDF”)

%python
from pyspark.sql.functions import struct
complexDF = df\
.select(struct(“Description”, “InvoiceNo”).alias(“complex”))
complexDF.createOrReplaceTempView(“complexDF”)
DATA ENGINEER S GUIDE T O 73
A PAC HE S PA R K A ND D ELTA L A K E

split
We now have a DataFrame with a column complex. We can query it just as we might We do this with the split function and specify the delimiter.
another DataFrame, the only difference is that we use a dot syntax to do so or the
column method getField. %scala
import org.apache.spark.sql.functions.split
complexDF.select(“complex.Description”) df.select(split(col(“Description”), “ “)).show(2)
complexDF.select(col(“complex”).getField(“Description”)
%python
We can also query all values in the struct with *. This brings up all the columns to the from pyspark.sql.functions import split
top level DataFrame. df.select(split(col(“Description”), “ “)).show(2)

complexDF.select(“complex.*”) %sql
SELECT
%sql split(Description, ‘ ‘)
SELECT FROM
complex.* dfTable
FROM
complexDF +---------------------+
|split(Description, )|
+---------------------+
Arrays
| [WHITE, HANGING, ...|
To define arrays, let’s work through a use case. With our current data, our object is to | [WHITE, METAL, LA...|
take every single word in our Description column and convert that into a row in our +---------------------+
DataFrame.

The first task is to turn our Description column into a complex type, an array.
DATA ENGINEER S GUIDE T O 74
A PAC HE S PA R K A ND D ELTA L A K E

Array Length
This is quite powerful because Spark will allow us to manipulate this complex type as We can query the array’s length by querying for its size.
another column. We can also query the values of the array with a python-like syntax.
%scala
%scala import org.apache.spark.sql.functions.size
df.select(split(col(“Description”), “ “).alias(“array_col”)) df.select(size(split(col(“Description”), “ “))).show(2) // shows 5 and 3
.selectExpr(“array_col[0]”)
.show(2) %python
from pyspark.sql.functions import size
%python df.select(size(split(col(“Description”), “ “))).show(2) # shows 5 and 3
df.select(split(col(“Description”), “ “).alias(“array_col”))\
.selectExpr(“array_col[0]”)\ Array Contains
.show(2) For instance we can see if this array contains a value.

%sql %scala
SELECT import org.apache.spark.sql.functions.array_contains
split(Description, ‘ ‘)[0] df.select(array_contains(split(col(“Description”), “ “), “WHITE”)).
FROM show(2)
dfTable
%python
+------------+ from pyspark.sql.functions import array_contains
|array_col[0]| df.select(array_contains(split(col(“Description”), “ “), “WHITE”)).
+------------+
show(2)
| WHITE|
| WHITE|
+------------+
DATA ENGINEER S GUIDE T O 75
A PAC HE S PA R K A ND D ELTA L A K E

Explode
This is quite powerful because Spark will allow us to manipulate this complex type as The explode function takes a column that consists of arrays and creates one row (with
another column. We can also query the values of the array with a python-like syntax. the rest of the values duplicated) per value in the array. The following figure illustrates
the process.
%sql
SELECT
SPLIT EXPLODE
array_contains(split(Description, ‘ ‘), ‘WHITE’)
“Hello World” , “other col” [ “Hello” , “World” ] , “other col” “Hello” , “other col”
FROM
“World” , “other col”
dfTable
LIMIT 2
%scala
+--------------------------------------------+ import org.apache.spark.sql.functions.{split, explode}
|array_contains(split(Description, ), WHITE)| df.withColumn(“splitted”, split(col(“Description”), “ “))
+--------------------------------------------+
.withColumn(“exploded”, explode(col(“splitted”)))
| true|
| true| .select(“Description”, “InvoiceNo”, “exploded”)
+--------------------------------------------+ .show(2)

However this does not solve our current problem. In order to convert a complex type %python
into a set of rows (one per value in our array), we use the explode function. from pyspark.sql.functions import split, explode
df.withColumn(“splitted”, split(col(“Description”), “ “))\
.withColumn(“exploded”, explode(col(“splitted”)))\
.select(“Description”, “InvoiceNo”, “exploded”)\
.show(2)
DATA ENGINEER S GUIDE T O 76
A PAC HE S PA R K A ND D ELTA L A K E

Maps
%sql Maps are used less frequently but are still important to cover. We create them with
SELECT the map function and key value pairs of columns. Then we can select them just like
Description, we might select from an array.
InvoiceNo,
exploded %scala
FROM import org.apache.spark.sql.functions.map
(SELECT df.select(map(col(“Description”), col(“InvoiceNo”)).alias(“complex_
*, map”))
split(Description, “ “) as splitted .selectExpr(“complex_map[‘Description’]”)
FROM .show(2)
dfTable)
LATERAL VIEW explode(splitted) as exploded %python
LIMIT 2 from pyspark.sql.functions import create_map
df.select(create_map(col(“Description”), col(“InvoiceNo”)).alias(“com-
+--------------------+---------+--------+ plex_map”))\
| Description|InvoiceNo|exploded| .show(2)
+--------------------+---------+--------+
|WHITE HANGING HEA...| 536365| WHITE|
|WHITE HANGING HEA...| 536365| HANGING| %sql
+--------------------+---------+--------+ SELECT
map(Description, InvoiceNo) as complex_map
FROM
dfTable
WHERE
Description IS NOT NULL
DATA ENGINEER S GUIDE T O 77
A PAC HE S PA R K A ND D ELTA L A K E

+--------------------+ We can also explode map types which will turn them into columns.
| complex_map|
+--------------------+
|Map(WHITE HANGING...| %scala
|Map(WHITE METAL L...| df.select(map(col(“Description”), col(“InvoiceNo”)).alias(“complex_
+--------------------+ map”))
.selectExpr(“explode(complex_map)”)
We can query them by using the proper key. A missing key returns null. .show(2)

%scala %python
df.select(map(col(“Description”), col(“InvoiceNo”)).alias(“complex_map”)) df.select(map(col(“Description”), col(“InvoiceNo”)).alias(“complex_
.selectExpr(“complex_map[‘WHITE METAL LANTERN’]”) map”))\
.show(2) .selectExpr(“explode(complex_map)”)\
.show(2)
%python
df.select(map(col(“Description”), col(“InvoiceNo”)).alias(“complex_map”))\ +--------------------+------+
.selectExpr(“complex_map[‘WHITE METAL LANTERN’]”)\ | key| value|
.show(2) +--------------------+------+
|WHITE HANGING HEA...|536365|
| WHITE METAL LANTERN|536365|
+--------------------------------+ +--------------------+------+
|complex_map[WHITE METAL LANTERN]|
+--------------------------------+
| null|
| 536365|
+--------------------------------+
DATA ENGINEER S GUIDE T O 78
A PAC HE S PA R K A ND D ELTA L A K E

Working with JSON

Spark has some unique support for working with JSON data. You can operate directly %python
on strings of JSON in Spark and parse from JSON or extract JSON objects. Let’s start from pyspark.sql.functions import get_json_object, json_tuple
by creating a JSON column. jsonDF.select(
get_json_object(col(“jsonString”), “$.myJSONKey.myJSONValue[1]”),
%scala json_tuple(col(“jsonString”), “myJSONKey”))\
val jsonDF = spark.range(1) .show(2)
.selectExpr(“””
‘{“myJSONKey” : {“myJSONValue” : [1, 2, 3]}}’ as jsonString The equivalent in SQL would be.
“””)

jsonDF.selectExpr(“json_tuple(jsonString, ‘$.myJSONKey.myJSONValue[1]’)
%python as res”)
jsonDF = spark.range(1)\
.selectExpr(“”” +------+--------------------+
‘{“myJSONKey” : {“myJSONValue” : [1, 2, 3]}}’ as jsonString |column| c0|
“””) +------+--------------------+
| 2|{“myJSONValue”:[1...|
+------+--------------------+
We can use the get_json_object to inline query a JSON object, be it a dictionary or
array. We can use json_tuple if this object has only one level of nesting.

%scala
import org.apache.spark.sql.functions.{get_json_object, json_tuple}
jsonDF.select(
get_json_object(col(“jsonString”), “$.myJSONKey.myJSONValue[1]”),
json_tuple(col(“jsonString”), “myJSONKey”))
.show(2)
DATA ENGINEER S GUIDE T O 79
A PAC HE S PA R K A ND D ELTA L A K E

We can also turn a StructType into a JSON string using the to_json function. %scala
import org.apache.spark.sql.functions.from_json

%scala import org.apache.spark.sql.types._

import org.apache.spark.sql.functions.to_json val parseSchema = new StructType(Array(

df.selectExpr(“(InvoiceNo, Description) as myStruct”) new StructField(“InvoiceNo”,StringType,true),

.select(to_json(col(“myStruct”))) new StructField(“Description”,StringType,true)))

df.selectExpr(“(InvoiceNo, Description) as myStruct”)

%python .select(to_json(col(“myStruct”)).alias(“newJSON”))

from pyspark.sql.functions import to_json .select(from_json(col(“newJSON”), parseSchema), col(“newJSON”))

df.selectExpr(“(InvoiceNo, Description) as myStruct”)\

.select(to_json(col(“myStruct”))) %python
from pyspark.sql.functions import from_json

This function also accepts a dictionary (map) of parameters that are the same as the from pyspark.sql.types import *

JSON data source. We can use the from_json function to parse this (or other json) parseSchema = StructType((

back in. This naturally requires us to specify a schema and optionally we can specify StructField(“InvoiceNo”,StringType(),True),

a Map of options as well. StructField(“Description”,StringType(),True)))

df.selectExpr(“(InvoiceNo, Description) as myStruct”)\
.select(to_json(col(“myStruct”)).alias(“newJSON”))\
.select(from_json(col(“newJSON”), parseSchema), col(“newJSON”))\

+----------------------+--------------------+
|jsontostructs(newJSON)| newJSON|
+----------------------+--------------------+
| [536365,WHITE HAN...|{“InvoiceNo”:”536...|
| [536365,WHITE MET...|{“InvoiceNo”:”536...|
+----------------------+--------------------+
DATA ENGINEER S GUIDE T O 80
A PAC HE S PA R K A ND D ELTA L A K E

User-Defined Functions
One of the most powerful things that you can do in Spark is define your own functions. %python
These allow you to write your own custom transformations using Python or Scala and udfExampleDF = spark.range(5).toDF(“num”)
even leverage external libraries like numpy in doing so. These functions are called def power3(double_value):
user defined functions or UDFs and can take and return one or more columns return double_value ** 3
as input. Spark UDFs are incredibly powerful because they can be written in several power3(2.0)
different programming languages and do not have to be written in an esoteric format
or DSL. They’re just functions that operate on the data, record by record. By default, In this trivial example, we can see that our functions work as expected. We are able to
these functions are registered as temporary functions to be used in that specific provide an individual input and produce the expected result (with this simple test case).
SparkSession or Context. Thus far our expectations for the input are high, it must be a specific type and cannot
be a null value. See the section in this chapter titled “Working with Nulls in Data”.
While we can write our functions in Scala, Python, or Java, there are performance
considerations that you should be aware of. To illustrate this, we’re going to walk Now that we’ve created these functions and tested them, we need to register them
through exactly what happens when you create UDF, pass that into Spark, and then with Spark so that we can used them on all of our worker machines. Spark will
execute code using that UDF. serialize the function on the driver and transfer it over the network to all executor
processes. This happens regardless of language.
The first step is the actual function, we’ll just a take a simple one for this example.
We’ll write a power3 function that takes a number and raises it to a power of three. Once we go to use the function, there are essentially two different things that occur. If
the function is written in Scala or Java then we can use that function within the JVM.
%scala This means there will be little performance penalty aside from the fact that we can’t
val udfExampleDF = spark.range(5).toDF(“num”) take advantage of code generation capabilities that Spark has for built-in functions.
def power3(number:Double):Double = { There can be performance issues if you create or use a lot of objects which we will
number * number * number cover in the optimization section.
}
power3(2.0)
DATA ENGINEER S GUIDE T O 81
A PAC HE S PA R K A ND D ELTA L A K E

If the function is written in Python, something quite

different happens. Spark will start up a python process
EXECUTOR PROCESSES WORKER PYTHON PROCESS
on the worker, serialize all of the data to a format that
python can understand (remember it was in the JVM
before), execute the function row by row on that data in SPARK SESSION

the python process, before finally returning the results

of the row operations to the JVM and Spark. Driver

SCALA UDF
W A R N I N G | Starting up this Python process is
expensive but the real cost is in serializing the data to PYTHON UDF
Python. This is costly for two reasons, it is an expensive
computation but also once the data enters Python,
Spark cannot manage the memory of the worker. This
means that you could potentially cause a worker to fail
if it becomes resource constrained (because both the
JVM and python are competing for memory on the same
machine). We recommend that you write your UDFs in 1. Function serialized 2. Spark starts Python process 3. Python returns
and sent to workers and sends data answer
Scala - the small amount of time it should take you to
write the function in Scala will always yield significant
speed ups and on top of that, you can still use the
function from Python!
DATA ENGINEER S GUIDE T O 82
A PAC HE S PA R K A ND D ELTA L A K E

Now that we have an understanding of the process, let’s work through our example. +-----------+
|power3(num)|
First we need to register the function to be available as a DataFrame function.
+-----------+
| 0|
%scala | 1|

import org.apache.spark.sql.functions.udf +-----------+

val power3udf = udf(power3(_:Double):Double)

Now as of now, we can only use this as DataFrame function. That is to say, we can’t
Now we can use that just like any other DataFrame function. use it within a string expression, only on an expression. However, we can also register
this UDF as a Spark SQL function. This is valuable because it makes it simple to use
%scala this function inside of SQL as well as across languages.
udfExampleDF.select(power3udf(col(“num”))).show()
Let’s register the function in Scala.
The same applies to Python, we first register it.
%scala

%python spark.udf.register(“power3”, power3(_:Double):Double)

from pyspark.sql.functions import udf udfExampleDF.selectExpr(“power3(num)”).show(2)

power3udf = udf(power3)
Now because this function is registered with Spark SQL, and we’ve learned that any
Then we can use it in our DataFrame code. Spark SQL function or epxression is valid to use as an expression when working with
DataFrames, we can turn around and use the UDF that we wrote in Scala, in Python.
%python However rather than using it as a DataFrame function we use it as a SQL expression.
from pyspark.sql.functions import col
udfExampleDF.select(power3udf(col(“num”))).show() %python
udfExampleDF.selectExpr(“power3(num)”).show(2)
# registered in Scala
DATA ENGINEER S GUIDE T O 83
A PAC HE S PA R K A ND D ELTA L A K E

We can also register our Python function to be available as SQL function and use that Naturally we can use either of these from SQL too once we register them.
in any language as well.
%sql
One thing we can also do to make sure that our functions are working correctly is specify SELECT
a return type. As we saw in the beginning of this section, Spark manages its own power3py(12), -- doesn’t work because of return type
type information that does not align exactly with Python’s types. Therefore it’s a best power3(12)
practice to define the return type for your function when you define it. It is important
to note that specifying the return type is not necessary but is a best practice. When you want to optionally return a value from a UDF, you should return None in
python and an Option type in Scala.
If you specify the type that doesn’t align with the actual type returned by the function —
Spark will not error but rather just return null to designate a failure. You can see this ## Hive UDFs
if you were to switch the return type in the below function to be a DoubleType. As a last note, users can also leverage UDF/UDAF creation via a Hive syntax. To allow
for this, first you must enable Hive support when they create their SparkSession (via
%python SparkSession.builder().enableHiveSupport()) then you can register UDFs in
from pyspark.sql.types import IntegerType, DoubleType SQL. This is only supported with pre-compiled Scala and Java packages so you’ll have
spark.udf.register(“power3py”, power3, DoubleType()) to specify them as a dependency.

%python %sql
udfExampleDF.selectExpr(“power3py(num)”).show(2) CREATE TEMPORARY FUNCTION myFunc AS
# registered via Python ‘com.organization.hive.udf.FunctionName’

This is because the range above creates Integers. When Integers are operated on in Additionally, you can register this as a permanent function in the Hive Metastore by
Python, Python won’t convert them into floats (the corresponding type to Spark’s removing TEMPORARY.
Double type), therefore we see null. We can remedy this by ensuring our Python
function returns a float instead of an Integer and the function will behave correctly.
DATA ENGINEER S GUIDE T O 84
A PAC HE S PA R K A ND D ELTA L A K E

CHAPTER 3: Delta Lake Quickstart Delta Lake is an open-source storage layer that brings data reliability to data lakes. Delta Lake provides ACID
transactions, can handle metadata at scale, and can unify streaming and batch data processing. Delta Lake runs on
top of your existing data lake and is fully compatible with Apache Spark APIs.

How to start using Delta Lake

The Delta Lake package is available as with the--packages option. In our example, we will also demonstrate
the ability to VACUUM files and execute Delta Lake SQL commands within Apache Spark. As this is a short
demonstration, we will also enable the following configurations:
• spark.databricks.delta.retentionDurationCheck.enabled=false to allow us to vacuum files shorter
than the default retention duration of 7 days. Note, this is only required for the SQL command VACUUM.
•
spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension to enable Delta Lake SQL commands
within Apache Spark; this is not required for Python or Scala API calls.

# Using Spark Packages

./bin/pyspark --packages io.delta:delta-core_2.11:0.4.0 --conf “spark.databricks.
delta.retentionDurationCheck.enabled=false” --conf “spark.sql.extensions=io.delta.sql.
DeltaSparkSessionExtension”
DATA ENGINEER S GUIDE T O 85
A PAC HE S PA R K A ND D ELTA L A K E

Loading and saving our Delta Lake data

This scenario will be using the On-time flight performance or Departure Delays # Save flight delay data into Delta Lake format
dataset generated from the RITA BTS Flight Departure Statistics; some examples of departureDelays \
this data in action include the 2014 Flight Departure Performance via d3.js Crossfilter .write \
and On-Time Flight Performance with GraphFrames for Apache Spark™. This dataset .format(“delta”) \
can be downloaded locally from this github location. Within pyspark, start by reading .mode(“overwrite”) \
the dataset. .save(“departureDelays.delta”)

# Location variables N O T E | This approach is similar to how you would normally save Parquet data;
tripdelaysFilePath = “/root/data/departuredelays.csv” instead of specifying format(“parquet”), you will now specify format(“delta”). If
pathToEventsTable = “/root/deltalake/departureDelays.delta” you were to take a look at the underlying file system, you will notice four files created
# Read flight delay data for the departureDelays Delta Lake table.
departureDelays = spark.read \
.option(“header”, “true”) \ /departureDelays.delta$ ls -l
.option(“inferSchema”, “true”) \ .
.csv(tripdelaysFilePath) ..
_delta_log
Next, let’s save our departureDelays dataset to a Delta Lake table. By saving this table part-00000-df6f69ea-e6aa-424b-bc0e-f3674c4f1906-c000.snappy.parquet
to Delta Lake storage, we will be able to take advantage of its features including ACID part-00001-711bcce3-fe9e-466e-a22c-8256f8b54930-c000.snappy.parquet
transactions, unified batch and streaming, and time travel. part-00002-778ba97d-89b8-4942-a495-5f6238830b68-c000.snappy.parquet
part-00003-1a791c4a-6f11-49a8-8837-8093a3220581-c000.snappy.parquet

N O T E | The _delta_log is the folder that contains the Delta Lake transaction log.
For more information, refer to Diving Into Delta Lake: Unpacking The Transaction Log.
DATA ENGINEER S GUIDE T O 86
A PAC HE S PA R K A ND D ELTA L A K E

Now, let’s reload the data but this time our DataFrame will be backed by Delta Lake.

# Load flight delay data in Delta Lake format

delays_delta = spark \
.read \
.format(“delta”) \
.load(“departureDelays.delta”)
# Create temporary view
delays_delta.createOrReplaceTempView(“delays_delta”)
# How many flights are between Seattle and San Francisco
park.sql(“select count(1) from delays_delta where origin = ‘SEA’ and destination = ‘SFO’”).show()

Finally, let’s determine the number of flights originating from Seattle to San Francisco; in this dataset, there are 1698 flights.

In-place Conversion to Delta Lake

If you have existing Parquet tables, you have the ability to perform in-place conversions your tables to Delta Lake thus not needing to rewrite your table. To convert the table, you
can run the following commands.

from delta.tables import *

# Convert non partitioned parquet table at path ‘/path/to/table’
deltaTable = DeltaTable.convertToDelta(spark, “parquet.`/path/to/table`”)
# Convert partitioned parquet table at path ‘/path/to/table’ and partitioned by integer column named ‘part’ partitionedDeltaTable = DeltaTable.convert-
ToDelta(spark, “parquet.`/path/to/table`”, “part int”)
park.sql(“select count(1) from delays_delta where origin = ‘SEA’ and destination = ‘SFO’”).show()

For more information, including how to do this conversion in Scala and SQL, refer to Convert to Delta Lake.
DATA ENGINEER S GUIDE T O 87
A PAC HE S PA R K A ND D ELTA L A K E

Delete our Flight Data

To delete data from your traditional Data Lake table, you will need to: After we delete (more on this below) all of the on-time and early flights, as you can
1. Select all of the data from your table not including the rows you want to delete see from the preceding query there are 837 late flights originating from Seattle
2. Create a new table based on the previous query to San Francisco. If you review the file system, you will notice there are more files
3. Delete the original table even though you deleted data.
4. Rename the new table to the original table name for downstream dependencies.
Instead of performing all of these steps, with Delta Lake, we can simplify this process /departureDelays.delta$ ls -l
by running a DELETE statement. To show this, let’s delete all of the flights that had _delta_log
arrived early or on-time (i.e. delay < 0). part-00000-a2a19ba4-17e9-4931-9bbf-3c9d4997780b-c000.snappy.parquet
part-00000-df6f69ea-e6aa-424b-bc0e-f3674c4f1906-c000.snappy.parquet
from delta.tables import * part-00001-711bcce3-fe9e-466e-a22c-8256f8b54930-c000.snappy.parquet
from pyspark.sql.functions import * part-00001-a0423a18-62eb-46b3-a82f-ca9aac1f1e93-c000.snappy.parquet
# Access the Delta Lake table part-00002-778ba97d-89b8-4942-a495-5f6238830b68-c000.snappy.parquet
deltaTable = DeltaTable.forPath(spark, pathToEventsTable part-00002-bfaa0a2a-0a31-4abf-aa63-162402f802cc-c000.snappy.parquet
) part-00003-1a791c4a-6f11-49a8-8837-8093a3220581-c000.snappy.parquet
# Delete all on-time and early flights part-00003-b0247e1d-f5ce-4b45-91cd-16413c784a66-c000.snappy.parquet
deltaTable.delete(“delay < 0”)
# How many flights are between Seattle and San Francisco In traditional data lakes, deletes are performed by re-writing the entire table
spark.sql(“select count(1) from delays_delta where origin = ‘SEA’ and excluding the values to be deleted. With Delta Lake, deletes instead are performed
destination = ‘SFO’”).show() by selectively writing new versions of the files containing the data be deleted
and only marks the previous files as deleted. This is because Delta Lake uses
multiversion concurrency control to do atomic operations on the table: for
example, while one user is deleting data, another user may be querying the
previous version of the table. This multi-version model also enables us to travel
back in time (i.e. time travel) and query previous versions as we will see later.
DATA ENGINEER S GUIDE T O 88
A PAC HE S PA R K A ND D ELTA L A K E

Update our Flight Data

To update data from your traditional Data Lake table, you will need to:
1. Select all of the data from your table not including the rows you want to modify
2. Modify the rows that need to be updated/changed
3. Merge these two tables to create a new table
4. Delete the original table
5. Rename the new table to the original table name for downstream dependencies.

Instead of performing all of these steps, with Delta Lake, we can simplify this process by running an UPDATE statement. To show this, let’s update all of the flights originating
from Detroit to Seattle.

Update all flights originating from Detroit to now be originating from Seattle
deltaTable.update(“origin = ‘DTW’”, { “origin”: “’SEA’” } )
# How many flights are between Seattle and San Francisco
spark.sql(“select count(1) from delays_delta where origin = ‘SEA’ and destination = ‘SFO’”).show()

With the Detroit flights now tagged as Seattle flights, we now have 986 flights originating from Seattle to San Francisco. If you were to list the file system for
your departureDelays folder (i.e. $../departureDelays/ls -l), you will notice there are now 11 files (instead of the 8 right after deleting the files and the
four files after creating the table)
DATA ENGINEER S GUIDE T O 89
A PAC HE S PA R K A ND D ELTA L A K E

Merge our Flight Data

A common scenario when working with a data lake is to continuously append data to your table. This often results in duplicate data (rows you do not want inserted into your
table again), new rows that need to be inserted, and some rows that need to be updated. With Delta Lake, all of this can be achieved by using the merge operation (similar to the
SQL MERGE statement).

Let’s start with a sample dataset that you will want to be updated, inserted, or deduplicated with the following query.

# What flights between SEA and SFO for these date periods
spark.sql(“select * from delays_delta where origin = ‘SEA’ and destination = ‘SFO’ and date like ‘1010%’ limit 10”).show()

The output of this query looks like the table at left. Note, the color-coding has been added to this blog to clearly identify
which rows are deduplicated (blue), updated (yellow), and inserted (green).

Next, let’s generate our own merge_table that contains data we will insert, update or de-duplicate with the following
code snippet.

items = [(1010710, 31, 590, ‘SEA’, ‘SFO’), (1010521, 10, 590, ‘SEA’, ‘SFO’), (1010822, 31, 590, ‘SEA’, ‘SFO’)]
cols = [‘date’, ‘delay’, ‘distance’, ‘origin’, ‘destination’]
merge_table = spark.createDataFrame(items, cols)
merge_table.toPandas()
DATA ENGINEER S GUIDE T O 90
A PAC HE S PA R K A ND D ELTA L A K E

In the preceding table (merge_table), there are three rows that with a unique date value:
1. 1010521: this row needs to update the flights table with a new delay value (yellow)
2. 1010710: this row is a duplicate (blue)
3. 1010832: this is a new row to be inserted (green)

With Delta Lake, this can be easily achieved via a merge statement as noted in the following code snippet.

# Merge merge_table with flights

deltaTable.alias(“flights”) \
.merge(merge_table.alias(“updates”),”flights.date = updates.date”) \
.whenMatchedUpdate(set = { “delay” : “updates.delay” } ) \
.whenNotMatchedInsertAll() \
.execute()
# What flights between SEA and SFO for these date periods
spark.sql(“select * from delays_delta where origin = ‘SEA’ and destination = ‘SFO’ and date like ‘1010%’ limit 10”).show()

All three actions of de-duplication, update, and insert was efficiently completed with one statement.
DATA ENGINEER S GUIDE T O 91
A PAC HE S PA R K A ND D ELTA L A K E

View Table History

As previously noted, after each of our transactions (delete, update), there were more files created within the file system. This is because for each transaction, there are different
versions of the Delta Lake table. This can be seen by using the DeltaTable.history() method as noted below.

N O T E | You can also perform the same task with SQL: spark.sql(“DESCRIBE HISTORY ‘” + pathToEventsTable + “’”).show()

As you can see, there are three rows representing the different versions of the table (below is an abridged version to help make it easier to read) for each of the operations
(create table, delete, and update):
DATA ENGINEER S GUIDE T O 92
A PAC HE S PA R K A ND D ELTA L A K E

Travel Back in Time with Table History

With Time Travel, you can see review the Delta Lake table as of the version or timestamp. For more information, refer to Delta Lake documentation > Read older versions of data
using Time Travel. To view historical data, specify the version or Timestamp option; in the code snippet below, we will specify the version option

# Load DataFrames for each version

dfv0 = spark.read.format(“delta”).option(“versionAsOf”, 0).load(“departureDelays.delta”)
dfv1 = spark.read.format(“delta”).option(“versionAsOf”, 1).load(“departureDelays.delta”)
dfv2 = spark.read.format(“delta”).option(“versionAsOf”, 2).load(“departureDelays.delta”)
# Calculate the SEA to SFO flight counts for each version of history
cnt0 = dfv0.where(“origin = ‘SEA’”).where(“destination = ‘SFO’”).count()
cnt1 = dfv1.where(“origin = ‘SEA’”).where(“destination = ‘SFO’”).count()
cnt2 = dfv2.where(“origin = ‘SEA’”).where(“destination = ‘SFO’”).count()
# Print out the value
print(“SEA -> SFO Counts: Create Table: %s, Delete: %s, Update: %s” % (cnt0, cnt1, cnt2))
## Output
SEA -> SFO Counts: Create Table: 1698, Delete: 837, Update: 986

Whether for governance, risk management, and compliance (GRC) or rolling back errors, the Delta Lake table contains both the metadata (e.g. recording the fact that a delete
had occurred with these operators) and data (e.g. the actual rows deleted). But how do we remove the data files either for compliance or size reasons?
DATA ENGINEER S GUIDE T O 93
A PAC HE S PA R K A ND D ELTA L A K E

Cleanup Old Table Versions with Vacuum

The Delta Lake vacuum method will delete all of the rows (and files) by default that are N O T E | You perform the same task via SQL syntax:
older than 7 days (reference: Delta Lake Vacuum). If you were to view the file system,
you’ll notice the 11 files for your table. # Remove all files older than 0 hours old
spark.sql(“VACUUM ‘” + pathToEventsTable + “‘ RETAIN 0 HOURS”)
/departureDelays.delta$ ls -l
_delta_log Once the vacuum has completed, when you review the file system you will notice
part-00000-5e52736b-0e63-48f3-8d56-50f7cfa0494d-c000.snappy.parquet fewer files as the historical data has been removed.
part-00000-69eb53d5-34b4-408f-a7e4-86e000428c37-c000.snappy.parquet
part-00000-f8edaf04-712e-4ac4-8b42-368d0bbdb95b-c000.snappy.parquet /departureDelays.delta$ ls -l
part-00001-20893eed-9d4f-4c1f-b619-3e6ea1fdd05f-c000.snappy.parquet _delta_log
part-00001-9b68b9f6-bad3-434f-9498-f92dc4f503e3-c000.snappy.parquet part-00000-f8edaf04-712e-4ac4-8b42-368d0bbdb95b-c000.snappy.parquet
part-00001-d4823d2e-8f9d-42e3-918d-4060969e5844-c000.snappy.parquet part-00001-9b68b9f6-bad3-434f-9498-f92dc4f503e3-c000.snappy.parquet
part-00002-24da7f4e-7e8d-40d1-b664-95bf93ffeadb-c000.snappy.parquet part-00002-24da7f4e-7e8d-40d1-b664-95bf93ffeadb-c000.snappy.parquet
part-00002-3027786c-20a9-4b19-868d-dc7586c275d4-c000.snappy.parquet part-00003-b9292122-99a7-4223-aaa9-8646c281f199-c000.snappy.parquet
part-00002-f2609f27-3478-4bf9-aeb7-2c78a05e6ec1-c000.snappy.parquet
part-00003-850436a6-c4dd-4535-a1c0-5dc0f01d3d55-c000.snappy.parquet N O T E | The ability to time travel back to a version older than the retention period is lost
part-00003-b9292122-99a7-4223-aaa9-8646c281f199-c000.snappy.parquet after running vacuum.

To delete all of the files so that you only keep the current snapshot of data, you will What’s Next
specify a small value for the vacuum method (instead of the default retention of 7 days). Try out Delta Lake today by trying out the preceding code snippets on your Apache
Spark 2.4.3 (or greater) instance. By using Delta Lake, you can make your data lakes
# Remove all files older than 0 hours old. more reliable (whether you create a new one or migrate an existing data lake). To learn
deltaTable.vacuum(0) more, refer to delta.io and join the Delta Lake community via Slack and Google Group.
You can track all the upcoming releases and planned features in github milestones.
DATA ENGINEER S GUIDE T O
A PAC HE S PA R K A ND D ELTA L A K E

The datasets used in the book are also available for you to explore:

S PA R K : T H E D E F I N I T I V E G U I D E D ATA S E T S

Get started with a free trial of Databricks and take your

big data and data science projects to the next level today.

S TA R T YO U R F R E E T R I A L

Contact us for a personalized demo

C O N TA C T

© Databricks 2020. All rights reserved. Apache, Apache Spark, Spark, and the Spark Logo are trademarks of the Apache Software Foundation.
Delta Lake is a trademark of the Linux Foundation.