Learn how to use SQL queries to prepare, clean, and transform data for analysis!

One of the first tasks performed when doing data analytics is to create clean the dataset you’re working with. The insights you draw from your data are only as good as the data itself, so it’s no surprise that an estimated 80% of the time spent by analytics professionals involves preparing data for use in analysis.

SQL can help expedite this important task. In this tutorial, we will discuss different functions commonly used to clean, transform, and remove duplicate data from query outputs that may not be in the form we would like. This means you’ll learn about:

• CASE WHEN
• COALESCE
• NULLIF
• LEAST / GREATEST
• Casting
• DISTINCT

We will be using the following sample table, employees, throughout this tutorial to illustrate how our functions work:

This data is preloaded into a Next Tech sandbox for you to experiment and test the below queries. Connect to the database here for free!

Let’s get started!

This tutorial is adapted from Next Tech’s full SQL for Data Analysis course, which includes an in-browser sandboxed environment and interactive activities and challenges using real datasets. You can get started with this course here!

CASE WHEN

CASE WHEN is a function that allows a query to map various values in a column to other values. The general format of a CASE WHEN statement is:

CASE
    WHEN condition1 THEN value1
    WHEN condition2 THEN value2
    ...
    WHEN conditionX THEN valueX
    ELSE else_value
END

Here, condition1 and condition2, through conditionX, are Boolean conditions; value1 and value2, through valueX, are values to map the Boolean conditions; and else_value is the value that is mapped if none of the Boolean conditions are met.

For each row, the program starts at the top of the CASE WHEN statement and evaluates the first Boolean condition. The program then runs through each Boolean condition from the first one. For the first condition from the start of the statement that evaluates as true, the statement will return the value associated with that condition. If none of the statements evaluate as true, then the value associated with the ELSE statement will be returned.

As an example, let’s say you wanted to return all rows for employees from the employees table. Additionally, you would like to add a column that labels an employee as being an New employee if they were hired after 2019-01-01. Otherwise, it will mark the employee as a Standard employee. This column will be called employee_type. We can create this table by using a CASE WHEN statement as follows:

SELECT
    *,
    CASE
        WHEN hire_date >= '2019-01-01' THEN 'New'
        ELSE 'Standard'
    END AS employee_type
FROM
    employees;

This query will give the following output:

The CASE WHEN statement effectively mapped a hire date to a string describing the employee type. Using a CASE WHEN statement, you can map values in any way you please.

COALESCE

Another useful technique is to replace NULL values with a standard value. This can be accomplished easily by means of the COALESCE function. COALESCE allows you to list any number of columns and scalar values, and, if the first value in the list is NULL, it will try to fill it in with the second value. The COALESCE function will keep continuing down the list of values until it hits a non-NULL value. If all values in the COALESCE function are NULL, then the function returns NULL.

To illustrate a simple usage of the COALESCE function, let's say we want a list of the names and titles of our employees. However, for those with no title, we want to instead write the value 'NO TITLE'. We can accomplish this request with COALESCE:

SELECT
    first_name,
    last_name,
    COALESCE(title, 'NO TITLE') AS title
FROM
    employees;

This query produces the following results:

When dealing with creating default values and avoiding NULL, COALESCE will always be helpful.

NULLIF

NULLIF is, in a sense, the opposite of COALESCE. NULLIF is a two-value function and will return NULL if the first value equals the second value.

As an example, imagine that we want a list of the names and titles of our employees. However, this time, we want to replace the title 'Honorable' with NULL. This could be done with the following query:

SELECT
    first_name,
    last_name,
    NULLIF(title, 'Honorable') AS title
FROM
    employees;

This will blot out all mentions of 'Honorable' from the title column and give the following output:

LEAST / GREATEST

Two functions often come in handy for data preparation are the LEAST and GREATEST functions. Each function takes any number of values and returns the least or the greatest of the values, respectively.

A simple use of this variable would be to replace the value if it’s too high or low. For example, say the minimum wage increased to $15/hour and we need to change the wages of any employee earning less than that. We can create this using the following query:

SELECT
    id,
    first_name,
    last_name,
    title,
    age,
    GREATEST(15, wage) as wage,
    hire_date
FROM
    employees;

This query will give the following output:

As you can see, Bill Sadat’s wage has increased from $12 to $15.

Casting

Another useful data transformation is to change the data type of a column within a query. This is usually done to use a function only available to one data type, such as text, while working with a column that is in a different data type, such as a numeric.

To change the data type of a column, you simply need to use the column::datatype format, where column is the column name, and datatype is the data type you want to change the column to. For example, to change the age in the employees table to a text column in a query, use the following query:

SELECT
    first_name,
    last_name,
    age::TEXT
FROM
    employees;

This will convert the age column from an integer to text. You can now apply text functions to this transformed column. There is one final catch; not every data type can be cast to a specific data type. For instance, datetime cannot be cast to float types. Your SQL client will throw an error if you ever make an unexpected strange conversion.

DISTINCT

Often, when looking through a dataset, you may be interested in determining the unique values in a column or group of columns. This is the primary use case of the DISTINCT keyword. For example, if you wanted to know all the unique first names in the employees table, you could use the following query:

SELECT
    DISTINCT first_name
FROM
    employees;

This gives the following result:

You can also use DISTINCT with multiple columns to get all distinct column combinations present.

I hope you enjoyed this tutorial on data cleaning and transformation with SQL. This is just the beginning of what you can use SQL for in data analysis. If you’d like to learn more, Next Tech’s SQL for Data Analysis course covers:

• More functions used for data preparation and cleaning
• Aggregate functions and window functions
• Importing and exporting data
• Analytics using complex data types
• Writing performant queries

You can get started for free here!

Cleaning and Transforming Data with SQL was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Machine learning (ML) and deep learning (DL) techniques are being applied in a variety of fields, and data scientists are being sought after in many different industries.

With machine learning, we identify the processes through which we gain knowledge that is not readily apparent from data in order to make decisions. Applications of machine learning techniques may vary greatly, and are found in disciplines as diverse as medicine, finance, and advertising.

Deep learning makes use of more advanced neural networks than those used during the 1980s. This is not only a result of recent developments in the theory, but also advancements in computer hardware.

In this crash course, we will learn about deep learning and deep neural networks (DNNs), that is, neural networks with multiple hidden layers. We will cover the following topics:

• Introduction to deep learning
• Deep learning algorithms
• Applications of deep learning

Let’s get started!

This crash course is adapted from Next Tech’s Python Deep Learning Projects (Part 1: The Fundamentals) course, which explores the basics of deep learning, setting up a DL environment, and building an MLP. It includes an in-browser sandboxed environment with all the necessary software and libraries pre-installed. You can get started here!

Introduction to Deep Learning

Given the universal approximation theorem, you may wonder what the point of using more than one hidden layer is. This is in no way a naive question, and for a long time neural networks were used in this way.

One reason for multiple hidden layers is that approximating a complex function might require a huge number of neurons in the hidden layer, making it impractical to use. A more important reason for using deep networks, which is not directly related to the number of hidden layers but to the level of learning, is that a deep network does not simply learn to predict output Y given input X: it also understands basic features of the input.

Let’s take a look at an example.

In Proceedings of the International Conference on Machine Learning (ICML) (2009) by H. Lee, R. Grosse, R. Ranganath, and A. Ng, the authors train a neural network with pictures of different categories of either objects or animals. In the following image we can see how the different layers of the network learn different characteristics of the input data. In the first layer the network learns to detect some basic features, such as lines and edges, which are common to all images in all categories:

In the next layers, shown in the image below, it combines those lines and edges to compose more complex features that are specific to each category:

In the top row, we can see how the network detects different features of each category. Eyes, noses, and mouths for human faces, doors and wheels for cars, and so on. These features are abstract. That is, the network has learned the generic shape of a feature, such as a mouth or a nose, and can detect this feature in the input data despite variations it might have.

In the second row of the preceding image, we can see how the deeper layers of the network combine these features into even more complex ones, such as faces and whole cars. A strength of deep neural networks is that they can learn these high-level abstract representations themselves by deducing them from the training data.

Deep Learning Algorithms

We could define deep learning as a class of machine learning techniques where information is processed in hierarchical layers to understand representations and features from data in increasing levels of complexity. In practice, all deep learning algorithms are neural networks, which share some common basic properties. They all consist of interconnected neurons that are organized in layers. Where they differ is network architecture (the way neurons are organized in the network), and sometimes the way they are trained.

With that in mind, let’s look at the main classes of neural networks. The following list is not exhaustive, but it represents the vast majority of algorithms in use today.

Multi-Layer Perceptrons (MLPs)

A neural network with feedforward propagation, fully-connected layers, and at least one hidden layer.

The diagram demonstrates a 3-layer fully connected neural network with two hidden layers. The input layer has k input neurons, the first hidden layer has n hidden neurons, and the second hidden layer has m hidden neurons. The output, in this example, is the two classes y₁ and y₂. On top is the always-on bias neuron. A unit from one-layer is connected to all units from the previous and following layers (hence fully connected).

Convolutional Neural Networks (CNNs)

A CNN is a feedforward neural network with several types of special layers. For example, convolutional layers apply a filter to the input image (or sound) by sliding that filter all across the incoming signal, to produce an n-dimensional activation map. There is some evidence that neurons in CNNs are organized similarly to how biological cells are organized in the visual cortex of the brain. Today, they outperform all other ML algorithms on a large number of computer vision and natural language processing tasks.

Recurrent Neural Networks (RNNs)

This type of network has an internal state (or memory), which is based on all or part of the input data already fed to the network. The output of a recurrent network is a combination of its internal state (memory of previous inputs) and the latest input sample. At the same time, the internal state changes, to incorporate newly input data. Because of these properties, recurrent networks are good candidates for tasks that work on sequential data, such as text or time-series data.

Autoencoders

A class of unsupervised learning algorithms, in which the output shape is the same as the input, that allows the network to better learn basic representations. It consists of an input, hidden (or bottleneck), and output layers. Although it’s a single network, we can think of it as a virtual composition of two components:

• Encoder: Maps the input data to the network’s internal representation.
• Decoder: Tries to reconstruct the input from the network’s internal data representation.

Reinforcement Learning (RL)

Reinforcement algorithms learn how to achieve a complex objective over many steps by using penalties when they make a wrong decision and rewards when they make a correct decision. It is a method often used to teach a machine how to interact with an environment, similar to the way human behaviour is shaped by negative and positive feedback. RL is often used in building computer games and autonomous vehicles.

Applications of Deep Learning

Machine learning, particularly deep learning, is producing more and more astonishing results in terms of the quality of predictions, feature detection, and classification. Many of these recent results have even made the news! Here are just a few of the ways that these techniques can be applied today or in the near future:

Autonomous Vehicles

Nowadays, new cars have a suite of safety and convenience features that aim to make the driving experience safer and less stressful. One such feature is automated emergency braking if the car sees an obstacle. Another one is lane-keeping assist, which allows the vehicle to stay in its current lane without the driver needing to make corrections with the steering wheel. To recognize lane markings, other vehicles, pedestrians, and cyclists, these systems use a forward-facing camera. We can speculate that future autonomous vehicles will also use deep networks for computer vision.

Image and Text Recognition

Both Google’s Vision API and Amazon’s Rekognition services use deep learning models to provide various computer vision capabilities. These include recognizing and detecting objects and scenes in images, text recognition, face recognition, and so on.

Medical Imaging

Medical imaging is an umbrella term for various non-invasive methods of creating visual representations of the inside of the body. Some of these include Magnetic Resonance Images (MRIs), ultrasound, Computed Axial Tomography (CAT) scans, X-rays, and histology images. Typically, such an image is analyzed by a medical professional to determine the patient’s condition. Machine learning, computer vision, in particular, is enabling computer-aided diagnosis which can help specialists by detecting and highlighting important features of images.

For example, to determine the degree of malignancy of colon cancer a pathologist would have to analyze the morphology of the glands using histology imaging. This is a challenging task because morphology can vary greatly. A deep neural network could segment the glands from the image automatically, leaving the pathologist to verify the results. This would reduce the time needed for analysis, making it cheaper and more accessible.

Medical History Analysis

Another medical area that could benefit from deep learning is the analysis of medical history records. Before a doctor diagnoses a condition and prescribes treatment they consult the patient’s medical history for additional insight. A deep learning algorithm could extract the most relevant and important information from those extensive records, even if they are handwritten. In this way, the doctor’s job can be made easier while also reducing the risk of errors.

Language Translation

Google’s Neural Machine Translation API uses — you guessed it — deep neural networks for machine translation.

Speech Recognition and Generation

Google Duplex is another impressive real-world demonstration of deep learning. It’s a new system that can carry out natural conversations over the phone. For example, it can make restaurant reservations on a user’s behalf. It uses deep neural networks to both to understand the conversation and to generate realistic, human-like replies.

Siri, Google Assistant, and Amazon Alexa also rely on deep networks for speech recognition.

Gaming

Finally, AlphaGo is an artificial intelligence (AI) machine based on deep learning, that made the news in March 2016 for beating the world Go champion, Lee Sedol. AlphaGo had already made the news in January 2016, when it beat the European champion, Fan Hui. Although, at the time, it seemed unlikely that it could go on to beat the world champion. Fast-forward a couple of months and AlphaGo was able to achieve this remarkable feat by sweeping its opponent in a 4–1 victory series.

This was an important milestone because Go has many more possible game variations than other games, such as chess, and it’s impossible to consider every possible move in advance. Also, unlike chess, in Go it’s very difficult to even judge the current position or value of a single stone on the board. In 2017, DeepMind released an updated version of AlphaGo called AlphaZero.

In this crash course, we explained what deep learning is and how it’s related to deep neural networks. We discussed the different types of networks and some real-world applications of deep learning.

This is only the beginning — if you’d like to learn more about deep learning, Next Tech has a Python Deep Learning Projects series that explores real-world deep learning projects across computer vision, natural language processing (NLP), and image processing. Part 1 of the series covers the fundamental concepts of DL, how to set up a DL environment, and how to build an MLP. You can get started here!

Deep Learning Basics: A Crash Course was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learn the difference between mutable and immutable data types and how to find out which is which!

By now you may have heard the phrase “everything in Python is an object”. Objects are abstraction for data, and Python has an amazing variety of data structures that you can use to represent data, or combine them to create your own custom data.

A first fundamental distinction that Python makes on data is about whether or not the value of an object changes. If the value can change, the object is called mutable, while if the value cannot change, the object is called immutable.

In this crash course, we will explore:

• The difference between mutable and immutable types
• Different data types and how to find out whether they are mutable or immutable

It is very important that you understand the distinction between mutable and immutable because it affects the code you write.

Let’s get started!

This crash course is adapted from Next Tech’s Learn Python Programming course that uses a mix of theory and practicals to explore Python and its features, and progresses from beginner to being skilled in Python. It includes an in-browser sandboxed environment with all the necessary software and libraries pre-installed. You can get started for free here!

Mutable vs Immutable

To get started, it’s important to understand that every object in Python has an ID (or identity), a type, and a value, as shown in the following snippet:

age = 42
print(id(age))    # id
print(type(age))  # type
print(age)        # value

[Out:]
10966208
<class ‘int’>
42

Once created, the ID of an object never changes. It is a unique identifier for it, and it is used behind the scenes by Python to retrieve the object when we want to use it.

The type also never changes. The type tells what operations are supported by the object and the possible values that can be assigned to it.

The value can either change or not. If it can, the object is said to be mutable, while when it cannot, the object is said to be immutable.

Let’s take a look at an example:

age = 42
print(id(age))
print(type(age))
print(age)

age = 43
print(age)
print(id(age))

[Out:]
10966208
<class ‘int’>
42
43
10966240

Has the value of age changed? Well, no. 42 is an integer number, of the type int, which is immutable. So, what happened is really that on the first line, age is a name that is set to point to an int object, whose value is 42.

When we type age = 43, what happens is that another object is created, of the type int and value 43 (also, the id will be different), and the name age is set to point to it. So, we didn’t change that 42 to 43. We actually just pointed age to a different location.

As you can see from printing id(age) before and after the second object named age was created, they are different.

Now, let’s see the same example using a mutable object.

x = [1, 2, 3]
print(x)
print(id(x))

x.pop()
print(x)
print(id(x))

[Out:]
[1, 2, 3]
139912816421064
[1, 2]
139912816421064

For this example, we created a list named m that contains 3 integers, 1, 2, and 3. After we change m by “popping” off the last value 3, the ID of m stays the same!

So, objects of type int are immutable and objects of type list are mutable. Now let’s discuss other immutable and mutable data types!

Mutable Data Types

Mutable sequences can be changed after creation. Some of Python’s mutable data types are: lists, byte arrays, sets, and dictionaries.

Lists

As you saw earlier, lists are mutable. Here’s another example using the append() method:

a = list(('apple', 'banana', 'clementine'))
print(id(a))

a.append('dates')
print(id(a))

[Out:]
140372445629448
140372445629448

Byte Arrays

Byte arrays represent the mutable version of bytes objects. They expose most of the usual methods of mutable sequences as well as most of the methods of the bytes type. Items are integers in the range [0, 256).

Let’s see a quick example with the bytearray type to show that it is mutable:

b = bytearray(b'python')
print(id(b))

b.replace(b'p', b'P')
print(id(b))

[Out:]
139963525979808
139963525979808

Sets

Python provides two set types, set and frozenset. They are unordered collections of immutable objects.

c = set((‘San Francisco’, ‘Sydney’, ‘Sapporo’))
print(id(c))
c.pop()
print(id(c))

[Out:]
140494031990344
140494031990344

As you can see, sets are indeed mutable. Later, in the Immutable Data Types section, we will see that frozensets are immutable.

Dictionaries

d = {
    'a': 'alpha',
    'b': 'bravo',
    'c': 'charlie',
    'd': 'delta',
    'e': 'echo'
}
print(id(d))

d.update({
    'f': 'foxtrot'
})
print(id(d))

[Out:]
14007111431940
14007111431940

Immutable Data Types

Immutable data types differ from their mutable counterparts in that they can not be changed after creation. Some immutable types include numeric data types, strings, bytes, frozen sets, and tuples.

Numeric Data Types

You have already seen that integers are immutable; similarly, Python’s other built-in numeric data types such as booleans, floats, complex numbers, fractions, and decimals are also immutable!

Strings and Bytes

Textual data in Python is handled with str objects, more commonly known as strings. They are immutable sequences of Unicode code points. Unicode code points can represent a character.

When it comes to storing textual data though, or sending it on the network, you may want to encode it, using an appropriate encoding for the medium you’re using. The result of an encoding produces a bytes object, whose syntax and behavior is similar to that of strings.

Both strings and bytes are immutable, as shown in the following snippet:

# string
e = 'Hello, World!'
print(id(e))

e = 'Hello, Mars!'
print(id(e))

[Out:]
140595675113648
140595675113776

# bytes
unicode = 'This is üŋíc0de'  # unicode string: code points
print(type(unicode))
f = unicode.encode('utf-8')  # utf-8 encoded version
print(type(f))
print(id(f))

f = b'A bytes object'        # a bytes object
print(id(f))

[Out:]
<class 'str'>
<class 'bytes'>
140595675068152
140595675461360

In the bytes section, we first defined f as an encoded version of our unicode string. As you can see from print(type(f)) this is a bytes type. We then create another bytes object named f whose value is b'A bytes object'. The two f objects have different IDs, which shows that bytes are immutable.

Frozen Sets

As discussed in the previous section, frozensets are similar to sets. However, frozenset objects are quite limited in respect of their mutable counterpart since they cannot be changed. Nevertheless, they still prove very effective for membership test, union, intersection, and difference operations, and for performance reasons.

Tuples

The last immutable sequence type we’re going to see is the tuple. A tuple is a sequence of arbitrary Python objects. In a tuple, items are separated by commas. These, too, are immutable, as shown in the following example:

g = (1, 3, 5)
print(id(g))

g = (42, )
print(id(g))

[Out:]
139952252343784
139952253457184

I hope you enjoyed this crash course on the difference between immutable and mutable objects and how to find out which an object is! Now that you understand this fundamental concept of Python programming, you can now explore the methods you can use for each data type.

If you’d like to learn about this and continue to advance your Python skills, Next Tech has a full Learn Python Programming course that covers:

• Functions
• Conditional programming
• Comprehensions and generators
• Decorators, object-oriented programming, and iterators
• File data persistence
• Testing, including a brief introduction to test-driven development
• Exception handling
• Profiling and performances

You can get started here for free!

Immutable vs Mutable Data Types in Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Normalization is a technique for organizing data in a database. It is important that a database is normalized to minimize redundancy (duplicate data) and to ensure only related data is stored in each table. It also prevents any issues stemming from database modifications such as insertions, deletions, and updates.

The stages of organization are called normal forms. In this tutorial we will be redesigning a database for a construction company and ensuring that it satisfies the three normal forms:

First Normal Form (1NF):

• Data is stored in tables with rows uniquely identified by a primary key
• Data within each table is stored in individual columns in its most reduced form
• There are no repeating groups

Second Normal Form (2NF):

• Everything from 1NF
• Only data that relates to a table’s primary key is stored in each table

Third Normal Form (3NF):

• Everything from 2NF
• There are no in-table dependencies between the columns in each table

Note that there are actually six levels of normalization; however, the third normal form is considered the highest level necessary for most applications so we will only be discussing the first three forms.

Let’s get started!

This tutorial is adapted from Next Tech’s Database Fundamentals course which comes with an in-browser MySQL database and interactive tasks and projects to complete. You can get started here for free!

Our Database: Codey’s Construction

The database we will be working with in this tutorial is for Codey’s Construction company (Codey is a helpful coding bot that works with you in the course mentioned earlier). As you can see from the schema above, the database contains the tables projects, job_orders, employees, and project_employees. Recently, the customers table was added to store customer data.

Unfortunately, this table has not designed in a way that satisfies the three forms of normalization… Let’s fix that!

First Normal Form

First normal form relates to the duplication and over-grouping of data in tables and columns.

Codey’s Construction’s table customers violates all three rules of 1NF.

1. There is no primary key! A user of the database would be forced to look up companies by their name, which is not guaranteed to be unique (since unique company names are registered on a state-by-state basis).
2. The data is not in its most reduced form. The column contact_person_and_role can be further divided into two columns, such as contact_person and contact_role.
3. There are two repeating groups of columns — (project1_id, project1_feedback) and (project2_id, project2_feedback).

The following SQL statement was used to create the customers table:

By modifying some columns, we can help redesign this table so that it satisfies 1NF.

First, we need to add a primary key column called id with data type INT(6):

With this statement, we added an automatically incrementing primary key as the first column in the table.

To satisfy the second condition, we need to split the contact_person_and_role column:

Here, we simply renamed it as contact_person, and added a column contact_person_role immediately after it.

To satisfy the third condition, we need to move the columns containing project IDs and project feedback to a new table called project_feedbacks. First, let’s drop these columns from the customers table:

And then create the project_feedbacks table:

Here’s what the database schema looks like now:

As you can see, there are no more repeating groups in either the project_feedbacks table or the customers table. We still know which customer said what since project_feedbacks.customer_id refers back to the customers table.

Now our customers table satisfies 1NF! Let’s move on to second normal form.

Second Normal Form

To achieve second normal form, a database must first satisfy all the conditions for 1NF. After this, satisfying 2NF requires that all data in each table relates directly to the record that the primary key of the table identifies.

We are in violation of 2NF because the contact_person, contact_person_role and phone_number columns track data that relate to the contact person, not the customer. If the contact person for a customer changes, we would have to edit all of these columns, running the risk that we will change the values in one of the columns but forget to change another.

To help Codey’s Construction fix this table to satisfy 2NF, these columns should be moved to a table containing data on the contact person. First, let’s remove the columns in `customers` that are not related to our primary key:

Note that we kept the contact_person_id so we still know who to contact. Now, let’s create our new table contact_persons so we have somewhere to store data about each contact.

Codey’s Construction’s database schema now looks like this:

Now, if the contact person for a customer changes, the construction company just has to insert a record into the contact_persons table and change the contact_person_id in the customers table.

Third Normal Form

For a database to be in third normal form, it must first satisfy all the criteria for 2NF (and therefore, also 1NF).

Then, each column must be non-transitively dependent on the table’s primary key. This means that all columns in a table should rely on the primary key and no other column. If column_a relies on the primary key and also on column_b then column_a is transitively dependent on the primary key so the table does not satisfy 3NF.

Does your brain hurt from reading that? Don’t worry! It’s explained more below.

This is how the customers table looks after we have satisfied 1NF and 2NF:

The table currently has transitively dependent columns. The transitively dependent relationship is between city and zip. The city in which a customer is located relies on the customer, so this satisfies 2NF; however, the city also depends on the zip code. If a customer relocates, there may be a chance we update one column but not the other. Because this relationship exists, the database is not in 3NF.

To fix our database to satisfy 3NF, we need to drop the city column from customers, and create a new table zips to store this data:

That’s it! Finding issues that violate 3NF can be difficult, but it’s worth it to ensure that your database is resilient to errors caused by only partially updating data.

I hope you enjoyed this tutorial on database normalization! Codey’s Construction’s database now satisfies the three forms of normalization.

If you’d like to continue learning about databases, Next Tech’s Database Fundamentals course covers all you need to know to get started with databases and SQL. By helping an interactive coding bot named Codey, you will learn how to create and design databases, modify data, and write SQL queries to answer business questions. You can get started for free here!

Database Normalization Explained was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learn how to build and train a multilayer perceptron using TensorFlow’s high-level API Keras!

The development of Keras started in early 2015. As of today, it has evolved into one of the most popular and widely used libraries built on top of Theano and TensorFlow. One of its prominent features is that it has a very intuitive and user-friendly API, which allows us to implement neural networks in only a few lines of code.

Keras is also integrated into TensorFlow from version 1.1.0. It is part of the contrib module (which contains packages developed by contributors to TensorFlow and is considered experimental code).

In this tutorial we will look at this high-level TensorFlow API by walking through:

• The basics of feedforward neural networks
• Loading and preparing the popular MNIST dataset
• Building an image classifier
• Train a neural network and evaluate its accuracy

Let’s get started!

This tutorial is adapted from Part 4 of Next Tech’s Python Machine Learning series, which takes you through machine learning and deep learning algorithms with Python from 0 to 100. It includes an in-browser sandboxed environment with all the necessary software and libraries pre-installed, and projects using public datasets. You can get started for free here!

Multilayer Perceptrons

Multilayer feedforward neural networks are a special type of fully connected network with multiple single neurons. They are also called Multilayer Perceptrons (MLP). The following figure illustrates the concept of an MLP consisting of three layers:

The MLP depicted in the preceding figure has one input layer, one hidden layer, and one output layer. The units in the hidden layer are fully connected to the input layer, and the output layer is fully connected to the hidden layer. If such a network has more than one hidden layer, we also call it a deep artificial neural network.

We can add an arbitrary number of hidden layers to the MLP to create deeper network architectures. Practically, we can think of the number of layers and units in a neural network as additional hyperparameters that we want to optimize for a given problem task.

As shown in the preceding figure, we denote the ith activation unit in the ith layer as a_i^(l). To make the math and code implementations a bit more intuitive, we will use the in superscript for the input layer, the h superscript for the hidden layer, and the o superscript for the output layer.

For instance, a_i^(in)​ refers to the ith value in the input layer, a_i^(h)​ refers to the ith unit in the hidden layer, and a_i^(out)​ refers to the ith unit in the output layer. Here, the activation units a_0^(in) and a_0^(out)​ are the bias units, which we set equal to 1. The activation of the units in the input layer is just its input plus the bias unit:

Each unit in layer l is connected to all units in layer l + 1 via a weight coefficient. For example, the connection between the kth unit in layer l to the jth unit in layer l + 1 will be written as w_{k, j}^(l)​. Referring back to the previous figure, we denote the weight matrix that connects the input to the hidden layer as W^(h)​, and we write the matrix that connects the hidden layer to the output layer as W^(out)​.

We summarize the weights that connect the input and hidden layers by a matrix:

where d is the number of hidden units and m is the number of input units including the bias unit. Since it is important to internalize this notation to follow the concepts later in this tutorial, let’s summarize what we have just learned in a descriptive illustration of a simplified 3–4–3 multilayer perceptron:

The MNIST dataset

To see what neural network training via the tensorflow.keras (tf.keras) high-level API looks like, let's implement a multilayer perceptron to classify the handwritten digits from the popular Mixed National Institute of Standards and Technology (MNIST) dataset that serves as a popular benchmark dataset for machine learning algorithm.

To follow along with the code snippets in this tutorial, you can use this Next Tech sandbox, which has the MNIST dataset and all necessary packages installed. Otherwise, you can use your local environment and download the dataset here.

The MNIST dataset in four parts, as listed here:

• Training set images: train-images-idx3-ubyte.gz — 60,000 samples
• Training set labels: train-labels-idx1-ubyte.gz — 60,000 labels
• Test set images: t10k-images-idx3-ubyte.gz — 10,000 samples
• Test set labels: t10k-labels-idx1-ubyte.gz — 10,000 labels

The training set consists of handwritten digits from 250 different people (50% high school students, 50% employees from the Census Bureau). The test set contains handwritten digits from different people.

Note that TensorFlow also provides the same dataset as follows:

However, we will work with the MNIST dataset as an external dataset to learn all the steps of data preprocessing separately. This way, you learn what you need to do with your own dataset.

The first step is to unzip the four parts of the MNIST dataset by running the following commands in your Terminal:

The load_mnist function returns two arrays, the first being an n x m dimensional NumPy array (images), where n is the number of samples and m is the number of features (here, pixels). The images in the MNIST dataset consist of 28 x 28 pixels, and each pixel is represented by a gray scale intensity value. Here, we unroll the 28 x 28 pixels into one-dimensional row vectors, which represent the rows in our images array (784 per row or image). The second array (labels) returned by the load_mnist function contains the corresponding target variable, the class labels (integers 0-9) of the handwritten digits.

Then, the dataset is loaded and prepared as follows:

[Out:]
 Rows: 60000,  Columns: 784
 Rows: 10000,  Columns: 784
 (60000, 784) (60000,)
 (10000, 784) (10000,)

To get an idea of how those images in MNIST look, let’s visualize examples of the digits 0–9 via Matplotlib’s imshowfunction:

We should now see a plot of the 2 x 5 subfigures showing a representative image of each unique digit:

Now let’s start building our model!

Building an MLP using TensorFlow’s Keras API

First, let’s set the random seed for NumPy and TensorFlow so that we get consistent results:

To continue with the preparation of the training data, we need to convert the class labels (integers 0–9) into the one-hot format. Fortunately, Keras provides a convenient tool for this:

First 3 labels:  [5 0 4]

First 3 labels (one-hot):
 [[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]]

Now, let’s implement our neural network! Briefly, we will have three layers, where the first two layers (the input and hidden layers) each have 50 units with the tanh activation function and the last layer (the output layer) has 10 layers for the 10 class labels and uses softmax to give the probability of each class. Keras makes these tasks very simple:

First, we initialize a new model using the Sequential class to implement a feedforward neural network. Then, we can add as many layers to it as we like. However, since the first layer that we add is the input layer, we have to make sure that the input_dim attribute matches the number of features (columns) in the training set (784 features or pixels in the neural network implementation).

Also, we have to make sure that the number of output units (units) and input units (input_dim) of two consecutive layers match. Our first two layers have 50 units plus one bias unit each. The number of units in the output layer should be equal to the number of unique class labels — the number of columns in the one-hot-encoded class label array.

Note that we used glorot_uniform to as the initialization algorithm for weight matrices. Glorot initialization is a more robust way of initialization for deep neural networks. The biases are initialized to zero, which is more common, and in fact the default setting in Keras.

Before we can compile our model, we also have to define an optimizer. We chose a stochastic gradient descent optimization. Furthermore, we can set values for the weight decay constant and momentum learning to adjust the learning rate at each epoch. Lastly, we set the cost (or loss) function to categorical_crossentropy.

The binary cross-entropy is just a technical term for the cost function in the logistic regression, and the categorical cross-entropy is its generalization for multiclass predictions via softmax.

After compiling the model, we can now train it by calling the fit method. Here, we are using mini-batch stochastic gradient with a batch size of 64 training samples per batch. We train the MLP over 50 epochs, and we can follow the optimization of the cost function during training by setting verbose=1.

The validation_split parameter is especially handy since it will reserve 10% of the training data (here, 6,000 samples) for validation after each epoch so that we can monitor whether the model is overfitting during training:

Printing the value of the cost function is extremely useful during training to quickly spot whether the cost is decreasing during training and stop the algorithm earlier. Otherwise, hyperparameter values will need to be tuned.

To predict the class labels, we can then use the predict_classes method to return the class labels directly as integers:

First 3 predictions: [5 0 4]

Finally, let’s print the model accuracy on training and test sets:

Training accuracy: 98.81
Test accuracy: 96.27

I hope you enjoyed this tutorial on using TensorFlow's keras API to build and train a multilayered neural network for image classification! Note that this is just a very simple neural network without optimized tuning parameters.

In practice you need to know how to optimize the model by tweaking learning rate, momentum, weight decay, and number of hidden units. You also need to learn how to deal with the vanishing gradient problem, wherein error gradients become increasingly small as more layers are added to a network.

We cover these topics in Next Tech’s Python Machine Learning (Part 4) course, as well as:

• Breaking down the mechanics of TensorFlow, such as tensors, activation functions computation graphs, variables, and placeholders
• Low-level TensorFlow and another high-level API, Layers
• Modeling sequential data using recurrent neural networks (RNN) and long short-term memory (LSTM) networks
• Classifying images with deep convolutional neural networks (CNN).

You can get started here for free!

Introduction to Multilayer Neural Networks with TensorFlow’s Keras API was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Clustering (or cluster analysis) is a technique that allows us to find groups of similar objects, objects that are more related to each other than to objects in other groups. Examples of business-oriented applications of clustering include the grouping of documents, music, and movies by different topics, or finding customers that share similar interests based on common purchase behaviors as a basis for recommendation engines.

In this tutorial, we will learn about one of the most popular clustering algorithms, k-means, which is widely used in academia as well as in industry. We will cover:

• The basic concepts of k-means clustering
• The mathematics behind the k-means algorithm
• The advantages and disadvantages of k-means
• How to implement the algorithm on a sample dataset using scikit-learn
• How to visualize clusters
• How to choose the optimal k using the elbow method

Let’s get started!

This tutorial is adapted from Part 3 of Next Tech’s Python Machine Learning series, which takes you through machine learning and deep learning algorithms with Python from 0 to 100. It includes an in-browser sandboxed environment with all the necessary software and libraries pre-installed, and projects using public datasets. You can get started for free here!

Fundamentals of K-Means Clustering

As we will see, the k-means algorithm is extremely easy to implement and is also computationally very efficient compared to other clustering algorithms, which might explain its popularity. The k-means algorithm belongs to the category of prototype-based clustering.

Prototype-based clustering means that each cluster is represented by a prototype, which can either be the centroid (average) of similar points with continuous features, or the medoid (the most representative or most frequently occurring point) in the case of categorical features.

While k-means is very good at identifying clusters with a spherical shape, one of the drawbacks of this clustering algorithm is that we have to specify the number of clusters, k, a priori. An inappropriate choice for k can result in poor clustering performance — we will discuss later in this tutorial how to choose k.

Although k-means clustering can be applied to data in higher dimensions, we will walk through the following examples using a simple two-dimensional dataset for the purpose of visualization.

You can follow along with the code in this tutorial by using a Next Tech sandbox, which has all the necessary libraries pre-installed, or if you’d prefer, you can run the snippets in your own local environment.

Once your sandbox loads, let’s import the toy dataset from scikit-learn and visualize the datapoints:

The dataset that we just created consists of 150 randomly generated points that are roughly grouped into three regions with higher density, which is visualized via a two-dimensional scatterplot.

In real-world applications of clustering, we do not have any ground truth category information (information provided as empirical evidence as opposed to inference) about those samples; otherwise, it would fall into the category of supervised learning. Thus, our goal is to group the samples based on their feature similarities, which can be achieved using the k-means algorithm that can be summarized by the following four steps:

1. Randomly pick k centroids from the sample points as initial cluster centers.
2. Assign each sample to the nearest centroid μ^(j), j ∈ {1, …, k}.
3. Move the centroids to the center of the samples that were assigned to it.
4. Repeat steps 2 and 3 until the cluster assignments do not change or a user-defined tolerance or maximum number of iterations is reached.

Now, the next question is how do we measure similarity between objects? We can define similarity as the opposite of distance, and a commonly used distance for clustering samples with continuous features is the squared Euclidean distance between two points x and y in m-dimensional space:

Note that in the preceding equation, the index j refers to the jth dimension (feature column) of the sample points x and y. We will use the superscripts i and j to refer to the sample index and cluster index, respectively.

Based on this Euclidean distance metric, we can describe the k-means algorithm as a simple optimization problem, an iterative approach for minimizing the within-cluster Sum of Squared Errors(SSE), which is sometimes also called cluster inertia:

Here,

and

Note that when we are applying k-means to real-world data using a Euclidean distance metric, we want to make sure that the features are measured on the same scale and apply z-score standardization or min-max scaling if necessary.

K-means clustering using scikit-learn

Now that we have learned how the k-means algorithm works, let’s apply it to our sample dataset using the KMeans class from scikit-learn's cluster module:

Using the preceding code, we set the number of desired clusters to 3. We set n_init=10 to run the k-means clustering algorithms 10 times independently with different random centroids to choose the final model as the one with the lowest SSE. Via the max_iter parameter, we specify the maximum number of iterations for each single run (here, 300).

Note that the k-means implementation in scikit-learn stops early if it converges before the maximum number of iterations is reached. However, it is possible that k-means does not reach convergence for a particular run, which can be problematic (computationally expensive) if we choose relatively large values for max_iter.

One way to deal with convergence problems is to choose larger values for tol, which is a parameter that controls the tolerance with regard to the changes in the within-cluster sum-squared-error to declare convergence. In the preceding code, we chose a tolerance of 1e-04 (= 0.0001).

A problem with k-means is that one or more clusters can be empty. However, this problem is accounted for in the current k-means implementation in scikit-learn. If a cluster is empty, the algorithm will search for the sample that is farthest away from the centroid of the empty cluster. Then it will reassign the centroid to be this farthest point.

Now that we have predicted the cluster labels y_km, let’s visualize the clusters that k-means identified in the dataset together with the cluster centroids. These are stored under the cluster_centers_ attribute of the fitted KMeans object:

In the resulting scatterplot, we can see that k-means placed the three centroids at the center of each sphere, which looks like a reasonable grouping given this dataset.

The Elbow Method

Although k-means worked well on this toy dataset, it is important to reiterate that a drawback of k-means is that we have to specify the number of clusters, k, before we know what the optimal k is. The number of clusters to choose may not always be so obvious in real-world applications, especially if we are working with a higher dimensional dataset that cannot be visualized.

The elbow method is a useful graphical tool to estimate the optimal number of clusters k for a given task. Intuitively, we can say that, if k increases, the within-cluster SSE (“distortion”) will decrease. This is because the samples will be closer to the centroids they are assigned to.

The idea behind the elbow method is to identify the value of k where the distortion begins to decrease most rapidly, which will become clearer if we plot the distortion for different values of k:

As we can see in the resulting plot, the elbow is located at k = 3, which is evidence that k = 3 is indeed a good choice for this dataset.

I hope you enjoyed this tutorial on the k-means algorithm! We explored the basic concepts and mathematics behind the k-means algorithm, how to implement k-means, and how to select an optimal number of clusters, k.

If you’d like to learn more, Next Tech’s Python Machine Learning (Part 3) course further explores clustering algorithms and techniques such as:

• Silhouette plots, another method used to select the optimal k
• k-means++, a variant of k-means, that improves clustering results through more clever seeding of the initial cluster centers.
• Other categories of clustering algorithms, such as hierarchical and density-based clustering, that do not require us to specify the number of clusters upfront or assume spherical structures in our dataset.

The course also explores regression analysis, sentiment analysis, and how to deploy a dynamic machine learning model to a web application. You can get started here!

K-Means Clustering with scikit-learn was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the modern age of technology, increasing amounts of data are produced and collected. In machine learning, however, too much data can be a bad thing. At a certain point, more features or dimensions can decrease a model’s accuracy since there is more data that needs to be generalized — this is known as the curse of dimensionality.

Dimensionality reduction is way to reduce the complexity of a model and avoid overfitting. There are two main categories of dimensionality reduction: feature selection and feature extraction. Via feature selection, we select a subset of the original features, whereas in feature extraction, we derive information from the feature set to construct a new feature subspace.

In this tutorial we will explore feature extraction. In practice, feature extraction is not only used to improve storage space or the computational efficiency of the learning algorithm, but can also improve the predictive performance by reducing the curse of dimensionality — especially if we are working with non-regularized models.

Specifically, we will discuss the Principal Component Analysis (PCA) algorithm used to compress a dataset onto a lower-dimensional feature subspace with the goal of maintaining most of the relevant information. We will explore:

• The concepts and mathematics behind PCA
• How to execute PCA step-by-step from scratch using Python
• How to execute PCA using the Python library scikit-learn

Let’s get started!

This tutorial is adapted from Part 2 of Next Tech’s Python Machine Learning series, which takes you through machine learning and deep learning algorithms with Python from 0 to 100. It includes an in-browser sandboxed environment with all the necessary software and libraries pre-installed, and projects using public datasets. You can get started for free here!

Introduction to Principal Component Analysis

Principal Component Analysis (PCA) is an unsupervised linear transformation technique that is widely used across different fields, most prominently for feature extraction and dimensionality reduction. Other popular applications of PCA include exploratory data analyses and de-noising of signals in stock market trading, and the analysis of genome data and gene expression levels in the field of bioinformatics.

PCA helps us to identify patterns in data based on the correlation between features. In a nutshell, PCA aims to find the directions of maximum variance in high-dimensional data and projects it onto a new subspace with equal or fewer dimensions than the original one.

The orthogonal axes (principal components) of the new subspace can be interpreted as the directions of maximum variance given the constraint that the new feature axes are orthogonal to each other, as illustrated in the following figure:

In the preceding figure, x1 and x2 are the original feature axes, and PC1 and PC2 are the principal components.

If we use PCA for dimensionality reduction, we construct a d x k–dimensional transformation matrix W that allows us to map a sample vector x onto a new k–dimensional feature subspace that has fewer dimensions than the original d–dimensional feature space:

As a result of transforming the original d-dimensional data onto this new k-dimensional subspace (typically k ≪ d), the first principal component will have the largest possible variance, and all consequent principal components will have the largest variance given the constraint that these components are uncorrelated (orthogonal) to the other principal components — even if the input features are correlated, the resulting principal components will be mutually orthogonal (uncorrelated).

Note that the PCA directions are highly sensitive to data scaling, and we need to standardize the features prior to PCA if the features were measured on different scales and we want to assign equal importance to all features.

Before looking at the PCA algorithm for dimensionality reduction in more detail, let’s summarize the approach in a few simple steps:

1. Standardize the d-dimensional dataset.
2. Construct the covariance matrix.
3. Decompose the covariance matrix into its eigenvectors and eigenvalues.
4. Sort the eigenvalues by decreasing order to rank the corresponding eigenvectors.
5. Select k eigenvectors which correspond to the k largest eigenvalues, where k is the dimensionality of the new feature subspace (k ≤ d).
6. Construct a projection matrix W from the “top” k eigenvectors.
7. Transform the d-dimensional input dataset X using the projection matrix W to obtain the new k-dimensional feature subspace.

Let’s perform a PCA step by step, using Python as a learning exercise. Then, we will see how to perform a PCA more conveniently using scikit-learn.

Extracting the Principal Components Step By Step

We will be using the Wine dataset from The UCI Machine Learning Repository in our example. This dataset consists of 178 wine samples with 13 features describing their different chemical properties. You can find out more here.

In this section we will tackle the first four steps of a PCA; later we will go over the last three. You can follow along with the code in this tutorial by using a Next Tech sandbox, which has all the necessary libraries pre-installed, or if you’d prefer, you can run the snippets in your own local environment.

Once your sandbox loads, we will start by loading the Wine dataset directly from the repository:

Next, we will process the Wine data into separate training and test sets — using a 70:30 split — and standardize it to unit variance:

After completing the mandatory preprocessing, let’s advance to the second step: constructing the covariance matrix. The symmetric d x d-dimensional covariance matrix, where d is the number of dimensions in the dataset, stores the pairwise covariances between the different features. For example, the covariance between two features x_j and x_k on the population level can be calculated via the following equation:

Here, μ_j and μ_k are the sample means of features j and k, respectively.

Note that the sample means are zero if we standardized the dataset. A positive covariance between two features indicates that the features increase or decrease together, whereas a negative covariance indicates that the features vary in opposite directions. For example, the covariance matrix of three features can then be written as follows (note that Σ stands for the Greek uppercase letter sigma, which is not to be confused with the sum symbol):

The eigenvectors of the covariance matrix represent the principal components (the directions of maximum variance), whereas the corresponding eigenvalues will define their magnitude. In the case of the Wine dataset, we would obtain 13 eigenvectors and eigenvalues from the 13 x 13-dimensional covariance matrix.

Now, for our third step, let’s obtain the eigenpairs of the covariance matrix. An eigenvector v satisfies the following condition:

Here, λ is a scalar: the eigenvalue. Since the manual computation of eigenvectors and eigenvalues is a somewhat tedious and elaborate task, we will use the linalg.eig function from NumPy to obtain the eigenpairs of the Wine covariance matrix:

Using the numpy.cov function, we computed the covariance matrix of the standardized training dataset. Using the linalg.eig function, we performed the eigendecomposition, which yielded a vector (eigen_vals) consisting of 13 eigenvalues and the corresponding eigenvectors stored as columns in a 13 x 13-dimensional matrix (eigen_vecs).

Total and Explained Variance

Since we want to reduce the dimensionality of our dataset by compressing it onto a new feature subspace, we only select the subset of the eigenvectors (principal components) that contains most of the information (variance). The eigenvalues define the magnitude of the eigenvectors, so we have to sort the eigenvalues by decreasing magnitude; we are interested in the top k eigenvectors based on the values of their corresponding eigenvalues.

But before we collect those k most informative eigenvectors, let’s plot the variance explained ratios of the eigenvalues. The variance explained ratio of an eigenvalue λ_j is simply the fraction of an eigenvalue λ_j and the total sum of the eigenvalues:

Using the NumPy cumsum function, we can then calculate the cumulative sum of explained variances, which we will then plot via matplotlib’s step function:

The resulting plot indicates that the first principal component alone accounts for approximately 40% of the variance. Also, we can see that the first two principal components combined explain almost 60% of the variance in the dataset.

Feature Transformation

After we have successfully decomposed the covariance matrix into eigenpairs, let’s now proceed with the last three steps of PCA to transform the Wine dataset onto the new principal component axes.

We will sort the eigenpairs by descending order of the eigenvalues, construct a projection matrix from the selected eigenvectors, and use the projection matrix to transform the data onto the lower-dimensional subspace.

We start by sorting the eigenpairs by decreasing order of the eigenvalues:

Next, we collect the two eigenvectors that correspond to the two largest eigenvalues, to capture about 60% of the variance in this dataset. Note that we only chose two eigenvectors for the purpose of illustration, since we are going to plot the data via a two-dimensional scatter plot later in this subsection. In practice, the number of principal components has to be determined by a trade-off between computational efficiency and the performance of the classifier:

[Out:]
Matrix W:
 [[-0.13724218  0.50303478]
 [ 0.24724326  0.16487119]
 [-0.02545159  0.24456476]
 [ 0.20694508 -0.11352904]
 [-0.15436582  0.28974518]
 [-0.39376952  0.05080104]
 [-0.41735106 -0.02287338]
 [ 0.30572896  0.09048885]
 [-0.30668347  0.00835233]
 [ 0.07554066  0.54977581]
 [-0.32613263 -0.20716433]
 [-0.36861022 -0.24902536]
 [-0.29669651  0.38022942]]

By executing the preceding code, we have created a 13 x 2-dimensional projection matrix W from the top two eigenvectors.

Using the projection matrix, we can now transform a sample x (represented as a 1 x 13-dimensional row vector) onto the PCA subspace (the principal components one and two) obtaining x′, now a two-dimensional sample vector consisting of two new features:

Similarly, we can transform the entire 124 x 13-dimensional training dataset onto the two principal components by calculating the matrix dot product:

Lastly, let’s visualize the transformed Wine training set, now stored as an 124 x 2-dimensional matrix, in a two-dimensional scatterplot:

As we can see in the resulting plot, the data is more spread along the x-axis — the first principal component — than the second principal component (y-axis), which is consistent with the explained variance ratio plot that we created previously. However, we can intuitively see that a linear classifier will likely be able to separate the classes well.

Although we encoded the class label information for the purpose of illustration in the preceding scatter plot, we have to keep in mind that PCA is an unsupervised technique that does not use any class label information.

PCA in scikit-learn

Although the verbose approach in the previous subsection helped us to follow the inner workings of PCA, we will now discuss how to use the PCA class implemented in scikit-learn. The PCA class is another one of scikit-learn’s transformer classes, where we first fit the model using the training data before we transform both the training data and the test dataset using the same model parameters.

Let’s use the PCA class on the Wine training dataset, classify the transformed samples via logistic regression:

Now, using a custom plot_decision_regions function, we will visualize the decision regions:

By executing the preceding code, we should now see the decision regions for the training data reduced to two principal component axes.

For the sake of completeness, let’s plot the decision regions of the logistic regression on the transformed test dataset as well to see if it can separate the classes well:

After we plotted the decision regions for the test set by executing the preceding code, we can see that logistic regression performs quite well on this small two-dimensional feature subspace and only misclassifies very few samples in the test dataset.

If we are interested in the explained variance ratios of the different principal components, we can simply initialize the PCA class with the n_components parameter set to None, so all principal components are kept and the explained variance ratio can then be accessed via the explained_variance_ratio_ attribute:

Note that we set n_components=None when we initialized the PCA class so that it will return all principal components in a sorted order instead of performing a dimensionality reduction.

I hope you enjoyed this tutorial on principal component analysis for dimensionality reduction! We covered the mathematics behind the PCA algorithm, how to perform PCA step-by-step with Python, and how to implement PCA using scikit-learn. Other techniques for dimensionality reduction are Linear Discriminant Analysis (LDA) and Kernel PCA (used for non-linearly separable data).

These other techniques and more topics to improve model performance, such as data preprocessing, model evaluation, hyperparameter tuning, and ensemble learning techniques are covered in Next Tech’s Python Machine Learning (Part 2) course.

You can get started here for free!

Principal Component Analysis for Dimensionality Reduction was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

A decision tree is a supervised machine learning model used to predict a target by learning decision rules from features. As the name suggests, we can think of this model as breaking down our data by making a decision based on asking a series of questions.

Let's consider the following example in which we use a decision tree to decide upon an activity on a particular day:

Based on the features in our training set, the decision tree model learns a series of questions to infer the class labels of the samples. As we can see, decision trees are attractive models if we care about interpretability.

Although the preceding figure illustrates the concept of a decision tree based on categorical targets (classification), the same concept applies if our targets are real numbers (regression).

In this tutorial, we will discuss how to build a decision tree model with Python’s scikit-learn library. We will cover:

• The fundamental concepts of decision trees
• The mathematics behind the decision tree learning algorithm
• Information gain and impurity measures
• Classification trees
• Regression trees

Let’s get started!

This tutorial is adapted from Next Tech’s Python Machine Learning series which takes you through machine learning and deep learning algorithms with Python from 0 to 100. It includes an in-browser sandboxed environment with all the necessary software and libraries pre-installed, and projects using public datasets. You can get started here!

The Fundamentals of Decision Trees

A decision tree is constructed by recursive partitioning — starting from the root node (known as the first parent), each node can be split into left and right child nodes. These nodes can then be further split and they themselves become parent nodes of their resulting children nodes.

For example, looking at the image above, the root node is Work to do? and splits into the child nodes Stay in and Outlook based on whether or not there is work to do. The Outlook node further splits into three child nodes.

So, how do we know what the optimal splitting point is at each node?

Starting from the root, the data is split on the feature that results in the largest Information Gain (IG) (explained in more detail below). In an iterative process, we then repeat this splitting procedure at each child node until the leaves are pure — i.e. samples at each node all belong to the same class.

In practice, this can result in a very deep tree with many nodes, which can easily lead to overfitting. Thus, we typically want to prune the tree by setting a limit for the maximal depth of the tree.

Maximizing Information Gain

In order to split the nodes at the most informative features, we need to define an objective function that we want to optimize via the tree learning algorithm. Here, our objective function is to maximize the information gain at each split, which we define as follows:

Here, f is the feature to perform the split, Dp, Dleft, and Dright are the datasets of the parent and child nodes, I is the impurity measure, Np is the total number of samples at the parent node, and Nleft and Nright are the number of samples in the child nodes.

We will discuss impurity measures for classification and regression decision trees in more detail in our examples below. But for now, just understand that information gain is simply the difference between the impurity of the parent node and the sum of the child node impurities — the lower the impurity of the child nodes, the larger the information gain.

Note that the above equation is for binary decision trees — each parent node is split into two child nodes only. If you have a decision tree with multiple nodes, you would simply sum the impurity of all nodes.

Classification Trees

We will start by talking about classification decision trees (also known as classification trees). For this example, we will be using the Iris dataset, a classic in the field of machine learning. It contains the measurements of 150 Iris flowers from three different species — Setosa, Versicolor, and Virginica. These will be our targets. Our goal is to predict which category an Iris flower belongs to. The petal length and width in centimeters are stored as columns, which we also call the features of the dataset.

Let’s first import the dataset and assign the features as X and the target as y:

Using scikit-learn, we will now train a decision tree with a maximum depth of 4. The code is as follows:

Notice that we set the criterion as ‘entropy’. This criterion is known as the impurity measure (mentioned in the previous section). In classification, entropy is the most common impurity measure or splitting criteria. It is defined by:

Here, p(i|t) is the proportion of the samples that belong to class c for a particular node t. The entropy is therefore 0 if all samples at a node belong to the same class, and the entropy is maximal if we have a uniform class distribution.

For a more visual understanding of entropy, let’s plot the impurity index for the probability range [0, 1] for class 1. The code is as follows:

As you can see, entropy is 0 if ​p(i=1|t) = 1. If the classes are distributed uniformly with p(i=1|t) = 0.5​, entropy is 1.

Now, returning to our Iris example, we will visualize our trained classification tree and see how entropy decides each split.

A nice feature in scikit-learn is that it allows us to export the decision tree as a .dot file after training, which we can visualize using GraphViz, for example. In addition to GraphViz, we will use a Python library called pydotplus, which has capabilities similar to GraphViz and allows us to convert .dot data files into a decision tree image file.

You can install pydotplus and graphviz by executing the following commands in your Terminal:

pip3 install pydotplus
apt install graphviz

The following code will create an image of our decision tree in PNG format:

Looking at the resulting decision tree figure saved in the image file tree.png, we can now nicely trace back the splits that the decision tree determined from our training dataset. We started with 150 samples at the root and split them into two child nodes with 50 and 100 samples, using the petal width cut-off ≤ 1.75 cm. After the first split, we can see that the left child node is already pure and only contains samples from the setosa class (entropy = 0). The further splits on the right are then used to separate the samples from the versicolor and virginica class.

Looking at the final entropy we see that the decision tree with a depth of 4 does a very good job of separating the flower classes.

Regression Trees

We will be using the Boston Housing dataset for our regression example. This is another very popular dataset which contains information about houses in the suburbs of Boston. There are 506 samples and 14 attributes. For simplicity and visualization purposes, we will only use two — MEDV (median value of owner-occupied homes in $1000s) as the target and LSTAT (percentage of lower status of the population) as the feature.

Let’s first import the necessary attributes from scikit-learn into a pandas DataFrame.

Let’s use the DecisionTreeRegressor implemented in scikit-learn to train a regression tree:

Notice that our criterion is different from the one we used for our classification tree. Entropy as a measure of impurity is a useful criteria for classification. To use a decision tree for regression, however, we need an impurity metric that is suitable for continuous variables, so we define the impurity measure using the weighted mean squared error (MSE) of the children nodes instead:

Here, Nt is the number of training samples at node t, Dt is the training subset at node t, y(i) is the true target value, and ŷt​ is the predicted target value (sample mean):

Now, let’s model the relationship between MEDV and LSTAT to see what the line fit of a regression tree looks like:

As we can see in the resulting plot, the decision tree of depth 3 captures the general trend in the data.

I hope you enjoyed this tutorial on decision trees! We discussed the fundamental concepts of decision trees, the algorithms for minimizing impurity, and how to build decision trees for both classification and regression.

In practice, it is important to know how to choose an appropriate value for a depth of a tree to not overfit or underfit the data. Knowing how to combine decision trees to form an ensemble random forest is also useful as it usually has a better generalization performance than an individual decision tree due to randomness, which helps to decrease the model’s variance. It is also less sensitive to outliers in the dataset and doesn’t require much parameter tuning.

We cover these techniques in our Python Machine Learning series, as well as diving into other machine learning models such as perceptrons, Adaline, linear and polynomial regression, logistic regression, SVMs, kernel SVMs, k-nearest-neighbors, models for sentiment analysis, k-means clustering, DBSCAN, convolutional neural networks, and recurrent neural networks.

We also look at other topics such as regularization, data processing, feature selection and extraction, dimensionality reduction, model evaluation, ensemble learning techniques, and deploying a machine learning model.

You can get started here!

Classification and Regression Analysis with Decision Trees was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

All data analysis must be guided by some key questions or objectives. Before starting any data analysis tasks, you should have a clear goal in mind. When your goal allows you to understand your data and the problem, you will be in a good position to get meaningful results out of your analysis!

In this tutorial, we will learn how to perform EDA using data visualization. Specifically, we will focus on seaborn, a Python library that is built on top of matplotlib and has support for NumPy and pandas.

seaborn allows us to make attractive and informative statistical graphics. Although matplotlib makes it possible to visualize essentially anything, it is often difficult and tedious to make the plots visually attractive. seaborn is often used to make default matplotlib plots look nicer, and also introduces some additional plot types.

We will cover how to visually analyze:

• Numerical variables with histograms,
• Categorical variables with count plots,
• Relationships between numerical variables with scatter plots, joint plots, and pair plots, and
• Relationships between numerical and categorical variables with box-and-whisker plots and complex conditional plots.

By effectively visualizing a dataset’s variables and their relationships, a data analyst or data scientist is able to quickly understand trends, outliers, and patterns. This understanding can then be used to tell a story, drive decisions, and create predictive models.

This brief tutorial is adapted from Next Tech’s full Data Analysis with Python course, which includes an in-browser sandboxed environment with Python, Jupyter Notebooks, and seaborn pre-installed. You can get started with this course for free!

Data Preparation

Data preparation is the first step of any data analysis to ensure data is cleaned and transformed in a form that can be analyzed.

We will be performing EDA on the Ames Housing dataset. This dataset is popular among those beginning to learn data science and machine learning as it contains data about almost every characteristic of different houses that were sold Ames, Iowa. This data can then be used to try to predict sale prices.

This dataset is already cleaned and ready for analysis. All we will be doing is filtering some variables to simplify our task. Let’s begin by reading our data as a pandas DataFrame:

If you run this code in Next Tech’s sandbox which already has the dataset imported, or in a Jupyter notebook, you can see that there are 1,460 observations and 81 columns. Each column represents a variable in the DataFrame. We can see from the data type of each column what type of variable it is.

We will only be working with some of the variables — let’s filter and store their names in two lists called numerical and categorical, then redefine our housing DataFrame to contain only these variables:

Out: (1460, 14)

From housing.shape, we can see that our DataFrame now only has 14 columns. Let’s move onto some analysis!

Analyzing Numerical Variables

Our EDA objective will be to understand how the variables in this dataset relate to the sale price of the house.

Before we can do that, we need to first understand our variables. Let’s start with numerical variables, specifically our target variable, SalePrice.

Numerical variables are simply those for which the values are numbers. The first thing that we do when we have numerical variables is to understand what values the variable can take, as well as the distribution and dispersion. This can be achieved with a histogram:

Note that, due to an inside joke, the seaborn library is imported as sns.

With just one method sns.set(), we are able to style our figure, change the color, increase font size for readability, and change the figure size.

We use distplot to plot histograms in seaborn. This by default plots a histogram with a kernel density estimation (KDE). You can try changing the parameter kde=True to see what this looks like.

Taking a look at the histogram, we can see that very few houses are priced below 100,000, most of the houses sold between 100,000 and 200,000, and very few houses sold for above 400,000.

If we want to create histograms for all of our numerical variables, pandas offers the simplest solution:

From this visualization, we get a lot of information. We can see that 1stFlrSF (square footage of the first floor) is heavily skewed right, most houses do not have a second floor, and have 3 BedroomAbvGr (bedrooms above ground). Most houses were sold at an OverallCond of 5 and an OverallQual of 5 or higher. The LotArea visual is more difficult to decipher — however we can tell that there is one or more outliers that may need to be removed before any modeling.

Note that the figure keeps the style that we set previously using seaborn.

Analyzing Categorical Variables

Categorical variables are those for which the values are labeled categories. The values, distribution, and dispersion of categorical variables are best understood with bar plots. Let’s analyze the SaleCondition variable. seaborn gives us a very simple method to show the counts of observations in each category: the countplot.

From the visualization, we can easily see that most houses were sold in Normal condition, and very few were sold in AjdLand (adjoining land purchase), Alloca (allocation: two linked properties with separate deeds), and Family (sale between family members) conditions.

In order to visualize all the categorical variables in our dataset, just as we did with the numerical variables, we can loop through pandas series to create subplots.

Using plt.subplots, we can create a figure with a grid of 2 rows and 4 columns. Then we iterate over every categorical variable to create a countplot with seaborn:

The second for loop simply gets each x-tick label and rotates it 90 degrees to make the text fit on the plots better (you can remove these two lines if you want to know how the text looks without rotation).

As with our numerical variable histograms, we can gather lots of information from this visual — most houses have RL (Residential Low Density) zoning classification, have Regular lot shape, and have CentralAir. We can also see that houses were sold more frequently during the summer months, the most houses were sold in the NAmes (North Ames) neighborhood, and there was a dip in sale in 2010.

However, if we inspect the YrSold variable further, we can see that this “dip” is actually due to the fact that only data up to July was collected.

As you can see, thorough exploration of variables and their values is incredibly important — if we built a model to predict sale prices under the assumption that there was a decrease in sales in 2010, this model would likely be very inaccurate.

Now that we have explored our numerical and categorical variables, let’s take a look at the relationship between these variables — more importantly, how these variables impact our target variable, SalePrice!

Analyzing Relationships Between Numerical Variables

Plotting relationships between variables allows us to easily get a visual understanding of patterns and correlations.

The scatter plot is often used for visualizing relationships between two numerical variables. The seaborn method to create a scatter plot is very simple:

From the scatter plot, we see here that we have a positive relationship between the 1stFlrSF of the house and the SalePrice of the house. In other words, the larger the first floor of a house, the higher the likely sale price.

You can also see that the axis labels are added for us by default, and the markers are automatically outlined to make them clearer — this is opposed to matplotlib in which these are not the default.

seaborn also provides us with a nice function called jointplot which will give you a scatter plot showing the relationship between two variables along with histograms of each variable in the margins — also known as a marginal plot.

Not only can you see the relationships between the two variables, but also how they are distributed individually.

Analyzing Relationships Between Numerical and Categorical Variables

The box-and-whisker plot is commonly used for visualizing relationships between numerical variables and categorical variables, and complex conditional plots are used to visualize conditional relationships.

Let’s start by creating box-and-whisker plots with seaborn’s boxplot method:

Here, we have iterated through every subplot to produce the visualization between all categorical variables and the SalePrice.

We can see that houses with FV (Floating Village Residential) zoning classification have a higher average SalePrice than other zoning classifications, as do houses with CentralAir, and houses with a Partial (home not completed when last assessed) SaleCondition. We can also see that there is little variance in average SalePrice between houses with different LotShapes, or between MoSold and YrSold.

Let’s take a closer look at the Neighborhood variable. We see that there is definitely a different distribution for different neighborhoods, but the visualization is a bit difficult to decipher. Let’s sort our box plots by cheapest neighborhood to most expensive (by median price) using the additional argument order.

In the above snippet, we sorted our neighborhoods by median price and stored this in sorted_nb. Then, we passed this list of neighborhood names into the order argument to create a sorted box plot.

This figure gives us a lot of information. We can see that in the cheapest neighborhoods houses sell for a median price of around $100,000, and in the most expensive neighborhoods houses sell for around $300,000. We can also see that for some neighborhoods, dispersion between the prices is very low, meaning that all the prices are close to each other. In the most expensive neighborhood NridgHt, however, we see a large box — there is large dispersion in the distribution of prices.

Finally, seaborn also allows us to create plots that show conditional relationships. For example, if we are conditioning on the Neighborhood, using the FacetGrid function we can visualize a scatter plot between the OverallQual and the SalePrice variables:

For each individual neighborhood we can see the relationship between OverallQual and SalePrice.

We also added another categorical variable CentralAir to the (optional) hue parameter — the orange points correspond to houses that do not have CentralAir. As you can see, these houses tend to sell at a lower price.

The FacetGrid method makes it incredibly easy to produce complex visualizations and to get valuable information. It is good practice to produce these visualizations to get quick insights about variable relationships.

I hope you’ve enjoyed this brief tutorial on exploratory data analysis and data visualization with seaborn! I covered how to create histograms, count plots, scatter plots, marginal plots, box-and-whisker plots, and conditional plots.

During our exploration, we discovered outliers and trends within individual variables, and relationships between variables. This knowledge can be used to build a model to predict the SalePrice of houses in Ames. For example, since we found a correlation between SalePrice and the variables CentralAir, 1stFlrSf, SaleCondition, and Neighborhood, we can start with a simple model using these variables.

If you are interesting in learning how to do this and more, we have a full Data Analysis with Python course available at Next Tech. Get started for free here!

This course explores vectorizing operations with NumPy, EDA using pandas, data visualization with matplotlib, additional EDA and visualization techniques using seaborn, statistical computing with SciPy, and machine learning with scikit-learn.

How to Perform Exploratory Data Analysis with Seaborn was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Linear regression is a basic predictive analytics technique that uses historical data to predict an output variable. It is popular for predictive modelling because it is easily understood and can be explained using plain English.

Linear regression models have many real-world applications in an array of industries such as economics (e.g. predicting growth), business (e.g. predicting product sales, employee performance), social science (e.g. predicting political leanings from gender or race), healthcare (e.g. predicting blood pressure levels from weight, disease onset from biological factors), and more.

Understanding how to implement linear regression models can unearth stories in data to solve important problems. We’ll use Python as it is a robust tool to handle, process, and model data. It has an array of packages for linear regression modelling.

The basic idea is that if we can fit a linear regression model to observed data, we can then use the model to predict any future values. For example, let’s assume that we have found from historical data that the price (P) of a house is linearly dependent upon its size (S) — in fact, we found that a house’s price is exactly 90 times its size. The equation will look like this:

P = 90*S

With this model, we can then predict the cost of any house. If we have a house that is 1,500 square feet, we can calculate its price to be:

P = 90*1500 = $135,000

In this blog post, we cover:

1. The basic concepts and mathematics behind the model
2. How to implement linear regression from scratch using simulated data
3. How to implement linear regression using statsmodels
4. How to implement linear regression using scikit-learn

This brief tutorial is adapted from the Next XYZ Linear Regression with Python course, which includes an in-browser sandboxed environment, tasks to complete, and projects using public datasets.

Basic concepts and mathematics

There are two kinds of variables in a linear regression model:

• The input or predictor variable is the variable(s) that help predict the value of the output variable. It is commonly referred to as X.
• The output variable is the variable that we want to predict. It is commonly referred to as Y.

To estimate Y using linear regression, we assume the equation:

Yₑ = α + β X

where Yₑ is the estimated or predicted value of Y based on our linear equation.

Our goal is to find statistically significant values of the parameters α and β that minimise the difference between Y and Yₑ.

If we are able to determine the optimum values of these two parameters, then we will have the line of best fit that we can use to predict the values of Y, given the value of X.

So, how do we estimate α and β? We can use a method called ordinary least squares.

Ordinary Least Squares

The objective of the least squares method is to find values of α and β that minimise the sum of the squared difference between Y and Yₑ. We will not go through the derivation here, but using calculus we can show that the values of the unknown parameters are as follows:

where X̄ is the mean of X values and Ȳ is the mean of Y values.

If you are familiar with statistics, you may recognise β as simply
Cov(X, Y) / Var(X).

Linear Regression From Scratch

In this post, we’ll use two Python modules:

• statsmodels — a module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration.
• scikit-learn — a module that provides simple and efficient tools for data mining and data analysis.

Before we dive in, it is useful to understand how to implement the model from scratch. Knowing how the packages work behind the scenes is important so you are not just blindly implementing the models.

To get started, let’s simulate some data and look at how the predicted values (Yₑ) differ from the actual value (Y):

If the above code is run (e.g. in a Jupyter notebook), this would output something like:

To estimate y using the OLS method, we need to calculate xmean and ymean, the covariance of X and y (xycov), and the variance of X (xvar) before we can determine the values for alpha and beta.

Out:
alpha = 2.0031670124623426
beta = 0.32293968670927636

Great, we now have an estimate for alpha and beta! Our model can be written as Yₑ = 2.003 + 0.323 X,​ and we can make predictions:

Out:
array([3.91178282, 2.81064315, 3.27775989, 4.29675991, 3.99534802,
       1.69857201, 3.25462968, 2.36537842, 2.40424288, 2.81907292,
       ...
       2.16207195, 3.47451661, 2.65572718, 3.2760653 , 2.77528867,
       3.05802784, 2.49605373, 3.92939769, 2.59003892, 2.81212234])

Let’s plot our prediction ypred against the actual values of y, to get a better visual understanding of our model.

The blue line is our line of best fit, Yₑ = 2.003 + 0.323 X. We can see from this graph that there is a positive linear relationship between X and y. Using our model, we can predict y from any values of X!

For example, if we had a value X = 10, we can predict that:
Yₑ = 2.003 + 0.323 (10) = 5.233.

Linear Regression with statsmodels

Now that we have learned how to implement a linear regression model from scratch, we will discuss how to use the ols method in the statsmodels library.

To demonstrate this method, we will be using a very popular advertising dataset about various costs incurred on advertising by different mediums and the sales for a particular product. You can download this dataset here.

We will only be looking at the TV variable in this example — we will explore whether TV advertising spending can predict the number of sales for the product. Let’s start by importing this csv file as a pandas dataframe using read_csv():

First, we use statsmodels’ ols function to initialise our simple linear regression model. This takes the formula y ~ X, where X is the predictor variable (TV advertising costs) and y is the output variable (Sales). Then, we fit the model by calling the OLS object’s fit() method.

We no longer have to calculate alpha and beta ourselves as this method does it automatically for us! Calling model.params will show us the model’s parameters:

Out:
Intercept    7.032594
TV           0.047537
dtype: float64

In the notation that we have been using, α is the intercept and β is the slope i.e. α = 7.032 and β = 0.047.

Thus, the equation for the model will be: Sales = 7.032 + 0.047*TV

In plain English, this means that, on average, if we spent $100 on TV advertising, we should expect to sell 11.73 units.

Now that we’ve fit a simple regression model, we can try to predict the values of sales based on the equation we just derived using the .predict method.

We can also visualise our regression model by plotting sales_pred against the TV advertising costs to find the line of best fit:

We can see that there is a positive linear relationship between TV advertising costs and Sales — in other words, spending more on TV advertising predicts a higher number of sales!

With this model, we can predict sales from any amount spent on TV advertising. For example, if we increase TV advertising costs to $400, we can predict that sales will increase to 26 units:

Out:
0    26.04725
dtype: float64

Linear Regression with scikit-learn

We’ve learnt to implement linear regression models using statsmodels…now let’s learn to do it using scikit-learn!

For this model, we will continue to use the advertising dataset but this time we will use two predictor variables to create a multiple linear regression model. This is simply a linear regression model with more than one predictor, and is modelled by:

Yₑ = α + β₁X₁ + β₂X₂ + … + βₚXₚ, where p is the number of predictors.

In our example, we will be predicting Sales using the variables TV and Radio i.e. our model can be written as:

Sales = α + β₁*TV + β₂*Radio.

First, we initialise our linear regression model, then fit the model to our predictors and output variables:

Again, there is no need to calculate the values for alpha and betas ourselves – we just have to call .intercept_ for alpha, and .coef_ for an array with our coefficients beta1 and beta2:

Out:
alpha = 2.921099912405138
betas = [0.04575482 0.18799423]

Therefore, our model can be written as:

Sales = 2.921 + 0.046*TV + 0.1880*Radio.

We can predict values by simply using .predict():

Out:
array([20.555464, 12.345362, 12.337017, 17.617115, 13.223908,
       12.512084, 11.718212, 12.105515,  3.709379, 12.551696,
       ...
       12.454977,  8.405926,  4.478859, 18.448760, 16.4631902,
        5.364512,  8.152375, 12.768048, 23.792922, 15.15754285])

Now that we’ve fit a multiple linear regression model to our data, we can predict sales from any combination of TV and Radio advertising costs! For example, if we wanted to know how many sales we would make if we invested $300 in TV advertising and $200 in Radio advertising…all we have to do is plug in the values!

Out:
[54.24638977]

This means that if we spend $300 on TV advertising and $200 on Radio advertising, we should expect to see, on average, 54 units sold.

I hope you enjoyed this brief tutorial about the basics of linear regression!

We covered how to implement linear regression from scratch and by using statsmodels and scikit-learn in Python. In practice, you will have to know how to validate your model and measure efficacy, how to select significant variables for your model, how to handle categorical variables, and when and how to perform non-linear transformations.

We have a full course covering all these topics (and more!) here at Next XYZ if you’re interested in learning more about linear regression with Python!

Introduction to Linear Regression in Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.