Scribd
Upload
190 views

Case Study DataCamp

The document describes building a recommendation engine for courses at DataCamp. It involves extracting rating data for courses from a database, transforming the data by calculating average ratings and filtering for eligible user-course pairs, and loading the recommendations into a target database. The transformations are performed daily and loaded into Postgres using Airflow to schedule it as a recurring job. Matrix factorization and other recommendation techniques can then be used to calculate estimated ratings for new courses to recommend.

Uploaded by

Raka Featheris
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
190 views

Case Study DataCamp

The document describes building a recommendation engine for courses at DataCamp. It involves extracting rating data for courses from a database, transforming the data by calculating average ratings and filtering for eligible user-course pairs, and loading the recommendations into a target database. The transformations are performed daily and loaded into Postgres using Airflow to schedule it as a recurring job. Matrix factorization and other recommendation techniques can then be used to calculate estimated ratings for new courses to recommend.

Uploaded by

Raka Featheris
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 30

Course ratings

I N T R O D U C T I O N T O D ATA E N G I N E E R I N G

Vincent Vankrunkelsven
Data Engineer @ DataCamp
Ratings at DataCamp

INTRODUCTION TO DATA ENGINEERING


Recommend using ratings

Get rating data

Clean and calculate top-recommended courses

Recalculate daily

Example usage: user's dashboard

INTRODUCTION TO DATA ENGINEERING


As an ETL process
It's an ETL process!

INTRODUCTION TO DATA ENGINEERING


The database

Course Rating
course_id user_id
title
course_id
description
rating
programming_language

INTRODUCTION TO DATA ENGINEERING


The database relationship

Course Rating
course_id user_id

title course_id

description rating

programming_language

INTRODUCTION TO DATA ENGINEERING


Let's practice!
I N T R O D U C T I O N T O D ATA E N G I N E E R I N G
From ratings to
recommendations
I N T R O D U C T I O N T O D ATA E N G I N E E R I N G

Vincent Vankrunkelsven
Data Engineer @ DataCamp
The recommendations table

user_id course_id rating


1 1 4.8
1 74 4.78
1 21 4.5
2 32 4.9

The estimated rating of a course the user hasn't taken yet.

INTRODUCTION TO DATA ENGINEERING


Recommendation techniques

Matrix factorization

Building Recommendation Engines with PySpark

INTRODUCTION TO DATA ENGINEERING


Common sense transformation
Course Recommendations
course_id

title user_id course_id rating


description 1 1 4.8
programming_language
1 74 4.78
1 21 4.5
2 32 4.9
Rating
user_id

course_id

rating

INTRODUCTION TO DATA ENGINEERING


Average course ratings
Average course rating

course_id avg_rating
1 4.8
74 4.78
21 4.5
32 4.9

We want to recommend highly rated courses

INTRODUCTION TO DATA ENGINEERING


Use the right programming language
Rating

user_id course_id programming_language rating


1 1 r 4.8
1 74 sql 4.78
1 21 sql 4.5
1 32 python 4.9

Recommend SQL course for user with id 1

INTRODUCTION TO DATA ENGINEERING


Recommend new courses
Rating

user_id course_id programming_language rating


1 1 r 4.8
1 74 sql 4.78
1 21 sql 4.5
1 32 python 4.9

Don't recommend the combinations already in the rating table

INTRODUCTION TO DATA ENGINEERING


Our recommendation transformation

Use technology that user has rated most

Don't recommend courses that user already rated

Recommend three highest rated courses from remaining combinations

INTRODUCTION TO DATA ENGINEERING


Rating

user_id course_id programming_language rating


1 12 sql 4.78
1 52 sql 4.5
1 32 r 4.9

Recommend three highest rated SQL courses which are not 12 and 52.

INTRODUCTION TO DATA ENGINEERING


Let's practice!
I N T R O D U C T I O N T O D ATA E N G I N E E R I N G
Scheduling daily
jobs
I N T R O D U C T I O N T O D ATA E N G I N E E R I N G

Vincent Vankrunkelsven
Data Engineer @ DataCamp
What you've done so far

Extract using extract_course_data() and extract_rating_data()

Clean up using NA using transform_fill_programming_language()

Average course ratings per course: transform_avg_rating()

Get eligible user and course id pairs: transform_courses_to_recommend()

Calculate the recommendations: transform_recommendations()

INTRODUCTION TO DATA ENGINEERING


Loading to Postgres

Use the calculations in data products

Update daily

Example use case: sending out e-mails with recommendations

INTRODUCTION TO DATA ENGINEERING


The loading phase

recommendations.to_sql(
"recommendations",
db_engine,
if_exists="append",
)

INTRODUCTION TO DATA ENGINEERING


def etl(db_engines):
# Extract the data
courses = extract_course_data(db_engines)
rating = extract_rating_data(db_engines)
# Clean up courses data
courses = transform_fill_programming_language(courses)
# Get the average course ratings
avg_course_rating = transform_avg_rating(rating)
# Get eligible user and course id pairs
courses_to_recommend = transform_courses_to_recommend(
rating,
courses,
)
# Calculate the recommendations
recommendations = transform_recommendations(
avg_course_rating,
courses_to_recommend,
)
# Load the recommendations into the database
load_to_dwh(recommendations, db_engine))

INTRODUCTION TO DATA ENGINEERING


Creating the DAG
from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator

dag = DAG(dag_id="recommendations",
scheduled_interval="0 0 * * *")
task_recommendations = PythonOperator(
task_id="recommendations_task",
python_callable=etl,
)

INTRODUCTION TO DATA ENGINEERING


Let's practice!
I N T R O D U C T I O N T O D ATA E N G I N E E R I N G
Congratulations
I N T R O D U C T I O N T O D ATA E N G I N E E R I N G

Vincent Vankrunkelsven
Data Engineer @ DataCamp
Introduction to data engineering

Identify the tasks of a data engineer

What kind of tools they use

Cloud service providers

INTRODUCTION TO DATA ENGINEERING


Data engineering toolbox

Databases

Parallel computing & frameworks (Spark)

Work ow scheduling with Air ow

INTRODUCTION TO DATA ENGINEERING


Extract, Load and Transform (ETL)

Extract: get data from several sources

Transform: perform transformations using parallel computing

Load: load data into target database

INTRODUCTION TO DATA ENGINEERING


Case study: DataCamp

Fetch data from multiple sources

Transform to form recommendations

Load into target database

INTRODUCTION TO DATA ENGINEERING


Good job!
I N T R O D U C T I O N T O D ATA E N G I N E E R I N G

You might also like