Scribd
Upload
274 views

Data Pipeline

The document discusses data pipelines, which are a series of automated processes that move data from source systems like a company's POS system or website to destinations like a data warehouse or data lake. This allows a company to collect all its sales data in one place for analysis. Common components of a data pipeline include extraction, validation, transformation, loading, and quality checks. Pipelines help automate data flows, provide flexible integration, reduce costs, and improve insights by maintaining consistent data. Pipelines can be either batch or streaming, and either ETL or ELT depending on whether the data is extracted then transformed and loaded or extracted and loaded then transformed.

Uploaded by

wan nur anisah
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
274 views

Data Pipeline

The document discusses data pipelines, which are a series of automated processes that move data from source systems like a company's POS system or website to destinations like a data warehouse or data lake. This allows a company to collect all its sales data in one place for analysis. Common components of a data pipeline include extraction, validation, transformation, loading, and quality checks. Pipelines help automate data flows, provide flexible integration, reduce costs, and improve insights by maintaining consistent data. Pipelines can be either batch or streaming, and either ETL or ELT depending on whether the data is extracted then transformed and loaded or extracted and loaded then transformed.

Uploaded by

wan nur anisah
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

DATA

PIPELINE

SIMPLIFIED

BY NISCHAY THAPA
Imagine a retail company that
wants to analyse its sales data
to understand customer
behaviours.
The company collects data and
stores it from different systems

POS System

Website

Social Media

CRM System

These are known as 'Sources'


For analysis, they would like
to collect and store all this
data at one place.

Data Warehouse Data Lake

These are known as 'Destinations'


How will they build the connection?

How do I move
the data? Destination

Source
Data Pipeline
A data pipeline is a series of
automated processes that
move data from one system or
stage to another.

Destination

Source
The processes in a data pipeline
can include

Extraction

Validation

Transformation

Loading

Quality Checks

Monitoring
Without a data pipeline,

The company will have to manually


transfer data, perform multiple
extracts and transformations on the
same data, which makes it difficult to
track changes, and becomes time-
consuming leading to poor data
quality and insights.
With data pipelines, they can

Automate data flows

Have flexible integration

Be cost-effective

Produce better insights

Maintain data consistency


Common types of data pipeline

Batch Data Pipeline:


Processes data in large chunks
At specific intervals.
Used for non-time-sensitive
data

Streaming Data Pipeline:


Processes data in real-time
Commonly used for time-
sensitive data
financial transactions
social media feeds.
ETL Pipeline:
Extracts data from various
sources
Transforms it
Loads it into the destination
system.

ELT Pipeline:
Extracts data from various
sources
Loads it into a destination
system
Transforms it
Data pipelines are an efficient means for
managing and processing data enabling
automation, improved governance, and
providing accurate insights to inform
decision-making.

Quality
Check Load
Validate

Transform

Destination

Monitor

Source Extract
RESOURCES

Data Pipeline | IBM

Design a data pipeline

Build a data pipeline | GCP

Build a data pipeline | Azure

Build a data pipeline | AWS

Click them to find out more!


Nischay Thapa
Data Scientist | Mentor

If you enjoyed this post


Comment below what you want to
learn next

@nischaythapa for tips around


Data Science, Data Engineering,
Machine Learning & Cloud

You might also like