DATA ENGINEERING

Dr. sachin kumar yadav

 Data
Data refers to raw, unorganized facts that need to be processed to become meaningful.
It can take various forms, such as numbers, words, measurements, observations, or
descriptions. Data is often the foundation for generating insights, making decisions, and
solving problems.
• Types of Data:
Structured Data: Organized in a specific format, like tables in databases.
Unstructured Data: Lacks a predefined format, such as text, images, or videos.
Semi-structured Data: Has some organizational properties, like JSON or XML files.
• Characteristics:
Volume: The quantity of data.
Variety: Different types of data.
Velocity: The speed at which data is generated.
Veracity: The reliability of data.
 Engineering
Engineering is the application of scientific principles, mathematics, and technology
to design, build, and maintain systems, structures, machines, or processes. It
involves problem-solving, innovation, and optimization to meet specific needs or
challenges.
Branches of Engineering:
• Civil Engineering: Designing and constructing buildings, roads, and infrastructure.
• Mechanical Engineering: Creating machines and mechanical systems.
• Electrical Engineering: Working with electrical systems and electronics.
• Software Engineering: Developing and maintaining software applications.
• Data Engineering: Building systems to collect, process, and analyze large datasets.
Core Concepts:
• Design and Innovation.
• Optimization and Efficiency.
• Problem-Solving and Analysis.
 What is Data Engineering?
Data engineering is the process of designing and
building systems to collect, store, transform, and
analyze large amounts of data:
• Data collection: Data engineers collect data from a
variety of sources.
• Data storage: Data engineers store the collected
data.
• Data transformation: Data engineers transform the
data into usable core data sets.
• Data analysis: Data engineers analyze the data to
provide predictive models and show trends.
• Bridges the gap between raw data and actionable
insights.
Data engineering is the process of designing, building,
and maintaining systems for collecting, storing,
processing, and analyzing large-scale data.
Data engineering involves working with various data
sources, such as databases, APIs, web scraping,
streaming data, etc., and transforming them into a
unified and consistent format that can be used for
Data engineering involves a wide range of tasks, including data modeling, data integration, data
transformation, data quality, and data governance. The goal is to provide a reliable and efficient
data infrastructure that supports the organization’s data-driven decision-making processes.
"Data engineering is the development, implementation, and maintenance of
systems and processes that take in raw data and produce high-quality, consistent
information that supports downstream use cases, such as analysis and machine
learning.

Data engineering is the intersection of security, data management, DataOps, data

architecture, orchestration, and software engineering. A data engineer manages
the data engineering lifecycle, beginning with getting data from source systems
and ending with serving data for use cases, such as analysis or machine learning."
Data statistics is the process of collecting, analyzing, and
presenting data to understand and interpret it. Statistics can
be used to answer questions about relationships between
variables, such as whether income level and education are
correlated.
To get started with data engineering, you can learn the basics by:
• Learning programming languages: Python and SQL are essential for data
engineering. You can also learn Java and Scala.
• Getting familiar with tools: Experiment with relational databases like MySQL,
NoSQL databases like MongoDB, and data processing frameworks like Apache
Spark.
• Building simple projects: Create data pipelines using open-source tools or
cloud-based services.
• Understanding data warehousing: Learn how to build and work with a data
warehouse to aggregate unstructured data from multiple sources.
• Understanding database systems: Gain a deep understanding of SQL and
NoSQL database systems, including how to design, query, and manage them.
Data engineers manage and organize data, and look for trends. They typically
have a background in data science, software engineering, math, or a business-
related field.
 Key Components of Data Engineering

Data Collection:
•Gathering data from various sources such as databases, APIs, IoT devices, or web scraping.
•Ensuring data integrity during the collection process.
Data Transformation (ETL/ELT):
•ETL (Extract, Transform, Load): Data is extracted, cleaned, and transformed before loading into a data
warehouse.
•ELT (Extract, Load, Transform): Data is first loaded into storage, then transformed for analysis.
Data Storage:
•Designing and managing storage systems like relational databases (SQL), NoSQL databases, and data lakes.
•Optimizing storage for scalability, cost-efficiency, and performance.
Data Pipelines:
•Automating the flow of data from source to destination.
•Tools: Apache Airflow, Apache Kafka, AWS Glue.
Data Security:
•Encrypting data in transit and at rest.
•Controlling access and monitoring for vulnerabilities.
Data Quality Management
Ensuring that data meets predefined standards of accuracy, completeness,
consistency, timeliness, and reliability. Data engineers implement validation
checks, monitoring, and error-handling processes to maintain high data quality
throughout the pipeline.
Data Governance
Establishing policies, procedures, and standards for managing data assets
effectively. Governance ensures proper oversight of data use, accountability,
data ownership, and compliance with organizational and legal requirements.
Ensuring compliance with regulations like GDPR or HIPAA.
Data Monitoring and Maintenance
Continuous monitoring of data pipelines, storage systems, and processes to
ensure smooth operations. Maintenance involves troubleshooting issues,
updating systems, optimizing performance, and minimizing downtime to sustain
data infrastructure reliability.
1. Accuracy
•Refers to how closely the data values align with the true or real-world values they represent.
•Accurate data is free from errors, distortions, or misrepresentations.
•Example: If a dataset records a person's age as 30 but they are actually 40, the data is inaccurate.

2. Completeness
•Refers to whether all required data is present and populated.
•A dataset is considered complete if no critical data points, attributes, or records are missing.
•Example: A customer record missing an email address or phone number lacks completeness.
3. Consistency
•Ensures that data remains uniform across all systems and sources.
•Inconsistent data occurs when the same value or entity is represented differently in various
locations or datasets.
•Example: A customer's name recorded as "John Do" in one system and "J. Do" in another is
inconsistent.
4. Timeliness
•Measures whether data is up-to-date and available when needed.
•Timely data reflects current conditions or the most recent updates.
•Example: Stock price data used in trading systems must be real-time; delayed data can lead
to wrong decisions.
5. Data insights are the understanding that comes from analyzing and interpreting
data. They can be used to make strategic decisions, such as forecasting customer
needs.
Role of a Data Engineer
A Data Engineer focuses on building and maintaining the infrastructure, systems, and pipelines required to
collect, store, and process data efficiently. They enable data to flow seamlessly to support analysis and
decision-making.
Key Responsibilities:
• Data Pipeline Development:
Design and build scalable, automated pipelines to extract, transform, and load (ETL) data from various
sources.
• Data Architecture:
Create and manage data storage systems (e.g., data lakes, warehouses) ensuring scalability, efficiency, and
security.
• Data Integration:
Consolidate and harmonize data from multiple systems into a unified format for analysis.
• Data Quality Assurance:
Implement processes to clean, validate, and monitor data for accuracy, completeness, and consistency.
• Performance Optimization:
Optimize databases, pipelines, and processing frameworks to handle large-scale data efficiently.
• Tooling & Technology:
Use tools like Apache Spark, Kafka, Hadoop, SQL, and cloud platforms (AWS, Azure, GCP).
• Collaboration:
Role of a Data Scientist
A Data Scientist focuses on analyzing and interpreting complex data to extract insights, develop models, and solve
business problems using statistical and machine learning techniques.
Key Responsibilities:
• Data Analysis:
Explore and analyze data to uncover trends, patterns, and anomalies that can inform decisions.
• Statistical Modeling:
Develop predictive models, such as regression, classification, or clustering, to address business challenges.
• Machine Learning:
Build, train, and deploy machine learning models for tasks like recommendation systems, fraud detection, or
forecasting.
• Data Visualization:
Create visual representations (e.g., dashboards, charts) to communicate findings effectively to stakeholders.
• Experimentation:
Conduct A/B testing or other experiments to validate hypotheses and assess the impact of changes.
• Feature Engineering:
Select or create relevant data features to improve model accuracy and performance.
• Collaboration:
Work with data engineers to access clean and structured data, and with business teams to align models with
organizational goals.
• Programming & Tools:
 Aspect Data Engineering Data Science

Focus Data infrastructure, pipelines, and processing Data analysis, modeling, and insights

Objective Prepare, transform, and manage data for use Extract insights, build predictive models

Data Handling Raw data cleaning, integration, storage Analyzing, exploring, visualizing data

Tools and Apache Hadoop, Spark, Kafka, SQL/NoSQL Python/R, Jupyter Notebooks, Machine
Technologies databases Learning libraries

Programming (Python, Java), ETL, database Statistics, Machine Learning, Data

Skills
management Visualization

Clean, structured data ready for analysis and Predictive models, insights, actionable
Output reporting recommendations

Develop and maintain data pipelines, ensure Analyze data, build ML models,
Role data quality communicate findings

Data integration, ETL processes, data Predictive analytics, recommendation

Use Cases warehousing systems
 Responsibilities of Data
Engineer
• Design and develop data processing pipelines to extract, transform, and load
data from various sources into a centralized data warehouse or data lake
• Create and maintain databases, data warehouses, and data lakes
• Develop scripts and code for data processing, manipulation, and transformation
• Ensure data quality and consistency across all data sources and data pipelines
• Implement and maintain data security and access controls
• Collaborate with data scientists, data analysts, and other stakeholders to identify
and implement data solutions that meet business needs
• Monitor and troubleshoot the performance of data infrastructure and address
any issues that arise
• Develop and maintain documentation for data processes, pipelines, and systems
• Stay up-to-date with new and emerging technologies related to data
engineering and recommend ways to improve existing data infrastructure
What are the skills and tools required for data engineering?
To become a data engineer, you need to have a strong background in computer science,
mathematics, and statistics, as well as a good understanding of the data domain and the
business context. You also need to have a variety of technical skills and tools, such as:
• Programming languages, such as Python, Java, Scala, etc., that can be used for data
manipulation, analysis, and automation.
• Databases, such as SQL, NoSQL, or graph databases, that can store and query structured,
semi-structured, or unstructured data.
• Cloud computing, such as AWS, Azure, or Google Cloud, that can provide scalable and cost-
effective data services and platforms.
• BigData tools, such as Hadoop, Spark, Kafka, etc., that can handle distributed and parallel
data processing and streaming.
• Orchestration tools, such as Airflow, Luigi, or Prefect, that can orchestrate and schedule data
pipelines and workflows.
• Data warehouse platforms, such as Snowflake, Redshift, or BigQuery, that can provide data
warehousing and analytics capabilities.
• Data visualization tools, such as Tableau, Power BI, or Dash, that can create interactive and
 Data Ingestion
Data ingestion is the process of collecting data from various sources and loading
it into a centralized storage system or computing system. It is a fundamental step
in data management and analytics workflows.
 Importance of Data Engineering
Data engineering plays a critical role in the modern data-driven world, enabling
organizations to extract meaningful insights, make informed decisions, and
achieve operational excellence.
1. Foundation for Data Analytics and Data Science
• Clean and Reliable Data: Data engineers prepare and structure raw data to
ensure accuracy, consistency, and completeness, making it usable for data
scientists and analysts.
• Optimized Data Pipelines: Efficient pipelines enable seamless data flow,
critical for building predictive models and performing analytics.
2. Handling Large Volumes of Data
• Scalability: Data engineering frameworks like Apache Spark and Hadoop
handle massive datasets across distributed systems.
• Real-Time Processing: Streaming tools like Kafka and Flink enable real-time
data ingestion and analysis, essential for applications like fraud detection
and recommendation systems.
3. Improved Decision-Making
• Data engineering enables businesses to access well-structured and timely
data, driving informed strategic and operational decisions.
Examples include market trend analysis, performance tracking, and supply
chain optimization.
4. Enabling Automation
• Automated Workflows: Data engineers create automated data pipelines,
reducing manual efforts and errors.
• Continuous Data Processing: Automated pipelines support continuous
integration and delivery of data, ensuring up-to-date information.
5. Integration of Diverse Data Sources
• Businesses collect data from multiple sources like databases, IoT devices, and
APIs.
• Data engineering ensures seamless integration, providing a unified view for
analytics and reporting.
6. Cost Efficiency
• Optimized Storage: Engineers design systems to store data cost-effectively,
leveraging solutions like cloud-based data lakes and warehouses.
• Efficient Processing: Streamlined pipelines reduce computational overhead and
resource wastage.
7. Enhanced Data Security and Compliance
• Implementing security protocols, such as encryption and role-based access controls,
safeguards sensitive data.
• Ensures compliance with data regulations like GDPR, HIPAA, and CCPA.
8. Support for Advanced Technologies
• AI and Machine Learning: Data engineering provides the necessary infrastructure to train and
deploy machine learning models.
• IoT and Real-Time Analytics: Enables handling and processing of high-velocity IoT data for
insights and decision-making.
9. Competitive Advantage
• Organizations with robust data engineering capabilities can outpace competitors by making
faster, data-backed decisions.
• Predictive analytics, customer personalization, and operational insights give a strategic edge.
10. Future-Proofing Data Infrastructure
• With the exponential growth in data, scalable and robust engineering ensures systems remain
relevant and effective.
• Adaptability to new tools and technologies ensures longevity and efficiency.
 Applications of Data Engineering
Data engineering is essential in various industries, enabling data-driven
strategies, real-time decision-making, and operational efficiency. Below are
some key applications of data engineering:
• Business Intelligence: Building data warehouses and dashboards for decision-
making.
• Machine Learning: Preparing data for training and deploying predictive
models.
• Real-Time Analytics: Enabling live insights for fraud detection or dynamic
pricing.
• IoT Management: Processing data from smart devices for automation and
insights.
• Personalized Recommendations: Powering suggestion engines for e-commerce
and streaming.
• Marketing Optimization: Aggregating customer data for targeted campaigns.
• Healthcare: Analyzing patient data for research, diagnostics, and resource
planning.
• Finance: Fraud detection, compliance, and real-time trade analysis.
• Supply Chain: Route optimization and inventory management.
• Energy: Monitoring consumption and predictive maintenance.
• Gaming: Player behavior analysis and in-game recommendations.
• Education: Personalized learning and performance tracking.
• Public Services: Traffic management and resource allocation.
Data engineering has rapidly evolved over the past decade, becoming a cornerstone of modern data-
driven organizations. As businesses increasingly rely on data to inform decisions, the role of the data
engineer is more crucial than ever.
Why data processing is critical
 Batch processing, streaming, web scraping
Batch Processing: Batch processing involves processing a large volume of data collected over time. It’s ideal for use
cases where real-time processing is unnecessary, and latency is acceptable.
Key Characteristics:
• High Latency: Processes data in bulk with some delay.
• Efficient: Optimized for throughput and large datasets.
• Fault Tolerance: Built-in mechanisms for retrying and ensuring data integrity.
Use Cases: ETL (Extract, Transform, Load) pipelines, data warehousing, financial reporting, and periodic data
aggregation.
Popular Tools:
• Apache Hadoop: Distributed storage (HDFS) - (Hadoop Distributed File System) and processing (MapReduce).
• Apache Spark: Efficient distributed processing with in-memory computing.
• AWS Glue/Data Pipeline: Cloud-based batch data processing.
• Databricks: Unified analytics for batch and streaming.
• Google Dataflow: Handles batch and stream processing with the same API.
Examples
• Payroll processing: Processing employee payroll at the end of a pay period
• Data warehousing: Aggregating and processing large datasets for reporting and analysis
• Credit card transaction processing: Processing credit card transactions in batches
 Key Features of Batch
Processing
•Processing Large Volumes:
Efficiently handles significant
amounts of data in one go.
•Scheduled Intervals: Data is
processed at predetermined times,
such as daily, weekly, or monthly.
•Higher Latency: Results are
delivered after the entire batch is
processed, leading to a delay
Batch processing is a method of simultaneously processing compared to real-time processing.
large volumes of data, typically at scheduled intervals. This
approach is suitable for tasks that do not require immediate
results, focusing instead on handling data in groups or
batches.

Use Cases of Batch Processing

•Payroll Processing: Companies process employee payroll in batches at the end of a pay period.
•Data Warehousing: Large datasets are aggregated and processed in batches for reporting and
analysis.
•Reporting: Generating periodic reports, such as monthly sales summaries, is often done through
Streaming: Streaming processes data in real time or near-real-time as it arrives. This paradigm is ideal for low-latency
applications requiring immediate insights.
How it works
• Streaming continuously collects data from sources like mobile apps, social networks, and financial trading floors.
• It processes the data as it arrives, instead of waiting to process it in batches.
• It immediately transforms the data into streams of outputs.
Key Characteristics:
• Low Latency: Processes data in milliseconds or seconds.
• Continuous Processing: Data is handled as a continuous flow.
• Event-Driven: Relies on events triggering specific actions.
Use Cases: Fraud detection, real-time analytics, recommendation engines, and IoT data processing.
Popular Tools:
• Apache Kafka: Distributed event streaming for real-time pipelines.
• Apache Flink: Low-latency, distributed processing for stream and batch data.
• Apache Spark Streaming/Structured Streaming: Real-time data processing on top of Spark.
• Amazon Kinesis: Scalable stream processing in the cloud.
• Google Pub/Sub: Real-time messaging for stream processing.
Examples:
• Real-time anomaly detection in transaction data.
 Key Features of Real-Time
Processing
1.Low Latency: One of the defining
characteristics of real-time processing is
its minimal delay between data input and
output. This allows for prompt decision-
making and action.
2.Continuous Data Input: Data is
continuously fed into the system,
enabling constant monitoring and real-
time analysis.
3.Immediate Output: When the data is
processed, the system generates
responses or results, ensuring that users
or systems can take timely actions based
on the latest information.
What is Real-Time Processing?
Real-time processing refers to the immediate or near-immediate handling of data as it is
received.
Stream processing, then, is useful for tasks
like fraud detection and cybersecurity. If
transaction data is stream-processed,
fraudulent transactions can be identified and
stopped before they are even complete.
Key Differences Between
1. Speed and Latency
Real-Time and Batch •Real-Time Processing: Designed for minimal latency,

Processing real-time processing handles data as it arrives, providing

immediate results. This makes it ideal for applications
where quick response times are critical, such as financial
trading or live monitoring systems.
•Batch Processing: Involves processing data in bulk at
scheduled intervals, leading to higher latency. Outputs
are generated after the entire batch is processed, which
may take minutes, hours, or even days, depending on the
data volume and complexity.
2. Data Volume
•Real-Time Processing: Manages
continuous streams of data, handling small amounts of
data at a time but processing it immediately. This allows
for ongoing updates and instant reactions.
•Batch Processing: Suited for large volumes of data
collected over time and processed together. This
approach is efficient for tasks like report generation or
4. Cost data consolidation, where immediate processing is
•Real-Time Processing: Typically more expensive unnecessary.
3. Complexity
due to the need for advanced technology, •Real-Time Processing: Requires more complex
infrastructure, and resources to maintain low infrastructure to ensure data is processed quickly and
latency and high availability. accurately as it arrives. Systems must be robust and
•Batch Processing: More cost-effective, capable of handling high input rates without delays.
particularly for non-time-sensitive tasks, as it can •Batch Processing: Simpler to implement and manage,
as it processes data in bulk at specific times. The system
utilize less expensive hardware and requires fewer only needs to be active during processing periods,
Web Scraping: Web scraping involves extracting data from websites to make it available for analysis or
integration.
Key Characteristics:
• Data Extraction: Automated collection of data from web pages.
• Scalability: Can handle scraping a few pages or thousands.
• Unstructured Data: Often deals with raw HTML and needs parsing( a process used in computer
science to break down data into smaller components that are more manageable)/cleaning.
• Ethical Concerns: Must adhere to website terms and avoid scraping restricted data.
Popular Tools:
• Beautiful Soup: Python library for parsing HTML and XML.
• Scrapy: Framework for large-scale web scraping projects.
• Selenium: For dynamic websites requiring JavaScript execution.
• Playwright: Modern alternative with multi-browser support.
Use Cases:
• Aggregating product prices from e-commerce sites.
• Collecting news articles or financial data for sentiment analysis.
• Monitoring competitor websites for changes.
 What Can We “Scrape” From The Web?
It’s possible to scrape all kinds of web data. From
search engines and RSS feeds to government
information, most websites make their data
publicly available to scrapers, crawlers, and other
forms of automated data gathering.

What Is Web Scraping?

Web scraping is a collection of practices
used to automatically extract — or
“scrape” — data from the web.
Batch + Streaming:
• Use Lambda Architectures to combine batch and streaming data for low-latency and
accurate processing.
• Example: Batch processes historical sales data, while streaming tracks live purchases.
Streaming + Web Scraping:
• Use streaming pipelines to scrape and process web data in real-time.
• Example: Live tracking of stock market prices from multiple financial websites.
Batch + Web Scraping:
• Perform scraping periodically (batch mode) and store results for further analysis.
• Example: Collect product reviews weekly and analyze sentiment trends.
 Scenario: Batch Processing in a Data Warehouse

Problem1: A retail company collects daily sales transactions from stores nationwide. The
goal is to generate a weekly report that aggregates sales by store, region, and product
category to inform inventory decisions and business strategies. Real-time processing is
unnecessary, as the report is generated weekly.
Solution: Implementing a Batch Processing Pipeline
1. Define Requirements
• Aggregate daily sales data into weekly summaries.
• Store aggregated results in a data warehouse for reporting.
• Ensure fault tolerance and data consistency.
2. Choose Tools
• Apache Hadoop for distributed storage (HDFS) and processing (MapReduce).
• Apache Spark for faster, distributed in-memory data processing.
• AWS Glue for ETL pipelines and integration with cloud storage.
3. Design the Pipeline
• Data Ingestion:
• Collect daily sales data from stores. Example Pipeline: Weekly
• Store the raw data in a distributed storage system (e.g., HDFS or Amazon S3). Sales Report
1.Data Source: Daily CSV files
• ETL (Extract, Transform, Load):
of sales transactions stored in
• Extract: Load raw sales data from storage.
Amazon S3.
• Transform: Use Apache Spark or AWS Glue to:
2.Processing: Apache Spark
• Clean the data (handle missing values, remove duplicates).
• Aggregate daily sales data by store, region, and product category.
processes data in batches:
• Load: Write the aggregated weekly data into a data warehouse like Amazon Redshift or 1. Aggregates total sales
Google BigQuery. and quantities sold
per store and
• Reporting:
product.
• Use a business intelligence (BI) tool like Tableau or Power BI to generate weekly sales
3.Output: Aggregated results
reports.
are written to Amazon
4. Ensure Fault Tolerance Redshift.
• Retries: Use built-in mechanisms (e.g., Spark’s DAG recovery) to retry failed tasks. 4.Visualization: Tableau
• Checkpoints: Enable checkpoints in Spark to recover intermediate states. generates visual reports for
stakeholders.
• Data Integrity: Validate input data and logs to ensure no records are lost.
5. Monitor and Optimize
• Monitoring: Use tools like AWS CloudWatch to track pipeline performance.
• Optimization: Tune Spark jobs for better resource utilization (e.g., partitioning,
2. A mid-sized company employs thousands of employees across multiple locations. Employee salaries are
calculated based on fixed pay, hourly wages, overtime, and other factors like bonuses, taxes, and deductions.
Payroll data includes timecards, attendance logs, and tax details.
The company needs to process payroll once a month to ensure employees are paid accurately and on time. Since
payroll is not time-sensitive until payday, real-time processing is unnecessary.
Solution using Batch Processing
Data Collection
• Attendance logs, timecards, and tax information are collected daily from various systems (e.g., biometric devices, time-
tracking software).
• Data is stored in a centralized system like a relational database or data warehouse.
Batch Processing Workflow
• At the end of the payroll cycle (e.g., last day of the month), a batch job processes the data.
• Workflow:
• Extract: Retrieve attendance, tax, and payroll details.
• Transform: Calculate:
• Regular hours worked and overtime.
• Taxes, bonuses, and deductions.
• Final net salary for each employee.
• Load: Store the payroll results in the payroll system.
Scheduling
• Batch jobs are triggered on the last day of the month using tools like Apache Airflow, Control-M, or Windows Task
Scheduler.
Output
3. An e-commerce company collects customer purchase data, website interactions,
and marketing campaign results. They want to understand customer behavior
trends over each quarter to refine their marketing strategies.
Generate a quarterly insights report that includes customer segmentation,
purchase trends, and the effectiveness of marketing campaigns.
Solution:
 Scenario: Real-Time Traffic Monitoring and Incident
Detection
Problem 1: A transportation agency wants to monitor traffic flow in real time across a city's
network of highways to detect accidents, traffic jams, or other incidents. The goal is to trigger
immediate alerts and provide actionable insights to dispatch emergency services or manage traffic
signals effectively.
Solution: Real-Time Streaming Traffic Data Processing
1. Define Requirements
• Continuously monitor traffic data from sensors or cameras (e.g., vehicle count, speed, GPS data).
• Detect traffic incidents in real-time.
• Trigger alerts or actions immediately (e.g., notifying traffic control systems or dispatching
emergency responders).
2. Choose Tools
• Apache Kafka: For real-time data streaming from sensors or traffic cameras.
• Apache Flink or Spark Streaming: For processing and analyzing incoming data in near-real-time.
• Amazon Kinesis: If using cloud-based stream processing.
• Grafana or Kibana: For real-time dashboards to visualize traffic conditions.
3. Data Ingestion and Processing Pipeline
Data Sources:
• Traffic sensors or cameras are deployed across highways to collect vehicle count, speed, and GPS data.
• The data is continuously sent as real-time streams into the Kafka topics.
Stream Processing:
• Apache Kafka ingests the traffic data in real-time from the sensors.
• Apache Flink (or Spark Streaming) processes the data:
• Detect anomalies (e.g., vehicle speed drops significantly or vehicle counts suddenly increase).
• Identify traffic jams, accidents, or sudden slowdowns.
Real-Time Actions and Alerts:
• When an incident is detected (e.g., a crash or traffic jam), an alert is triggered, notifying traffic management systems or emergency
services.
• Automated traffic control systems can adjust signals in real-time to manage traffic flow and reduce congestion.
Dashboard Visualization:
• Grafana displays live traffic conditions on an interactive map or dashboard, showing average speeds, congestion, and real-time
incidents.
4. Monitor and Scale the System
• Ensure the stream processing system scales horizontally to handle data from thousands of sensors or cameras across
the city.
• Use monitoring tools to keep track of processing latency and system health.
5. Ensure Fault Tolerance
• Use Kafka’s message replication to ensure the integrity of the data stream.
2. A bank processes thousands of transactions per second from customers worldwide. Some transactions may involve fraudulent
activities, such as credit card theft, money laundering, or unauthorized access.
The bank aims to detect potential fraud in real time to prevent unauthorized transactions and protect customer accounts.
Immediate alerts should be triggered if suspicious patterns are identified, enabling swift action.
Solution using Streaming Processing
Key Requirements for Real-Time Fraud Detection
To effectively detect fraud in real-time, the bank must implement a system with the following key capabilities:
• High-Speed Data Processing: Ability to analyze thousands of transactions per second.
• Pattern Recognition: Identifying unusual patterns that may indicate fraud.
• Machine Learning & AI: Using historical data to improve fraud detection accuracy.
• Rule-Based Systems: Implementing predefined rules to flag suspicious transactions.
• Immediate Alerts & Action: Notifying customers and freezing transactions instantly when fraud is detected.
• Scalability & Reliability: Ensuring the system can handle global transactions without failure.
Fraud Detection Mechanisms
Several approaches can be used to detect fraudulent activities in real-time:
A. Rule-Based Fraud Detection
• A rule-based system applies predefined rules to flag suspicious transactions. Examples of such rules include:
• Unusual Transaction Amount: Transactions exceeding a predefined threshold (e.g., $10,000 withdrawal in one transaction).
• Geographical Anomalies: Transactions from unexpected locations (e.g., a user from New York suddenly transacting from
Russia).
• Rapid Consecutive Transactions: Multiple transactions in a short time from different locations or merchants.
B. Machine Learning-Based Fraud Detection
Machine learning (ML) models can improve fraud detection by analyzing historical
transaction data and identifying complex fraud patterns. Some common techniques include:
• Supervised Learning: Training a model using labeled transaction data (fraudulent vs. non-
fraudulent).
• Unsupervised Learning: Detecting anomalies in transactions without predefined labels.
• Deep Learning: Using neural networks to recognize hidden fraud patterns.
• Ensemble Models: Combining multiple models (e.g., decision trees, random forests,
neural networks) to improve accuracy.
C. Real-Time Anomaly Detection
Anomaly detection techniques help identify unusual behaviors that deviate from a
customer's normal spending pattern. Common approaches include:
• Statistical Methods: Analyzing deviations in spending behavior.
• Time-Series Analysis: Detecting sudden spikes or drops in transactions.
• Graph-Based Detection: Identifying unusual connections between accounts or
transactions.
3. Real-Time Fraud Detection Architecture
A scalable fraud detection system typically consists of the following components:
A. Data Ingestion Layer
• Sources: Transactions originate from ATMs, mobile apps, online banking, POS terminals, etc.
• Streaming Frameworks: Apache Kafka, Apache Flink, or AWS Kinesis are used to collect and process
transaction data in real time.
B. Processing & Analytics Layer
• Stream Processing: Apache Flink, Spark Streaming, or Google Dataflow analyze transaction streams.
• Machine Learning Models: Trained ML models (e.g., TensorFlow, Scikit-Learn) evaluate each transaction for
fraud likelihood.
• Rule-Based Engine: Simple rule-based checks are performed for immediate detection.
C. Decision & Alerting Layer
• Real-Time Alerts: If a transaction is flagged, the system generates an alert via SMS, email, or push
notification.
• Automated Response: The system may block the transaction, request additional authentication, or freeze the
account.
D. Storage & Logging Layer
• Databases: Fraud-related transactions are stored in NoSQL databases (e.g., MongoDB) or data lakes for
further analysis.
4. Response & Action Mechanisms
Once fraud is detected, the bank must take immediate action:
• Send Alerts: Notify the customer via SMS, email, or phone call.
• Step-Up Authentication: Request additional verification (e.g., OTP, biometrics).
• Block or Hold Transactions: Prevent unauthorized transactions from going through.
• Freeze Accounts: Temporarily disable suspicious accounts until verification.
• Investigate & Escalate: Analysts review flagged transactions and escalate if necessary.
5. Future Trends in Fraud Detection
• AI-Driven Adaptive Security: AI models will continuously learn and evolve to detect new
fraud techniques.
• Behavioral Biometrics: Analyzing typing speed, mouse movements, and mobile gestures
for authentication.
• Blockchain & Decentralized Identity: Using blockchain to enhance security and prevent
identity theft.
• Federated Learning: Enabling banks to train fraud detection models without sharing
sensitive data.
Scenario: A multinational bank processes thousands of financial transactions per
second from customers worldwide. Transactions include credit card payments,
wire transfers, ATM withdrawals, and online banking activities. However, some
transactions may involve fraudulent activities such as:
• Credit card theft – Unauthorized purchases from stolen card details.
• Money laundering – Large or structured transactions designed to hide illicit
funds.
• Unauthorized access – Account takeovers leading to unapproved fund transfers
Solution:
3. A manufacturing company uses IoT sensors to monitor equipment in its factories. These
sensors track metrics like temperature, pressure, vibration, and energy consumption.
Equipment failures can lead to downtime, production delays, and increased costs.
The company aims to detect anomalies or equipment failures in real time to prevent costly
downtime and optimize maintenance schedules.
Solution using Streaming Processing
Data Ingestion
• IoT sensors continuously send data to a central streaming platform via protocols like MQTT or HTTP.
• The data includes timestamped sensor readings and equipment status.
Streaming Workflow
• A real-time processing engine (e.g., Apache Spark Streaming, Azure Stream Analytics, or Google
Cloud Dataflow) processes the incoming data.
• Workflow:
• Ingest: Sensor data streams into the system in real time.
• Process:
• Detect anomalies using thresholds or predictive models (e.g., abnormal temperature or vibration spikes).
• Correlate multiple sensor readings to identify patterns indicating potential failures.
• Filter noise to focus on actionable insights.
• Alert: Generate real-time notifications or visual dashboards highlighting anomalies and their potential impact.
Scenario: Price Monitoring for E-commerce Competitors

Problem: A retail company wants to monitor the prices of its competitors’ products on
their e-commerce websites to adjust pricing strategies dynamically. The challenge is that
the data is scattered across multiple web pages and some websites load prices dynamically
using JavaScript.
Solution: Web Scraping Pipeline
1. Define Requirements
• Extract product names, prices, and availability from competitor websites.
• Handle dynamic content loading.
• Store the data for analysis and comparison.
2. Choose Tools
• Playwright or Selenium: To handle websites with dynamic JavaScript rendering.
• Beautiful Soup: For parsing static HTML content.
• Pandas: For organizing and cleaning scraped data.
•
3. Steps to Implement the Solution
Inspect Target Websites:
• Identify product details (name, price, availability) and their corresponding HTML tags or classes.
• Check if the data is loaded dynamically via JavaScript.
Set Up the Scraper:
• Use Playwright for JavaScript-rendered pages to ensure dynamic content loads fully before scraping.
• Use Beautiful Soup for simpler, static pages
Store and Analyze the Data:
• Save the scraped data to a database (e.g., SQLite, PostgreSQL) for further analysis.
• Use Python’s Pandas or visualization tools (e.g., Tableau) to compare pricing trends.
Schedule Regular Scraping:
• Automate the scraper to run daily or weekly using a task scheduler like Cron or Apache
Airflow.
4. Handle Ethical Concerns
• Check the website’s robots.txt file to ensure scraping is allowed.
• Respect the website’s terms of service.
• Avoid overloading servers by setting delays between requests.
2. A travel agency wants to monitor flight prices across multiple airline and travel booking websites to
offer competitive deals. Airline prices fluctuate frequently based on demand, availability, and other
factors, and the data is often dynamically loaded via JavaScript.
• Scrape flight price data in near real time to:
• Identify price drops or limited-time offers.
• Adjust travel package pricing dynamically.
• Notify customers of the best deals.
Solution: ·
Challenges
Airline websites often use JavaScript to load flight prices, requiring tools that can render JavaScript.
Frequent changes to website structure may break scrapers.
Approach
• Use web scraping tools such as Playwright, Selenium, or Puppeteer for rendering JavaScript and
scraping dynamic content.
• Set up a proxy pool (e.g., with Scrapy or Bright Data) to avoid IP bans due to frequent requests.
Workflow
Identify Data Sources: Determine key websites to monitor, such as airline booking pages and travel
Scraping:
• Launch browser instances to render pages and extract flight details like departure, arrival,
price, and availability.
• Use CSS selectors or XPath to extract elements containing the flight prices and metadata.
Store and Process: Save scraped data in a database (e.g., PostgreSQL or
MongoDB).
Integration:
• Compare scraped prices with the travel agency's existing price database.
• Notify stakeholders or customers via email or an app for significant price changes.
Scheduling
• Use a scheduler like Cron Jobs, Apache Airflow, or AWS Lambda to run scrapers
periodically (e.g., every hour).
Output
• Create dashboards to display current flight prices and price trends.
• Highlight competitive deals.
Step Tools/Technologies

Architecture Design Lucidchart, Draw.io, Visio

Data Ingestion Apache Nifi, Talend, Fivetran, Kafka, AWS Glue

Data Processing Apache Spark, Apache Flink, Python, SQL

AWS S3, Azure Data Lake, Google Cloud Storage, Snowflake,

Storage
BigQuery

Workflow Orchestration Apache Airflow, Prefect, Dagster

Deployment Docker, Kubernetes, Jenkins, GitHub Actions

Monitoring Prometheus, Grafana, AWS CloudWatch

Visualization Tableau, Power BI, Looker

A data warehouse is a centralized storage system that allows for the storing, analyzing, and
interpreting of data in order to facilitate better decision-making.
a schema is a graphical representation that makes it easy to organize information or knowledge.
APIs (Application Programming Interfaces)
APIs serve as bridges to extract data from systems, applications, or services, often in a
structured and programmatically accessible way. They are essential for real-time and large-
scale data extraction.

Key Characteristics of APIs:

• Structured Data: APIs typically return data in formats like JSON, XML, or CSV.
• Programmatic Access: Developers can write scripts or applications to query APIs directly.
• Authentication & Security: Most APIs require keys, OAuth, or tokens to control access
and ensure security.
• Rate Limits: APIs often restrict the number of requests per time interval to prevent abuse.
Steps followed in the working of APIs

•The client initiates the requests via the

APIs URI (Uniform Resource Identifier)

•The API makes a call to the server after

receiving the request

•Then the server sends the response back

to the API with the information

•Finally, the API transfers the data to the

client
Application Programming Interface protocol types
They use a few different protocols, but the most common ones are HTML, XML, and JSON. They
are used to define how data is formatted when exchanged between systems.

HTML: HTML is used for web-based APIs, and it allows you to embed images, videos, and other
types of content in your messages. It’s easy to use, but it can be restrictive since you’re limited
to what you can include on a web page.

XML: XML is more versatile than HTML and creates custom tags and attributes. This makes it a
popular choice for SOAP APIs, but it can be more complex to use.

JSON: JSON is a lightweight alternative to XML, and it’s easy to read and write. It’s famous for
RESTful APIs because it’s fast and efficient.
Types of APIs:
REST APIs (Representational State Transfer):
• Stateless, lightweight, and widely used.
• Data is typically exchanged in JSON format.
• Example: Twitter API, GitHub API.
GraphQL APIs:
• Flexible querying allows clients to request only the data they need.
• Example: Shopify API, GitHub GraphQL API.
SOAP APIs (Simple Object Access Protocol):
• XML-based, often used in enterprise systems.
• Example: Payment gateways like PayPal.
Streaming APIs:
• Send continuous data streams in real time.
• Example: Twitter's live feed API.
Custom APIs:
• Built for internal systems or specific use cases.
Data Extraction Using APIs
APIs are a reliable source for extracting structured or semi-structured data.
Workflow:
1.Understand API Documentation:
•Learn the available endpoints, authentication methods, rate limits, and request/response formats.
2.Authentication:
•Generate API keys or tokens to authenticate requests.
3.Make API Requests:
•Use tools like Postman, curl, or Python libraries (e.g., requests) to query endpoints.
4.Parse Responses:
•Parse JSON or XML data into structured formats using libraries like Python’s json or
xml.etree.ElementTree.
5.Store Data:
•Save the extracted data into databases, data lakes, or files (e.g., CSV, Parquet).
Tools & Libraries:
•Postman: API testing and request automation.
•Python:
•requests and httpx for API calls.
•pandas for data manipulation.
•Node.js:
•Axios is a Javascript library used to make HTTP requests from node.
•API Gateways: AWS API Gateway for managing APIs.
Data Extraction (Beyond APIs)
When APIs are unavailable, or additional data sources are needed, these methods come into play:
Web Scraping:
Extract data from web pages when no API is available.
•Tools: Beautiful Soup, Scrapy, Selenium, Playwright.
•Challenges: Handling CAPTCHA, JavaScript-heavy websites, or dynamic content.

Database Extraction:
Directly extract data from relational or NoSQL databases.
•Tools: SQL queries, ODBC/JDBC drivers(The JDBC-ODBC bridge driver uses ODBC driver to connect to the
database).
•Example: Query a MySQL database for historical sales records.

File-based Extraction:
Retrieve data stored in files.
•Formats: CSV, Excel, JSON, Parquet.
•Tools: Python’s pandas, openpyxl, csv, and pyarrow.

Cloud and Storage Systems:

Fetch data stored in cloud storage or data lakes.
•Tools: AWS S3, Google Cloud Storage, Azure Blob Storage.
•Formats: ORC, Parquet, Avro.
 Overview of the Data Engineering Lifecycle
The data engineering lifecycle involves stages that enable the collection, processing,
and preparation of data for analysis and decision-making. Here’s an overview of the key
phases:
• Data Collection: Gather data from sources like APIs, IoT devices, and databases.
• Data Ingestion: Move data into storage systems (real-time or batch).
• Data Storage: Save data securely in databases, data lakes, or warehouses.
• Data Transformation (ETL/ELT): Clean, standardize, and organize raw data.
• Data Integration: Combine data from multiple sources into a unified view.
• Data Validation: Ensure data accuracy, completeness, and consistency.
• Data Governance: Secure data and ensure compliance with regulations.
• Data Pipelines: Automate workflows for seamless data processing.
• Data Delivery: Provide data for analytics, BI tools, or APIs.
• Monitoring: Track and maintain pipeline performance and reliability.
• Generation
• Ingestion
• Storage
• Transformation
• Serving (ML, Analytics, Reverse ETL)
1. Generation: Data is created or captured from various sources (e.g., IoT devices, APIs, web apps,
logs, databases, etc.).
Examples: Sensor readings, user activity logs, financial transactions.
2. Ingestion: Data is collected and brought into a system for further processing.
• Common methods: Batch ingestion (nightly ETL jobs) or streaming ingestion (e.g., Kafka, Kinesis).
Tools: Apache Kafka, AWS Glue, Flume.
3. Storage:The ingested data is stored in a system that supports the desired level of accessibility, scalability, and
durability.
Storage systems vary based on structure:
•Relational databases (MySQL, PostgreSQL).
•NoSQL databases (MongoDB, Cassandra).
•Data lakes (S3, HDFS).
•Data warehouses (Snowflake, BigQuery).
4. Transformation: Raw data is cleaned, enriched, and converted into a format suitable for analysis or
machine learning.
Techniques include:
•Data cleaning (handling nulls, duplicates).
•Aggregation.
•Feature engineering for ML.
Tools: Apache Spark, Airflow.
5. Serving: The processed data is made accessible for various purposes like analytics, machine learning
models, or operational use cases.
Includes:
•Analytics: Powering dashboards (e.g., Tableau, Looker).
•Machine Learning: Serving data to models for inference or training.
•Reverse ETL: Moving processed data back into operational tools (e.g., Salesforce, marketing
platforms).
“Reverse ETL is the process of moving data from a data warehouse (or data lake) back into operational systems
like CRMs, marketing platforms, support tools, and SaaS applications”
How Reverse ETL Works
• Extract: Data is pulled from a warehouse (e.g., Snowflake, BigQuery, Redshift).
• Transform (Optional): Data may be cleaned, formatted, or enriched before syncing.
• Load: The processed data is pushed into operational tools (e.g., Salesforce, HubSpot, Zendesk, Marketo).
Tools: Redis (low-latency serving), Snowflake (analytics), Feature Stores (for ML).
 Enterprise Resource Planning (ERP) and Customer
Data Platform (CDP)

ERP is used for internal processes, while CDP is

used for customer-related information.
Scenario: A large retail company operates both online and in physical
stores, processing thousands of customer orders, payments, and
inventory updates daily. This data is critical for tracking sales, managing
stock levels, optimizing supply chains, and understanding customer
behavior.
As a data engineer, your role is to efficiently manage this data
throughout its lifecycle, ensuring it is collected, stored, processed,
secured, and transformed for business insights.
Solution: Stages of the Data Lifecycle & Data Engineer's Role
1. Data Collection
Sources: Customer purchases, payment transactions, inventory updates, website logs, customer
feedback, and third-party integrations (e.g., shipping providers, marketing platforms).
Data Engineer's Role:
• Design and maintain ETL (Extract, Transform, Load) pipelines to gather data from various sources.
• Implement real-time streaming (Apache Kafka, Apache Flink) for immediate updates.
• Ensure data consistency and completeness by handling missing or duplicate records.
2. Data Storage
Types of Data: Structured (orders, payments), semi-structured (JSON logs, API responses), and
unstructured (customer reviews, images).
Storage Solutions:
• Use Cloud Data Warehouses (AWS Redshift, Snowflake, BigQuery) for structured data.
• Implement Data Lakes (S3, Delta Lake) for raw and semi-structured data.
• Use NoSQL Databases (MongoDB, DynamoDB) for flexible, high-speed access.
Data Engineer's Role:
• Design efficient data storage strategies balancing cost, performance, and scalability.
• Implement partitioning, indexing, and data retention policies.
3. Data Processing & Transformation
• Batch Processing: Aggregating daily sales, summarizing stock levels, and preparing reports.
• Stream Processing: Detecting fraud in transactions, real-time stock updates, dynamic
pricing.
• Data Engineer's Role:
• Build pipelines using Spark, Flink, or DBT to clean, normalize, and enrich data.
• Implement data quality checks to eliminate errors.
• Optimize query performance for faster analytics.

4. Data Security & Governance

• Security Measures:
• Encrypt sensitive data (credit card details, PII) using AES, TLS/SSL.
• Implement role-based access control (RBAC) to restrict access.
• Comply with regulations like GDPR, CCPA, PCI-DSS.
• Data Engineer's Role:
• Ensure audit logs and data masking for compliance.
• Design backup and disaster recovery solutions.
5. Data Analysis & Business Insights
• Data Transformation:
• Create aggregations (total sales, customer retention rates).
• Develop customer segmentation for targeted marketing.
• Enable real-time dashboards for executives using BI tools (Tableau, Power BI, Looker).
6. Data Governance & Lifecycle Management
• Implement data retention policies (cold storage, archival).
• Use data cataloging tools (Apache Atlas, AWS Glue Data Catalog) for metadata
management.
• Set up data quality checks using Great Expectations or Soda.

How Data Helps Business Teams:

• Sales Team: Identifies best-selling products and seasonal trends.
• Inventory Team: Optimizes stock levels to prevent shortages or overstocking.
• Marketing Team: Personalizes campaigns based on customer behavior.
A leading e-commerce company wants to provide a personalized shopping
experience to millions of customers across its website and mobile app. Every
second, thousands of users browse products, add items to their carts, and make
purchases. The company aims to leverage real-time data to recommend relevant
products, optimize pricing, and improve customer engagement.
As a data engineer, your job is to design and maintain a data pipeline that collects,
processes, and analyzes user interactions in real time to enable personalized
recommendations.
Data Engineering Process
Extract, Transform, Load (ETL) Process is an automated process which
include:
1. Gathering raw data
2. Extracting information needed for reporting and analysis
3. Cleaning standardizing, and transforming data into usable format
4. Loading data into a data repository
ETL\ELT\Data Pipelines
Data pipelines are generally structured as either an Extract, Transform, Load or an
Extract, Load, Transform (ETL vs ELT) of course, there are other design patterns we
may take on, such as event pipelines, streaming and change data capture (CDC)

• E - Extract The extract step involves connecting to a data source such as an API,
automated reporting system or file store, and pulling the data out of it.
For example, one project I worked on required me to pull data from Asana. This
meant I needed to create several components to interact with Asana’s multiple API
endpoints, pull out the JSON and store it in a file service.
• T - Transform The transform step (especially in an initial pipeline) will likely
standardize the data (format dates, standardize booleans, etc.,) as well as
sometimes starting to integrate data by adding in IDs which are also
standardized, deduplicating data and adding in more human readable categories.
Data Engineering Process
Extract, Transform, Load (ETL) Process is an automated process which
include:

Extraction can be through:

• Bath processing – large chunks of data moved from

Staging Area
source to destination at scheduled intervals.
Extract • Stream processing – data pulled in real-time from
source, transformed in transit, and loaded into data
repository
Data Engineering Process
Extract, Transform, Load (ETL) Process is an automated process which
include:
Transforming data:
• Standardizing data formats and
units of measurement
• Removing duplicate data
Transform
Staging Area
• Filtering out data that is not
Extract and load required
• Fill missing data
• Enriching data
• Establishing key relationships
across tables
• Applying business rules and data
validations
Data Engineering Process
Extract, Transform, Load (ETL) Process is an automated process which
include:

Loading is the transportation of processed data

into a data repository. It can be
Transform • Initial loading – populating all of the data in the
Staging Area

and load repository

• Incremental loading – applying updates and
modifications periodically
• Full refresh – erasing a data table and reloading
fresh data
Data Engineering Process
Extract, Transform, Load (ETL) Process is an automated process which
include:

Load verification includes checks for:

• Missing or null values
Transform
Staging Area

• Server performance
and load • Load failures
Data Engineering Process
Extract, Transform, Load (ETL) Process is
an automated process which include:

Transform
Staging Area
Extract and load

Data Repository Analytics

Data Engineering Process
Extract, Load, Transform (ELT) Process
• Helps process large sets of unstructured data and non-relational data
• Is ideal for data lakes

Transformations

Extract
and load Data lake

Data Repository

Data warehouse
Data Engineering Process
Advantages of ELT process
• Shortens the cycle between extraction and delivery
• Allows you to ingest volumes of raw data as immediately as the data becomes
available
• Affords greater flexibility to analysts and data scientists for exploratory data
analysis
• Transforms only that data which is required for a particular analysis so it can be
leveraged for multiple use cases
• Is more suited to work with Big Data
Data Engineering Process
Data Pipeline
• Encompasses the entire journey of moving data from one system to
another, including the ETL/ELT process
• Combining a transactional database, a programming language, a processing
engine, and a data warehouse results in a pipeline.
• Can be used for both batch and streaming data
• Supports both long-running batch queries and smaller interactive queries
• Typically loads data into a data lake but can also load data into a variety of
target destinations – including other application and visualization tools
 What is Data Modeling?
Data modeling is the process of conceptualizing and visualizing how data will be captured, stored, and used by an
organization. The ultimate aim of data modeling is to establish clear data standards for your entire organization.
Diagrams are used to present information visually, making it easier to understand, compare, and remember. They
can be used to represent data, ideas, processes, relationships, and more.

• Simplify complex data: Diagrams can represent large amounts of data in a way that's easy to understand.
• Compare data: Diagrams can help you visually compare two sets of data.
• Reveal hidden facts: Diagrams can help you identify patterns and hidden facts in data.
• Express ideas: Diagrams can help you express ideas that are difficult to put into words.
• Show probabilities and risk: Diagrams can help you show the probability of success or risk in a situation.
• Show spatial relations: Diagrams can help you show spatial relations like hierarchy, proximity, and
connectedness.

Examples of diagrams
• Bar charts: Used to represent statistics
• Pie charts: Used to represent statistics
• Tree diagrams: Used to visualize topics like company roles, family relationships, and evolutionary relationships
There are 3 main types of Data Models:
1. Conceptual Data Model: These models are designed in an effort to communicate with stakeholders
showing relationships between different entities and defining their essential attributes according to the
business requirements. It is an abstract version represented by ER or UML diagrams to confirm with the
goal and scope of the data project.
Characteristics:
• Independent of any specific technology or database.
• Focuses on what data is needed and its meaning.
• Simple representation without technical details like keys or constraints.
• Purpose:
• To understand and document business requirements.
• To serve as the foundation for logical and physical models.
• Example:
• Entities: Customer, Product, Order.
• Attributes:
• Customer: Name, Email, Phone Number.
• Product: Name, Price, Category.
• Relationships:
• A Customer places Orders.
2. Logical Data Model: A logical data model is a detailed representation of the data structure that
specifies the attributes, data types, relationships, and rules governing the data. It is independent of the
database management system (DBMS) and defines how the data will be organized.
Characteristics:
• Specifies how the data will be organized logically.
• Includes primary keys, foreign keys, and relationships.
• Focuses on data integrity, normalization, and business rules.
Purpose:
• To act as a bridge between business needs and the technical implementation.
• To standardize the structure for implementation in any DBMS.
Example:
• Entities: Customer, Order, Product.
• Attributes and Keys:
• Customer: CustomerID (Primary Key), Name, Email.
• Order: OrderID (Primary Key), OrderDate, CustomerID (Foreign Key).
• Product: ProductID (Primary Key), Name, Price.
Relationships:
• A Customer places one or more Orders (1:M).
3. Physical Data Model: A physical data model describes the actual
implementation of the data in a specific database system, including tables,
columns, data types, indexes, and storage details. It defines how the data will be
stored and accessed in the DBMS.
Characteristics:
• Tied to a specific DBMS (e.g., MySQL, PostgreSQL, Oracle).
• Includes how the data will be stored and accessed physically.
• Contains technical details like partitioning, indexing, and storage mechanisms.
Purpose:
• To define the exact database schema for implementation.
• To optimize the database for performance, scalability, and reliability.
Key Differences
Entity, Relationship, and E-R
Diagram
• A database can be modeled as:
• a collection of entities,
• relationship among entities.
• A database can be illustrated by an E-R diagram
E-R Diagrams

 Rectangles represent entity sets.

 Diamonds represent relationship sets.
 Lines link attributes to entity sets and entity sets to relationship sets.
 Ellipses represent attributes
 Double ellipses represent multivalued attributes. (will study later)
 Dashed ellipses denote derived attributes. (will study later)
 Underline indicates primary key attributes (will study later)
 Entity Sets
• An entity is an object that exists and is distinguishable from other objects.
• Example: specific person, company, event, plant
• Entities have attributes
• Example: people have names and addresses
• An entity set is a set of entities of the same type that share the same properties.
• Example: set of all persons, companies, trees, holidays
 Attributes
• An entity is represented by a set of attributes, that is descriptive properties possessed by
all members of an entity set.
Example:
customer = (customer-id, customer-name,
customer-street, customer-city)
loan = (loan-number, amount)

• Domain – the set of permitted values for each attribute

• Attribute types:
• Simple and composite attributes.
• Single-valued and multi-valued attributes
• E.g. multivalued attribute: phone-numbers
• Derived attributes
• Can be computed from other attributes
• E.g. age, given date of birth
Composite Attributes
E-R Diagram With Composite, Multivalued, and
Derived Attributes
 Weak Entity and Regular/Strong
Entity
 A weak entity is an entity that is existence-dependent on some other entity. By
contrast, a regular entity (or “a strong entity”) is an entity which is not weak.
 The existence of a weak entity set depends on the existence of a identifying entity set
it must relate to the identifying entity set via a total, one-to-many relationship set
from the identifying to the weak entity set
 E.g. An employee’s dependents might be weak entities --- they can’t exist (so far as
the database is concerned) if the relevant employee does not exist.
 A weak entity type can be related to more than one regular entity type.
Weak Entity and Regular/Strong
Entity We depict a weak entity by double rectangles.
 The identifying relationship is depicted using a
double diamond.
E-R Diagram with a Ternary
Relationship
Roles • Entity sets of a relationship need not be distinct
o The labels “manager” and “worker” are called roles; they specify how
employee entities interact via the works-for relationship set.
o Roles are indicated in E-R diagrams by labeling the lines that connect
diamonds to rectangles.
o Role labels are optional, and are used to clarify semantics of the relationship
Mapping Cardinalities
• Express the number of entities to which another
entity can be associated via a relationship set.
• Most useful in describing binary relationship
sets.
• For a binary relationship set the mapping
cardinality must be one of the following types:
• One to one
• One to many
• Many to one
• Many to many
Mapping Cardinalities

One to one One to many

Note: Some elements in A and B may not be mapped to any
elements in the other set
Mapping Cardinalities

Many to one Many to many

Note: Some elements in A and B may not be mapped to any
elements in the other set
Mapping Cardinality
• We express cardinality constraints by drawing either a
directed line (), signifying “one,” or an undirected line
(—), signifying “many,” between the relationship set and
the entity set.
• E.g.: One-to-one relationship:
• A customer is associated with at most one loan via the
relationship borrower
• A loan is associated with at most one customer via borrower
 One-To-Many Relationship

In the one-to-many relationship a loan is associated with at most one

customer via borrower, a customer is associated with several (including 0)
loans via borrower
Many-To-One Relationships

• In a many-to-one relationship a loan is associated

with several (including 0) customers via borrower, a
customer is associated with at most one loan via
borrower
Many-To-Many Relationship

• A customer is associated with several (possibly 0) loans via

borrower
• A loan is associated with several (possibly 0) customers via
borrower
Mapping Cardinalities affect ER Design
 Can make access-date an attribute of account, instead of a
relationship attribute, if each account can have only one customer
 I.e., the relationship from account to customer is many to one,
or equivalently, customer to account is one to many
Relationship Sets with Attributes
 Participation of an Entity Set in a
Relationship Set
 Total participation (indicated by double line): every entity in the
entity set participates in at least one relationship in the
relationship set
 E.g. participation of loan in borrower is total
 every loan must have a customer associated to it via
borrower
 Partial participation: some entities may not participate in any
relationship in the relationship set
 E.g. participation of customer in borrower is partial
Attribute of a Relationship Type is:
Hours of WORKS_ON
COMPANY ER Schema Diagram using (min, max) notation
ER DIAGRAM FOR A BANK DATABASE

© The Benjamin/Cummings Publishing Company, Inc. 1994, Elmasri/Navathe, Fundamentals of Database Systems, Second Edition
Specialization Example
 Normalization and Denormalization

1. Normalization:
The process of organizing data to eliminate redundancy and improve integrity.
Benefits:
•Reduces data redundancy.
•Ensures data consistency.
Forms of Normalization:
1.1NF (First Normal Form): Eliminate duplicate columns and ensure atomicity (no
repeating groups).
2.2NF (Second Normal Form): Ensure 1NF and eliminate partial dependency (non-
prime attributes depending only on part of a composite key).
3.3NF (Third Normal Form): Ensure 2NF and eliminate transitive dependency
(attributes depending indirectly on the primary key).
4.BCNF (Boyce-Codd Normal Form): A stricter version of 3NF ensuring every
determinant is a candidate key.
2. Denormalization:
The process of combining tables to improve query performance, often at the cost of
redundancy.
Benefits:
•Faster read queries in OLAP systems.
•Simplifies complex joins.

3. OLAP (Online Analytical Processing)

•Designed for data analysis and reporting.
•Focuses on historical data and aggregates.
Use case: Business intelligence, dashboards, trends.
Characteristics:
• High query latency is acceptable.
• Data is stored in a star or snowflake schema.
• Example tools: Snowflake, Google BigQuery, Apache Druid.
4. OLTP (Online Transaction Processing)
•Designed for handling transactional data in real time.
•Focuses on current, detailed data.
Use case: Banking systems, e-commerce platforms.
Characteristics:
• Fast query performance (low latency).
• Relational database design with normalized tables.
• Example tools: MySQL, PostgreSQL, MongoDB.
1.Normalization is the technique of dividing the data into multiple tables to
reduce data redundancy and inconsistency and to achieve data integrity. On
the other hand, Denormalization is the technique of combining the data into
a single table to make data retrieval faster.
2.Normalization is used in an OLTP system, which emphasizes making the
insert, delete and update anomalies faster. As the opposite, Denormalization
is used in an OLAP system, which emphasizes making the search and
analysis faster.
3.Data integrity is maintained in the normalization process while in
denormalization data integrity harder to retain.
4.Redundant data is eliminated when normalization is performed whereas
denormalization increases the redundant data.
5.Normalization increases the number of tables and joins. In contrast,
denormalization reduces the number of tables and joins.
6.Disk space is wasted in denormalization because the same data is stored in
different places. On the contrary, disk space is optimized in a normalized
table.
 Scenario based Questions
1. Real-Time Product Recommendation System

Problem: An e-commerce platform wants to recommend products in real-time

based on user behavior.
Approach:
• Data Generation: Capture clickstream data (e.g., pages visited, searches).
• Data Collection: Use Apache Kafka to stream user events.
• Data Storage: Store raw data in a data lake (e.g., S3).
• Data Processing: Use Apache Flink to process clickstream events and generate
recommendations in real time.
• Data Serving: Use a NoSQL database (e.g., Redis) to store recommendations for
quick retrieval.
• Tools: Kafka, Flink, Redis, S3.
 2. Data Lake for Unstructured Media Files
3. Problem: A media company wants to manage unstructured media files like
videos and images while enabling metadata searches.
Approach:
• Data Generation: Videos and images are uploaded by users.
• Data Collection: Upload files to a data lake (e.g., AWS S3) using an API or
CLI.
• Data Storage: Store metadata (e.g., file name, tags) in NoSQL databases like
DynamoDB.
• Data Processing: Use AWS Lambda to extract metadata (e.g., video
duration) and enrich the data.
• Analysis: Use Elasticsearch for metadata search.
• Tools: S3, DynamoDB, Elasticsearch, AWS Lambda.
 3. Web Scraping Competitor Pricing Data
Problem: A retail company needs daily competitor pricing data for analysis.
Approach:
• Data Collection: Use Python (BeautifulSoup/Scrapy) to scrape competitor
websites.
• Data Storage: Store raw scraped data in CSV files or a database (SQLite for
small-scale, PostgreSQL for larger scale).
• Data Processing: Clean and transform the data using pandas.
• Analysis: Visualize price trends using tools like Tableau or Power BI.
• Tools: Scrapy, pandas, PostgreSQL, Tableau.
4. Batch Processing for Daily Sales Reports
Problem: A retailer wants to generate daily sales summary reports from
transaction logs.
Approach:
• Data Collection: Gather logs from a point-of-sale system.
• Data Storage: Store logs in a relational database (e.g., MySQL).
• Data Processing: Use Apache Spark or pandas for daily ETL to aggregate
sales data.
• Data Serving: Push the summarized data into a data warehouse (e.g.,
Snowflake).
• Tools: MySQL, Apache Spark, Snowflake.
 5. Fraud Detection System for Banking
Problem: A bank wants to detect fraudulent transactions in real time.
Approach:
• Data Generation: Stream transaction data from banking systems.
• Data Collection: Use Apache Kafka for ingestion.
• Data Storage: Store raw data in a data lake and processed data in a
NoSQL database (e.g., MongoDB).
• Data Processing: Use Apache Spark Streaming or Flink to apply fraud
detection models.
• Data Serving: Use the processed data to alert users in real time.
• Tools: Kafka, Flink, MongoDB, Spark Streaming.
 6. ETL Pipeline for Marketing Campaign
Analysis
Problem: A marketing team wants insights into email campaign performance.
Approach:
• Data Collection: Extract email engagement data from APIs (e.g., Mailchimp).
• Data Storage: Store raw data in a data lake (e.g., Google Cloud Storage).
• Data Processing: Use Apache Airflow for ETL to clean and aggregate
campaign metrics.
• Data Analysis: Load processed data into a data warehouse (e.g., BigQuery)
for BI reporting.
• Tools: Airflow, Google Cloud Storage, BigQuery.
 7. Real-Time Weather Data Analytics
Problem: A weather company wants to analyze real-time weather data from
IoT devices.
Approach:
• Data Generation: Weather sensors stream temperature, humidity, etc.
• Data Collection: Use AWS Kinesis or Kafka for streaming ingestion.
• Data Storage: Store raw data in a time-series database (e.g., InfluxDB).
• Data Processing: Use AWS Lambda to process data for aggregation and
anomaly detection.
• Data Analysis: Serve insights via dashboards built with Grafana.
• Tools: Kafka, InfluxDB, AWS Lambda, Grafana.
 8. Migration to a Cloud Data Warehouse

Problem: A company wants to migrate its on-premise database to a cloud-

based data warehouse.
Approach:
• Data Collection: Extract data from the on-premise relational database.
• Data Storage: Migrate raw data to a cloud platform (e.g., AWS S3).
• Data Processing: Use AWS Glue or Apache Nifi to clean and transform
data during migration.
• Data Serving: Load data into a cloud warehouse like Amazon Redshift or
Snowflake.
• Tools: AWS Glue, Redshift, Nifi.
 9. Building a Reverse ETL System
Problem: A marketing team wants customer analytics data pushed back into
operational tools like Salesforce.
Approach:
• Data Storage: Store analytics data in a data warehouse (e.g., BigQuery).
• Data Processing: Use dbt to prepare data for operational use.
• Data Serving: Push the processed data back into Salesforce using a
reverse ETL tool like Hightouch or Census.
• Tools: BigQuery, dbt, Hightouch.
 10. Building a Batch and Streaming Data Pipeline
for Customer Insights
Problem:
A retail company wants to combine historical sales data with real-time customer behavior (e.g., page
views and clicks) to generate actionable insights for improving the shopping experience.
Solution:
Step 1: Data Collection
• Batch Data (Historical Sales):
• Pull historical sales data from a relational database (e.g., MySQL).
• Export the data into a data lake (e.g., AWS S3).
• Streaming Data (Real-Time Customer Behavior):
• Use Apache Kafka or AWS Kinesis to collect and stream clickstream data.
• Events include customer page views, search queries, and cart actions.
Step 2: Data Storage
• Batch Data: Store raw historical data in a data lake for processing.
• Streaming Data: Store raw clickstream data in a time-series database (e.g., Apache Druid) for quick
analysis.
Step 3: Data Processing
• Batch Pipeline:
• Use Apache Spark to clean and transform historical sales data.
• Aggregate sales data by customer (e.g., total spend, frequency of purchases).
• Write the processed data into a data warehouse (e.g., Snowflake).
• Streaming Pipeline:
• Use Apache Flink to process clickstream events in real-time.
• Aggregate behavior data (e.g., number of pages viewed in a session).
• Enrich streaming data by joining with customer profiles from the warehouse.
Step 4: Data Serving
• Combine historical and real-time data in the data warehouse (e.g., BigQuery or Snowflake).
• Create a dashboard (using Tableau or Looker) to display real-time insights, such as:
• Top viewed products.
• Customers likely to abandon their cart.
• Tools Used:
• Batch: MySQL → S3 → Spark → Snowflake
• Streaming: Kafka → Flink → Druid
• Visualization: Tableau or Looker
 11. Data Warehouse for Multi-Source Analytics
Problem:
A logistics company collects data from various sources (e.g., shipment tracking, vehicle sensors, customer orders) and wants to
centralize it into a data warehouse for business intelligence and predictive analytics.
Solution:
Step 1: Data Collection
• Shipment Tracking Data:
• Extract data from an API provided by the shipment management system.
• Store raw API responses in a data lake (e.g., Azure Blob Storage).
• Vehicle Sensor Data:
• Stream IoT sensor data (e.g., GPS, fuel consumption) using Apache Kafka.
• Customer Order Data:
• Extract order records from a relational database (e.g., PostgreSQL).
Step 2: Data Storage
• Use a data lake (Azure Blob Storage) to store raw data from all sources.
• Load processed data into a cloud data warehouse (e.g., Snowflake or Redshift).
Step 3: Data Processing
• ETL Pipeline:
• Use dbt (data build tool) for transformations:
• Clean data from the shipment API (e.g., fix missing fields, parse JSON).
• Aggregate vehicle sensor data to calculate metrics like fuel efficiency.
• Join customer order data with shipment data for delivery insights.
Step 4: Data Modeling
• Implement a star schema in the data warehouse:
• Fact Table: Shipment facts (e.g., delivery times, costs).
• Dimension Tables: Customers, vehicles, shipment status, regions.
Step 5: Data Analysis
• Build a BI dashboard for:
• On-time delivery rates by region.
• Top-performing vehicles (based on fuel efficiency).
• Order volume trends.
Step 6: Predictive Analytics
• Export prepared data to a machine learning pipeline for:
• Predicting delivery delays based on weather, vehicle data, and order location.
• Optimizing routes using historical shipment data.
• Tools Used:
• Data Collection: API → Blob Storage, Kafka → Blob Storage
• ETL: dbt, Apache Airflow
• Storage: Azure Blob Storage, Snowflake
• Visualization: Power BI or Tableau
• ML: Python (scikit-learn)