Unit 4: Mining Data Streams (11 Hours)

Overview
Unit 4 focuses on mining data streams, which are continuous, high-speed, and often
infinite flows of data generated in real time, such as social media posts, online trans-
actions, stock market feeds, or sensor data from IoT devices. Unlike traditional static
datasets, data streams are dynamic, meaning they evolve over time, arrive at a rapid
pace, and cannot be fully stored due to memory constraints. This unit explores spe-
cialized techniques to process and analyze such data, focusing on methodologies for
stream processing, frequent and sequential pattern mining, handling class imbalance, and
applying these methods to graph mining and social network analysis. The 11-hour dura-
tion allows for an in-depth study of these complex topics, which are critical for real-time
decision-making in modern applications.

1 Methodologies for Stream Data Processing and Stream Data

Systems
1.1 What is Stream Data Processing?
• Definition: Stream data processing refers to the real-time analysis of data that
arrives continuously, like a stream of water, without the ability to store it all.
The goal is to process data as it arrives, often in a single pass, to extract insights
immediately.
• Characteristics of Stream Data:
– Infinite Volume: Data never stops coming (e.g., tweets on X during a global
event).
– High Velocity: Arrives at a very fast rate (e.g., thousands of transactions per
second in an online store).
– Transient Nature: Cannot be stored entirely due to limited memory, so only
recent or summarized data is kept.
– Time-Sensitive: Insights must be generated quickly to be useful (e.g., detect-
ing fraud in real time).

Example: A smart city system receiving live traffic data from sensors on
roads to monitor congestion instantly.

1.2 Why Stream Data Processing is Challenging

• Volume and Speed: The sheer amount and speed of data make it impossible to
process everything using traditional methods.
• Memory Constraints: Systems have limited memory, so the entire stream cannot
be stored for later analysis.

1
 • Evolving Patterns: Data patterns change over time (e.g., trending topics on
social media shift hourly).
• One-Pass Requirement: Data must be processed in a single pass, as revisiting
past data is often not feasible.

1.3 Methodologies for Stream Data Processing

• Sampling:
– What is it?: Selecting a small, representative portion of the stream to analyze
instead of the whole dataset.
– Types:
∗ Random Sampling: Randomly picking data points (e.g., selecting 1% of
tweets to analyze).
∗ Reservoir Sampling: Maintaining a fixed-size sample that updates as new
data arrives.

Example: A news app samples 1 out of every 100 articles to identify

trending topics.

– Pros: Reduces processing load.

– Cons: May miss rare but important events.
• Load Shedding:
– What is it?: Intentionally dropping some data when the system is overloaded
to keep up with the stream.

Example: During a peak shopping event, an e-commerce platform

drops some low-priority data (e.g., user clicks) to focus on processing
payments.

– Pros: Prevents system crashes during high load.

– Cons: Loss of data can lead to incomplete analysis.
• Approximation:
– What is it?: Using approximate calculations instead of exact ones to save time
and resources.

Example: Estimating the average number of website visitors per

minute instead of calculating the exact number.

– Techniques:
∗ Histograms: Summarizing data into buckets (e.g., grouping visitor counts
into ranges).

2
 ∗ Sketches: Data structures like Count-Min Sketch to estimate frequencies.
– Pros: Fast and memory-efficient.
– Cons: Results are not 100% accurate.
• Sliding Window:
– What is it?: Focusing on a small, recent portion of the stream (e.g.,
the last 5 minutes or 1000 records).
– Types:
∗ Time-Based Window: Based on a time period (e.g., last 10 seconds).
∗ Count-Based Window: Based on a number of records (e.g., last 100 trans-
actions).

Example: A stock trading app analyzes the last 1 minute of trades to

detect price trends.

– Pros: Focuses on recent, relevant data.

– Cons: Ignores older data, which may still be useful.
• Synopsis Structures:
– What is it?: Creating compact summaries of the stream to save memory while
retaining key information.
– Examples:
∗ Bloom Filters: To check if an item exists in the stream (e.g., checking if
a user ID has been seen).
∗ HyperLogLog: To estimate the number of unique items (e.g., unique visi-
tors to a website).

Example: A streaming platform uses HyperLogLog to estimate unique

viewers of a live event.

– Pros: Highly memory-efficient.

– Cons: Provides approximate results, not exact.

1.4 Stream Data Systems

• What are they?: Specialized software frameworks designed to handle the
unique challenges of stream data processing.
• Key Features:
– Single-Pass Processing: Analyzes data as it arrives without revisiting.
– Low Latency: Processes data with minimal delay for real-time insights.
– Scalability: Can handle increasing data volumes by adding more resources.

3
 – Fault Tolerance: Recovers from failures without losing data (e.g., if a server
crashes).
• Popular Stream Data Systems:
– Apache Kafka: A platform for storing and distributing streams of data in real
time.

Example: Used by a messaging app to handle live chat messages.

– Apache Flink: A framework for processing stream data with low latency and
high throughput.

Example: Used by a financial firm to analyze stock trades in real

time.

– Apache Storm: A system for real-time computation on streams.

Example: Used for live traffic monitoring in a smart city.

– Spark Streaming: An extension of Apache Spark for processing streams in

micro-batches.

Example: Used by a video platform to analyze live viewer

engagement.

• How They Work:

– Ingestion: Collect data from sources (e.g., sensors, apps).
– Processing: Apply computations (e.g., filtering, aggregating).
– Output: Send results to dashboards, databases, or other systems.
• Applications:
– Real-time fraud detection in banking (e.g., spotting unusual transactions).
– Live monitoring of patient health data in hospitals.
– Real-time analytics for social media trends (e.g., trending hashtags on X).

1.5 Challenges in Stream Data Processing

• High Throughput: Systems must handle thousands or millions of records per
second.
• Memory Constraints: Limited memory requires efficient data structures like
sketches.
• Evolving Data: Patterns change over time, requiring adaptive algorithms (e.g., a
sudden spike in traffic during an event).

4
 • Fault Tolerance: Systems must handle failures (e.g., network issues) without
losing data.
• Latency: Delays in processing can make insights outdated (e.g., a fraud alert
arriving too late).

2 Frequent Pattern Mining in Stream Data

2.1 What is Frequent Pattern Mining in Stream Data?
• Definition: Identifying items, events, or patterns that appear frequently in a
stream, such as items often bought together or recurring events.

Example: In a stream of online purchases, noticing that customers often

buy a phone and a phone case together.

• Why Its Important: Helps uncover trends and associations in real time,
which can be used for recommendations or anomaly detection.

2.2 Challenges in Stream Data

• One-Pass Constraint: Data can only be processed once as it arrives.
• Memory Limitations: Cannot store the entire stream to count frequencies.
• Concept Drift: Frequent patterns change over time (e.g., seasonal shopping
trends).
• High Speed: Patterns must be updated quickly as new data arrives.

2.3 Methods for Frequent Pattern Mining in Streams

• Lossy Counting Algorithm:
– How It Works: Keeps track of items and their approximate counts, allowing
small errors to save memory. It periodically removes items with low counts.

Example: In a stream of retail transactions, tracking how often a

product is bought with an error margin of 1%.

– Pros: Memory-efficient and scalable.

– Cons: May miss some rare but important patterns.
• FP-Stream (Frequent Pattern Stream):
– How It Works: An extension of the FP-growth algorithm, which builds a tree
(FP-tree) to store frequent patterns and updates it as new data arrives.

Example: Building a tree of items bought together in an e-commerce

stream and updating it with each new transaction.

5
 – Pros: Captures both frequent and time-evolving patterns.
– Cons: Requires more memory than simpler methods.
• Sliding Window Approach:
– How It Works: Focuses on a recent portion of the stream (e.g., last 1,000
transactions) to find frequent patterns.

Example: Finding frequently purchased items in the last hour of sales

data.

– Pros: Prioritizes recent data, which is often more relevant.

– Cons: Ignores older patterns that might still be useful.
• Count-Min Sketch:
– What is it?: A probabilistic data structure that estimates item frequencies
using minimal memory.

Example: Estimating how often a hashtag appears in a stream of

tweets without storing every tweet.

– Pros: Very memory-efficient and fast.

– Cons: Provides approximate counts, not exact.
• Sticky Sampling:
– How It Works: Samples items with a probability that increases for frequent
items, ensuring they are tracked.

Example: Tracking popular search terms on a website by sampling

queries.

– Pros: Good for identifying frequent items with low memory usage.
– Cons: May miss items that become frequent later.

2.4 Applications
• E-commerce: Recommending products based on frequent purchases (e.g., sug-
gesting a phone case when someone buys a phone).
• Network Security: Detecting frequent patterns in network traffic to identify
attacks (e.g., repeated login attempts).
• Social Media: Identifying trending topics or hashtags in real time (e.g., #World-
Cup trends during a match).
• IoT: Monitoring frequent patterns in sensor data (e.g., frequent temperature spikes
in a factory).

6
2.5 Challenges
• Accuracy vs. Efficiency: Approximate methods may miss rare patterns or over-
estimate frequencies.
• Evolving Patterns: Patterns change over time, requiring constant updates (e.g.,
a product becoming popular during a sale).
• Scalability: Handling high-speed streams with millions of records per second.
• Noise: Irrelevant data (e.g., spam transactions) can distort frequent patterns.

3 Sequential Pattern Mining in Data Streams

3.1 What is Sequential Pattern Mining in Stream Data?
• Definition: Finding sequences of events that occur in a specific order in a stream,
such as a series of actions or events over time.

Example: In a stream of website clicks, noticing that users often visit the
homepage, then a product page, then the checkout page.

• Why Its Important: Helps understand and predict sequences of behavior,

which is useful for recommendations, forecasting, and anomaly detection.

3.2 Challenges in Stream Data

• Dynamic Nature: Sequences evolve over time (e.g., user navigation patterns
change during holidays).
• One-Pass Processing: Must identify sequences in a single pass without revisiting
past data.
• Memory Constraints: Cannot store all sequences, requiring efficient summaries
or approximations.
• High Velocity: Sequences must be detected quickly as data streams in.

3.3 Methods for Sequential Pattern Mining in Streams

• PrefixSpan for Streams:
– How It Works: An adaptation of the PrefixSpan algorithm, which finds se-
quential patterns by building prefix-based patterns and updating them as new
data arrives.

Example: In a stream of website clicks, tracking sequences like

"Homepage → Product → Checkout" and updating with new user
actions.

– Pros: Efficient for finding sequential patterns.

– Cons: Requires more memory than simpler methods.

7
 • Sliding Window Approach:
– How It Works: Focuses on a recent time window (e.g., last 10 minutes) to find
sequences.

Example: Analyzing the last 1,000 user actions on a website to find

common navigation sequences.

– Pros: Focuses on recent, relevant sequences.

– Cons: May miss long-term sequences outside the window.
• Approximate Sequential Mining:
– How It Works: Uses summaries or sketches to estimate sequences instead of
tracking them exactly.

Example: Estimating common sequences of purchases in a retail

stream without storing every transaction.

– Pros: Saves memory and processes data faster.

– Cons: May miss some sequences due to approximations.
• SPADE (Sequential PAttern Discovery using Equivalence classes):
– How It Works: An algorithm adapted for streams that finds frequent sequences
by dividing them into equivalence classes.

Example: Finding sequences of events in a stream of IoT sensor data

(e.g., temperature rise → pressure drop → alarm).

– Pros: Efficient for large datasets.

– Cons: Complex to implement in a streaming context.
• IncSpan (Incremental Sequential Pattern Mining):
– How It Works: Updates sequential patterns incrementally as new data
arrives in the stream.

Example: Updating sequences of user actions on a streaming platform

as new viewers join.

– Pros: Handles evolving patterns well.

– Cons: Requires careful tuning to avoid memory overload.

3.4 Applications
• Web Navigation: Predicting the next page a user will visit to improve website
design (e.g., suggesting products after a user views a category).

8
 • Stock Market: Analyzing sequences of trades to predict price movements (e.g.,
buy → sell → buy pattern).
• Healthcare: Tracking sequences of symptoms in real-time patient data to predict
outcomes (e.g., fever → cough → diagnosis).
• IoT Systems: Detecting sequences in sensor data (e.g., a sequence of events lead-
ing to a machine failure).

3.5 Challenges
• Speed: Sequences must be identified in real time as data arrives.
• Memory: Storing all possible sequences is impossible, so approximations are
needed.
• Concept Drift: Sequences change over time (e.g., user behavior shifts during a
sale).
• Noise: Irrelevant sequences (e.g., random clicks) can distort results.

4 Class Imbalance Problem

4.1 What is the Class Imbalance Problem?
• Definition: A problem in data mining where one category (class) has signifi-
cantly fewer instances than another, making it hard to predict the minority
class accurately.

Example: In a stream of 10,000 credit card transactions, 9,900 are normal,

and 100 are fraudulent. A model might focus on the majority (normal) and
miss the minority (fraud).

• Why Its a Problem: Models trained on imbalanced data often perform poorly
on the minority class, which is often the more important one (e.g., fraud, rare
diseases).

4.2 Why Its Challenging in Streams

• Real-Time Requirement: Decisions must be made quickly, but imbalance can
lead to missing critical events.
• Evolving Imbalance: The ratio of classes may change over time (e.g., fraud rates
increase during a holiday season).
• Limited Data Access: Cannot revisit past data to rebalance classes, as in static
datasets.

4.3 Methods to Address Class Imbalance in Streams

• Oversampling:

9
 – How It Works: Increases the number of minority class instances by duplicating
them or generating synthetic data.

Example: Duplicating fraud transactions in a stream to make them

more frequent for the model to learn.

– Pros: Improves model focus on the minority class.

– Cons: Can lead to overfitting (model over-learns the minority class).
• Undersampling:
– How It Works: Reduces the number of majority class instances to balance the
dataset.

Example: Ignoring some normal transactions to focus on fraud cases

in a stream.

– Pros: Simplifies the dataset for better balance.

– Cons: Loss of majority class data can reduce overall accuracy.
• SMOTE (Synthetic Minority Oversampling Technique):
– How It Works: Creates synthetic (fake) minority class instances by interpo-
lating between existing ones.

Example: Generating synthetic fraud transactions based on patterns

in real fraud cases.

– Pros: Adds variety to the minority class without simple duplication.

– Cons: Synthetic data may not always reflect real-world patterns.
• Cost-Sensitive Learning:
– How It Works: Assigns a higher cost to misclassifying the minority
class, making the model prioritize it.

Example: Making the model penalize missing a fraud case more than
missing a normal case.

– Pros: Focuses on the minority class without changing the data.

– Cons: Requires careful tuning of costs.
• Adaptive Methods:
– How It Works: Continuously adjusts the model as the stream evolves to handle
changing imbalances.

Example: Updating the model daily to account for new fraud patterns
in a transaction stream.

10
 – Pros: Adapts to evolving data distributions.
– Cons: Computationally expensive due to frequent updates.
• Ensemble Methods:
– How It Works: Combines multiple models to improve prediction on the mi-
nority class.

Example: Using a combination of models to detect fraud, where each

model focuses on different aspects of the data.

– Pros: Improves overall performance on imbalanced data.

– Cons: More complex and resource-intensive.

4.4 Applications
• Fraud Detection: Identifying rare fraudulent transactions in a stream of payments
(e.g., credit card fraud).
• Healthcare: Detecting rare medical events in patient data streams (e.g., heart
attacks in vital sign data).
• Network Security: Spotting rare cyber-attacks in a stream of network traffic
(e.g., DDoS attacks).
• Marketing: Identifying rare but high-value customer behaviors (e.g., big purchases
in a stream of sales).

4.5 Challenges
• Real-Time Processing: Balancing classes in a fast-moving stream is computa-
tionally intensive.
• Overfitting: Oversampling or SMOTE can cause the model to overfocus on the
minority class.
• Concept Drift: Imbalance patterns change over time, requiring constant adapta-
tion.
• Data Quality: Noise in the stream (e.g., incorrect labels) can worsen the imbalance
problem.

5 Graph Mining
5.1 What is Graph Mining in Stream Data?
• Definition: Analyzing stream data represented as graphs, where nodes (e.g.,
users, items) and edges (e.g., relationships, transactions) evolve over time.

Example: In a stream of social media interactions, nodes are users, and

edges are friendships, likes, or comments that change dynamically.

11
 • Why Its Important: Many real-world systems (e.g., social networks, financial
transactions) can be modeled as graphs, and mining them helps uncover evolving
relationships and patterns.

5.2 Challenges in Stream Data

• Dynamic Updates: Graphs change rapidly as new nodes and edges are added or
removed.
• Scalability: Large graphs with millions of nodes and edges are hard to process in
real time.
• Memory Constraints: Cannot store the entire graph history, requiring efficient
updates.

5.3 Methods for Graph Mining in Streams

• Dynamic Graph Updates:
– How It Works: Continuously updates the graph structure as new data arrives
in the stream.

Example: Adding a new user and their friendships to a social network

graph as they join.

– Pros: Keeps the graph up-to-date for real-time analysis.

– Cons: Computationally expensive for large graphs.
• Frequent Subgraph Mining:
– How It Works: Identifies smaller graphs (subgraphs) that appear frequently
in the stream.

Example: Finding a group of users who frequently interact in a

stream of social media messages.

– Techniques:
∗ gSpan Algorithm: Adapted for streams to find frequent subgraphs.
∗ Approximate Methods: Uses sketches to estimate frequent subgraphs.
– Pros: Uncovers recurring patterns in relationships.
– Cons: Approximate methods may miss some patterns.
• Graph Clustering:
– How It Works: Groups similar nodes in the graph as the stream evolves.

Example: Clustering users with similar interests based on their

interactions in a social media stream.

12
 – Techniques:
∗ Louvain Method: Adapted for streams to detect communities.
∗ Incremental Clustering: Updates clusters as new data arrives.
– Pros: Helps identify communities or groups in real time.
– Cons: Requires frequent updates as the graph changes.
• Anomaly Detection:
– How It Works: Spots unusual patterns in the graph that deviate from the
norm.

Example: Detecting a sudden spike in connections that might indicate

a bot attack in a social network.

– Techniques:
∗ Degree Analysis: Monitors nodes with unusual connection patterns.
∗ Subgraph Anomaly Detection: Looks for unexpected subgraphs.
– Pros: Useful for security and fraud detection.
– Cons: False positives due to noise in the stream.
• Graph Summarization:
– How It Works: Creates a smaller, summarized version of the graph to save
memory.

Example: Summarizing a large social network graph by grouping

similar users into clusters.

– Pros: Reduces memory usage while retaining key patterns.

– Cons: May lose fine-grained details.

5.4 Applications
• Social Media: Tracking evolving communities or influence in real time (e.g., iden-
tifying trending groups on X).
• Fraud Detection: Identifying suspicious patterns in a transaction graph (e.g., a
ring of accounts involved in money laundering).
• Network Monitoring: Analyzing network traffic to detect unusual behavior (e.g.,
a sudden spike in data transfers).
• Recommendation Systems: Suggesting connections or content based on graph
patterns (e.g., recommending friends on a social platform).

13
5.5 Challenges
• topia RegularDynamic Changes: Graphs change rapidly, making updates com-
plex and resource-intensive.
• Scalability: Large graphs require significant memory and processing power.
• Noise: Irrelevant or incorrect data in the stream can distort the graph (e.g., fake
accounts in a social network).
• Accuracy: Approximate methods may miss important patterns or relationships.

6 Social Network Analysis

6.1 What is Social Network Analysis in Stream Data?
• Definition: Studying social networks (e.g., X, Facebook) in a stream to understand
relationships, behaviors, and information flow as they evolve over time.

Example: Analyzing a stream of X posts to track how a hashtag spreads

during a global event like the Olympics.

• Why Its Important: Social networks generate massive real-time data, and an-
alyzing this data helps understand trends, influence, and community dynamics
dynamically.

6.2 Challenges in Stream Data

• Volume and Velocity: Social networks produce huge amounts of data at high
speed (e.g., millions of tweets per hour).
• Evolving Networks: Relationships change quickly (e.g., users follow or unfollow
others).
• Privacy Concerns: Mining social data raises ethical issues about user privacy.

6.3 Methods for Social Network Analysis in Streams

• Centrality Measures:
– What is it?: Identifies important users or nodes in the network as it evolves.
– Types:
∗ Degree Centrality: Measures the number of connections (e.g., users with
the most followers).
∗ Betweenness Centrality: Identifies users who connect different groups
(e.g., a user linking two communities).
∗ Closeness Centrality: Measures how quickly a user can reach others (e.g.,
how fast a post spreads).

14
 Example: Finding the most influential users on X during a live event
based on their follower count.

– Pros: Highlights key players in the network.

– Cons: Computationally expensive for large networks.
• Community Detection:
– What is it?: Groups users who interact frequently in the stream.
– Techniques:
∗ Louvain Method: Adapted for streams to detect communities.
∗ Incremental Community Detection: Updates communities as new inter-
actions occur.

Example: Identifying a group of friends who frequently comment on

each others posts on a social platform.

– Pros: Uncovers groups with shared interests or behaviors.

– Cons: Requires frequent updates as the network changes.
• Link Prediction:
– What is it?: Predicts future connections in the network based on current
patterns.
– Techniques:
∗ Common Neighbors: Predicts links between users with many mutual
friends.
∗ Preferential Attachment: Predicts links based on the popularity of nodes
(e.g., popular users attract more connections).

Example: Predicting who a user might follow next on X based on

their current connections.

– Pros: Useful for recommendations and network growth analysis.

– Cons: Predictions may be inaccurate due to noise or sparse data.
• Information Diffusion:
– What is it?: Studies how information (e.g., a viral post) spreads through the
network over time.
– Techniques:
∗ Cascade Models: Models how information spreads (e.g., Independent Cas-
cade Model).

15
 ∗ Diffusion Paths: Tracks the path of information (e.g., who retweeted a
post).

Example: Tracking how a hashtag spreads across X users during a

global event.

– Pros: Helps understand trends and influence dynamics.

– Cons: Hard to model accurately due to complex user behaviors.
• Sentiment Analysis in Networks:
– What is it?: Analyzes the sentiment (positive, negative) of interactions in the
stream.

Example: Detecting whether users are posting positive or negative

comments about a product launch on X.

– Pros: Provides insights into public opinion in real time.

– Cons: Sentiment analysis can be inaccurate due to sarcasm or context.

6.4 Applications
• Marketing: Identifying influencers in real time to promote products (e.g., finding
popular users to endorse a brand).
• Security: Detecting fake accounts or harmful groups in a stream of social interac-
tions (e.g., spotting bot networks).
• Trend Analysis: Spotting emerging trends or topics as they happen (e.g., a new
hashtag going viral).
• Public Health: Tracking the spread of misinformation about health issues (e.g.,
vaccine myths on social media).

6.5 Challenges
• Volume and Speed: Social network streams are massive and fast-moving, requir-
ing efficient processing.
• Evolving Networks: Relationships change quickly, making analysis complex (e.g.,
users unfollow others).
• Privacy: Mining social data raises ethical concerns about user consent and data
usage.
• Noise: Irrelevant or spam content (e.g., bot posts) can distort analysis.

7 Importance of Mining Data Streams

• Real-Time Decision-Making: Enables immediate actions, such as detecting
fraud or spotting trends as they happen.

16
 • Scalability for Big Data: Handles large, continuous data flows that traditional
methods cannot manage.
• Dynamic Applications: Supports industries like finance (e.g., stock trading),
healthcare (e.g., patient monitoring), and social media (e.g., trend analysis).
• Adaptability: Techniques like sliding windows and adaptive methods handle
evolving data patterns effectively.

8 Challenges in Mining Data Streams

• High Velocity: Data arrives too fast to process everything accurately, requiring
approximations.
• Limited Memory: Cannot store the entire stream, so summaries, sketches, or
windows are used.
• Concept Drift: Patterns change over time, requiring models to adapt (e.g., user
behavior shifts during a global event).
• Noise and Quality: Streams often contain irrelevant, incomplete, or incorrect
data (e.g., spam transactions or fake social media posts).
• Scalability: Systems must scale to handle millions of records per second without
crashing.
• Privacy and Ethics: Mining real-time data, especially social data, raises concerns
about user privacy and data security.

Conclusion
Unit 4 provides a thorough understanding of mining data streams, covering methodolo-
gies for processing and managing stream data, finding frequent and sequential patterns,
addressing class imbalance, and applying these techniques to graph mining and social
network analysis. Each topic is tailored to handle the unique challenges of streaming
data, such as high speed, limited memory, and evolving patterns. The 11-hour duration
ensures a deep dive into these concepts, preparing students for real-world applications
like fraud detection, trend analysis, network monitoring, and social media analytics. By
mastering these techniques, students can tackle the complexities of continuous, dynamic
data in modern systems effectively.

17