Association Rules and

Sequential Patterns
Road map

 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary

CS583, Bing Liu, UIC 2

Association rule mining
 Proposed by IBM researchers in 1993
 Agrawal et al, 1993.
 It is an important data mining model studied
extensively by the database and data mining
community. Hardly studied in ML.
 Assume all data are categorical.
 No good algorithm for numeric data.
 Initially used for Market Basket Analysis to find
how items purchased by customers are related.
Bread  Milk [sup = 5%, conf = 100%]
CS583, Bing Liu, UIC 3
The model: data

 I = {i1, i2, …, im}: a set of all items in the

system.
 Transaction t :
 t a set of items, and t  I.

 Transaction Database T: a set of transactions

T = {t1, t2, …, tn}.

CS583, Bing Liu, UIC 4

Transaction data: supermarket data
 Market basket transactions:
t1: {bread, cheese, milk}
t2: {apple, eggs, salt, yogurt}
… …
tn: {biscuit, eggs, milk}
 Concepts:
 An item: an item/article in a basket
 I: the set of all items sold in the store
 A transaction: items purchased in a basket; it may
have TID (transaction ID)
 A transactional dataset: A set of transactions
CS583, Bing Liu, UIC 5
Transaction data: a set of documents
 A text document data set. Each document
is treated as a “bag” of keywords
doc1: Student, Teach, School
doc2: Student, School
doc3: Teach, School, City, Game
doc4: Baseball, Basketball
doc5: Basketball, Player, Spectator
doc6: Baseball, Coach, Game, Team
doc7: Basketball, Team, City, Game

CS583, Bing Liu, UIC 6

The model: rules
 A transaction t contains X, a set of items
(itemset) in I, if X  t.
 An association rule is an implication of the
form:
X  Y, where X, Y  I, and X Y = 

 An itemset is a set of items.

 E.g., X = {milk, bread, cereal} is an itemset.
 A k-itemset is an itemset with k items.
 E.g., {milk, bread, cereal} is a 3-itemset

CS583, Bing Liu, UIC 7

Rule strength measures
 Support: The rule holds with support S in T (a
transaction data set) if S % of the
transactions in T contain X  Y.
 Confidence: The rule holds in T with
confidence C if C % of the transactions in T
that contain X also contain Y.

 An association rule is a pattern: stating when

X occurs, Y occurs with a certain probability.

CS583, Bing Liu, UIC 8

 Support and Confidence
 Support count: The support count of an
itemset X, denoted by X.count, in a data set
T is the number of transactions in T that
contain X. Assume T has n transactions.
 Then,
( X  Y ).count
support 
n
( X  Y ).count
confidence 
X .count
CS583, Bing Liu, UIC 9
Goal and key features
 Goal: Find all rules that satisfy the user-
specified minimum support (minsup) and
minimum confidence (minconf).
 Key Features
 Completeness: find all rules.
 No fixed target item(s) on the right-hand-side
 X  Y and Y  X
 Mining with data on hard disk (not in memory)

CS583, Bing Liu, UIC 10

An example t1:
t2:
Beef, Chicken, Milk
Beef, Cheese
t3: Cheese, Boots
t4: Beef, Chicken, Cheese
t5: Beef, Chicken, Clothes, Cheese,
 Transaction data Milk
t6: Chicken, Clothes, Milk
 Assume: t7: Chicken, Milk, Clothes
minsup = 30%
minconf = 80%
 An example frequent itemset (≥ minsup):
{Chicken, Clothes, Milk} [sup = 3/7]
 Association rules from the itemset:
Clothes  Milk, Chicken [sup = 3/7, conf = 3/3]
… …
Clothes, Chicken  Milk, [sup = 3/7, conf = 3/3]

CS583, Bing Liu, UIC 11

Transaction data representation
 A simplistic view of shopping baskets,
 Some important information not considered.
E.g,
 the quantity of each item purchased and
 the price paid.

CS583, Bing Liu, UIC 12

Many mining algorithms
 There are a large number of them!!
 They use different strategies and data structures.
 Their resulting sets of rules are all the same.
 Given a transaction data set T, and a minimum support
and a minimum confident, the set of association rules
existing in T is uniquely determined.
 Any algorithm should find the same set of rules
although their computational efficiencies and
memory requirements may be different.
 We study only one: the Apriori Algorithm

CS583, Bing Liu, UIC 13

Road map

 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary

CS583, Bing Liu, UIC 14

The Apriori algorithm
 The best known algorithm
 Two steps:
 Find all itemsets that have minimum support (called
frequent itemsets, large itemsets).
 Use frequent itemsets to generate rules.

 E.g., a frequent itemset

{Chicken, Clothes, Milk} [sup = 3/7]
and one rule from the frequent itemset
Clothes  Milk, Chicken [sup = 3/7, conf = 3/3]

CS583, Bing Liu, UIC 15

Step 1: Mining all frequent
itemsets
 Key idea: The apriori property (downward

closure property): any subset of a frequent

itemset is also a frequent itemset.
 Contraposition: If an itemset is not frequent, then
any of its superset cannot be frequent.
ABC ABD ACD BCD

AB AC AD BC BD CD

A B C D

CS583, Bing Liu, UIC 16

The Algorithm
 Iterative algo. (also called level-wise search):
Find all 1-item frequent itemsets; then all 2-item
frequent itemsets, and so on.
 In each iteration k, only consider itemsets that

contain some k-1 frequent itemset.

 Find frequent itemsets of size 1: F1
 From k = 2
 Ck = candidates of size k: those itemsets of size k
that could be frequent, given Fk-1
 Fk = those itemsets that are actually frequent, Fk
 Ck (need to scan the database once).
CS583, Bing Liu, UIC 17
 Dataset T
TID Items
Example – minsup=0.5 T100 1, 3, 4
Finding frequent itemsets T200 2, 3, 5
T300 1, 2, 3, 5
T400 2, 5
itemset:count
1. scan T  C1: {1}:2, {2}:3, {3}:3, {4}:1, {5}:3
 F1: {1}:2, {2}:3, {3}:3, {5}:3

 C2: {1,2}, {1,3}, {1,5}, {2,3}, {2,5}, {3,5}

2. scan T  C2: {1,2}:1, {1,3}:2, {1,5}:1, {2,3}:2, {2,5}:3, {3,5}:2

 F2: {1,3}:2, {2,3}:2, {2,5}:3, {3,5}:2

 C3: {2, 3,5}

3. scan T  C3: {2, 3, 5}:2  F3: {2, 3, 5}

CS583, Bing Liu, UIC 18

Details: ordering of items

 The items in I are sorted in lexicographic

order (which is a total order).
 The order is used throughout the algorithm in
each itemset.
 {w[1], w[2], …, w[k]} represents a k-itemset w
consisting of items w[1], w[2], …, w[k], where
w[1] < w[2] < … < w[k] according to the total
order.

CS583, Bing Liu, UIC 19

Details: the algorithm
Algorithm Apriori(T)
C1  init-pass(T);
F1  {f | f  C1, f.count/n  minsup}; // n: no. of transactions in T
for (k = 2; Fk-1  ; k++) do
Ck  candidate-gen(Fk-1);
for each transaction t  T do
for each candidate c  Ck do
if c is contained in t then
c.count++;
end
end
Fk  {c  Ck | c.count/n  minsup}
end
return F  k Fk;

CS583, Bing Liu, UIC 20

Apriori candidate generation

 The candidate-gen function takes Fk-1 and

returns a superset (called the candidates
Ck) of the set of all frequent k-itemsets. It
has two steps
 join step: Generate all possible candidate
itemsets Ck of length k
 prune step: Remove those candidates in Ck
that cannot be frequent.

CS583, Bing Liu, UIC 21

Candidate-gen function
Function candidate-gen(Fk-1)
Ck  ;
forall f1, f2  Fk-1
with f1 = {i1, … , ik-2, ik-1}
and f2 = {i1, … , ik-2, i’k-1}
and ik-1 < i’k-1 do
c  {i1, …, ik-1, i’k-1}; // join f1 and f2
Ck  Ck  {c};
for each (k-1)-subset s of c do
if (s  Fk-1) then
delete c from Ck; // prune
end
end
return Ck;

CS583, Bing Liu, UIC 22

An example
 F3 = {{1, 2, 3}, {1, 2, 4}, {1, 3, 4},
{1, 3, 5}, {2, 3, 4}}

 After join
 C4 = {{1, 2, 3, 4}, {1, 3, 4, 5}}
 After pruning:
 C4 = {{1, 2, 3, 4}}
because {1, 4, 5} is not in F3 ({1, 3, 4, 5} is removed)

CS583, Bing Liu, UIC 23

Step 2: Generating rules from
frequent itemsets
 Frequent itemsets  association rules
 One more step is needed to generate
association rules
 For each frequent itemset X,
For each proper nonempty subset A of X,
 Let B = X - A
 A  B is an association rule if
 Confidence(A  B) ≥ minconf,

support(A  B) = support(AB) = support(X)

confidence(A  B) = support(A  B) / support(A)
CS583, Bing Liu, UIC 24
Generating rules: an example
 Suppose {2,3,4} is frequent, with sup=50%
 Proper nonempty subsets: {2,3}, {2,4}, {3,4}, {2}, {3}, {4}, with
sup=50%, 50%, 75%, 75%, 75%, 75% respectively
 These generate these association rules:
 2,3  4, confidence=100%
 2,4  3, confidence=100%
 3,4  2, confidence=67%
 2  3,4, confidence=67%
 3  2,4, confidence=67%
 4  2,3, confidence=67%
 All rules have support = 50%

CS583, Bing Liu, UIC 25

Generating rules: summary

 To recap, in order to obtain A  B, we need

to have support(A  B) and support(A)
 All the required information for confidence
computation has already been recorded in
itemset generation. No need to see the data
T any more.
 This step is not as time-consuming as
frequent itemsets generation.

CS583, Bing Liu, UIC 26

On Apriori Algorithm

Seems to be very expensive

 Level-wise search
 K = the size of the largest itemset
 It makes at most K passes over data
 In practice, K is bounded (10).
 The algorithm is very fast. Under some
conditions, all rules can be found in linear time.
 Scale up to large data sets

CS583, Bing Liu, UIC 27

More on association rule mining
 Clearly the space of all association rules is
exponential, O(2m), where m is the number of
items in I.
 The mining exploits sparseness of data, and
high minimum support and high minimum
confidence values.
 Still, it always produces a huge number of
rules, thousands, tens of thousands, millions,
and more

CS583, Bing Liu, UIC 28

Road map

 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary

CS583, Bing Liu, UIC 29

Different data formats for mining
 The data can be in transaction form or tabular
form
Transaction form: a, b
a, c, d, e
a, d, f
Table form: Attr1 Attr2 Attr3
a, b, d
b, c, e
 Tabular data need to be converted to the
transaction form for association mining

CS583, Bing Liu, UIC 30

From a table to a set of transactions
Tabular form: Attr1 Attr2 Attr3
a, b, d
b, c, e

 Transaction form:
(Attr1, a), (Attr2, b), (Attr3, d)
(Attr1, b), (Attr2, c), (Attr3, e)

candidate-gen can be slightly improved. Why?

CS583, Bing Liu, UIC 31

Road map

 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary

CS583, Bing Liu, UIC 32

Problems with the association mining
 Single minsup: It assumes that all items in
the data are of the same nature and/or
have similar frequencies.
 Not true: In many applications, some items
appear very frequently in the data, while
others rarely appear.
E.g., in a supermarket, people buy food processor
and cooking pan much less frequently than they
buy bread and milk.

CS583, Bing Liu, UIC 33

 Rare Item Problem
 If the frequencies of items vary a great deal,
we will encounter two issues
 If minsup is set too high, those rules that involve
rare items will not be found.
 To find rules that involve both frequent and rare
items, minsup has to be set very low. This may
cause combinatorial explosion because those
frequent items will be associated with one another
in all possible ways.

CS583, Bing Liu, UIC 34

Multiple minsups model (1)
 We use 2 mechanisms to deal with the problem.
 (1) We assign each item i a minimum item support
MIS(i).
 We define the minsup of a rule R as the lowest MIS
value of the items in the rule.
 I.e., rule R: a1, a2, …, ak  ak+1, …, ar satisfies its
minimum support if its actual support is 
min(MIS(a1), MIS(a2), …, MIS(ar)).
 This prevents frequent items to be associated with
one another in all possible ways.
CS583, Bing Liu, UIC 35
Multiple minsups model (2)
 However, the first mechanism does not prevent
those meaningless associations involving rare
items and very frequent items, e.g.,
 TV -> bread
 To prevent such rules or itemsets from being
generated, we introduce the support
difference constraint (SDC).
 Let S be an itemset. We have SDC
maxis{sup(i)}  minjs{sup(j)} ≤ 
where sup(i) is the actual support of item i in dataset T.

CS583, Bing Liu, UIC 36

 An Example

 Consider the following items:

bread, shoes, clothes
The user-specified MIS values are as follows:
MIS(bread) = 2% MIS(shoes) = 0.1%
MIS(clothes) = 0.2%
The following rule doesn’t satisfy its minsup:
clothes  bread [sup=0.15%,conf =70%]
The following rule satisfies its minsup:
clothes  shoes [sup=0.15%,conf =70%]

CS583, Bing Liu, UIC 37

 Mining with multiple minimum
supps
 In the new model, the downward closure (DC)
property no longer holds:
 DC: If an itemset is not frequent, then any of its
superset cannot be frequent.
 E.g., Consider four items 1, 2, 3 and 4 in a
dataset. Their minimum item supports are
MIS(1) = 10% MIS(2) = 20%
MIS(3) = 5% MIS(4) = 6%
{1, 2} with actual support 9% is infrequent, but {1,
2, 3} and {1, 2, 4} could be frequent.

CS583, Bing Liu, UIC 38

To deal with the problem
 We can deal with the problem fairly easily.
 By sorting all items in I according to their MIS
values (make it a total order).
 The order is used in each itemset throughout the
mining algorithm.
 I.e., each itemset w is of the following form:
{w[1], w[2], …, w[k]},
where MIS(w[1])  MIS(w[2])  …  MIS(w[k]).

CS583, Bing Liu, UIC 39

The MSapriori algorithm
Algorithm MSapriori(T, MS, ) //  is for support difference constraint
M  sort(I, MS);
L  init-pass(M, T);
F1  {{i} | i  L, i.count/n  MIS(i)};
for (k = 2; Fk-1  ; k++) do
if k=2 then
Ck  level2-candidate-gen(L, )
else Ck  MScandidate-gen(Fk-1, );
end;
for each transaction t  T do
for each candidate c  Ck do
if c is contained in t then
c.count++;
if c – {c[1]} is contained in t then
c.tailCount++
end
end
Fk  {c  Ck | c.count/n  MIS(c[1])}
end
return F  kFk;

CS583, Bing Liu, UIC 40

Candidate itemset generation
 Special treatments needed:
 Sorting the items according to their MIS values
 First pass over data (the first three lines)
 Let us look at this in detail.
 Candidate generation at level-2
 Read it in the book.
 Pruning step in level-k (k > 2) candidate
generation.
 Read it in the book.

CS583, Bing Liu, UIC 41

First pass over data
 It makes a pass over the data to record the
support count of each item.
 It then follows the sorted order to find the first
item i in M that meets MIS(i).
 i is inserted into L.
 For each subsequent item j in M after i, if
j.count/n  MIS(i) then j is also inserted into L,
where j.count is the support count of j and n is
the total number of transactions in T. Why?
 L is used by function level2-candidate-gen
CS583, Bing Liu, UIC 42
First pass over data: an example
 Consider the four items 1, 2, 3 and 4 in a data set.
Their minimum item supports are (M = {3, 4, 1, 2}):
MIS(1) = 10% MIS(2) = 20%
MIS(3) = 5% MIS(4) = 6%
 Assume our data set has 100 transactions. The first
pass gives us the following support counts:
{3}.count = 6, {4}.count = 3,
{1}.count = 9, {2}.count = 25.
 Then L = {3, 1, 2}, and F1 = {{3}, {2}}
 Item 4 is not in L because 4.count/n < MIS(3) (= 5%),
 {1} is not in F1 because 1.count/n < MIS(1) (= 10%).

CS583, Bing Liu, UIC 43

Rule generation
 The following two lines in MSapriori algorithm
are important for rule generation, which are
not needed for the Apriori algorithm
if c – {c[1]} is contained in t then
c.tailCount++
 E.g., c = {3, 4, 2), c.tail = {4, 2}
 Many rules cannot be generated without
them.
 Why?
CS583, Bing Liu, UIC 44
On multiple minsup rule mining
 Multiple minsup model subsumes the single
support model.
 It is a more realistic model for practical
applications.
 The model enables us to find rare item rules yet
without producing a huge number of
meaningless rules with frequent items.
 By setting MIS values of some items to 100% (or
more), we effectively instruct the algorithms not
to generate rules only involving these items.

CS583, Bing Liu, UIC 45

Road map

 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary

CS583, Bing Liu, UIC 46

Mining class association rules (CAR)
 Normal association rule mining does not have
fixed targets.
 It finds all possible rules that exist in data, i.e., any
item can appear as a consequent or a condition of
a rule. E.g., X -> Y and Y-> X
 However, in some applications, the user is
interested in some targets.
 E.g, the user has a set of text documents from
some known topics. He/she wants to find out what
words are associated or correlated with each topic.

CS583, Bing Liu, UIC 47

Problem definition
 Let T be a transaction data set consisting of n
transactions.
 Each transaction is also labeled with a class y.
 Let I be the set of all items in T, Y be the set of all
class labels (targets) and I  Y = .
 A class association rule (CAR) is an implication of
the form
X  y, where X  I, and y  Y.
 The definitions of support and confidence are the
same as those for normal association rules.

CS583, Bing Liu, UIC 48

An example
 A text document data set
doc 1: Student, Teach, School : Education
doc 2: Student, School : Education
doc 3: Teach, School, City, Game : Education
doc 4: Baseball, Basketball : Sport
doc 5: Basketball, Player, Spectator : Sport
doc 6: Baseball, Coach, Game, Team : Sport
doc 7: Basketball, Team, City, Game : Sport

 Let minsup = 20% and minconf = 60%. The following are

two examples of class association rules:
Student, School  Education [sup= 2/7, conf = 2/2]
game  Sport [sup= 2/7, conf = 2/3]

CS583, Bing Liu, UIC 49

Mining algorithm
 Unlike normal association rules, CARs can be
mined directly in one step.
 The key operation is to find all ruleitems that have
support above minsup. A ruleitem is of the form:
(condset, y)
where condset is a set of items from I (i.e., condset
 I), and y  Y is a class label.
 Each ruleitem basically represents a rule:
condset  y,
 Apriori algorithm can be modified to generate CARs

CS583, Bing Liu, UIC 50

Multiple minimum class supports
 The multiple minimum support idea can also be
applied here.
 The user can specify different minimum supports to
different classes, which effectively assign a different
minimum support to rules of each class.
 E.g., we have a data set with two classes, Yes and No. We
may want
 rules of class Yes to have the minimum support of 5% and
 rules of class No to have the minimum support of 10%.
 By setting minimum class supports to 100% (or
more for some classes), we tell the algorithm not to
generate rules of those classes.
 This is a very useful trick in applications.

CS583, Bing Liu, UIC 51

Road map

 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary

CS583, Bing Liu, UIC 52

Sequential pattern mining

 Association rule mining does not consider the

order of transactions.
 In many applications such orderings are
significant. E.g.,
 in market basket analysis, it is interesting to know
whether people buy some items in sequence,
 e.g., buying bed first and then bed sheets later.
 In Web usage mining, it is useful to find navigation
patterns of users within a Web site from sequences
of page visits of users.
CS583, Bing Liu, UIC 53
Basic concepts
 Let I = {i1, i2, …, im} be a set of items.
 Sequence: An ordered list of itemsets.
 Itemset/element: A non-empty set of items X  I.
We denote a sequence s by a1a2…ar, where ai is
an itemset, also called an element of s.
 An element (or an itemset) of a sequence is denoted by
{x1, x2, …, xk}, where xj  I is an item.
 We assume without loss of generality that items in
an element of a sequence are in lexicographic
order.
CS583, Bing Liu, UIC 54
Basic concepts (contd)
 Size: The size of a sequence is the number of
elements (or itemsets) in the sequence.
 Length: The length of a sequence is the number of
items in the sequence.
 A sequence of length k is called k-sequence.
 A sequence s1 = a1a2…ar is a subsequence of
another sequence s2 = b1b2…bv, or s2 is a
supersequence of s1, if there exist integers 1 ≤ j1 <
j2 < … < jr1 < jr  v such that a1  bj1, a2  bj2, …, ar 
bjr. We also say that s2 contains s1.

CS583, Bing Liu, UIC 55

An example

 Let I = {1, 2, 3, 4, 5, 6, 7, 8, 9}.

 Sequence {3}{4, 5}{8} is contained in (or is a
subsequence of) {6} {3, 7}{9}{4, 5, 8}{3, 8}
 because {3}  {3, 7}, {4, 5}  {4, 5, 8}, and {8}  {3,
8}.
 However, {3}{8} is not contained in {3, 8} or vice
versa.
 The size of the sequence {3}{4, 5}{8} is 3, and the
length of the sequence is 4.

CS583, Bing Liu, UIC 56

Goal
 Given a set S of input data sequences (or
sequence database), the problem of mining
sequential patterns is to find all the
sequences that have a user-specified
minimum support.
 Each such sequence is called a frequent
sequence, or a sequential pattern.
 The support for a sequence is the fraction of
total data sequences in S that contains this
sequence.

CS583, Bing Liu, UIC 57

Example

CS583, Bing Liu, UIC 58

Example (cond)

CS583, Bing Liu, UIC 59

GSP mining algorithm
 Very similar to the Apriori algorithm

CS583, Bing Liu, UIC 60

Candidate generation

CS583, Bing Liu, UIC 61

An example

CS583, Bing Liu, UIC 62

Road map

 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary

CS583, Bing Liu, UIC 63

 Summary
 Association rule mining and sequential pattern mining
has been extensively studied in data mining.
 Many efficient algorithms and model variations exist.
 Other related work includes
 Multi-level or generalized rule mining
 Constrained rule mining
 Incremental rule mining
 Maximal frequent itemset mining
 Closed itemset mining
 Rule interestingness and visualization
 …

CS583, Bing Liu, UIC 64