Association

Rules
 What Is Frequent Pattern
Analysis?
• Frequent pattern: a pattern (a set of items, subsequences, substructures, etc.)
that occurs frequently in a data set
• First proposed by Agrawal, Imielinski, and Swami [AIS93] in the context of
frequent itemsets and association rule mining
• Motivation: Finding inherent regularities in data
• What products were often purchased together?— Coffee and diapers?!
• What are the subsequent purchases after buying a PC?
• What kinds of DNA are sensitive to this new drug?
• Can we automatically classify web documents?
• Applications
• Basket data analysis, cross-marketing, catalog design, sale campaign analysis,
Web log (click stream) analysis, and DNA sequence analysis.
2
 Why Is Freq. Pattern Mining
Important?
• Freq. pattern: An intrinsic and important property of datasets
• Foundation for many essential data mining tasks
• Association, correlation, and causality analysis
• Sequential, structural (e.g., sub-graph) patterns
• Pattern analysis in spatiotemporal, multimedia, time-series,
and stream data
• Classification: discriminative, frequent pattern analysis
• Cluster analysis: frequent pattern-based clustering
• Data warehousing: iceberg cube and cube-gradient
• Semantic data compression: fascicles
• Broad applications

July 8, 2025
3
 Association Rule Mining
• A very popular DM method in business
• Finds interesting relationships (affinities) between
variables (items or events)
• Part of machine learning family
• Employs unsupervised learning
• There is no output variable
• Also known as market basket analysis
• Often used as an example to describe DM to ordinary
people, such as the famous “relationship between
diapers and beers!”
Slide 4-4
 Association Rule Mining
• Input: the simple point-of-sale transaction data
• Output: Most frequent affinities among items
• Example: according to the transaction data…
“Customer who bought a lap-top computer and a virus
protection software, also bought extended service plan
70 percent of the time."
• How do you use such a pattern/knowledge?
• Put the items next to each other
• Promote the items as a package
• Place items far apart from each other!

Slide 4-5
 Association Rule Mining
• A representative application of association rule mining
includes
• In business: cross-marketing, cross-selling, store design,
catalog design, e-commerce site design, optimization of online
advertising, product pricing, and sales/promotion
configuration

• In medicine: relationships between symptoms and illnesses;

diagnosis and patient characteristics and treatments (to be
used in medical DSS); and genes and their functions (to be
used in genomics projects)
•…
Slide 4-6
 Association Rule Mining
• Are all association rules interesting and useful?
A Generic Rule: X  Y [S%, C%]
X, Y: products and/or services
X: Left-hand-side (LHS)
Y: Right-hand-side (RHS)
S: Support: how often X and Y go together
C: Confidence: how often Y go together with the X
Example: {Laptop Computer, Antivirus Software} 
{Extended Service Plan} [30%, 70%]
Slide 4-7
 Association Rule Mining

• Several algorithms are developed for discovering

(identifying) association rules
• Apriori
• Eclat
• FP-Growth
• + Derivatives and hybrids of the three
• The algorithms help identify the frequent itemsets,
which are then converted to association rules

Slide 4-8
Association Rule Mining

• Apriori Algorithm
• Finds subsets that are common to at least a
minimum number of the itemsets
• Uses a bottom-up approach
• frequent subsets are extended one item at a time
(the size of frequent subsets increases from one-
item subsets to two-item subsets, then three-item
subsets, and so on), and
• groups of candidates at each level are tested
against the data for minimum support
(see the figure)  --

Slide 4-9
 Association Rule Mining
Apriori Algorithm
• FIGURE 4.13 A Graphical Illustration of the Steps in the k-Means
Algorithm

Raw Transaction Data One-item Itemsets Two-item Itemsets Three-item Itemsets

Transaction SKUs Itemset Itemset Itemset

Support Support Support
No (Item No) (SKUs) (SKUs) (SKUs)

1001234 1, 2, 3, 4 1 3 1, 2 3 1, 2, 4 3
1001235 2, 3, 4 2 6 1, 3 2 2, 3, 4 3
1001236 2, 3 3 4 1, 4 3
1001237 1, 2, 4 4 5 2, 3 4
1001238 1, 2, 3, 4 2, 4 5
1001239 2, 4 3, 4 3

Slide 4-10
Basic Concepts: Frequent
Patterns
Tid Items bought • itemset: A set of one or more items
10 Coffee, Nuts, Diaper
• k-itemset X = {x1, …, xk}
20 Coffee, Coffee, Diaper
30 Coffee, Diaper, Eggs • (absolute) support, or, support count
40 Nuts, Eggs, Milk of X: Frequency or occurrence of an
50 Nuts, Coffee, Diaper, Eggs, itemset X
Milk
• (relative) support, s, is the fraction of
Customer Customer transactions that contains X (i.e., the
buys both buys diaper probability that a transaction
contains X)
• An itemset X is frequent if X’s
support is no less than a minsup
Customer
threshold
buys Coffee
July 8, 2025
11
 Association Rule Mining
• Given a set of transactions, find rules that will predict the
occurrence of an item based on the occurrences of other items
• Equivalent representation

Market-Basket transactions
Example of Association
Rules
{Diaper}  {coffee},
{Milk, Bread}  {Eggs,Coke},
{coffee, Bread}  {Milk},

July 8, 2025
12
 The model: data

•I = {i1, i2, …, im}: a set of items.

•Transaction t :
• t a set of items, and t  I.
•Transaction Database T: a set of
transactions T = {t1, t2, …, tn}.

13
 Transaction data:
supermarket data
• Market basket transactions:
t1: {bread, cheese, milk}
t2: {apple, eggs, salt, yogurt}
… …
tn: {biscuit, eggs, milk}
• Concepts:
• An item: an item/article in a basket
• I: the set of all items sold in the store
• A transaction: items purchased in a basket; it may
have TID (transaction ID)
• A transactional dataset: A set of transactions

14
Formal Model

•I = i1, i2, …, im: set of literals (items)

•D : database of transactions
•T  D : a transaction. T  I
• TID: unique identifier, associated with each T
•X: a subset of I
• T contains X if X  T.
Formal Model (Cont.)
• Association rule: X  Y
here X  I, Y  I and X Y = .
• Rule X  Y has a confidence c in D
if c% of transactions in D that contain X also
contain Y.
• Rule X  Y has a support s in D
if s% of transactions in D contain X  Y.
Example

• I: itemset
{cucumber, parsley, onion, tomato, salt, bread, olives,
cheese, butter}
• D: set of transactions
1 {{cucumber, parsley, onion, tomato, salt, bread},
2 {tomato, cucumber, parsley},
3 {tomato, cucumber, olives, onion, parsley},
4 {tomato, cucumber, onion, bread},
5 {tomato, salt, onion},
6 {bread, cheese}
7 {tomato, cheese, cucumber}
8 {bread, butter}}
Support and Confidence
• Support count: The support count of an itemset X,
denoted by X.count, in a data set T is the number of
transactions in T that contain X. Assume T has n
transactions.
• Then,
( X  Y ).count
support 
n
( X  Y ).count
confidence 
X .count

18
Transaction data: A set of
documents
• A text document data set. Each document is treated as a
“bag” of keywords
doc1: Student, Teach, School
doc2: Student, School
doc3: Teach, School, City, Game
doc4: Baseball, Basketball
doc5: Basketball, Player, Spectator
doc6: Baseball, Coach, Game, Team
doc7: Basketball, Team, City, Game

• PubMed Experiment

July 8, 2025
19
Definition: Frequent Itemset
• Itemset
• A collection of one or more items from the
item space
• Example: {Milk, Bread, Diaper}
• k-itemset
• An itemset that contains k items
• Support count ()
• Frequency of occurrence of an itemset
• E.g. ({Milk, Bread,Diaper}) = 2
• Support
• Fraction of transactions that contain an
itemset
• E.g. s({Milk, Bread, Diaper}) = 2/5
• Frequent Itemset
• An itemset whose support is greater than or
equal to a user-specified minsup threshold

July 8, 2025
20
 Definition: Association Rule
 Association Rule
 An implication of the form X 

Y, where X and Y are itemsets


Antecedent  Consequent

E.g. {Milk, Diaper}  {coffee}
 Rule Evaluation Metrics
 Support (s)


Fraction of transactions that
contain both X and Y … s(XUY)
Example:

Statistical significance
{Milk, Diaper}  Coffee
 Confidence (c)

Measures how often items in Y  (Milk , Diaper, Coffer ) 2
s  0.4
appear in transactions |T| 5
containing X

Statistical strength  (Milk, Diaper, Coffee) 2
c  0.67
 (Milk, Diaper ) 3
July 8, 2025
21
Association Rule Mining Task

• Given a set of transactions T, the goal of association rule

mining is to find all rules having
• support ≥ minsup threshold
• confidence ≥ minconf threshold

• Brute-force approach:
• List all possible association rules
• Compute the support and confidence for each rule
• Prune rules that fail the minsup and minconf thresholds
 Computationally prohibitive!

July 8, 2025
22
 Example of Rules:

•{Milk,Diaper}  {coffee}
Observations: •(s=0.4, c=0.67)
•All the above rules are binary •{Milk,coffee}  {Diaper}
partitions of •(s=0.4, c=1.0)
the same itemset: {Milk, Diaper, coffee} •{Diaper,coffee}  {Milk}
•(s=0.4, c=0.67)
•Rules originating from the same •{coffee}  {Milk,Diaper}
itemset have identical support but can •(s=0.4, c=0.67)
have different confidence •{Diaper}  {Milk,coffee}
•If Itemset is not frequent  don’t •(s=0.4, c=0.5)
bother with the rules •{Milk}  {Diaper,coffee}
•(s=0.4, c=0.5)
July 8, 2025
23
Mining Association Rules

• Two-step approach:
• Frequent Itemset Generation
• Generate all itemsets whose support  minsup
• Rule Generation
• Generate high confidence rules from each frequent itemset, where each rule is a
binary partitioning of a frequent itemset such that its confidence  minconf

• Frequent itemset generation is still computationally expensive

July 8, 2025
24
 Frequent Itemset Generation
null

A B C D E

AB AC AD AE BC BD BE CD CE DE

ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

ABCD ABCE ABDE ACDE BCDE Given d items, there

are 2d possible
candidate itemsets
ABCDE
July 8, 2025
25
 Frequent Itemset Generation
• Brute-force approach:
• Each itemset in the lattice is a candidate frequent itemset
• Count the support of each candidate by scanning the
database

• Complexity ~ O(NMw) => Expensive since M = 2d !!!

July 8, 2025
26
Reducing Candidates

• Apriori principle:
• If an itemset is frequent, then all of its subsets must also be
frequent

• Apriori principle holds due to the following property of the

support measure:

X , Y : ( X  Y )  s( X ) s(Y )
• Support of an itemset never exceeds the support of its subsets

• This is known as the anti-monotone or downward-closure

property of support

July 8, 2025
27
 Illustrating Apriori Principle
null

A B C D E

AB AC AD AE BC BD BE CD CE DE

Found to be
Infrequent
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

Going down pays off

ABCD ABCE ABDE ACDE BCDE

Pruned
ABCDE
supersets
July 8, 2025
28
 Apriori Algorithm
• Method:
• Let k=1
• Generate frequent itemsets of length 1
• Repeat until no new frequent itemsets are identified
• Generate length (k+1) candidate itemsets from length k
frequent itemsets
• Prune candidate itemsets containing subsets of length k
that are infrequent
• Count the support of each candidate by scanning the DB
• Eliminate candidates that are infrequent, leaving only
those that are frequent
July 8, 2025 - 29
 TID Items
Minimum Support = 3
1 Bread, Milk
Illustrating Apriori Principle
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

Triplets (3-itemsets)
Itemset Count
{Bread,Milk,Diaper} 3
(No need to generate candidates
involving {Bread, coffee} or {Milk,
coffee})
July 8, 2025
30
The Apriori Algorithm
•Pseudo-code:
Ck: Candidate itemset of size k
Lk : frequent itemset of size k

L1 = {frequent items};
for (k = 1; Lk !=; k++) do begin
Ck+1 = candidates generated from Lk;
for each transaction t in database do
increment the count of all candidates in Ck+1
that are contained in t
Lk+1 = candidates in Ck+1 with min_support
end
return k Lk;

July 8, 2025 - 31
 Important Details of Apriori
• How to generate candidates?
• Step 1: self-joining Lk
• Join any two itemsets from Lk if they share the same (k-1) prefix (i.e. differ by last item
only)
• Step 2: pruning (omitted in most implementations)
• Prune any itemset from Ck+1 if any of its k-itemset subsets is not in Lk
• Example of Candidate-generation
• L3={abc, abd, acd, ace, bcd}
• Self-joining: L3*L3
• abcd from abc and abd
• acde from acd and ace
• Pruning:
• acde is removed because ade is not in L3
• CJuly
4={abcd}
8, 2025 - 32
 The Apriori Algorithm—An
Example
Supmin = 2 Itemset sup
Itemset sup
Database TDB {A} 2
L1 {A} 2
Tid Items C1 {B} 3
{B} 3
10 A, C, D {C} 3
1st scan {C} 3
20 B, C, E {D} 1
{E} 3
30 A, B, C, E {E} 3
40 B, E
C2 Itemset sup C2 Itemset
{A, B} 1
L2 Itemset sup
{A, C} 2
2nd scan {A, B}
{A, C} 2 {A, C}
{A, E} 1
{B, C} 2 {A, E}
{B, C} 2
{B, E} 3
{B, E} 3 {B, C}
{C, E} 2
{C, E} 2 {B, E}
{C, E}

C3 Itemset
3rd scan L3 Itemset sup
{B, C, E} {B, C, E} 2
July 8, 2025 - 33
Factors Affecting Complexity
• Dimensionality (number of items) of the data set
• more space is needed to store support count of each item
• if number of frequent items also increases, both computation and I/O costs may
also increase
• Size of database
• since Apriori makes multiple passes, runtime of algorithm may increase with
number of transactions
• How many passes?
• Average transaction width
• transaction width increases with denser data sets
• This may increase max length of frequent itemsets and traversals as number of
subsets in a transaction increases with its width
• Choice of minimum support threshold
• lowering support threshold results in more frequent itemsets
• this may increase number of candidates and max length of frequent itemsets

July 8, 2025 - 34
 Rule Generation

• Given a frequent itemset L, find all non-empty subsets A  L

such that A  C = L – A satisfies the minimum confidence
requirement
• If {A,B,C,D} is a frequent itemset, candidate rules (how
many)?:
ABC D, ABD C, ACD B, BCD A
AB CD, AC  BD, AD  BC, BC AD,
BD AC, CD AB
A BCD, B ACD, C ABD, D ABC

July 8, 2025 - 35
Rule Generation

• How to efficiently generate rules from frequent

itemsets?
• In general, confidence does not have an anti-
monotone property
c(ABC D) can be larger or smaller than c(AB D)

• But confidence of rules generated from the same

itemset has an anti-monotone property
e.g., L = {A,B,C,D}:
c(ABC  D)  c(AB  CD)  c(A  BCD)
•Confidence is anti-monotone w.r.t. number of items on
the RHS of the rule
July 8, 2025 - 36
 Rule Generation for Apriori
Algorithm
ABCD=>{ } Lattice of rules

Low Confidence
Rule BCD=>A ACD=>B ABD=>C ABC=>D

CD=>AB BD=>AC BC=>AD AD=>BC AC=>BD AB=>CD

D=>ABC C=>ABD B=>ACD A=>BCD

Pruned
Rules

July 8, 2025 - 37
 Dataset T
Example – Finding frequent
itemsets TID Items
T100 1, 3, 4
T200 2, 3, 5

itemset: count T300 1, 2, 3, 5

T400 2, 5
1. scan T  C1: {1}:2, {2}:3, {3}:3, {4}:1, {5}:3
 F1: {1}:2, {2}:3, {3}:3, {5}:3
minsup=0.5

 C2: {1,2}, {1,3}, {1,5}, {2,3}, {2,5}, {3,5}

2. scan T  C2: {1,2}:1, {1,3}:2, {1,5}:1, {2,3}:2, {2,5}:3, {3,5}:2

 F2: {1,3}:2, {2,3}:2, {2,5}:3, {3,5}:2

 C3: {2, 3,5}

3. scan T  C3: {2, 3, 5}:2  F3: {2, 3, 5}

38
Example Question