Data Mining-Knowledge

Presentation 2
Prof. Sin-Min Lee
 Overview

Association rules are useful in that they suggest

●

hypotheses for future research

Association rules integrated into the generic
●

actual argument model can assist in identifying the

most plausible claim from given data items in a
forward inference way or the likelihood of missing
data values in a backward inference way
 What is data mining ? What is knowledge
discovery from databases KDD?

knowledge discovery in databases (KDD)

●

is the 'non trivial extraction of nontrivial of

implicit, previously unknown, and
potentially useful information from data
KDD encompasses a number of different
●

technical approaches, such as clustering, data

summarization, learning classification rules,
finding dependency networks, analyzing
changes, and detecting anomalies

KDD has only recently emerged because we

●

only recently have been gathering vast

quantities of data
●
 Examples of KDD studies
●Mangasarian et al (1997) Breast Cancer diagnosis. A sample from breast lump
mass is assessed by:
●mammagrophy (not sensitive 68%-79%)

●data mining from FNA test results and visual inspection (65%-98%)

●surgery (100% but invasive, expensive)

●Basket analysis. People who buy nappies also buy beer

●NBA. National Basketball Association of America. Player pattern profile.

Bhandary et al (1997)
●Credit card fraud detection
●Stranieri/Zeleznikow (1997) predict family law property outcomes

●Rissland and Friedman (1997) discovers a change in the concept of ‘good faith’
in US Bankruptcy cases
● Pannu (1995) discovers a prototypical case from a library of cases

• Wilkins and Pillaipakkamnatt (1997) predicts the time a case takes to be heard
• Veliev et al (1999) association rules for economic analaysis
 Overview of process of knowledge discovery in
databases ?

Pre-proces Transform
Raw data Target data patterns knowledge
sed data ed data

Pre Trans Data Interpret

Select
process form mining patterns

from Fayyad, Pitatetsky-Shapiro, Smyth

(1996)
 Phase 4. Data mining

Finding patterns in data or fitting models to data

●

Categories of techniques
●

●
Predictive (classification: neural networks, rule induction,
linear, multiple regression)
●
Segmentation (clustering, k-means, k-median)
●
Summarisation (associations, visualisation)
●
Change detection/modelling
 What Is Association
Mining?
• Association rule mining:
– Finding frequent patterns, associations, correlations, or
causal structures among sets of items or objects in
transaction databases, relational databases, and other
information repositories.
• Applications:
– Basket data analysis, cross-marketing, catalog design, loss-
leader analysis, clustering, classification, etc.
• Examples.
– Rule form: “Body → Ηead [support, confidence]”.
– buys(x, “diapers”) → buys(x, “beers”) [0.5%, 60%]
– major(x, “CS”) ^ takes(x, “DB”) → grade(x, “A”) [1%,
75%]
 More examples

– age(X, “20..29”) ^ income(X, “20..29K”) 

buys(X, “PC”) [support = 2%, confidence = 60%]
– contains(T, “computer”)  contains(x, “software”)
[1%, 75%]
 Association rules are a data mining technique
• An association rules tell us something about the association
between two attributes
• Agrawal et al (1993) developed the first association rule
algorithm, Apriori
• A famous (but unsubstantiated AR) from a hypothetical
supermarket transaction database is if nappies then beer (80%)
Read this as nappies are bought implies beer are bought 80% of
the time
• Association rules have only recently been applied to law with
promising results
• Association rules can automatically discover rules that may
prompt an analyst to think of hypothesis they would otherwise
have considered
 Rule Measures: Support
Support and confidence
and Confidence
Customer are two independent notions
buys both
Customer • Find all the rules X & Y ⇒ Z
buys diaper
with minimum confidence and
support
– support, s, probability that a
transaction contains {X  Y  Z}
Customer – confidence, c, conditional
buys beer
probability that a transaction having
{X  Y} also contains Z
Transaction ID Items Bought Let minimum support 50%,
2000 A,B,C and minimum confidence
1000 A,C 50%, we have
4000 A,D
– A ⇒ C (50%, 66.6%)
5000 B,E,F
– C ⇒ A (50%, 100%)
 Mining Association Rules—An
Example
Transaction ID Items Bought Min. support 50%
2000 A,B,C Min. confidence 50%
1000 A,C
4000 A,D Frequent Itemset Support
{A} 75%
5000 B,E,F
{B} 50%
{C} 50%
{A,C} 50%

For rule A ⇒ C:
support = support({A C}) = 50%
confidence = support({A C})/support({A}) = 66.6%
 Two Step Association Rule
Mining

Step 1: Frequent itemset generation – use Support

Step 2: Rule generation – use Confidence

{milk, bread} is a frequent item set.
Folks buying milk, also buy bread.
Is it also true?: “Folks buying bread also buy milk.”
 Confidence and support of an association rule
• 80% is the confidence of the rule if nappies then beer (80%). This is
calculated by n2/n1 where:
•n1 = no of records where nappies are bought
•n2 = no of records where nappies were bought and beer was also
bought.
• if 1000 transactions for nappies, and of those, 800 also had beer then
confidence is 80%.
• A rule may have a high confidence but not be interesting because it
doesn’t apply to many records in the database. i.e. no. of records where
nappies were bought with beer / total records.
• Rules that may be interesting have a confidence level and support level
above a user set threshold
 Interesting rules: Confidence and support of an
association rule
• if 1000 transactions for nappies, and of those, 800 also had beer
then confidence is 80%.
• A rule may have a high confidence but not be interesting because
it doesn’t apply to many records in the database. i.e. no. of records
where nappies were bought with beer / total records.
• Rules that may be interesting have a confidence level and support
level above a user set threshold
 Association rule screen shot with A-Miner from
Split Up data set

• In 73.4% of cases where the wife's needs are some to high then the
husband's future needs are few to some.
• Prompts an analyst to posit plausible hypothesis e.g. it may be the case
that the rule reflects the fact that more women remain custodial parents of
the children following divorce than men do. The women that have some to
high needs may do so because of their obligation to children.
 Mining Frequent Itemsets:
the Key Step
• Find the frequent itemsets: the sets of items that
have minimum support
– A subset of a frequent itemset must also be a frequent
itemset – Apriori principle
• i.e., if {AB} is a frequent itemset, both {A} and {B} should be a
frequent itemset
– Iteratively find frequent itemsets with cardinality from 1
to k (k-itemset)
• Use the frequent itemsets to generate association
rules.
 The Apriori Algorithm
• Join Step: C is generated by joining L with itself
k k-1

• Prune Step: Any (k-1)-itemset that is not frequent cannot

be a subset of a frequent k-itemset
• Pseudo-code:
Ck: Candidate itemset of size k
Lk : frequent itemset of size k

L1 = {frequent items};
for (k = 1; Lk !=∅; k++) do begin
Ck+1 = candidates generated from Lk;
for each transaction t in database do
increment the count of all candidates in Ck+1
that are contained in t
Lk+1 = candidates in Ck+1 with min_support
end
return ∪k Lk;
 Association rules in law
• Association rules generators are typically packaged with very expensive
data mining suites. We developed A-Miner (available from authors) for a
PC platform.
• Typically, too many association rules are generated for feasible analysis.
So, our current research involves exploring metrics of interesting to restrict
numbers of rules that might be interesting
• In general, structured data is not collected in law as it is in other domains
so very large databases are rare
• Our current research involves 380,000 records from a Legal Aid
organization data base that contains data on client features.
• ArgumentDeveloper shell that can be used by judges to structure their
reasoning in a way that will facilitate data collection and reasoning
 The Apriori Algorithm —
Example Support = 2
Database D itemset sup.
L1 itemset sup.
TID Items C1 {1} 2 {1} 2
100 134 {2} 3 {2} 3
200 235 Scan D {3} 3 {3} 3
300 1235 {4} 1 {5} 3
400 25 {5} 3

C2 itemset sup C2 itemset

L2 itemset sup {1 2} 1 Scan D {1 2}
{1 3} 2 {1 3} 2 {1 3}
{2 3} 2 {1 5} 1 {1 5}
{2 3} 2 {2 3}
{2 5} 3
{2 5} 3 {2 5}
{3 5} 2
{3 5} 2 {3 5}
 Join Operation — Example
Infrequent
L2 join L2 Subset
itemset sup itemset sup
{1 3} 2 {1 3} 2 {1 3} {1 3} null
{2 3} 2 {2 3} 2 {1 3} {2 3} {1 2 3} {1 2}
{2 5} 3 {2 5} 3 {1 3} {2 5} null
{3 5} 2 {3 5} 2 {1 3} {3 5} {1 3 5} {1 5}
L2 L2 {2 3} {2 3} null
{2 3} {2 5} {2 3 5}
{2 3} {3 5} {2 3 5}
{2 5} {2 5} null
{2 5} {3 5} {2 3 5}

C3 itemset Scan D L3 itemset sup

{2 3 5} {2 3 5} 2
 Anti-Monotone Property

If a set cannot pass a test, all of its supersets

will fail the same test as well.

If {2 3} does not have a support, nor will

{1 2 3}, {2 3 5}, {1 2 3 5}, etc.

If {2 3} occurs only in 5 times, can {2 3 5}

occur in 8 times?
 How to Generate Candidates?
• Suppose the items in Lk-1 are listed in an order
• Step 1: self-joining Lk-1
insert into Ck
select p.item1, p.item2, …, p.itemk-1, q.itemk-1
from Lk-1 p, Lk-1 q
where p.item1=q.item1, …, p.itemk-2=q.itemk-2, p.itemk-1 < q.itemk-1

• Step 2: pruning
forall itemsets c in Ck do
forall (k-1)-subsets s of c do
if (s is not in Lk-1) then delete c from Ck
 Example of Generating
Candidates
• L3={abc, abd, acd, ace, bcd}

• Self-joining: L3*L3
Problem of
– abcd from abc and abd
generate-&-test
– acde from acd and ace heuristic

• Pruning:
– acde is removed because ade is not in L3

• C4={abcd}
 extremely serious pattern of priors
very serious pattern of priors
Severity of prior
I.3 serious pattern of priors
convictions constellation
not so serious pattern of priors
no prior convictions
serious offender status yes
no
I.2 major psychiatric illness
Offender's health some psychiatric illness
drug dependency

Association rules can

major disability
no major health issues
Offender's age >0 yrs extremely serious
very serious
serious

be used for forward

I.A Seriousness of armed robbery as an
not so serious
offense relative to other offenses trifling
very high
Moral culpability of high
average

and backward I.4 offender low

very low

extreme

inferences in the
Degree of remorse high
displayed by offender some
little I.1
I.5 Offender lone
none penalty

generic/actual seriousness of the offence

relative to other armed
extreme serious
very serious
serious
Imprisonment
Combined custody and treatment order

argument model for

I.6 robberies not so serious
trifling Hospital security order
Intensive correction order
I.7 extreme Suspended sentence
high Youth training centre detention

sentencing armed Co-operation some

little
none
Community based order
Fine
Adjournment on conditions

robbery
Extent to which retribution is an very appropriate Discharge offender
appropriate purpose somewhat appropriate Dismiss offence
not appropriate at all Defer sentence
Extent to which specific deterrence very appropriate
somewhat appropriate
is an appropriate purpose not appropriate at all
Extent to which general deterrence very appropriate
is an appropriate purpose somewhat appropriate
not appropriate at all
Extent to which rehabilitation is an very appropriate
somewhat appropriate
appropriate purpose not appropriate at all
Extent to which community very appropriate
protection is an appropriate somewhat appropriate
purpose not appropriate at all

guilt plea early

Offender's plea guilty plea during
not guilty throughout
 Generic/actual argument model for sentencing armed
robbery Page-1 19 May, 2001 Page 1 of 1
extreme impact
serious impact
some impact extremely serious pattern of priors
Personal background little impact prior offence name ? very serious pattern of priors
Severity of prior
bipolar disorder I.3 serious pattern of priors
other psychiatric imprisonment, ico, convictions constellation
cbo not so serious pattern of priors
Psychiatric illness other psychological prior offence type no prior convictions
none etc
extreme addiction serious offender status yes
serious addiction prior offence sentence no
Gambling I.2 major psychiatric illness
some gambling
none date of prior offence Offender's health some psychiatric illness
drug dependency
extremely pertinent Victoria major disability
Personal crisis somewhat pertinent prior sentence jurisdiction no major health issues
Other Australia >0 yrs
not an issue Other Offender's age extremely serious certainly exist
Reasons to
serious offender status at time very serious probably exist
extremely pertinent Seriousness of armed robbery as an serious depart from possibly exist
Cultural adjustment somewhat pertinent I.A not so serious don't exist

:
not an issue offense relative to other offenses from parity
trifling
extreme addiction with co-
serious addiction very high
Drug dependence high offender
some addiction Moral culpability of
none average penalty
I.4 offender low
extremely pertinent very low
Intellectual disability somewhat pertinent
not an issue guilt plea early
extreme
guilty plea during
Plea Degree of remorse high
not guilty throughout
displayed by offender some
Remarks to police indicate remorse little I.1
neutral I.5 Offender lone
indicate no remorse none Penalty
penalty
Apology offered yes
no extreme serious
Restitution made yes seriousness of the offence very serious Imprisonment
extremely significant
Degree of no relative to other armed serious Combined custody and treatment order
significant
I.B violence not so significant I.6 robberies not so serious
not significant at all trifling Hospital security order
Intensive correction order
extremely significant verry highe extreme Suspended sentence
Degree of significant Degree of assistence high I.7
I.C planning high Youth training centre detention
not so significant offered to police by the average Co-operation
not significant at all low some Community based order
offender very low little Fine
extremely significant none
Extent to which significant full admission Adjournment on conditions
Assisted victim not so significant Police interview partial admission Extent to which retribution is an very appropriate Discharge offender
not significant at all complete denial appropriate purpose somewhat appropriate Dismiss offence
positive defense offerred not appropriate at all Defer sentence
Impact of the extreme Extent to which specific deterrence very appropriate
crime on high no instructions somewhat appropriate
I.C some is an appropriate purpose not appropriate at all Co-
victims little Assistance to Crown very important offender's
none important Extent to which general deterrence very appropriate penalty
Impact of the extreme provided but not important is an appropriate purpose somewhat appropriate
high not appropriate at all
I.D crime on the some not provided None
Extent to which rehabilitation is an very appropriate
community little somewhat appropriate
none appropriate purpose not appropriate at all Imprisonment
Value of major psychiatric illness Extent to which community Combined custody and treatment order
Offender's health very appropriate
property stolen drug dependency protection is an appropriate somewhat appropriate Hospital security order
purpose not appropriate at all Intensive correction order
over many days/months or years no major health issues
Duration of Suspended sentence
over many hours
offence over many minutes guilt plea early Youth training centre detention
Offender's plea guilty plea during Community based order
not guilty throughout Fine
Adjournment on conditions
Discharge offender
Dismiss offence
Defer sentence
 Forward inference: confidence
extremely serious pattern of priors

I.3
Severity of prior
convictions constellation
very serious pattern of priors
serious pattern of priors
not so serious pattern of priors
• In the sentence actual argument
no prior convictions

I.2
serious offender status yes
no
major psychiatric illness
database the following outcomes
some psychiatric illness

were noted for the inputs suggested:

Offender's health drug dependency
major disability
no major health issues
Offender's age >0 yrs extremely serious
very serious
Seriousness of armed robbery as an serious
I.A not so serious
offense relative to other offenses trifling
very high
Moral culpability of high
I.4 offender
average
low
very low Imprisonment 57%
Degree of remorse
extreme
high Combined custody and treatment order 0.1%
0%
displayed by offender some
I.1
Hospital security order
I.5 little Offender lone
none penalty

seriousness of the offence

extreme serious
very serious Imprisonment Intensive correction order 12%
I.6
relative to other armed
robberies
serious
not so serious
Combined custody and treatment order
Hospital security order
Suspended sentence 2%
trifling

I.7 extreme
Intensive correction order
Suspended sentence
Youth training centre detention
Youth training centre detention 10%
high
Co-operation some
little
Community based order
Fine Community based order 16%
Fine 0%
none Adjournment on conditions
Extent to which retribution is an very appropriate Discharge offender
somewhat appropriate Dismiss offence
appropriate purpose
Extent to which specific deterrence
not appropriate at all
very appropriate
Defer sentence Adjournment on conditions 0%
Discharge offender
somewhat appropriate
is an appropriate purpose not appropriate at all
0%
Dismiss offence
Extent to which general deterrence very appropriate
is an appropriate purpose somewhat appropriate
not appropriate at all
Extent to which rehabilitation is an
appropriate purpose
very appropriate
somewhat appropriate
not appropriate at all Defer sentence
Extent to which community very appropriate
protection is an appropriate somewhat appropriate
purpose not appropriate at all

guilt plea early

Offender's plea guilty plea during
not guilty throughout
 Backward inference: constructing the strongest argument

If all the items you suggest AND

If extremely serious pattern of priors then imprisonment 90% 2%
If very serious pattern of priors then imprisonment 75% 7%
If serious pattern of priors then imprisonment 68% 17%
If not so serious pattern of priors then imprisonment 78% 17%
If no prior convictions then imprisonment 2% 3%
 Conclusion

●
Data mining or Knowledge discovery from databases has not been
appropriately exploited in law to date.
●
Association rules are useful in that they suggest hypotheses for
future research
●
Association rules integrated into the generic actual argument model
can assist in identifying the most plausible claim from given data items
in a forward inference way or the likelihood of missing data values in
a backward inference way
 Generating Association Rules
• For each nonempty subset s of l, output the rule:
s => (l - s)
if support_count(l) / support_count(s) >= min_conf

where min_conf is the minimum confidence

l threshold.
= {2 3 5}, s of l are {2 3}, {3 5}, {2 5}, {2}, {3}, & {5}.
Candidate rules:
{2 3} => {5} {2} => {3 5}
{3 5} => {2} {3} => {2 5}
{2 5} => {3} {5} => {2 3}
 Generating Association Rules
if support_count(l) / support_count(s) >= min_conf
(e.g,75%),

itemset sup
then introduce the rule s => (l - s).
{1 2} 1 itemset sup.
{1 3} 2 {1} 2 itemset sup
{1 5} 1 {2} 3 {2 3 5} 2
{2 3} 2 {3} 3
{2 5} 3 {4} 1
{3 5} 2 {5} 3

l = {2 3 5} s = {2 3} {3 5} {2 5} {2} {3} {5}

{2 3} => {5} : 2/2 {2} => {3 5} : 2/3
{3 5} => {2} : 2/2 {3} => {2 5} : 2/3
{2 5} => {3} : 2/3 {5} => {2 3} : 2/3
Presentation of Association
Rules (Table Form )
Visualization of Association Rule Using Plane Graph
Visualization of Association Rule Using Rule Graph
Decision tree is a classifier in the form of a tree structure
where each node is either:
• a leaf node, indicating a class of instances, or
• a decision node that specifies some test to be
carried out on a single attribute 　 value, with one
branch and sub-tree for each possible outcome of the
test.

A decision tree can be used to classify an instance by

starting at the root of the tree and moving through it until
a leaf node, which provides the classification of the
instance.
Example: Decision making in the London stock market

Suppose that the major factors affecting the London stock

market are:

• what it did yesterday;

• what the New York market is doing today;
• bank interest rate;
• unemployment rate;
• England’s prospect at cricket.
The process of predicting an instance by this decision tree
can also be expressed by answering the questions in the
following order:

Is unemployment high?
YES: The London market will rise today
NO: Is the New York market rising today?
YES: The London market will rise today
NO: The London market will not rise today.
Decision tree induction is a typical inductive approach to learn
knowledge on classification. The key requirements to do mining
with decision trees are:
• Attribute-value description: object or case must be
expressible in terms of a fixed collection of properties or attributes.
• Predefined classes: The categories to which cases are to be
assigned must have been established beforehand (supervised
data).
• Discrete classes: A case does or does not belong to a
particular class, and there must be for more cases than classes.
• Sufficient data: Usually hundreds or even thousands of
training cases.
• “Logical” classification model: Classifier that can be only
expressed as decision trees or set of production rules
An appeal of market analysis comes from the clarity and
utility of its results, which are in the form of association
rules. There is an intuitive appeal to a market analysis
because it expresses how tangible products and services
relate to each other, how they tend to group together. A
rule like, “if a customer purchases three way calling, then
that customer will also purchase call waiting” is clear.
Even better, it suggests a specific course of action, like
bundling three-way calling with call waiting into a single
service package. While association rules are easy to
understand, they are not always useful.
The following three rules are examples of real rules
generated from real data:
∀• On Thursdays, grocery store consumers often
purchase diapers and beer together.
∀• Customers who purchase maintenance agreements
are very likely to purchase large appliances.
∀• When a new hardware store opens, one of the most
commonly sold items is toilet rings.
These three examples illustrate the three common types
of rules produced by association rule analysis: the useful,
the trivial, and the inexplicable.
OLAP (Summarization) Display Using MS/Excel 2000
Market-Basket-Analysis (Association)—Ball graph
Display of Association Rules in Rule Plane Form
Display of Decision Tree (Classification Results)
Display of Clustering (Segmentation) Results
3D Cube Browser