GyanData Private Limited

STATISTICAL MODELLING

Data Analytics 1
 GyanData Private Limited

Contents

1 Random Variables, Probability Density Functions

1.1 Introduction
1.2 Random Variables
1.3 Probability & Probability density functions (pdfs)
1.4 Some common pdfs
1.5 Moments of a pdf
2 Estimation, Hypotheses Testing
2.1 Sample statistics
2.2 Estimation
2.3 Hypothesis testing

Data Analytics 2
GyanData Private Limited

RANDOM VARIABLES AND

PROBABILITY DISTRIBUTIONS

Data Analytics 3
 GyanData Private Limited

Random Phenomena
 Deterministic phenomenon: Phenomenon whose outcome can be
predicted with a very high degree of confidence
◦ Example: Age of a person (using date of birth stated in Aadhaar card)
 Stochastic phenomenon: Phenomenon which can have many possible
outcomes for same experimental conditions. Outcome can be predicted
with limited confidence
◦ Example: Outcome of a coin toss

Data Analytics 4
 GyanData Private Limited

Characterizing random phenomena

 Sources of error in observed outcomes
◦ Lack of knowledge of generating process (model error)
◦ Errors in sensors used for observing outcomes (measurement error)
 Types of random phenomena
◦ Discrete: Outcomes are finite
 Coin toss outcomes {H, T}
 Throw of a dice {1, 2, 3, 4, 5, 6}
◦ Continuous: Infinite number of outcomes
 Body temperature measurement in deg F

Data Analytics 5
GyanData Private Limited

Sample space and events

 Sample space
◦ Set of all possible outcomes of a random phenomenon
 Coin Toss : S = {H, T}
 Two coin tosses: S = {HH, HT, TH, TT}
 Event
◦ Subset of the sample space
 Occurrence of a head in first toss of a two coin toss experiment A = {HH, HT}

Data Analytics 6
GyanData Private Limited

Exclusive and Independent Events

 Mutually exclusive events
◦ Two events are mutually exclusive if occurrence of one precludes occurrence of
the other
 eg. In a two coin toss experiment, events {HH} and {HT} are mutually exclusive
 Independent events
◦ Two events are independent if occurrence of one has no influence on occurrence
of other
 eg. In a two coin toss experiment, the occurrence of head in second toss is independent of
occurrence of head or tail in first toss

Data Analytics 7
GyanData Private Limited

Probability Measure
 Probability measure is a function that assigns a real value to every
outcome of a random phenomena which satisfies following axioms
 P(S) = 1 (one of the outcomes should occur)
 0 ≤ P(A) ≤ 1 (Probabilities are non-negative and less than 1 for any event A)
 For two mutually exclusive events A and B
 P(A U B) = P(A) + P(B)

Data Analytics 8
GyanData Private Limited

Some rules of probability

 Following important probability rules can be proved
using Venn diagrams
S= A= B=
All outcomes are equally likely
TH
HH HT
If Ac is the complement of event Ac,
TT
P(A) = 1 – P(A) = 1 – 0.5 = 0.5
TH
If B  A, P(B) ≤ P(A); 0.25 < 0.5 HT HH
TT
P(AB) = P(A) + P(B) – P(AB)
For two independent events A and
HT HH TH
B, P(AB) = P(A)P(B)
TT
0.5 + 0.5 – 0.5*0.5 = 0.75
Data Analytics 9
GyanData Private Limited

Conditional Probability
 If two events A and B are not independent, then information
available about the outcome of event A can influence the
predictability of event B
 Conditional probability
◦ P(B | A) = P(AB)/P(A) if P(A) > 0
◦ P(A | B)P(B) = P(B | A)P(A) - Bayes formula
◦ P(A) = P(A | B)P(B) + P(A | Bc)P(Bc)
 Example: two (fair) coin toss experiment TH
◦ Event A : First toss is head = {HT, HH} HT HH
◦ Event B : Two successive heads ={HH} TT
◦ Pr(B) = 0.25 (no information)
◦ Given event A has occurred Pr(B|A) = 0.5 = 0.25/0.5 = P(AB)/P(A)

Data Analytics 10
GyanData Private Limited

Example
In a manufacturing process 1000 parts are produced of
which 50 are defective. We randomly take a part from
the day’s production
◦ Outcomes : {A=Defective part B = Non-defective part}
◦ P(A) = 50/1000, P(B) = 950/1000
 Suppose we draw a second part without replacing the
first part
◦ Outcomes : {C=Defective part D = Non-defective part}
◦ P(C | A) = 49/999
◦ P(C) = 49/999*50/1000 + 50/999*950/1000 = 50/1000
◦ P(A | C) = P(A  C)/P(C) = P(C | A)P(A)/P(C) = 49/999

Data Analytics 11
GyanData Private Limited

Random Variable
 A random variable (RV) is a map from sample space to a real line such
that there is a unique real number corresponding to every outcome of
sample space
◦ eg. Coin toss sample space [H T] mapped to [0 1]. If the sample space outcomes are real
valued no need for this mapping (eg. throw of a dice)
◦ Allows numerical computations such as finding expected value of a RV
◦ Discrete RV (throw of a dice or coin)
◦ Continuous RV (sensor readings, time interval between failures)
◦ Associated with the RV is also a probability measure

Data Analytics 12
GyanData Private Limited

Probability Mass/Density Functions

 For a discrete RV, the probability mass function
assigns a probability measure to every discrete
outcome of sample space
◦ eg. Coin toss sample space [H T] mapped to [0 1]. X is discrete
RV whose outcome can be 0 or 1.
◦ P(X = 0) = 0.5, P(X = 1) = 0.5
 For a continuous RV, the probability density function
assigns a probability measure to every interval on
real line
𝑏
◦ P(a < x < b) = ‫( 𝑥𝑑)𝑥(𝑓 𝑎׬‬Area under the curve of f(x))
 Cumulative density function:
𝑏
◦ F(b) = P(- < x < b) = ‫׬‬−∞ 𝑓(𝑥)𝑑𝑥
Data Analytics 13
GyanData Private Limited

Binomial mass density function

 Probability of obtaining k
heads in n successive tosses
of a coin. p is the probability
of obtaining a head in a toss
𝑛 𝑘
◦ 𝑓 𝑥=𝑘 = 𝑝 1 − 𝑝 𝑛−𝑘
𝑘
◦ Density is characterized by
parameter p
◦ For large n it tends to Gaussian
distribution

Data Analytics 14
GyanData Private Limited

Gaussian density function

 Frequently used
(𝑥−𝜇)2
1 −
◦ 𝑓 𝑥 = 𝑒 2𝜎2
2𝜋𝜎
◦ Density is characterized by two
parameters  and 
◦ Density function is symmetric
◦ Maximum entropy distribution
◦ Central limit theorem motivates
use of this distribution

Data Analytics 15
GyanData Private Limited

Chi-square density function

1
◦ 𝑓 𝑥 = 𝑛
𝑛
𝑥 𝑛/2−1 𝑒 −𝑥/2
2 2 Γ( )
2

◦ Density is characterized by
parameter n (degrees of
freedom)
◦ Distribution of sum of squares
of n independent standard
normal RVs
◦ Distribution of sample variance

Data Analytics 16
GyanData Private Limited

Moments of a pdf
 Similar to describing a function using derivatives, a
pdf can be described by its moments
◦ For continuous distributions
∞
 𝐸 𝑥 𝑘 = ‫׬‬−∞ 𝑥 𝑘 𝑓 𝑥 𝑑𝑥
◦ For discrete distributions
 𝐸 𝑥 𝑘 = σ𝑁 𝑘
𝑖=1 𝑖 𝑝(𝑥𝑖 )
𝑥
 Mean : 𝜇 = 𝐸[𝑥]
 Variance : 𝜎 2 = 𝐸 𝑥 − 𝜇 2 = E 𝑥 2 − 𝜇2
 Standard deviation = Square root of variance = 

Data Analytics 17
GyanData Private Limited

Properties of Gaussian RVs

 For a Gaussian RV x
◦ Mean : 𝐸 𝑥 = 𝜇
◦ Variance : 𝐸 𝑥 − 𝜇 2 = 𝜎 2
◦ Symbolically 𝑥~𝒩(𝜇, 𝜎 2 )
 Standard Gaussian RV 𝑧~𝒩(0,1)
 If 𝑥~𝒩(𝜇, 𝜎 2 ) and 𝑦 = 𝑎𝑥 + 𝑏 then
◦ 𝑦~𝒩(𝑎𝜇 + 𝑏, 𝑎2 𝜎 2 )
 Standardization
(𝑥−𝜇)
◦ If 𝑥~𝒩(𝜇, 𝜎 2 ), then 𝑧 = ~𝒩(0,1)
𝜎

Data Analytics 18
GyanData Private Limited

Computation of probability using R

 Functions pxxx(p, parameters, lower.tail =
TRUE/FALSE)
◦ xxx refers to the distribution (eg norm, chisq, exp,
unif)
◦ parameters: Parameters of the distribution (eg.
mean and std for normal distribution)
◦ lower.tail = TRUE:

Data Analytics 19
GyanData Private Limited

Joint pdf of two RVs

 Joint pdf of two RVS x and y: f(x,y)
𝑏 𝑎
◦ 𝑃 𝑥 ≤ 𝑎, 𝑦 ≤ 𝑏 = ‫׬‬−∞ ‫׬‬−∞ 𝑓 𝑥, 𝑦 𝑑𝑥𝑑𝑦
◦ Covariance between x and y: 𝜎𝑥𝑦 = 𝐸[(𝑥 − 𝜇𝑥 )(𝑦 − 𝜇𝑦 )]
𝜎𝑥𝑦
◦ Correlation between x and y: 𝑥𝑦 = 𝜎
𝑥 𝜎𝑦

 Two RVs x and y are uncorrelated if xy = 0

 Two RVs x and y are independent if f(x,y) = f(x)f(y)

Data Analytics 20
GyanData Private Limited

Multivariate Normal Distribution

 A vector of RVs x = 𝑥1 𝑥2 ⋯ 𝑥𝑛 𝑇
 Multivariate Gaussian Distribution : x~𝒩(𝝁, 𝜮)
◦ E[x] =  : Mean vector
◦ E[(x- )(x- )T] = Σ : Variance-covariance matrix
1
◦ 𝑓 x =
1
𝑒 −
2
x−𝝁 𝑇 Σ−1 x−𝝁 : pdf
(2𝜋)𝑛/2 Σ 1/2

 Structure of Σ
𝜎𝑥21 𝜎𝑥1 𝑥2 ⋯ 𝜎𝑥1 𝑥𝑛
𝜎𝑥2 𝑥1 𝜎𝑥22 ⋯
Σ=
⋮ ⋮ ⋮
𝜎𝑥𝑛 𝑥1 2
⋯ ⋯ 𝜎𝑥𝑛

Data Analytics 21
GyanData Private Limited

Some Properties of Multivariate RVs

 If x~𝒩(𝝁, 𝜮)
◦ x − 𝝁 𝑇 Σ−1 x − 𝝁 ~𝜒𝑛2
◦ 𝑁 xത − c 𝑇 S−1 xത − c ~𝑇𝑛,𝑁−𝑛
2

𝑁−1 𝑛
◦ 2
𝑇𝑛,𝑁−𝑛 = 𝐹𝑛,𝑁−𝑛
(𝑁−𝑛)

Data Analytics 22
GyanData Private Limited

SAMPLE STATISTICS

Data Analytics 23
 GyanData Private Limited

Need for sampling

 PDFs of RVs establishes theoretical framework. But
◦ Entire sample space may not be known
◦ Parameters of distribution may not be known
 From a finite sample derive conclusions about the pdf and its
parameters
 Sample (or observation) set is assumed to be sufficiently representative
of the entire sample space
◦ Proper sampling procedures and design of experiments to be used for obtaining
the sample

Data Analytics 24
 GyanData Private Limited

Basic Concepts
 Population: Set of all possible outcomes of a
random experiment characterized by f(x)
 Sample set (realization) : Finite set of observations
obtained through an experiment
 Inference: Conclusion derived regarding the
population (pdf, parameters) from the sample set
◦ Inference made from a sample set is also uncertain since it
depends on the sample set which is one of many possible
realizations

Data Analytics 25
 GyanData Private Limited

Statistical Analysis
 Descriptive Statistics (Analysis)
◦ Graphical : Organizing and presenting the data (eg. box
plots, probability plots)
◦ Numerical: Summarizing the sample set (eg. mean, mode,
range, variance, moments)
 Inferential
◦ Estimation: Estimate parameters of the pdf along with its
confidence region
◦ Hypotheses testing: Making judgements about f(x) and its
parameters

Data Analytics 26
GyanData Private Limited

Measures of Central Tendency - Mean

 Represent sample set by a single value
1 𝑁
◦ Mean (or average): 𝑥ҧ = σ 𝑥
𝑁 𝑖=1 𝑖
 Best estimate in least squares criterion
 Unbiased estimate of population mean: E[𝑥]ҧ = 𝜇
 Affected by outliers
 Eg: Sample heights of 20 cherry trees
[55 55 59 60 63 65 66 67 67 67 71 71 72 73 75 75 78 81 82 83]
 Mean = 69.25 (population mean used to generate random sample
was 70)
 Mean = 71.75 (after a bias of 50 was added to first sample value)

Data Analytics 27
GyanData Private Limited

Measures of Central Tendency – Median

 Represent sample set by a single value
◦ Median: Value of xi such that 50% of the values are less
than xi and 50% of observations are greater than xi
 Robust with respect to outliers in data
 Best estimate in least absolute deviation sense
 Eg: Sample heights of 20 cherry trees
[55 55 59 60 63 65 66 67 67 67 71 71 72 73 75 75 78 81 82 83]
 Median = 69 (population mean used to generate random sample was
70)
 Median = 69 (after a bias of 50 was added to first sample value)

Data Analytics 28
GyanData Private Limited

Measures of Central Tendency -Mode

 Represent sample set by a single value
◦ Mode: Value that occurs most often (Most probable value)
◦ eg. Sample heights of 20 cherry trees

[55 55 59 60 63 65 66 67 67 67 71 71 72 73 75 75 78 81 82 83]

◦ Mode: 67 (three occurrences)

Data Analytics 29
 GyanData Private Limited

Measures of Spread
 Represents spread of sample set
1
◦ Sample variance : 𝑠 =
2 σ𝑁 (𝑥 − 𝑥)ҧ 2
𝑁−1 𝑖=1 𝑖
 Unbiased estimate of population variance : E[𝑠 2 ] = 𝜎 2
 Standard deviation is sqrt of variance
1 𝑁
◦ Mean absolute deviation : 𝑑 = 𝑁 σ𝑖=1 𝑥𝑖 − 𝑥ҧ
ҧ
◦ Range : 𝑅 = 𝑥𝑚𝑎𝑥 − 𝑥𝑚𝑖𝑛
◦ Eg. Sample heights of 20 cherry trees
s2 = 70.5132 and 212.25 with outlier
s = 8.392 (population std used for generating numbers was 10)
MAD = 6.85 and 9.5 with outlier
Range = 83 - 55 = 28
Data Analytics 30
GyanData Private Limited

Distribution of sample mean and variance

 Sample mean
◦ For any distribution sample mean is an unbiased estimate of
population mean
◦ If 𝑥𝑖 ~𝒩(𝜇, 𝜎 2 ) and all observations are mutually
𝜎2
independent, then 𝑥~𝒩(𝜇,
ҧ )
𝑁
 Sample Variance
◦ For any distribution sample variance is an unbiased
estimate of the population variance
◦ If 𝑥𝑖 ~𝒩(𝜇, 𝜎 2 ) and all observations are mutually
(𝑁−1)𝑆 2 2
independent, then ~𝜒𝑁−1
𝜎2

Data Analytics 31
 GyanData Private Limited

Graphical Analysis - Histograms

 Histograms
◦ Divide the range of values in sample set into small intervals and
count how many observations fall within each interval.
◦ For each interval plot a rectangle with width = interval size and
height equal to number of observations in interval
◦ eg. Sample of 20 heights of black cherry trees
[73 75 55 60 66 71 81 67 83 75 82 71 63 55 72 78 67 65 67 59]

Data Analytics 32
 GyanData Private Limited

Graphical Analysis - Box Plot

 Box plot
◦ Find quartiles (Q1, Q2 and Q3), minimum and maximum
values in range
◦ Box is between Q1 and Q3, and whiskers is between min and
max values
◦ eg. Sorted values of heights of 20 cherry trees
[55 55 59 60 63 65 66 67 67 67 71 71 72 73 75 75 78 81 82 83]
Q1: 64, Q2 (median): 69, Q3: 75, min: 55, max: 83

Data Analytics 33
GyanData Private Limited

Graphical Analysis – Probability Plot

 Probability plot (p-p or q-q plot)
◦ Determine different quantile values from sample set. Plot computed
quantiles vs theoretical quantile values from chosen distribution
◦ Same example (standardized and sorted values)
[-1.697 -1.697 -1.2206 -1.1016 -0.7443 -0.5061 -0.3870 -0.2679
-0.2679 -0.2679 0.2084 0.2084 0.3275 0.4466 0.6848 0.6848
1.0420 1.3993 1.5184 1.6374]

Data Analytics 34
GyanData Private Limited

Graphical Analysis – Scatter Plot

 Scatter plot
◦ Plot of one RV (y) against another RV (x) to examine whether
there is any dependence
◦ Example: Marks obtained vs study time for 100 students

Data Analytics 35
GyanData Private Limited

HYPOTHESES TESTING

Data Analytics 36
 GyanData Private Limited

Motivation for Hypotheses Testing

 Business: Will an investment in a mutual fund yield
annual returns greater than desired value? (based on
past performance of the fund)
 Medical: Is the incidence of diabetes greater among
males than females?
 Social: Are women more likely to change mobile
service provider than men?
 Engineering: Has the efficiency of the pump ()
decreased from its original value due to aging?

Data Analytics 37
 GyanData Private Limited

Hypotheses Testing
 The hypotheses is generally converted to a test of the
mean or variance parameter of a population (or
differences in means or variances of populations)
 A hypothesis a statement or postulate about the
parameters of a distribution (or model)
◦ Null hypothesis H0 : The default or status quo postulate that
we wish to reject if the sample set provides sufficient
evidence (eg.  = 0)
◦ Alternative hypothesis H1 : The alternative postulate that is
accepted if the null hypothesis is rejected (eg.  < 0)

Data Analytics 38
 GyanData Private Limited

Hypotheses Testing Procedure

 Identify the parameter of interest (mean, variance,
proportion) which you wish to test
 Construct the null and alternative hypotheses
 Compute a test statistic which is a function of the
sample set of observations
 Derive the distribution of the test statistic under the
null hypothesis assumption
 Choose a test criterion (threshold) against which the
test statistic is compared to reject/not reject the null
hypothesis

Data Analytics 39
 GyanData Private Limited

Two-sided and one-sided tests

 Two sided test  One sided test
H0 : µ = 0 H0 : µ = 0
H1 : µ ≠ 0 H1 : µ > 0
 Test statistic standard normal RV z  Test statistic standard normal RV z

 Reject H0 if z ≤ -2 or z ≥ 2  Reject H0 if z ≥ 1.5

Data Analytics 40
 GyanData Private Limited

Hypotheses Testing Procedure

 No hypotheses test is perfect. There are inherent
errors since it is based on observations which are
random
 The performance of a hypotheses test depends on
◦ Extent of variability in data
◦ Number of observations (Sample size)
◦ Test statistic (function of observations)
◦ Test criterion (threshold)

Data Analytics 41
 GyanData Private Limited

Errors in Hypotheses Testing

 Two Types of errors (Type I and Type II)

Decision → H0 is not rejected H0 is rejected

Truth 
H0 is true Correct Decision Type I error
Pr = 1 -  Pr = 
H1 is true Type II error Correct Decision
Pr =  Pr = 1 - 

◦ Typically the Type 1 error probability  (also called as level

of significance of the test) is controlled by choosing the
criterion from the distribution of the test statistic under the
null hypothesis

Data Analytics 42
 GyanData Private Limited

Errors in Hypotheses Testing

 Type I and Type II error probabilities

Threshold
Distribution if
Distribution if null
alternative hypothesis
hypothesis is true
is true

 Statistical test Power = 1 – Type II error probability

 Trade-off : If we decrease Type I error probability,
then Type II error probability will increase

Data Analytics 43
 GyanData Private Limited

HYPOTHESES TESTING METHODS

 Compute test statistic

 Rejection or acceptance of null hypothesis based on
◦ Specified level of significance
◦ Confidence interval of estimate
◦ p-value of test statistic

Data Analytics 44
GyanData Private Limited

Test for Mean : Solid Propellant example

For a given application the burning rate of a solid
propellant should be 50 cm/s.
◦ 25 samples of the solid propellant are taken and their burning rate
noted. The average burning rate is computed to be 51.3 cm/s. The
standard deviation in the burning rate is known to be 2 cm/s
◦ Null hypothesis :  = 50 cm/s
◦ Alternative hypothesis :   50 cm/s (lower or higher burning rate
propellants are both unsatisfactory) – Two sided test
ҧ
𝑥−50
◦ Test statistic z = ~𝒩(0,1); z = 3.25
2/ 25
◦ Critical value for  = 0.05 is 1.96
◦ Decision: Reject null hypothesis

Data Analytics 45
GyanData Private Limited

Test for Differences in Means : Training example

Two groups of teachers of similar capabilities are
trained by two methods A and B. Is Method B more
effective than Method A?
◦ 10 teachers in each group. Average scores and standard deviation of
scores after training are Group 1: 𝑥1ҧ = 70, 𝑠1 = 3.3665 Group 2: 𝑥1ҧ =
74, 𝑠1 = 5.3955
◦ Null hypothesis : 1 - 2 = 0
◦ Alternative hypothesis : 1 - 2 < 0 – one sided test
◦ Test statistic (assuming unknown but equal variances for two groups)
𝑥ҧ 1 −𝑥ҧ 2 (𝑁1 −1)𝑠12 +(𝑁2 −1)𝑠22
t= ~𝑡𝑁1+𝑁2−2 ; 𝑆𝑝 = t = -1.989
𝑁1 +𝑁2 −2
𝑠2
𝑝 𝑠𝑝
2
+
𝑁1 𝑁2

◦ Critical value for  = 0.05 is -1.73

◦ Decision: Reject null hypothesis (Method B is better)

Data Analytics 46
GyanData Private Limited

Test for Differences in Variances : Process Yields

The variability in yields from two different processes
are to be compared to decide whether they are identical
or not
◦ 50 samples for each process taken. Yield variances are found to be
𝑠12 = 2.05 and 𝑠22 = 7.64
𝜎12
◦ Null hypothesis : =1
𝜎22
𝜎12
◦ Alternative hypothesis : ≠ 1 – two sided test
𝜎22

◦ Test statistic (assuming unknown but equal variances for two groups)
𝑠12
f= ~F(𝑁1 − 1, 𝑁2 − 1); f = 0.27
𝑠22

◦ Critical value for  = 0.025 is 0.567 and  = 0.975 is 1.762

◦ Decision: Reject null hypothesis (Process 2 has higher variability)

Data Analytics 47
 GyanData Private Limited

Summary of useful hypotheses tests

Type of test Characteristic Example Application
z-test Sum of independent Test for a mean or Test coefficients of a
normal variables comparison between two regression model
group means (variance
known)

t-test Ratio of a standard Test for a mean or Test coefficients of a

normal variable and comparison between two regression model
chi-square variables group means (variance
with p degrees of unknown)
freedom

chi-square test Sum of p independent Test for variance Test quality of regression
(p degrees of standard normal model
freedom) variables

F-test (p1 and p2 Ratio of two chi- Test for comparing Choose between
degrees of square variables variances of two groups regression models having
freedom) different number of
parameters

Data Analytics 48
GyanData Private Limited

End of session

Data Analytics 49