MAB 8102 Multivariate Analysis
PCA and FA Assignment -Group 1
Description of the methods for estimation in Factor Analysis Models Including
the advantages and disadvantages of each of the methods.
Group Members
01 AWINO DIANA 2018/HD07/3889U 1800743646
02 NAKALANZI VICTORIA 2020/HD07/19442U 2000719442
03 THERESA NAMUYIGE 2020/HD07/19452U 2000719452
04 KIDUBULE IBRAHIM 2020/HD07/19448U 2000719448
Group contributions
MEMBER CONTRIBUTIONS
Coordination
Principal component Analysis
Theresa Namuyige Principal factor Analysis
Organising the work, edits
Conclusion, Powerpoint Presentation.
Principal Axis Factor Analysis
Maximum-likelihood Factor Analysis
Nakalanzi Victoria
Organising the work, edits.
Conclusion, Powerpoint Presentation.
Introduction
Unweighted least-squares factor analysis
Ibrahim Kidubule
Alpha factor analysis
Conclusion, Powerpoint Presentation.
Image Factor Analysis analysis
Generalized least squares (GLS)
Awino Diana
Organising the work, final edits
Reading through the work
Introduction
Factor analysis is a statistical technique commonly used for evaluating the strength of the relationship of
individual items of a scale with the hidden concept, assessing the content or construct validity of an
instrument, determining plausible structures underlying a set of variables, and combining a set of
variables into one composite score. It assumes several assumptions like a linear relationship, no
multicollinearity, only relevant variables in the analysis, and a true correlation between variables and
factors. The purpose of this report is to describe the methods for estimation in Factor Analysis Models
and include the advantages and disadvantages of each method. The several methods available are
categorized into two: Classic models which assume that all error is random whereas the neoclassic
acknowledges both a random and a systematic component (Ferketich & Muller, 1990).
Methods of estimation
1. Classical Methods
1.1 Maximum Likelihood Method
Maximum likelihood (ML) estimation is popular for fitting factor analysis models, especially those having
restrictions on the parameters. [Chuanhai Liu And Donald B. Rubin]. This method finds estimates of the
factor loadings and unique variances by maximizing the likelihood function associated with the
multivariate normal model, assuming the data are independently sampled from a multivariate normal
distribution with mean vector μ, and variance-covariance matrix of the form (Σ = LL’ + Ψ), Where L is the
matrix of factor loadings and Ψ is the diagonal matrix of specific variances. The MLE procedure involves
the estimation of μ, the matrix of factor loadings L, and the specific variance Ψ, from the log-likelihood
function which is given by the following expression:
By maximizing the above log-likelihood function, the maximum likelihood estimators for μ, L, and Ψ are
obtained.
Advantages
1. Maximum likelihood provides a consistent approach to parameter estimation problems.
2. Several popular statistical software packages provide excellent algorithms for MLE. This helps
mitigate the computational complexity of maximum likelihood estimation.
Disadvantages
1. Maximum likelihood estimates can be heavily biased for small samples.
1
2. The likelihood equations need to be specifically worked out for a given distribution and
estimation problem. The mathematics is often non-trivial, particularly if confidence intervals for
the parameters are desired.
1.2 Principal Component Factor analysis
Principal components extracts “real” factors in the sample, those that are due to optimization of the data
at hand. Because these factors are “real” rather than hypothetical, some sources refer to them as
components or component factors rather than as factors (Kim & Mueller, 1978). PC assumes that there is
no unique variance, the total variance is equal to the common variance. The PC method is based on an
approximation 𝐐 of 𝐐, the factor loadings matrix. The sample covariance matrix is diagonalized, 𝑺 =
𝚪𝚲𝚪𝑻. Then the first K eigenvectors are retained to build 𝐐 = 𝝀𝟏𝜸𝟏,…, 𝝀𝒌𝜸𝒌.
Advantages
● It is best for the Uniform distribution of all variables and all samples.
● The principal Component factor explains more variance than the other factors.
Disadvantages
● Independent variables become less interpretable since principal components are not as readable
as the original features.
● Since PCF analysis is affected by scales; data standardization is a must before PCF occurs.
2. Neoclassical Methods
2.1 Principal factor method (Common Factor Analysis)
It seeks the fewest factors which can account for the common variance (correlation) of a set of variables.
The “problem” is that common factors can only be computed as combinations of the “common parts” of
the variables. Unfortunately, we can’t separate each person’s score on each variable into the “common”
and “unique” parts. So, common factor scores have to be “estimated”. The principal factor method
involves finding an approximation 𝚿 of Ψ, the matrix of specific variances, and then correcting R, the
correlation matrix of X, by 𝚿.
Advantages
1. PCA helps in overcoming the overfitting issue by reducing the number of features.
Disadvantages
1. You must standardize your data before implementing PCA, otherwise PCA will not be able to find
the optimal Principal Components.
2.2 Principal axis factor analysis
This method seeks the least number of factors that can account for the common variance of a set of
variables. Though the Principal axis factor analysis uses the principal component analysis strategy, it
2
applies it to a different version of the correlation matrix. Because the analysis of the data structure is
focused on common variance and not sources of error that are specific to individual measurements, the
correlation matrix has estimates of commonalities as its diagonal entries. Allowing for specific variance,
the correlation matrix is estimated as (ρ = LL’ + Ψ). Where Ψ is a diagonal matrix of specific variances.
The estimate of the specific variances is obtained as (Ψ = ρ - LL’). Where the matrix and diagonal entries
Ψ are estimated as
Advantages
1. PAF is the best for small variables and sample sizes.
2. The PAF approach does not require meeting specific assumptions regarding your data such as
multivariate normal distribution which is often violated in practice.
Disadvantages
1. This factoring approach does not give statistics regarding the tested model such as chi square.
2.3 Alpha Factoring Method
This technique is based on the maximization of the reliability of factors, assuming that the variables are
randomly sampled from a very large set of variables. It determines the factors which have the maximum
generalizability in the Kuder-Richardson sense. The AFA has the property of giving the same factors
regardless of the units of measurement of the observable variables. The AFA operates in the metric of
the common parts. In the establishment of the number of factors, the AFA retains only those alpha
factors with positive generalizability. Alpha factor analysis is derived from finding uncorrelated common
factors Xs (s = 1, 2, ..., q), each of which successively has maximum generalizability in the
coefficient-alpha sense, a psychometric measure of reliability in the generalized Kuder-Richardson sense.
McDonald (1970)
Advantages
1. No distributional assumptions.
2. Error is due to sampling symptoms, not individuals.
3. Emphasizes low (<0.40) communalities.
4. Always converges.
Disadvantages
1. Unlike other methods, this method does not assume sampled cases and fixed variables.
2.4 Unweighted least squares (ULS) factoring:
This technique is based upon minimizing the sum of squared differences between the observed and
estimated correlation matrices, without counting the diagonal. Ordinary or Unweighted least squares
3
(ULS) is the algorithm that directly aims at minimizing the residuals between the input correlation matrix
and the reproduced (by the factors) correlation matrix (while diagonal elements as the sums of
communality and uniqueness are aimed to restore 1s). ULS method can work with singular and even not
positive semidefinite matrix of correlations provided the number of factors is less than its rank, -
although it is questionable if theoretically FA is appropriate then. ULS method includes iterative eigen
decomposition of the reduced correlation matrix, like PAF, but within a more complex Newton-Raphson
optimization procedure aiming to find unique variances at which the correlations are reconstructed
maximally.
Advantages:
1. Provides more accurate and less variable parameter estimates,
2. Gives more precise standard errors and better coverage rates.
Disadvantage:
Least squares provide "best linear unbiased estimators if the response really does have a linear
relationship with any predictors. If it does not, then least squares estimators of the data are biased,
regardless of how much data you have.
2.5 Image Factor Analysis
Guttman's image theory is based on the assumption that each variable in the set of variables in the
domain of the characteristics of interest, can be split into two parts: the "image" of the variable (or, the
"common part") and the "anti-image" of the variable (or, the "unique part"). This method is based on
correlation matrix. Ordinary Least Squares Regression is used to predict the factors. Image factor
analysis is given by:
Where Diag[X] stands for a diagonal matrix formed from the diagonal elements of a square matrix X. J
Advantages
1. With image factor analysis, more factors can be extracted without yielding a perfect fit to the
observed data unlike in traditional factor analytic models.
2. No distribution assumptions
Disadvantages
1. It has little advantage over Principal Axis factoring.
4
2.6 Generalized least squares (GLS)
The generalized least squares estimator of y is the y that minimizes the residual quadratic form
where D is either a positive definite random matrix which converges in probability to a positive definite
matrix D as n approaches infinity, or D is a positive definite constant matrix (D = D). The generalized least
squares estimator will be denoted by ý . The estimator of y can be constructed by defining a matrix T
such that D = T’T and using a nonlinear least squares program with T s as the dependent variable.
Advantages
1. GLS generates a Chi-squared goodness-of-fit test to guide the number of factors.
Disadvantages
1. Requires a large sample.
Conclusions
An article by Fabrigar, Wegener, MacCallum, and Strahan (1999) argued that if data are relatively
normally distributed, maximum likelihood is the best choice because it allows for the computation of a
wide range of indexes of the goodness of fit of the model [and] permits statistical significance testing of
factor loadings and correlations among factors and the computation of confidence intervals. If the
assumption of multivariate normality is “severely violated” they recommend one of the principal factor
methods or the “principal axis factors" (Fabrigar et al., 1999). Other authors have argued that in
specialized cases, or for particular applications, other extraction techniques (e.g., alpha extraction) are
most appropriate, but the evidence of advantage is slim. In general, ML, PCA, or PAF will give you the
best results, depending on whether your data are generally normally-distributed or significantly
non-normal, respectively.
5
References
1. https://support.minitab.com/en-us/minitab/18/help-and-how-to/modeling-statistics/multivariat
e/how-to/factor-analysis/methods-and-formulas/methods-and-formulas/
2. A Comparison of Principal Component Analysis, Maximum Likelihood and the Principal Axis in
Factor Analysis by Onyekachi Akuoma Mabel, Olanrewaju Samuel Olayemi
3. Fabrigar, L.R., Wegener,D.T., MacCallum, R.C., and Strahan, E. J. (1999). Evaluating the use of
exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299.
4. Maximum likelihood estimation of factor analysis using the ecme algorithm with complete and
incomplete data. [Chuanhai Liu and Donald B. Rubin]
5. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3732178/
6. https://www.statisticssolutions.com/free-resources/directory-of-statistical-analyses/factor-analy
sis/
7. https://www.statisticssolutions.com/exploratory-factor-analysis/
8. https://www.koreascience.or.kr/article/JAKO201915658233382.pdf
9. https://conservancy.umn.edu/bitstream/handle/11299/107735/v14n1p029.pdf?sequence=1&is
Allowed=y
10. itl.nist.gov/div898/handbook/eda/section3/eda3652.htm
11. Identification of Cancer-Related Symptom Clusters: An Empirical Comparison of Exploratory
Factor Analysis Methods Helen M. Skerman, PhD, Patsy M. Yates, PhD, and Diana Battistutta, PhD
Institute of Health and Biomedical Innovation (H.M.S., P.M.Y., D.B.), School of Nursing (H.M.S.,
P.M.Y.), and School of Public Health (D.B), Queensland University of Technology, Brisbane,
Queensland, Australia
12. Generalized Least Squares Estimation of the Functional Multivariate Linear Errors-in-Variables
Model P. FRED DAHM AND WAYNE A. FULLER Texas A&M University and Iowa State University
