MAB 8102 Multivariate Analysis
Take-Home Group Assignment
Description of the methods for estimation in Factor Analysis Models Including the advantages
and disadvantages of each of the methods.
01 AWINO MARIAM 2018/HD07/3889U 1800743646
02 NAKALANZI VICTORIA 2020/HD07/19442U 2000719442
03 THERESA NAMUYIGE 2020/HD07/19452U 2000719452
04 KIDUBULE IBRAHIM 2020/HD07/19448U 2000719448
INTRODUCTION
Factor analysis is a statistical technique commonly used for evaluating the strength of the
relationship of individual items of a scale with the hidden concept, assessing the content or
construct validity of an instrument, determining plausible structures underlying a set of
variables, and combining a set of variables into one composite score. It assumes several
assumptions like a linear relationship, no multicollinearity, only relevant variables in the
analysis, and a true correlation between variables and factors. The purpose of this report is to
describe the methods for estimation in Factor Analysis Models and include the advantages and
disadvantages of each method. The several methods available are categorized into two: Classic
models which assumes that all error is random whereas the neoclassic acknowledges both a
random and a systematic component (Ferketich & Muller, 1990) and includes maximum
likelihood (ML) and principal components (PC) extraction whereas, principal axes factoring
(PAF), image factoring (IF), alpha factoring (AF), generalized least squares (GLS), and
unweighted least squares (ULS) are representative of the neoclassic model.
Methods of estimation
1. Maximum Likelihood Method
Maximum likelihood (ML) estimation is popular for fitting factor analysis models, especially
those having restrictions on the parameters. [Chuanhai Liu And Donald B. Rubin]. This method
finds estimates of the factor loadings and unique variances by maximizing the likelihood
function associated with the multivariate normal model, assuming the data are independently
sampled from a multivariate normal distribution with mean vector μ, and variance-covariance
matrix of the form (Σ = LL’ + Ψ), Where L is the matrix of factor loadings and Ψ is the diagonal
matrix of specific variances. The MLE procedure involves the estimation of μ, the matrix of
factor loadings L, and the specific variance Ψ, from the log-likelihood function which is given by
the following expression:
By maximizing the above log-likelihood function, the maximum likelihood estimators for μ, L,
and Ψ are obtained.
Advantages
1. Maximum likelihood provides a consistent approach to parameter estimation problems
i.e estimates can be developed for a large variety of estimation situations.
2. Several popular statistical software packages provide excellent algorithms for maximum
likelihood estimates. This helps mitigate the computational complexity of maximum
likelihood estimation.
Disadvantages
1. Maximum likelihood estimates can be heavily biased for small samples.
2. Maximum likelihood can be sensitive to the choice of starting values.
3. The likelihood equations need to be specifically worked out for a given distribution and
estimation problem. The mathematics is often non-trivial, particularly if confidence
intervals for the parameters are desired.
2. Principal Component Factor analysis
Principal components extracts “real” factors in the sample, those that are due to optimization
of the data at hand. Because these factors are “real” rather than hypothetical, some sources
refer to them as components or component factors rather than as factors (Kim & Mueller,
1978; NThe principal1978). Principal component assumes an assumption that there is no
unique variance, the total variance is equal to the common variance. The principal component
method is based on an approximation 𝐐 of 𝐐, the factor loadings matrix. The sample
covariance matrix is diagonalized, 𝑺 = 𝚪𝚲𝚪𝑻. Then the first K eigenvectors are retained to
build 𝐐 = 𝝀𝟏𝜸𝟏,…, 𝝀𝒌𝜸𝒌.
Advantages
It is best for the Uniform distribution of all variables and all samples.
The principal Component factor explains more variance than the other factors.
For large variables and sample sizes, Principle Component Factor analysis performs
better across all the distributions.
Disadvantages
Independent variables become less interpretable since principal components are not as
readable as the original features.
Since PC factor analysis is affected by scales; data standardization is a must before PCF
analysis occurs
PCF analysis can lead to loss of certain information especially if the number of factors
isn’t well selected.
3. Principal factor method (Common Factor Analysis)
It seeks the fewest factors which can account for the common variance (correlation) of a set of
variables. The “problem” is that common factors can only be computed as combinations of the
“common parts” of the variables. Unfortunately, we can’t separate each person’s score on each
variable into the “common” and “unique” parts. So, common factor scores have to be
“estimated”. The principal factor method involves finding an approximation 𝚿 of Ψ, the matrix
of specific variances, and then correcting R, the correlation matrix of X, by 𝚿.
Advantages
Disadvantages
4. Principal axis factor analysis
This method seeks the least number of factors that can account for the common variance of a
set of variables. Though the Principal axis factor analysis uses the principal component analysis
strategy, it applies it to a different version of the correlation matrix. Because the analysis of the
data structure is focused on common variance and not sources of error that are specific to
individual measurements, the correlation matrix rho has estimates of commonalities as its
diagonal entries. Allowing for specific variance, the correlation matrix is estimated as (ρ = LL’ +
Ψ). Where Ψ is a diagonal matrix of specific variances. The estimate of the specific variances is
obtained as (Ψ = ρ - LL’). Where the matrix and diagonal entries Ψ are estimated as
Advantages
1. PAF is the best for small variables and sample sizes.
2. The PAF approach does not require meeting specific assumptions regarding your data
such as multivariate normal distribution which is often violated in practice.
Disadvantages
ALPHA FACTORING METHOD
This technique is based on the maximization of the reliability of factors, assuming that the
variables are randomly sampled from a very large set of variables. It determines the factors
which have the maximum generalizability in the Kuder-Richardson sense. The AFA has the
property of giving the same factors regardless of the units of measurement of the observable
variables. The AFA operates in the metric of the common parts. In the establishment of the
number of factors, the AFA retains only those alpha factors with positive generalizability. Alpha
factor analysis is derived from finding uncorrelated common factors Xs (s = 1, 2, ..., q), each of
which successively has maximum generalizability in the coefficient-alpha sense, a psychometric
measure of reliability in the generalized Kuder-Richardson sense. McDonald (1970)
Advantages
1. In AFA, problems with convergence arise only if one or more communalities are very
small but the corresponding attributes should be removed.
2. In AFA, successive approximations to the communalities apparently converge on the
correct values with little meandering.
3. AFA is invariant under corrections for attenuation (unreliability). [Glass (1966)]
If the researcher considers the Thurstone’s common factor analysis as a psychometric method
then AFA is best choice since it treats the number-of-factors question more sensibly, perhaps
weighting the basic variables of common factor analysis more in accordance with that which is
important, and is numerically better behaved.
Disadvantages
1. Unlike other methods, this method does not assume sampled cases and fixed variables.
2. There are no distribution assumptions for AFA.
Unweighted least squares (ULS) factoring:
This technique is based upon minimizing the sum of squared differences between the observed
and estimated correlation matrices, without counting the diagonal. Ordinary or Unweighted
least squares (ULS) is the algorithm that directly aims at minimizing the residuals between the
input correlation matrix and the reproduced (by the factors) correlation matrix (while diagonal
elements as the sums of communality and uniqueness are aimed to restore 1s). ULS method
can work with singular and even not positive semidefinite matrix of correlations provided the
number of factors is less than its rank, - although it is questionable if theoretically FA is
appropriate then. ULS method includes iterative eigen decomposition of the reduced
correlation matrix, like PAF, but within a more complex, Newton-Raphson optimization
procedure aiming to find unique variances at which the correlations are reconstructed
maximally. In doing so ULS appears equivalent to method called MINRES (only loadings
extracted appear somewhat orthogonally rotated in comparison with MINRES) which is known
to directly minimize the sum of squared residuals of correlations.
Advantage:
1. provides more accurate and less variable parameter estimates,
2. Gives more precise standard errors and better coverage rates.
3. If the data are independent and identically distributed, the Central Limit Theorem
suggests that least squares estimators will be approximately normally distributed.
Disadvantage:
Least squares provide "best linear unbiased estimators if the response really does have a linear
relationship with any predictors. If it does not, then least squares estimators of the data are
biased, regardless of how much data you have.
Conclusions
An article by Fabrigar, Wegener, MacCallum, and Strahan (1999) argued that if data are
relatively normally distributed, maximum likelihood is the best choice because it allows for the
computation of a wide range of indexes of the goodness of fit of the model [and] permits
statistical significance testing of factor loadings and correlations among factors and the
computation of confidence intervals. If the assumption of multivariate normality is “severely
violated” they recommend one of the principal factor methods or the “principal axis factors"
(Fabrigar et al., 1999). Other authors have argued that in specialized cases, or for particular
applications, other extraction techniques (e.g., alpha extraction) are most appropriate, but the
evidence of advantage is slim. In general, ML, PCA, or PAF will give you the best results,
depending on whether your data are generally normally-distributed or significantly non-normal,
respectively.
Table showing applicability of the different factor estimation
methods across different distributions
Distribution sample variables Method
Uniform distribution all all PC
Normal distribution all small ML
Large all PC
all small PC and PAF
Gamma Distribution
large large ML
Exponential small small PAF
distribution Large all PC
small small ML then PAF
Laplace distribution
Large large PC
References
1. Methods and formulas for Factor Analysis...
https://support.minitab.com/en-us/minitab/18/help-and-how-to/modeling-statistics/
multivariate/how-to/factor-analysis/methods-and-formulas/methods-and-formulas/
2. A Comparison of Principal Component Analysis, Maximum Likelihood and the Principal
Axis in Factor Analysis by Onyekachi Akuoma Mabel, Olanrewaju Samuel Olayemi
3. Fabrigar, L.R., Wegener,D.T., MacCallum, R.C., and Strahan, E. J. (1999). Evaluating the
use of exploratory factor analysis in psychological research. Psychological Methods, 4(3),
272-299.
4. Maximum likelihood estimation of factor analysis using the ecme algorithm with
complete and incomplete data. [Chuanhai Liu and Donald B. Rubin]
5. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3732178/
6. https://www.statisticssolutions.com/free-resources/directory-of-statistical-analyses/
factor-analysis/
7. https://www.statisticssolutions.com/exploratory-factor-analysis/
8. https://www.koreascience.or.kr/article/JAKO201915658233382.pdf
9. https://conservancy.umn.edu/bitstream/handle/11299/107735/v14n1p029.pdf?
sequence=1&isAllowed=y
10. itl.nist.gov/div898/handbook/eda/section3/eda3652.htm
Group contributions
MEMBER CONTRIBUTIONS
Principle component Analysis
Theresa Namuyige
Principle factor Analysis
Principle Axis Factor Analysis
Nakalanzi Victoria
Maximum-likelihood Factor Analysis
Ibrahim Kidubule Unweighted least-squares factor analysis
Alpha factor analysis
Image component analysis
Awino Diana
Harris component analysis
