Newsom

Psy 521/621 Univariate Quantitative Methods, Fall 2023 1

Reliability Analysis Example

SPSS
This example comes from a study on health care coverage among low-income mothers conducted by Karen
Seccombe. 1 There were several items that were negatively scored that needed to be rescored, most on a 7-
point scale.

Q11 How satisfied with plan overall

Q13 Recommend health plan
Q14 Intend to switch plans
Q15 Days waiting for Routine Care
Q15b Days waiting for Minor Illness or Injury
Q15c Days waiting for Chronic Condition
Q15d Days waiting for Urgent Care

Syntax
get file='c:\jason\spsswin\usp532\ahca.sav'.

*some items needed to be reverse coded--this may or may not be necessary on the homework problem.
recode q11 (1=7) (2=6) (3=5) (4=4) (5=3) (6=2) (7=1) into q11r.
recode q15 (1=7) (2=6) (3=5) (4=4) (5=3) (6=2) (7=1) into q15r.
recode q15b (1=7) (2=6) (3=5) (4=4) (5=3) (6=2) (7=1) into q15br.
recode q15c (1=7) (2=6) (3=5) (4=4) (5=3) (6=2) (7=1) into q15cr.
recode q15d (1=7) (2=6) (3=5) (4=4) (5=3) (6=2) (7=1) into q15dr.
recode q13 (1=4) (2=3) (3=2) (4=1) into q13r.

frequencies vars=q11r q13r q14 q15r to q15dr

/histogram=normal.

descriptives vars=q11r q13r q14 q15r to q15dr

/statistics=default variance skewness kurtosis
/missing=listwise.

reliability vars=q11r q13r q14 q15r to q15dr

/scale(satisf)=q11r q13r q14 q15r to q15dr
/statistics=correlations scale
/summary=means corr total.
Menus
Analyze  Scale  Reliability Analysis
Drag over the desired variables. Click the Statistics button and check “Item,” “Scale,” and “Scale-if-item-
deleted” under Descriptives; check “Correlations” under Inter-Item, and “Means and Correlations” under
Summaries
It is always a good idea to examine the distributions of individual variables, but I have omitted the histograms
output here to save paper.

1
Seccombe, K., Hoffman, K., Hartley, H., Newsom, J., Marchand, G. C., Albo, C., Gordon, C., Zaback, T., Lockwood, R., & Pope, C. (2007). The
aftermath of Welfare Reform: Health, health insurance, and access to care among families leaving TANF in Oregon. Journal of Family Issues, 28, 151-
181.
Newsom
Psy 521/621 Univariate Quantitative Methods, Fall 2023 2

R
>#clear console
> cat("\014")
> #clear active frame from previous analyses
> rm(d)
> library(haven)
> d = read_sav("c:/jason/spsswin/usp532/ahca.sav")

> #make sure all variables are numeric

> d$Q11 <- as.numeric(d$Q11)
> d$Q13 <- as.numeric(d$Q13)
> d$Q14 <- as.numeric(d$Q14)
> d$Q15 <- as.numeric(d$Q15)
> d$Q15B <- as.numeric(d$Q15B)
> d$Q15C <- as.numeric(d$Q15C)
> d$Q15D <- as.numeric(d$Q15D)
> #reverse score items
> library(dplyr)
>
> #define 8 at missing for these variables
> d$Q15[d$Q15 == 8] <- NA
> d$Q15B[d$Q15B == 8] <- NA
> d$Q15C[d$Q15C == 8] <- NA
> d$Q15D[d$Q15D == 8] <- NA
>
Newsom
Psy 521/621 Univariate Quantitative Methods, Fall 2023 3

> d$Q11r <- recode(d$Q11, '7' = 1, '6' = 2, '5' = 3, '4' = 4, '3' = 5, '2' = 6, '1' = 7)
> d$Q15r <- recode(d$Q15, '7' = 1, '6' = 2, '5' = 3, '4' = 4, '3' = 5, '2' = 6, '1' = 7)
> d$Q15Br <- recode(d$Q15B,'7' = 1, '6' = 2, '5' = 3, '4' = 4, '3' = 5, '2' = 6, '1' = 7)
> d$Q15Cr <- recode(d$Q15C, '7' = 1, '6' = 2, '5' = 3, '4' = 4, '3' = 5, '2' = 6, '1' = 7)
> d$Q15Dr <- recode(d$Q15D, '7' = 1, '6' = 2, '5' = 3, '4' = 4, '3' = 5, '2' = 6, '1' = 7)
> d$Q13r <- recode(d$Q13, '4' = 1, '3' = 2, '2' = 3, '1' = 4)

> #always check to see that recode worked as expected

> #describe(d)

> #create data frame with just scale variables

> dsatis <- subset(d, select=c(Q11r,Q13r,Q14,Q15r,Q15Br,Q15Cr,Q15Dr))
> #listwise deletion--psych alpha procedure has odd use of pairwise deletion
> dsatis <- dsatis[complete.cases(dsatis), ]
> describe(dsatis)
vars n mean sd median trimmed mad min max range skew kurtosis se
Q11r 1 155 5.26 1.72 6 5.51 1.48 1 7 6 -1.10 0.29 0.14
Q13r 2 155 3.16 1.00 3 3.32 1.48 1 4 3 -1.05 -0.01 0.08
Q14 3 155 3.37 1.21 4 3.46 1.48 1 5 4 -0.60 -0.59 0.10
Q15r 4 155 5.43 1.44 6 5.65 1.48 1 7 6 -1.45 1.95 0.12
Q15Br 5 155 6.28 0.90 6 6.42 1.48 1 7 6 -2.25 9.16 0.07
Q15Cr 6 155 6.17 1.22 7 6.39 0.00 1 7 6 -2.24 6.28 0.10
Q15Dr 7 155 6.83 0.54 7 6.96 0.00 3 7 4 -4.14 21.00 0.04
> #use baseR function cor to get matrix of correlations of all vars
> cor(dsatis)
Q11r Q13r Q14 Q15r Q15Br Q15Cr Q15Dr
Q11r 1.00000000 0.79232345 0.231479257 0.09944457 0.188189396 0.14683192 0.2179690
Q13r 0.79232345 1.00000000 0.244745428 0.06896356 0.165548599 0.07350864 0.1853269
Q14 0.23147926 0.24474543 1.000000000 -0.01588291 -0.007066063 0.05042693 0.1491950
Q15r 0.09944457 0.06896356 -0.015882905 1.00000000 0.423925272 0.39995423 0.2729916
Q15Br 0.18818940 0.16554860 -0.007066063 0.42392527 1.000000000 0.60604222 0.4551304
Q15Cr 0.14683192 0.07350864 0.050426935 0.39995423 0.606042222 1.00000000 0.3836887
Q15Dr 0.21796900 0.18532694 0.149194975 0.27299159 0.455130387 0.38368869 1.0000000
> #use the psych package to get alpha and item statistics
> library(psych)
> alpha(dsatis)
Reliability analysis
Call: alpha(x = dsatis)
raw_alpha std.alpha G6(smc) average_r S/N ase mean sd median_r
0.65 0.69 0.75 0.24 2.3 0.042 5.2 0.68 0.19
95% confidence boundaries
lower alpha upper
Feldt 0.56 0.65 0.73
Duhachek 0.57 0.65 0.73
Reliability if an item is dropped:
raw_alpha std.alpha G6(smc) average_r S/N alpha se var.r med.r
Q11r 0.59 0.64 0.65 0.23 1.8 0.050 0.035 0.19
Q13r 0.58 0.65 0.66 0.24 1.9 0.051 0.033 0.22
Q14 0.67 0.72 0.77 0.30 2.6 0.041 0.044 0.22
Q15r 0.64 0.68 0.74 0.26 2.1 0.043 0.047 0.19
Q15Br 0.59 0.63 0.68 0.22 1.7 0.049 0.039 0.19
Q15Cr 0.60 0.64 0.70 0.23 1.8 0.048 0.041 0.19
Q15Dr 0.62 0.64 0.72 0.23 1.8 0.047 0.054 0.17
Item statistics
n raw.r std.r r.cor r.drop mean sd
Q11r 155 0.71 0.64 0.64 0.45 5.3 1.72
Q13r 155 0.65 0.61 0.60 0.50 3.2 1.00
Q14 155 0.41 0.40 0.21 0.17 3.4 1.21
Q15r 155 0.56 0.54 0.41 0.30 5.4 1.44
Q15Br 155 0.62 0.68 0.64 0.49 6.3 0.90
Q15Cr 155 0.61 0.64 0.58 0.41 6.2 1.22
Q15Dr 155 0.53 0.64 0.53 0.45 6.8 0.54
Non missing response frequency for each item
1 2 3 4 5 6 7 miss
Q11r 0.06 0.03 0.08 0.08 0.14 0.37 0.24 0
Q13r 0.12 0.06 0.34 0.47 0.00 0.00 0.00 0
Q14 0.12 0.11 0.22 0.40 0.15 0.00 0.00 0
Q15r 0.04 0.03 0.01 0.10 0.19 0.43 0.19 0
Q15Br 0.01 0.01 0.00 0.01 0.12 0.39 0.47 0
Q15Cr 0.03 0.00 0.01 0.03 0.14 0.27 0.52 0
Q15Dr 0.00 0.00 0.01 0.01 0.01 0.10 0.87 0
Newsom
Psy 521/621 Univariate Quantitative Methods, Fall 2023 4

Write-up
Internal reliability of the 7-item Health Care Satisfaction Scale was investigated using Cronbach's alpha. 2
Results indicated that the standardized alpha for the total scale was equal to .71. Examination of individual
item statistics suggested that elimination of one of the items, "Days waiting for routine care," would increase
the reliability of the scale. Subsequent analysis [omitted above] resulted in a standardized alpha for the six-
item scale that was low, although acceptable by some sources (e.g., Furr & Bacharach, 2013), α = .72, and
might be improved in future research with additional items. 3

Comment: There are few important points to keep in mind. What is an acceptable alpha values is debatable.
Many sources give .7 as minimally acceptable, 4 but, in many cases, the reliability of a measure can be
improved for a scale with an alpha of approximately this value. Note also that I removed only one item, and
removing multiple items on a step is potentially problematic, because it involves multiple changes in the scale.
In the end, the more changes that are made, the greater the risk that the final result will not be replicated in
another sample. Moreover, the change made to the scale was based on empirical findings (post hoc) rather
than guided by theory (a priori), and reviewers may criticize post hoc changes to a scale, particularly if there
are many. It may be unwise to eliminate items without some theoretical explanation for why the item does not
perform well, and it may be quite reasonable to decide to retain items purely for theoretical reasons (i.e., even
though the results suggest the item may be dropped). Preferably, a process like this one would be used with a
smaller pilot data set that would later be replicated with a larger study intended to be published. It is also
important to keep in mind that items may perform poorly, because they are related to another construct rather
than because they are necessary "bad" items. Factor analytic approaches are better suited to examining
whether there are multiple constructs underlying a set of items (I topic will address briefly next term).

References
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.

2 It is not necessary any longer to cite Cronbach's paper, as the statistic is widely known, but the usual citation is to his
seminal paper published in 1951.
3 If items have different variances, the raw alpha will be smaller than the standardized alpha. It is fine to report the

standardized alpha, but, if there is much difference, it may be better to create any scale composites for use in further
analyses using standardized items.
4 The .7 standard seems to be traced to Nunnally (1978), who was actually stating .7 may not be high enough: "what a

satisfactory level of reliability is depends on how a measure is being used. In the early stages of research . . . one saves
time and energy by working with instruments that have only modest reliability, for which purpose reliabilities of .70 or
higher will suffice. . . . In contrast to the standards in basic research, in many applied settings a reliability of .80 is not
nearly high enough." (pp. 245-246)