WORKING MEMORY: THE CONCURRENT VALIDITY OF

THE BEHAVIOR RATING INVENTORY OF EXECUTIVE

FUNCTION AND THE RELATIONSHIP BETWEEN PARENT

AND TEACHER RATINGS AND PERFORMANCE-BASED MEASURES

W
A Dissertation
IE
Submitted to the School of Graduate Studies and Research

in Partial Fulfillment of the

EV
Requirements for the Degree

Doctor of Education
PR

Ray M. Milke

Indiana University of Pennsylvania

December 2014
 UMI Number: 3665330

All rights reserved

INFORMATION TO ALL USERS

The quality of this reproduction is dependent upon the quality of the copy submitted.

In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if material had to be removed,
a note will indicate the deletion.

W
IE
UMI 3665330
Published by ProQuest LLC (2014). Copyright in the Dissertation held by the Author.
EV
Microform Edition © ProQuest LLC.
All rights reserved. This work is protected against
unauthorized copying under Title 17, United States Code
PR

ProQuest LLC.
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, MI 48106 - 1346
 Indiana University of Pennsylvania
School of Graduate Studies and Research
Department of Educational and School Psychology

We hereby approve the dissertation of

Ray M. Milke

Candidate for the degree of Doctor of Education

W
___________________ _________________________________________
Mark Staszkiewicz, D.Ed.
Professor of Educational and
IE
School Psychology, Advisor

___________________ _________________________________________
Joseph F. Kovaleski, D.Ed.
EV
Professor of Educational and
School Psychology

___________________ _________________________________________
Lynanne Black, Ph.D.
PR

Associate Professor of Educational

and School Psychology

___________________ _________________________________________
John M. Garruto, D.Ed.
Adjunct Professor, SUNY Oswego

ACCEPTED

_________________________________________ ____________________
Timothy P. Mack, Ph.D.
Dean
School of Graduate Studies and Research

ii
Title: Working Memory: The Concurrent Validity of the
Behavior Rating Inventory of Executive Function and
the Relationship Between Parent and Teacher Ratings
and Performance-Based Measures

Author: Ray M. Milke

Dissertation Chair: Dr. Mark Staszkiewicz

Dissertation Committee Members: Dr. Joseph F. Kovaleski

Dr. Lynanne Black
Dr. John M. Garruto

The primary purposes of this study were to investigate

W
the concurrent validity of the Behavior Rating Inventory of

Executive Function (BRIEF) with respect to the Woodcock-

IE
Johnson III Tests of Cognitive Abilities (WJ-III Cog),

examine both teacher and parent ratings of working memory

EV
of at-risk students, and investigate parent and teacher

differences in the ratings of male and female students.

PR

More specifically, this study examined whether the BRIEF

rating scale can capture and accurately assess the complex

cognitive processes of working memory and yield similar

results to the established and accepted paradigm of working

memory assessment utilized by the Working Memory Cluster of

the WJ-III Cog. The results suggest that parents and

teachers were inconsistent and rather dissimilar in their

ratings, reflecting a lack of agreement of the level of

impairment demonstrated by the students. The results are

consistent with various studies that have demonstrated that

iii
the BRIEF assesses different constructs than that of

performance-based measures (Anderson et al., 2002; Bakar et

al., 2011; Bodnar et al., 2007; Conklin et al. 2008; Mahone

et al., 2002; McAuley et al., 2010; Rosenthal et al., 2005;

Toplak, et al., 2013; Vriezen & Pigott, 2002) and that

these results appear to be true for both parents and

teachers ratings. Until we have a better understanding of

the specific constructs measured by the BRIEF, caution

should be exercised when interpreting the results garnered

W
from this instrument.
IE
EV
PR

iv
 ACKNOWLEDGEMENTS

Completing this dissertation has been a long arduous

task and would not have been possible without the love and

support of the most important people in my life. First and

foremost I would like to thank God for providing me the

strength and perseverance to complete this journey and for

blessing me with a loving family, loyal friends, and

outstanding colleagues whose unwavering support has

W
sustained me.
IE
I would like to thank my parents Dr. Ray M. Milke and

Barbara M. Milke for all of their love, guidance, wisdom,

EV
and patience in raising me. It was your love and

upbringing that built the foundation from with which I

could grow, develop, and achieve this goal. I can only hope
PR

to provide for my children what you have provided and given

to me. I think that having both of your children achieve

their doctoral degrees in the same month is a testament to

what wonderful parents you are.

To my wife Antonia, I finally did it!!!!! You have

been my biggest supporter, my rock, my inspiration, my

blessing, my cheerleader, my motivator. I could not have

done this without you. I hope I can someday make up for

all of the sacrifices you have made, the selflessness you

v
have demonstrated, the responsibilities you have taken on,

the things you have put up with, and the stress you have

endured. You are inspirational in the degree to which you

have honored our wedding vows and promises to each other.

God has truly blessed me by allowing me to be married to

and spend my life with you.

I would like to thank my children Ramon and Rachel for

their unwavering support, understanding, and encouragement.

The way you two supported my countless days and nights in

W
the den, helping your mother around the house, bringing me
IE
snacks, and your understanding and acceptance of me missing

so many of your activities is a testament to your character

EV
and devotion to family, and demonstrates what truly

wonderful young adults you have become. I am so proud of

you. Thank you.

PR

I would also like to take this opportunity to thank my

dissertation committee for all of their support and efforts

in helping me complete this dissertation. Dr. Black, your

sustained encouragement, conceptualization of my project,

and assistance in restructuring and editing my dissertation

was invaluable. Thank you for being in my corner. Dr.

Kovaleski, thank you for supporting me throughout my

doctoral program. I am eternally grateful for your guidance

and assistance when I came across some pretty significant

vi
bumps in the road and thought that I might not be able to

finish my dissertation. Thank you for helping me overcome

those set-backs. Dr. Garruto, your expertise was

invaluable. I truly appreciate our in-depth discussions and

the challenges, inquiries, and perspectives that you

presented to me. I am proud to say that I was the first

person for whom you served on a dissertation committee.

To my chairperson, Dr. Staszkiewicz. I'm not sure I

can quite articulate the level of my gratitude and the

W
respect that I have for you. Without you as my chairperson
IE
I'm not sure I would have been able to complete my

dissertation. Your support and level of commitment are

EV
inspirational. I was pretty much at the point of

hopelessness thinking that I would not be able to finish my

dissertation. Then, while I was at a very low point and

PR

ready to give up, I received what I consider to be a

blessing; Dr. Kovaleski suggested I contact you. After

meeting you I regained a newfound hope and the possibility

to finish my dissertation was renewed. I can't begin to

thank you enough for your time and for all of the e-mails,

phone calls, and personal meetings. When I was confused you

provided clarity; when I was panicking you calmed me down;

when I lacked understanding you took the time to teach me.

You were the guiding force that allowed me to complete my

vii
dissertation. I hope that as the years go by you may read

this again and remember the wonderful impact you had on

making a difference and changing one doctoral student’s

life. Thank you from the bottom of my heart.

W
IE
EV
PR

viii
 TABLE OF CONTENTS

Chapter Page

I INTRODUCTION...................................1

Statement of Problem and Need for the Study....4

Hypothesis and Research Questions..............8
Definitions...................................11
Assumptions...................................13
Limitations and Delimitations.................14
Summary.......................................14

II REVIEW OF LITERATURE..........................16

W
Overview......................................16
Executive Function............................17
Developmental Trajectories....................20
IE
Information Processing Model..................24
Working Memory................................27
Neuroimaging Evidence of Working Memory.......32
Assessment of Working Memory..................41
EV
Working Memory and Academic Achievement.......44
Cattell-Horn-Carroll (CHC) Theory of
Cognitive Abilities...........................51
Woodcock-Johnson III Tests of
Cognitive Abilities (WJ-III Cog)..............52
PR

Working Memory, Attention, and Inhibition.....54

The Behavior Rating Inventory of
Executive Function (BRIEF)....................58
Factor Analysis of the BRIEF..................60
The BRIEF and Related Studies.................61
Measurement of Working Memory and the
BRIEF.........................................69
Parent and Teacher Rating Scales..............71
Summary.......................................76

III METHODOLOGY...................................78

Introduction..................................78
Population and Setting of the Study...........78
Sample........................................79
Assignment....................................80
Measurement...................................80
Procedures....................................83

ix
Chapter Page

Internal Threats..............................85
External Threats..............................85
Design........................................86
Participants..................................89
Procedures for Collecting Data................90
Statistical Analysis..........................92
Summary.......................................93

IV RESULTS.......................................95

Summary......................................101

V DISCUSSION...................................103

Implications for School Psychologists........105

W
Future Research..............................109
Considerations...............................111
Summary......................................113
IE
REFERENCES........................................115
EV
PR

x
 List of Tables

Table Page

1 Sample Size and Average t-Scores for Scales

by Sex of Student...........................96

2 Correlation Matrix of the WJ-III Cog

and the BRIEF...............................97

3 Correlation Matrix of the WJ-III Cog

and the BRIEF by Sex.......................100

W
IE
EV
PR

xi
 CHAPTER 1

INTRODUCTION

There is a growing body of evidence that demonstrates

the importance of working memory in learning and academic

achievement (Alloway & Alloway, 2010). Clinicians, when

evaluating students for disabilities, are trained and

required to evaluate for a broad variety of constructs.

Having accurate data is important and necessary to properly

W
advocate for students who may present with learning and

behavioral challenges. However, given the breadth of such

IE
areas that can be evaluated, it is imperative that

clinicians obtain accurate data and obtain such data in the

EV

most time-effective manner. Therefore, accuracy in

assessment and diagnostics is vital to providing

PR

appropriate interventions and instruction in order to

assist students in maximizing their academic attainment.

There has been an increase in the focus on the

importance of the construct of executive functioning when

assessing both typically developing and special needs

children (Toplak, West, & Stanovich, 2013). Likewise,

there has been a continual increase in the use of rating

scales by clinicians and currently, the most commonly used

rating scale to assess executive function is the Behavior

1
Rating Inventory of Executive Function (BRIEF) (Toplak, et

al., 2013). However, both clinicians and researchers have

expressed concern that the Behavior Rating Inventory of

Executive Function, when used as a stand-alone measure, may

provide misleading data by providing or indexing data that

are inconsistent with performance-based measures (McAuley,

Chen, Goos, Schachar, & Crosbie, 2010; Toplak, Bucciarelli,

Tain, & Tannock, 2009; Toplak, et al., 2013). If the BRIEF

is used as a stand-alone assessment or as a screener for

W
various executive functions, particularly working memory, a
IE
concern should be raised as to whether it yields similar

data regarding complex cognitive processes as a direct,

EV
performance-based assessment such as the working memory

cluster of the Woodcock-Johnson-III Tests of Cognitive

Abilities (WJ-III Cog). An extensive literature review did

PR

not yield any studies to date that have directly focused or

addressed this concern by solely targeting working memory.

Several studies have examined the validity and diagnostic

utility of the BRIEF by examining direct and/or

performance-based measures of executive functioning and

comparing those results with the performance levels

obtained and reflected on the BRIEF.

Although many studies have examined the broader

aspects of executive function, which include working

2
memory, none have focused specifically or directly on

working memory. For example, Mahone’s (2001) study of

validity of the BRIEF in children with Attention Deficit

Hyperactivity Disorder (ADHD) and/or Tourette Syndrome

indicated that the BRIEF index scores showed no significant

correlation with performance-based executive function or

psychoeducational measures. McAuley’s (2010) investigation

of whether the BRIEF is more strongly associated with

measures of impairment or executive function found that the

W
BRIEF indices were not associated with a student’s scores
IE
on performance-based tasks of executive function. Similar

findings regarding the differences between the performance

EV
levels on the BRIEF and performance-based measures of

executive functioning have been discussed (Anderson,

Northman, Jacobs, & Mikiewicz, 2002; Reddy, Hale, &

PR

Brodzinsky, 2011; Mahone, Martin, Kates, Hay, & Horska,

2009). In fact, Toplak et al. (2013) examined the

association between performance-based and rating scales

measures of executive function in 20 empirical studies.

They concluded that performance-based measures and ratings

scales assess different underlying constructs.

3
 Statement of the Problem and Need for the Study

Behavior rating scales such as the BRIEF appear to be

growing in popularity and clinicians are increasingly using

these instruments to make judgments regarding certain

aspects of executive functioning (Gioia, Isquith, Guy, &

Kenworthy, 2000; Della Toffalo, 2008). Moreover, the BRIEF

is now considered the most commonly used rating scale of

executive function (Toplak et al., 2013). However,

multiple scientist-practitioners in the field of assessment

W
have contended that the BRIEF may not measure these
IE
constructs as well, or may measure different constructs,

than a bona fide individual assessment designed to measure,

EV
with specificity, various aspects of executive functioning

(Anderson et.al., 2002; Della Toffalo, 2008; Mahone et al.,

2002; Mahone et.al., 2009; McAuley et al., 2010; Reddy et

PR

al., 2002).

The authors of the BRIEF indicate "working memory

capacity, whereby information is actively held ‘online’ in

the service of complex multi-step problem solving, is also

described as a key aspect of executive function” (Gioia, et

al., 2000, p.1). This definition of working memory is

similar to the accepted definition in the literature

(Baddeley, 1986, 2003, 2006; Baddeley and Hitch, 1974;

Torgesen, 1996; Norman & Shallice, 1986). The authors of

4
the BRIEF state that the behavioral descriptions that

comprise the working memory subscale items from their

assessment tool examine the ability to "hold information in

mind for the purpose of completing a task; stay with, or

stick to, an activity". (Gioia et al., 2000, p.2). They

also report that “two of the scales, Working Memory and

Inhibit, are clinically useful in differentiating the

diagnostic subtypes of Attention-Deficit/Hyperactivity

Disorder (ADHD)” (Gioia et al., 2000, p.1). The definition

W
of holding information in the mind and the function of
IE
differentiating subtypes of ADHD appear to employ only

observable behavioral indicators of the attention

EV
components of working memory, and not the process,

capacity, or proficiency of working memory. As such, the

construct validity of the BRIEF working memory subscale

PR

seems questionable because it appears to be confounded by

predominantly centering on the construct of attention. The

BRIEF seems to focus on attention and the diagnostic

classification/clarification of ADHD subtypes and

convergent validity with respect to the Child Behavior

Checklist (CBCL), Conners’ Rating Scales, and the Behavior

Assessment System for Children (BASC), but does not appear

to offer a clear focus on the mechanics, processes, or

capacity/output of working memory.

5
 Although these rating scale measures are commonly used

and demonstrate good psychometric properties (Achenbach &

Rescorla, 2001; Reynolds & Kamphaus, 1992), they are

designed to assess the behavioral output/symptoms of ADHD

and other emotional and behavioral disorders. However,

they are not intended to assess the cognitive underpinnings

of children’s executive function abilities and do not,

therefore, directly assess theoretical cognitive processes

and constructs proposed to explain the development of ADHD

W
(Jarret, Riccio, & Siekierski, 2005). Moreover, depending
IE
on which tool is used to evaluate working memory and the

potential for differences in the resultant conclusions, the

EV
learning implications and resultant interventions could

dramatically differ. This suggests that the results and

assessment conclusions drawn about working memory from the

PR

BRIEF may differ from the direct measures of working memory

that are applicable in the educational environment. In

fact, multiple scientist-practitioners have observed that

there are often significant differences in the parent

versus teacher ratings of students, and little

correspondence between BRIEF executive function and working

memory scores and performance on actual measures of

executive functioning and working memory (Anderson et.al.,

2002; Della Toffalo, 2008, Hale & Fiorello, 2004, Mahone et

6
al., 2009; McAuley et al., 2010; Reddy et al, 2002; Toplak

et al., 2013).

There is a growing body of evidence that demonstrates

the importance of working memory in learning and academic

achievement (Alloway & Alloway 2010). In fact, several

studies have indicated that working memory is one of the

strongest predictors of academic success (Alloway &

Alloway, 2010; Gathercole & Pickering, 2000; Westberg,

Hirvikoski, Forssberg, & Klingberg, 2004). Contemporary

W
evidence suggests that working memory may be one of the
IE
strongest predictors of a student's contemporary and long-

term academic outcomes (Gathercole Pickering, Ambridge &

EV
Wearing 2004; Jarvis & Gathercole, 2003). Moreover,

several studies have found that deficiencies in working

memory negatively impact learning and acquisition of core

PR

academic subjects such as English, mathematics, and science

(Alloway, Gathercole, Adams, Willis, Eaglen, & Lamont,

2005; Bull & Scerif, 2001; Gathercole, Pickering, Knight, &

Stegman, 2004; Jarvis & Gathercole, 2003). Therefore,

accuracy in assessment and diagnostics is vital to

providing appropriate interventions and instruction in

order to assist students in maximizing their academic

attainment.

7
 This study was designed to investigate concurrent

validity between the Behavior Rating Inventory of Executive

Function (BRIEF) and the Woodcock-Johnson-III Tests of

Cognitive Abilities. Parent and teacher ratings of working

memory on the BRIEF were compared to the Woodcock-Johnson-

III Tests of Cognitive Abilities Working Memory Cluster. An

examination of the differences between male and female

students was also conducted. This study, in other words,

examined whether a behavior rating scale (BRIEF) can

W
capture and assess the same complex cognitive process of
IE
working memory as a performance-based measure.

Hypothesis and Research Questions

EV
It was hypothesized that the standard to which the

construct of working memory is measured by a rating scale

such as the BRIEF would not be similar to that of a direct,

PR

performance-based measure such as the Woodcock-Johnson-III

Tests of Cognitive Abilities Working Memory Cluster. When

using subjective observational rating scales, which is the

structural basis and design of the BRIEF, inherent problems

arise such as objectivity of the rater, sensitivity of the

rater to the behavior in question, emotional status of the

rater, lack of insight into the student’s level of

motivation, contextual effects, and differences in the way

different observers/raters judge or interpret behavior

8
(Barkley, 2006). Assessing a complex cognitive construct

solely through the manifestation of observed behavior, and

making the assumption of cause and effect by assuming

multiple cognitive processes directly result in the same

behavior across students is questionable. Given the

aforementioned concerns regarding rating scales, the

prediction here was that working memory scores obtained on

the Behavior Rating Inventory of Executive Function would

reflect lower levels of working memory efficiency than the

W
scores obtained on the Woodcock-Johnson-III Tests of
IE
Cognitive Abilities (WJ-III Cog) Working Memory Cluster.

The present study of archival data attempted to address

EV
the following research questions and hypotheses:

1. What is the concurrent validity of the BRIEF with

respect to the WJ-III Cog Working Memory Cluster?

PR

Hypothesis 1a: Working memory, as rated by the

teacher on the BRIEF, will have significant, but

moderate correlation with the WJ-III Cog

Working Memory Cluster.

Hypothesis 1b: Working memory, as rated by the

parent on the BRIEF, will have significant, but

weak correlation with the WJ-III Cog Working

Memory Cluster.

9
 Hypothesis 1c: Correlations between working

memory measured by the teacher on the BRIEF and

the Working Memory Cluster measured by the WJ-III

Cog will be significantly higher than the

correlations between working memory measured by

the parent on the BRIEF and the Working Memory

Cluster measured by the WJ-III Cog.

2. Does the concurrent validity of the BRIEF vary as a

W
function of the sex of the child?

Hypothesis 2a: IECorrelations between the Working

Memory Cluster on the WJ-III Cog and the BRIEF

EV
Parent Form will be lower for male students than

female students. Parents’ ratings on the BRIEF

will reflect lower levels of impairment for

PR

female students than male students and will be

more strongly correlated with the WJ-III Cog.

Hypothesis 2b: Correlations between the Working

Memory Cluster on the WJ-III Cog and the BRIEF

Teacher Form will be lower for male students than

female students. Teachers’ ratings on the BRIEF

will reflect lower levels of impairment for

female students than male students and will be

more strongly correlated with the WJ-III Cog.

10
 DEFINITIONS

Executive Functions: An organized system of inter-

related abilities that facilitate purposeful, goal-

oriented, problem-solving behaviors (Lezak, 1995, Ardila,

2008, Jurado & Roselli, 2007).

BRIEF: Behavior Rating Inventory of Executive

Function is a questionnaire for parents and teachers of

school age children that assesses executive function

behaviors in the home and school environments (Gioia, et

W
al., 2000).

WJ-III-Cog:
IE
Woodcock Johnson-III Tests of Cognitive

Abilities is a norm-referenced, individually administered

EV
battery of tests that measures general intellectual ability

as well as specific cognitive abilities (Woodcock, McGrew,

& Mather, 2001).

PR

Working Memory: A cognitive process with a primary

function of facilitating and enhancing the capacity of

encoding, storage, and retrieval functions which are

essential for learning and higher level processing of

information through the management, manipulation, and

transformation of information drawn from short-term and

long-term memory (Dehn, 2008). Simply put, “working memory

refers to the ability to hold information in immediate

11
awareness while performing a mental operation on the

information” (Woodcock, et al., 2001).

Auditory Working Memory: The ability to hold auditory

information in immediate awareness, divide the information

into two groups, and shift resources to the new ordered

sequence (Woodcock, et al., 2001).

Inhibit (BRIEF): “The ability to inhibit, resist, or

not act on impulse and the ability to stop one's own

behavior at the appropriate time” (Gioia, et al., 2000).

W
Shift (BRIEF): “The ability to move freely from one
IE
situation, activity, or aspect of a problem to another as

the circumstances command” (Gioia, et al., 2000).

EV
Emotional Control (BRIEF): The ability to modulate

emotional responses (Gioia, et al., 2000).

Initiate (BRIEF): The ability to begin a task or

PR

activity, as well as independently generate ideas,

responses, or problem-solving strategies (Gioia, et al.,

2000).

Working Memory (BRIEF): The ability to hold

information in the mind for the purpose of completing a

task (Gioia, et al., 2000).

Plan/Organize (BRIEF): The ability to manage current

and future-oriented task demands (Gioia, et al., 2000).

Organization of Materials (BRIEF): Orderliness of

12