Program Evaluation

Ryan Rominger, Ph.D., PCLC

Associate Chair, Center for Leadership Studies & Organizational Research
Agenda

1. Introductions
2. CDC Program Evaluation Graphic as Overview
3. Transition to DBL Software & Program Evaluation
4. Conclusions/Question & Answers
Main text for this presentation:

Royse, D., Thyer, B. A., & Padgett, D. K. (Eds.). (2016). Program evaluation: An
introduction to an evidence-based approach (6th ed.). Boston, MA: Cengage.
E-Book version.
Also recommend:

Chyung, S. Y., Wisniewski, A., Inderbitzen, B., & Campbell, D. (2013). An improvement- and accountability-oriented
program evaluation: An evaluation of the Adventure Scouts program. Performance Improvement Quarterly,
26(3), 87-115. (5 different types of program evaluations)

Sridharan, S., & Nakaima, A. (2011). 10 steps to making evaluation matter. Evaluation and Program Planning, 34,
135-146. (10 helpful steps)

W.K. Kellogg Foundation. (2004). Using Logic Models to Bring Together Planning, Evaluation, and Action
Logic Model Development Guide. W. K. Kellogg Foundation. Retrieved at:
https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
(Logic model development)
Framework for Program Evaluation Presentation: Public Health & CDC Model

https://www.cdc.gov/eval/index.htm
What is the context or
organization within which the
program evaluation will take
place?
Definition: Organization or Context

The context or organization is the place where the program is being held. These
may be large organizations such as the Department of Defense, or smaller
organizations such as local schools, businesses, or community health centers.
Example #1 of context or organization
within a program evaluation

Morgenlander, J., & Walker, B. (2019). Evaluation of the Duke Neurology Advanced Practice Provider
Training Program. Neurology, 92(15). 5.9-075.

There is a current and projected shortage of Neurologists to meet the need of neurology services in
the United States. The use of Advanced Practice Providers (APPs) has been suggested as a possible
way to meet the need of patient access. While there are differences in the education of Physician
Assistants (PA) and Nurse Practitioners (NP, collectively APPs), both groups have little exposure to
neurology during their initial training. In 2016, the Department of Neurology at Duke established the
first ever formalized training program for APPs in neurology. This study attempts to capture the
experience of the Program APPs prior to and during their training as well as the perceptions of the
Neurology Physician Faculty during and after their training.
Is there a context or specific organization?

Specific
Context
Organization
Example #1 of context or organization
within a program evaluation

Morgenlander, J., & Walker, B. (2019). Evaluation of the Duke Neurology Advanced Practice Provider
Training Program. Neurology, 92(15). 5.9-075.

Abstract: There is a current and projected shortage of Neurologists to meet the need of neurology
services in the United States. The use of Advanced Practice Providers (APPs) has been suggested as
a possible way to meet the need of patient access. While there are differences in the education of
Physician Assistants (PA) and Nurse Practitioners (NP, collectively APPs), both groups have little
exposure to neurology during their initial training. In 2016, the Department of Neurology at Duke
established the first ever formalized training program for APPs in neurology. This study attempts to
capture the experience of the Program APPs prior to and during their training as well as the
perceptions of the Neurology Physician Faculty during and after their training.
Is there a context or specific organization?

Specific
Context
Organization

Department of Neurology at Duke

Example #2 of context or organization
within a program evaluation

Zimmerman, M. A., Eisman, A. B., Reischl, T. M., Morrel-Samuels, S., Stoddard, S., Miller, A. L., … Rupp,
L. (2018). Youth Empowerment Solutions: Evaluation of an After-School Program to Engage Middle
School Students in Community Change. Health Education & Behavior, 45(1), 20–31.
https://doi.org/10.1177/1090198117710491

We report on an effectiveness evaluation of the Youth Empowerment Solutions (YES) program. YES
applies empowerment theory to an after-school program for middle school students. YES is an active
learning curriculum designed to help youth gain confidence in themselves, think critically about their
community, and work with adults to create positive community change. …. Our sample included 367
youth from 13 urban and suburban middle schools. …. The results support both empowerment theory
and program effectiveness.
Is there a context or specific organization?

Specific
Context
Organization
Example #2 of context or organization
within a program evaluation

Zimmerman, M. A., Eisman, A. B., Reischl, T. M., Morrel-Samuels, S., Stoddard, S., Miller, A. L., … Rupp,
L. (2018). Youth Empowerment Solutions: Evaluation of an After-School Program to Engage Middle
School Students in Community Change. Health Education & Behavior, 45(1), 20–31.
https://doi.org/10.1177/1090198117710491

We report on an effectiveness evaluation of the Youth Empowerment Solutions (YES) program. YES
applies empowerment theory to an after-school program for middle school students. YES is an active
learning curriculum designed to help youth gain confidence in themselves, think critically about their
community, and work with adults to create positive community change. …. Our sample included 367
youth from 13 urban and suburban middle schools. …. The results support both empowerment theory
and program effectiveness.
Is there a context or specific organization?

Specific
Context
Organization

13 urban and suburban middle schools

Example #3 of context or organization
within a program evaluation

Zhou, Y., Liao, J., Feng, F., Ji, M., Zhao, C., & Wang, X. (2018). Effects of a Nurse-Led Phone Follow-up
Education Program Based on the Self-efficacy Among Patients With Cardiovascular Disease. The
Journal of Cardiovascular Nursing, 33(1), E15–E23. doi: 10.1097/JCN.0000000000000414

Objectives: The aim of this study is to determine the effectiveness of an NP-FEP in improving SE
(primary outcome) and achieving goals related to cardiovascular risk (secondary outcome) for
patients with cardiovascular disease.

Study Participants: Referrals for participation (403 in total) were obtained from the Inpatient
Cardiovascular Department of Shanghai East Hospital in June and July 2013 by the study nurses, who
collected demographic data before discharge.
Is there a context or specific organization?

Specific
Context
Organization
Example #3 of context or organization
within a program evaluation

Zhou, Y., Liao, J., Feng, F., Ji, M., Zhao, C., & Wang, X. (2018). Effects of a Nurse-Led Phone Follow-up
Education Program Based on the Self-efficacy Among Patients With Cardiovascular Disease. The
Journal of Cardiovascular Nursing, 33(1), E15–E23. doi: 10.1097/JCN.0000000000000414

Objectives: The aim of this study is to determine the effectiveness of an NP-FEP in improving SE
(primary outcome) and achieving goals related to cardiovascular risk (secondary outcome) for
patients with cardiovascular disease.

Study Participants: Referrals for participation (403 in total) were obtained from the Inpatient
Cardiovascular Department of Shanghai East Hospital in June and July 2013 by the study nurses, who
collected demographic data before discharge.
Is there a context or specific organization?

Specific
Context
Organization

Inpatient Cardiovascular Department of

Shanghai East Hospital
Upstream Constituents
Definition: Upstream Constituents

Constituents are the individuals and/or groups which are involved with an organization
and/or program. Constituents include those who run the organization, those who
implement programs, and those who benefit from the programs.

UPSTREAM CONSTITUENTS are those who make the decisions. They are the leaders of
the organization, including Board of Directors members. Sometimes those who
implement the program are also considered upstream constituents.
Example #1 of a program evaluation with
clearly defined upstream constituents.

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program: Outcomes
After One Year. Academic Medicine, 93(3), 430–434. doi: 10.1097/ACM.0000000000001825

“For program directors and clinical faculty members to optimally assess residents’ readiness for independent practice,
they need firsthand data on resident performance within the context of patient encounters.”

“When the program began in 2013, eight faculty coaches were recruited based on their teaching evaluations,
application to the program, and interviews with residency leaders. The program has subsequently expanded to 10
coaches, a coaching director (C.E.R.), and an associate director. Each coach is assigned 3 to 4 pediatric residents from
each training level for a total of 10 to 11 residents. Each coach receives 10% salary support for their coaching efforts
(the Department of Pediatrics funds the coaching program).”
Who were the upstream constituents?

Program
Residents
Directors

Department of
Students Pediatrics
Example #1 of a program evaluation with
clearly defined upstream constituents.

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program: Outcomes
After One Year. Academic Medicine, 93(3), 430–434. doi: 10.1097/ACM.0000000000001825

“For program directors and clinical faculty members to optimally assess residents’ readiness for independent practice,
they need firsthand data on resident performance within the context of patient encounters.”

“When the program began in 2013, eight faculty coaches were recruited based on their teaching evaluations,
application to the program, and interviews with residency leaders. The program has subsequently expanded to 10
coaches, a coaching director (C.E.R.), and an associate director. Each coach is assigned 3 to 4 pediatric residents from
each training level for a total of 10 to 11 residents. Each coach receives 10% salary support for their coaching efforts
(the Department of Pediatrics funds the coaching program).”

Those who fund programs are always “UPSTREAM” as they make the decision to
continue, alter, or discontinue the program under review!
Who were the upstream constituents?

Program
Residents
Directors

Department of
Students Pediatrics
Who were the upstream constituents?

What about FACULTY?

“eight faculty coaches were recruited” and “expanded to 10 coaches, a coaching director (C.E.R.), and an associate
director.”

YES, these individuals could be considered part of the upstream constituents

as they are helping administer (in this case, teach in) the program. However,
those administering the program might or might not be making decisions about
continuing, altering, or discontinuing the program.
Downstream Constituents
Definition: Downstream Constituents

Constituents are the individuals and/or groups which are involved with an organization
and/or program. Constituents include those who run the organization, those who
implement programs, and those who benefit from the programs.

DOWNSTREAM CONSTITUENTS are those who benefit from the program. They are the
individuals who are directly served, the families of those individuals, as well as potential
communities within which the program is in effect which may be altered due to the
program.
Example #1 of a program evaluation with
clearly defined downstream constituents.

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program: Outcomes
After One Year. Academic Medicine, 93(3), 430–434. doi: 10.1097/ACM.0000000000001825

“Program development: The Stanford Pediatric Residency Coaching Program, which is based on the conceptual
frameworks of reflective practice, self-determination theory, and lifelong learning and goal setting, began in 2013. We
developed a new conceptual model for coaching in medicine; that is, the coaching itself constitutes a continuous
improvement cycle in which both the coach and resident are active participants”

“All residents are assigned a dedicated faculty coach who coaches or mentors them in different clinical settings
throughout their years of training.”
Who were the downstream constituents?

Program
Residents
Directors

Department of
Students Pediatrics
Example #1 of a program evaluation with
clearly defined downstream constituents.

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program: Outcomes
After One Year. Academic Medicine, 93(3), 430–434. doi: 10.1097/ACM.0000000000001825

“Program development: The Stanford Pediatric Residency Coaching Program, which is based on the conceptual
frameworks of reflective practice, self-determination theory, and lifelong learning and goal setting, began in 2013. We
developed a new conceptual model for coaching in medicine; that is, the coaching itself constitutes a continuous
improvement cycle in which both the coach and resident are active participants”

“All residents are assigned a dedicated faculty coach who coaches or mentors them in different clinical settings
throughout their years of training.”
Who were the downstream constituents?
Residents receive the coaching

Program
Residents
Directors

Department of
Faculty
Coaches Pediatrics
Needs Assessment
Definition: Needs Assessment

A needs assessment is a study conducted by an organization in order to determine the

needs of its constituents. The needs assessment may be a formal study, such as a survey
of students, clients, or patients to gain their feedback on what might be missing in a
program or what could be better. Or, a needs assessment may be based off of existing
data, such as hospital records or reading scores for 5th grade students.

NOTE: Needs assessments are conducted PRIOR to a program implementation. It helps

the organization decide which program might be useful.
Example #1 of a program evaluation with a
needs assessment

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program: Outcomes
After One Year. Academic Medicine, 93(3), 430–434. doi: 10.1097/ACM.0000000000001825

“Problem: The ACGME requires all residency programs to assess residents on specialty-specific
milestones. Optimal assessment of competence is through direct observation of performance in
clinical settings, which is challenging to implement….”

“In a March 2013 needs assessment of the Stanford Pediatric Residency Program, residents reported
receiving insufficient feedback and written evaluations that did not always align with the feedback they
did receive. We recognized the opportunity to fundamentally change our approach ….”
Was there a needs assessment conducted?

Yes No
Example #1 of a program evaluation with a
needs assessment

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program: Outcomes
After One Year. Academic Medicine, 93(3), 430–434. doi: 10.1097/ACM.0000000000001825

“Problem: The ACGME requires all residency programs to assess residents on specialty-specific
milestones. Optimal assessment of competence is through direct observation of performance in
clinical settings, which is challenging to implement….”

“In a March 2013 needs assessment of the Stanford Pediatric Residency Program, residents reported
receiving insufficient feedback and written evaluations that did not always align with the feedback they
did receive. We recognized the opportunity to fundamentally change our approach ….”
Was a needs assessment conducted?

Yes No

In a March 2013 needs assessment of the

Stanford Pediatric Residency Program….
What constitutes a “program”
within a program evaluation?
Definition: Program

A program is an organized intervention (usually) created for a specific purpose

within an organization or community. A program is supported by upstream
constituents by their devoting resources to the creation and execution of the
program, and is utilized by downstream constituents who partake in the
program and benefit from the program activities. Programs contain within them
all aspects of the program (inputs, activities, outputs, and outcomes), which are
mapped within a Logic Model.
Definition: Logic Model

A logic model is a tool for identifying the processes and components that lead to the
proposed program outcomes. More precisely, a logic model is a diagram, generally a
one-page diagram, describing how the program should work conceptually or in theory to
achieve the desired outcomes for participants. That is, it points out the program’s inputs,
activities, outputs, and outcomes that visually show “how hypothesized processes will
lead to accomplishing an initiative’s objectives” (Herranz, 2010, p. 57).

Royse, D., Thyer, B. A., & Padgett, D. K. (Eds.). (2016). Program evaluation: An introduction to an evidence-based
approach (6th ed.). Boston, MA: Cengage. E-Book version.
Logic Model

Inputs: all of the resources, including money, trainings, staff, facilities, etc.

Activities: planned events or tasks of the program (what the facilitators DO)

Outputs: countable products (ex., number of sessions, pamphlets, etc.)

Outcomes: positive changes (ex., client or student changes) as a result of the

program
Royse, D., Thyer, B. A., & Padgett, D. K. (Eds.). (2016). Program evaluation: An introduction to an evidence-based
approach (6th ed.). Boston, MA: Cengage. E-Book version.
Example #1 of a program.

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program:
Outcomes After One Year. Academic Medicine, 93(3), 430–434. doi:
10.1097/ACM.0000000000001825

“Program development: The Stanford Pediatric Residency Coaching Program, which is

based on the conceptual frameworks of reflective practice, self-determination theory, and
lifelong learning and goal setting, began in 2013. We developed a new conceptual model
for coaching in medicine; that is, the coaching itself constitutes a continuous
improvement cycle in which both the coach and resident are active participants”
Example #1 of a program.

Rassbach, C., & Blankenburg, R. (2018). A Novel Pediatric Residency Coaching Program:
Outcomes After One Year. Academic Medicine, 93(3), 430–434. doi:
10.1097/ACM.0000000000001825

“Program development: The Stanford Pediatric Residency Coaching Program, which is

based on the conceptual frameworks of reflective practice, self-determination theory, and
lifelong learning and goal setting, began in 2013. We developed a new conceptual model
for coaching in medicine; that is, the coaching itself constitutes a continuous
improvement cycle in which both the coach and resident are active participants”
Type of Program Evaluation
Royse, D., Thyer, B. A., & Padgett, D. K. (Eds.). (2016). Program evaluation: An introduction to an evidence-based
approach (6th ed.). Boston, MA: Cengage. E-Book version.
 Formative / process Summative / conclusive / outcome

Royse, D., Thyer, B. A., & Padgett, D. K. (Eds.). (2016). Program evaluation: An introduction to an evidence-based
approach (6th ed.). Boston, MA: Cengage. E-Book version.
Formative Evaluation

Used to adjust a program

Solve problems along the way

Guide the program

Assess if it is being implemented appropriately (fidelity)

Formative Evaluation: A few approaches

1. Locate Model Standards (original program manual compared to current

activities)
2. Expert Consultation (see program in expert or new light)
3. Ad Hoc Evaluation Committee (staff, board members, professionals in
community, other stakeholders, who then interview/collect data on current
program, compare to other programs, hold focus groups)
Process Evaluation

“One major difference between formative evaluation and process

evaluation is that while a formative evaluation seeks to influence
the initial development of a program, a process evaluation can
be conducted anytime during a project-even at its end.” (Royse,
Thyer, & Padgett, 2016, p. 129).
Process Evaluation Overarching Goals

1. Program description
2. Program monitoring
3. Quality assurance

Asks the question, ‘Did the program really cause the

changes in participants?’
Summative / Conclusive / Outcome Evaluation

Goals:

1. To review data on program to determine if it had the impact desired

2. To determine if resources used where ‘adequately utilized’
3. To establish if program should continue, or if another program might better meet
needs of the specific community

May collect data throughout or at the end of the program, but data is summarized and
reviewed by upstream constituents to make organizational determinations.