International Education Studies Vol. 3, No.

2; May 2010

The Role for an Evaluator: A Fundamental Issue for Evaluation

of Education and Social Programs
Heng Luo
Department of Instructional Design, Development and Evaluation, Syracuse University
330 Huntington Hall, Syracuse University, Syracuse, NY, 13244, USA
E-mail: [email protected]
Abstract
This paper discusses one of the fundamental issues in education and social program evaluation: the proper role
for an evaluator. Based on respective and comparative analysis of five theorists’ positions on this fundamental
issue, this paper reveals how different perspectives on other fundamental issues in evaluation such as value,
methods, use and purposes can result in different roles for evaluators, and how such difference can affect
evaluators’ responsibilities in different stages of an evaluation. Then the paper proposes its own resolution of the
issue of evaluator’s role and discusses its implication and limitations.
Keywords: Role for an evaluator, Program evaluation, Fundamental issue
1. Introduction
Fundamental issues in evaluation include the purpose of evaluation, the nature of evaluation, the best methods,
strategies and tools of conducting evaluation, the practical concerns such as politics, clients and resources and
their influence on evaluation, as well as the roles, ethics and responsibilities of evaluators. Fundamental issues
are defined as “those underlying concerns, problems, or choices that continually resurface in different guises
throughout our evaluation work.” (Smith, 2008, p.2) Although those fundamental issues will resurface
periodically in the field of evaluation in new forms and cannot really be solved once and for all; the awareness of
the recurring nature of such fundamental issues can help one view the current problems in evaluation from a
better historical perspective. By identifying and examining such fundamental issues, evaluators can have a
deeper understanding of their importance, constraints and alternative solutions, thus propose a more effective,
yet still impermanent resolution for the existing problems.
1.1 Evaluator’s Role as a Fundamental Issue
The fundamental issue discussed in this paper is the role for an evaluator. Over the years, many evaluation
theorists have proposed different roles for evaluators. For example, Scriven sees evaluator as a “judge” who
justifies the value of an evaluand and offers his summative judgment in the final report; while Stake believes an
evaluator should be a “program facilitator” who works with different stakeholders and assists them to “discover
ideas, answers, and solutions within their own mind”. Campbell prefers a “methodologist” role for an evaluator,
advocating rigorous experiment design that yields strong causal inferences; but Wholey believes an evaluator
should be an “educator”, whose role is to infuse useful information to the potential users of the evaluation. The
emphasis resides not only in the immediate outcome of a program, but also in the inputs, implementation and
long-term outcome of the program.
However, terms such as “judge”, “methodologist” and “educator” are just metaphors to facilitate understanding
and cannot always accurately describe the role an evaluator plays during an evaluation. In fact, evaluators often
play different roles in different phases of an evaluation. For example, an evaluator can be a judge during the
phase of selecting criteria of merit, a methodologist when collecting data, a program facilitator during the
program implementation, and an educator during the results dissemination. The roles an evaluator takes during
an evaluation reflect his or her beliefs in other fundamental issues such as theories, values, methods, practice and
use, etc. Other issues such as education background, previous working experiences, the nature and setting of
social programs might also contribute to shaping the proper roles for an evaluator.
1.2 Importance of Evaluator’s Role as a Fundamental Issue
Just like any other fundamental issue in evaluation, there is no final resolution to defining the proper role for an
evaluator. However, studying the different roles an evaluator can play, as proposed by different theorists, is still
quite important for evaluators, clients and evaluation as a profession.
Turning first to evaluators, studying the different roles for an evaluator is actually studying the different
approaches of conducting evaluation. An evaluator’s role is not self-claimed; rather it is defined by the things an

1
International Education Studies www.ccsenet.org/ies

evaluator does during the evaluation. For instance, we wouldn’t use the metaphor “judge” to describe the
evaluator’s role in Scriven’s theory if evaluators’ job doesn’t include determining the criteria of merits, setting
comparative standards and giving a final summative judgment. Or we wouldn’t compare the role of evaluator
in Weiss’s theory to an educator if providing “enlightenment” (Note 1) is not a primary task for her evaluators.
The familiarity with different roles an evaluator can play allows one to take a more flexible approach to conduct
evaluation according to specific context, the nature of social program, available resources, and different client
expectations.
As for the clients of an evaluation, the awareness of the different roles of an evaluator can play will help them
select the right candidate according to their specific needs; and reach an agreement with the selected evaluator
about his/her job responsibilities as well as the obligations clients shall make in order to facilitate the evaluation
process. For example, for a program that does not welcome intrusion, an evaluator who prefers doing an
experiment might not be the best candidate. For an evaluator who prefers the role as a “program facilitator”,
program administrators should anticipate frequent meetings with the evaluator and make incremental changes
according to his/her feedback. The basic knowledge about evaluator’s roles is especially important in today’s
world of globalization, where evaluation in a different nation or culture becomes more common. Clarifying the
role an evaluator should play beforehand is a good way to avoid surprise, misunderstanding, and conflict later
on.
Finally, evaluation as a profession will benefit from a deeper understanding of the roles of evaluators. How is an
evaluator different from a social scientist? Can a methodologist be hired to do the job of an evaluator? What is
the difference between evaluation and research? What are the competencies that are unique for an evaluator?
Those questions are raised due to the lack of distinction between evaluation and other social science professions.
Explicating the proper roles for a professional evaluator will be a good approach to address such distinction and
solidify the status of evaluation as a profession.
2. Theorists’ Positions on the Fundamental Issue of Evaluator’s Role
Many theorists in the field of evaluation have different opinions regarding the proper roles for an evaluator.
Their opinions on this issue reflect their overall philosophy about doing evaluation as well as their stances on
other fundamental issues in evaluation. This section will first discuss the resolutions proposed by different
theorists regarding the role of evaluator, analyzing the strength and weakness of each resolution. Then a
comparative analysis will be conducted to study the positions across those theorists.
2.1 Scriven
Scriven believes that an evaluator’s role is to investigate and justify the value of an evaluand. Such investigation
and justification shall be supported with joining empirical facts and probative reasoning. “Bad is bad and good is
good and it is the job of evaluators to decide which is which” (Scriven, 1986, p.19). He rejects the notion that an
evaluator’s role is simply to provide information to decision-makers and claims that “the arguments for keeping
science value free are in general extremely bad” (Scriven, 1969, p.36). According to Scriven, an evaluator’s
responsibilities during an evaluation include:
 Determining criteria of merit from needs assessment. Criteria of merit of an evaluand should be its capacity
to meet needs. Although an evaluator can use the results of needs assessment conducted by a program developer,
sometimes he/she should do an independent needs-analysis. To avoid bias, Scriven advises evaluators to conduct
“goal-free” evaluation and formulate questions by ignoring the program goals and looking for all possible effects
an evaluand could have.
 Setting comparative evaluation standards. A set of standards should be created by evaluators to assess the
program performance. Such standards are used for comparison, either comparison with a set level of
performance, or with alternative programs. The latter comparison is preferred by Scriven since he believes that
an evaluator will usually make decisions about choosing among alternatives.
 Assess program performance. An evaluator will need to answer both the evaluative and non-evaluative
questions. Evaluative questions focus on the effects of the program and should be given top priority. The
evaluator should acquire the skills to collect and analyze both experimental and non-experimental data.
 Offering a final evaluative judgment. An evaluator should synthesize his findings into a final report and
offer his/her summative judgment.
Strength and Weakness of Scriven’s Position: Scriven differentiates evaluators from researchers or social
scientists by emphasizing that the value judgment is an integral part of an evaluator’s role and grounds such role

2
 International Education Studies Vol. 3, No. 2; May 2010

in the logic of evaluation. His “goal-free” evaluation allows evaluators to identify possible side effects of the
evaluand and address the concerns of underrepresented stakeholders.
However, besides giving evaluators higher authority over different stakeholders in value judgment, Scriven fails
to provide a solution to eliminate personal biases of evaluators. Metaevaluation proposed by Scriven is a good
attempt but still it is highly subjective and requires years of experiences and expertise for an evaluator to make a
non-biased judgment. For the novice evaluator, the decision of whose needs should be considered and which
merit should take higher priority can still be very arbitrary. Besides, a complete goal-free evaluation is also
highly unfeasible when an evaluator is hired by his/her clients and has an obligation to answer their specific
inquiries.
2.2 Campbell
Campbell believes that evaluators should play a role of methodologist during the program evaluation (Shadish,
1991, p.141). Evaluators should use scientific methodologies to design evaluative research that eliminate biases
and establish a causal inference about a program and its hypothesized effects. This role of methodologist
advocated by Campbell requires evaluators to employ a strong research design such as randomized experiment
or good quasi-experiment to determine the causal effectiveness of the program. (Shadish, 1991, p.129) An
evaluator should also distance him/herself from the program stakeholders and work independently to find out the
facts about the program. As for the dissemination of the evaluation findings, an evaluator should “write honest
reports for peers even if they cannot do so for funders or the public.” (Shadish, 1991, p.162) Last but not least, it
is also the obligation of evaluators to play an active role in scrutinizing, replicating, and debating the evaluation
results.
Campbell’s emphasis on methods of measuring the program outcome makes him less concerned about assigning
values to the program or facilitating the use of evaluation. As a result, he believes an evaluator is not responsible
for doing the following:
 An evaluator is not obligated to assign value to the program being evaluated. Valuing of evaluation results
should be left to the political process, not researchers. (Shadish, 1991, p.160).
 An evaluator shouldn’t promote use of her evaluation results actively “since this detracts from the
credibility of the more factlike findings.” (Shadish, 1991, p.162).
 It is up to the policy maker, stakeholders to decide how to interpret, disseminate and use the evaluation
results.
 An evaluator is not obligated to generate a different or modified program worth testing. Her job is simply
testing the efficacy of existing programs.
 An evaluator should avoid evaluating institutions, social organizations, or persons due to the almost
inevitable corruption pressure. (Campbell, 1984, p.41).
Strength and Weakness of Campbell’s Position: The methodologist role Campbell assigns to evaluators is
echoed with the proposal for conducting “scientifically based evaluation” as advocated by the Department of
Education. The role of a methodologist as defined by Campbell focuses on the internal validity of the causal
inference while is less concerned about the prescribing values and utility of the evaluation findings therefore is
quite suitable for an external evaluation regarding program outcome. Such role for an evaluator will also greatly
enhance the scientific nature of evaluation as a profession.
Nevertheless, the weaknesses are also quite obvious for such role of an evaluator. First of all, it is hard to
distinguish evaluation from other social science research if one sees an evaluator merely as a research
methodologist. Apparently, not every social scientist can do a good evaluation. Secondly, doing a rigorous
experiment design is preferable but not always feasible. The cost and time for doing a randomized control
experiment, as well as its intrusion into program might result in fewer and fewer evaluation being done due to
the reluctance from program administrators. Last but not least, the methodologist role restricts evaluators to
study only the outcome of the program while missing other key information such as how the program is
implemented, or which element of the program works and doesn’t work. As a result, an evaluator couldn’t give
advice about how to improve the program or adapt the program to fit other contexts.
2.3 Stake
Stake believes an evaluator should play a facilitator role during the evaluation. The evaluator should assist
different stakeholders to “discover ideas, answers, and solutions within their own mind” by conducting

3
International Education Studies www.ccsenet.org/ies

responsive evaluation. (Stake & Trumbull, 1982, p.1) According to Stake, the responsibilities of an evaluator
include:
 Identifying the stakeholders for whom the evaluation will be used: The evaluator should have a good sense
of whom he is working for and their concerns. (Stake, 1975, as cited in Shadish 1991, p.273). Minority
stakeholder groups should also be included to ensure justice and fairness.
 Spending more time observing the program and providing accurate portrayals of the program using case
studies. Because case studies reflect the complexities of the reality, they help readers to form their own opinions
and judgments about the case and they can be “useful in theory building”. (Stake, 1978, as cited in Shadish 1991,
p.289)
 Conducting responsive evaluation which allows evaluation questions and methods to emerge from
observing the program. In this approach, evaluators will orient evaluation directly to program activities than to
the program goals and respond promptly to audience information requests.
 Presenting his evaluation findings in the “natural ways in which people assimilate information and arrive at
understandings” so that the writings can reach maximal comprehensibility. (Stake, 1980, p.83)
Stake doesn’t believe an evaluator should make a summative value judgment since there is “no single true value”
for all the stakeholders of a program. (Stake, 1975, as cited in Shadish 1991, p.274) As a result, evaluators
shouldn’t blindly accept state and federal standards and impose treatments on local programs since such
standards are not pluralistic and might not be in the best interest of local people. (Shadish, 1991, 279) Stake also
believes the responsibility of synthesizing and interpreting case studies lies in the readers rather than evaluators
and it is up to the readers to resolve any conflicting arguments. (Shadish 1991, p.293)
Strength and Weakness of Stake’s Position: the facilitator role for an evaluator, as suggested by Stake, has two
major strengths. First, it indicates the interest shift among evaluators from giving a summative judgment,
whether it is a value judgment or an effect judgment, to generating useful information that can be used to
improve the program. Secondly, it justifies new ways to conduct an evaluation (e.g. responsive evaluation, case
study) and report its findings (e.g. narrative portrayal).
However, Stake fails to take into account clients’ expectations about the proper role for an evaluator. Will clients
accept the case study as the only approach of investigation? Will clients allow evaluators to start evaluations
without preordinate questions? Is it appropriate for an evaluator to completely ignore the state or federal
standards when evaluating local programs? All those doubts regarding the feasibility and validity of case studies
or responsive evaluation will also undermine the social acceptance of the evaluator role proposed by Stake.
2.4 Weiss
Weiss emphasizes the evaluator’s special role in promoting the use of his/her evaluation results, especially in the
policy-making process. She is frustrated about the fact that “evaluation results have generally not exerted
significant influence on program decisions”, and she argues that evaluation should start out with use in mind and
evaluators shouldn’t leave the use of evaluation to the natural processes of dissemination and application. (Weiss,
1972, as cited in Shadish 1991, p.182-183) Weiss claims that evaluation “should be continuing education for
program managers, planners and policy makers”. (Weiss, 1988, p.18) As a result, it seems that she sees the role
of evaluator more as an educator, who conducts evaluation not for giving an explicit solution to a social problem,
but for providing useful information to its potential users, policy-makers in particular.
She urges evaluators to look beyond the instrumental use of evaluation results and conduct “enlightenment”
research that “provides evidence that can be used by men and women of judgment in their efforts to research
solutions” (Weiss, 1978, p.76) so as to maximize the utility of evaluation results. By doing evaluation this way,
an evaluator should
 Assess the likelihood that evaluation results might be used. (Shadish, 1991, p.198)
 Ask questions that can “provide an intellectual setting of concepts, propositions, orientations, and empirical
generalizations” for policy making. (Weiss, 1978, as cited in Shadish 1991, p.202)
 Use well designed qualitative and quantitative methods to conduct evaluation study with emphasis not only
on the immediate outcome of a program, but also on the inputs, implementation and long-term outcome of the
program. (Shadish, 1991, p.205)

4
 International Education Studies Vol. 3, No. 2; May 2010

 Draw policy implications from evaluation research by compiling separate summaries to multiple
stakeholders with knowledge and information that best interest them. Make recommendations for future
programs from the data of evaluation results. (Shadish, 1991, p.205-206)
Strength and Weakness of Weiss’ Position: Weiss further differentiates the role of evaluator from the role of a
researcher by addressing the complex political context that besets social programs. She warns evaluators against
political naivety and urges them to do evaluation that can be used in policy-making, in the form of
“enlightenment” rather than “instrumental use”. The educator role she assigns to evaluators reflects her
pragmatic view of evaluation and suggests a new mode that evaluation can be used.
However, the role of evaluator proposed by Weiss has some intrinsic flaws. First, such role fails to take into
account of the variety of different contexts (ironically). For instance, the decision for the state or federal
government to hire an educator is often not for the purpose of “being educated”, but to get concrete data
regarding the program effect. The proposal to conduct “scientifically based evaluation” made by the Department
of Education is a good example of that. As a result, an evaluator who uses case studies to describe the program
input, implementation and long-term effect might not by appreciated by policy-makers in this context. Secondly,
her emphasis on providing information to policy-makers poses the danger of evaluators becoming the servant of
that particular stakeholder group. What should be the role of an evaluator when the interests of different
stakeholder groups conflict with each other and speaking for the underrepresented group might limit the use of
evaluation results in the policy-making process?
2.5 Rossi
Rossi didn’t give an explicit definition about the role of evaluators. Rather, the roles an evaluator shall play
might vary according to different stages of evaluation. For example, in the program conceptualization stage, an
evaluator sometimes takes the role of a social scientist, incorporating social science theories into the
development of an intervention model. (Shadish, 1991, p.389-391) In the stage of program implementation, an
evaluator works as a program administrator, making sure the program is implemented as expected so as to “rule
out faulty implementation as a culprit in poor program outcome”. (Shadish, 1991, p.381) Besides, the operational
data collected this way can also be useful for the future dissemination of the program. When determining the
program utility, an evaluator will take the roles of a methodologist and a project manager, who selects and
applies appropriate research methods to assess the impact of program intervention as well as conducts efficiency
analysis about the program such as cost-benefit and cost-effectiveness analysis. (Rossi & Freeman, 1985,
p.327-328)
The different types of social programs will also affect the roles an evaluator plays during the evaluation. Rossi
has categorized social programs into three types: innovative, established, and fine-tuning programs. For instance,
when evaluating innovative programs, much emphasis is given to the conceptualization of the program. (Shadish,
1991, p. 404) An evaluator’s responsibility will include setting program objectives and constructing an impact
model between program objectives and activities, which should be based on not only the stakeholder’s views,
but also the results of needs assessment and social science theories. However, conceptualization is rarely the
focus of evaluation of established programs since their conceptual frameworks already exist and are less likely to
change. (Shadish, 1991, p. 404) Instead, an evaluator will take a more summative approach and much of his/her
responsibility falls into judging the program accountability. The role of an evaluator is less summative in
fine-tuning programs, with emphasis on identifying the needs for change and formative modifications.
Rossi’s attempt to integrate the works of various theorists into one theoretical framework also helps to shape his
position on the issue of the proper role for an evaluator. Rossi appreciates the strengths of different roles
proposed by other theorists and assign such roles to the contexts that best fit them. The “good enough rule” of
doing evaluation proposed by Rossi also frees evaluators from the everlasting debate of internal validity vs.
external validity, quantitative methods vs. qualitative methods, descriptive value vs. prescriptive value. Instead,
it allows evaluators to choose the best possible design through assessing all kinds of “trade-offs”.
Strength and Weakness of Rossi’s position: In my opinion, Rossi’s stance on the role of evaluator is closest to
the reality, since evaluation by nature is highly context-based. The nature of different social programs, clients’
expectations, evaluators’ backgrounds and expertise, the employment status, available resources, and restraints,
as well as the influence of culture and politics can all result in quite different approaches towards doing an
evaluation. Even for the same evaluation, activities in different stages will require various competencies from an
evaluator. As a result, there shouldn’t be only one proper role for an evaluator and Rossi’s attempt to integrate
different roles into one theoretical framework is plausible. However, Rossi is less explicit when linking a
specific context to a specific role and linking the specific role to a set of responsibilities that relates to such role.

5
International Education Studies www.ccsenet.org/ies

2.6 A Comparative Analysis of Different Theorists’ Positions

Looking beyond the different metaphors used by theorists to express their opinions on the roles of evaluator, a
comparative analysis will be done in this section to dissect such roles into specific behaviors that an evaluator
should do in different phases of an evaluation. Those phases are listed in the table below as: program selection,
criteria selection, data collecting, evaluation findings, evaluation use and dissemination. (See table 1) As we can
see from the table, different roles an evaluator takes can result in quite different approaches of doing evaluation
in certain phases while still share a lot of similarity in the other phases. All the citations in the following table are
from the book Foundations of Program Evaluation. (Shadish et al, 1991)
Insert Table 1 Here
3. Proposed Resolution of the Fundamental Issue
The current debate among evaluation theorists regarding the proper roles for an evaluator reflects their different
stances on other fundamental issues such as the value of evaluation (descriptive vs. prescriptive), the methods of
evaluation (quantitative vs. qualitative), the use of evaluation (instrumental vs. enlightenment), the purpose of
evaluation (summative vs. formative). My own resolution on the issue of evaluator’s role will also be based on
my understanding of those fundamental issues.
Value: an evaluator should prioritize the values from different stakeholder groups when selecting the criteria of
merit for evaluands. Not only does the descriptive values approach reflect the concept of a plural democracy, it
also orients the evaluation questions towards the concerns of its stakeholders thus makes it more likely that the
evaluation finings will be used by them. By taking into account different opinions regarding the values of the
program, an evaluator can also reduce the personal bias, which is hard to avoid when the evaluator is the one
who assigns values. However, in case different stakeholders can’t reach an agreement on the issue of value, an
evaluator should take measures to make his/her judgment about the program value. For example, the evaluator
can conduct a needs-assessment to identify the primary stakeholder group who will be affected most by the
program and prioritize their values when selecting the criteria of merit for the evaluation.
Methods: an evaluator should be familiar with quantitative and qualitative methods and accept them both as
available methods for conducting evaluation. However, an evaluator should meet with clients before evaluation
and get their opinions on the suggested method. If the clients have a strong opinion to get some quantitative data
and have some nice charts in the final report, then the case study is not an ideal method. If the clients want the
program to suffer from minimum intrusion from the evaluation, then experiment or quasi-experiment design
shouldn’t be considered as first options. However, this is not to say that evaluators should discard the methods
they consider best for the evaluation immediately if such methods are not accepted by clients; but it is essential
for evaluators to convince their clients of the proposed methods for evaluation and reach an agreement before
applying any method.
Use: an evaluator should emphasize the instrumental use of his/her evaluation findings and actively promote the
dissemination of the evaluation results. It is hard to imagine that an evaluation will be initiated with no intent to
know the effect of a social program, especially when such program costs a lot of tax-payers’ money and impacts
a large population. In my opinion, ignoring the instrumental use of evaluation is highly irresponsible and has
detrimental impact on the profession of evaluation: why should I hire someone to evaluate a program if he/she
cannot tell me if the program is working or not? The enlightenment use of evaluation sounds promising but has
its limitation in actual practice. First, it is hard to determine the scope of data collecting. With potential users of
evaluation findings in obscurity, it is hard to know what specific data will be useful for them. Second, it is hard
to enlighten people without telling them what works in the program and what does not. We have to make a
judgment about success and failure if we want others to learn from the success or the failure. Last but not least,
clients’ expectation and time restraint often make conducting evaluation for enlightenment impossible. It takes
time to measure the program input, implementation, outcome, long-term impacts peoples’ attitude, etc. But most
clients do not have so much time for an evaluation.
Purpose: I prefer a more summative role for an evaluator for the following reasons. First of all, an evaluator is
hired for his/her expertise in making rational judgments. If clients want only the vicarious experiences or a set of
quantitative data, they can just hire some writers or statisticians to do the job. Second, even for the purpose to
make a program better there is a difference between an evaluator and a program manager. At some point, an
evaluator will give a summative opinion about what works and what does not in the program. An evaluator can
make recommendation about how to make improvement; but ultimately it is up to the program manager to make
that decision based on his/her knowledge about the program such as the available resources or people’s
commitment. Lastly, being too involved in improving a program might hinder the objective and rational

6
 International Education Studies Vol. 3, No. 2; May 2010

judgment from evaluators. A lot of personal bias is expected if the evaluator of a program becomes its advocate;
since it is hard for an evaluator to make a statement that a program is not working and should be terminated if
the evaluator believes in such program and has invested heavily in it.
However, by reviewing other theorists’ position on the issue of evaluator’s roles, we can see each different role
will have its unique strength and weakness and there is no specific role that fits all the evaluation. Instead of
using another metaphor to describe my position, I will define the role of evaluator in the things an evaluator will
do during an evaluation, as shown in the following table. (See table 2)
Insert Table 2 Here
Nevertheless, I understand my resolution for such fundamental issue is not perfect and is vulnerable to criticism
in several aspects. First of all, the role I proposed for evaluators might not work in certain contexts; or it is very
likely that evaluators who take different roles can still achieve great successes in conducting evaluations. Take
Stake, Weiss, Wholey and Cronbach for example, their difference regarding the proper role for an evaluator
doesn’t prevent them from presenting strong evaluation cases to support their positions. In other words, it is
almost like I propose a role for an evaluator and then caution evaluators to take such role while in the same time
indicate other roles might work even better in their contexts. Second, different cultural, racial and ethnic
backgrounds of evaluators could also affect their judgments about issues such as clients’ expectations, dynamics
between the program administrators and evaluators, the monetary relationship between evaluators, program
administrators, federal government and stakeholder groups, as those issues might vary significantly in different
background contexts. Since my resolution is based on my judgment of such issues, if my judgment is an
incorrect reflection of the reality elsewhere, then my resolution might not fit well in that context. Last but not
least, people can also attack the logic behind my approach to define the role for an evaluator. Maybe it is wrong
to define the evaluator’s role in terms of what they should do during the evaluation; maybe an evaluator’s role
should be associated with the purpose of an evaluation and such role actually defines what an evaluator should
do in the evaluation.
References
Campbell, D. T. (1984). Can we be scientific in applied social science. Evaluation studies review annual,
Volume 9. Beverly Hills, CA: Sage Publications.
Scriven, M. (1969). An introduction to meta-evaluation. Education Product Report, 2, 36-38.
Scriven, M. (1986). New frontiers of evaluation. Evaluation Practice, 7, 7-44
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage.
Shadish, W. R. Jr., Cook, T. D. & Leviton, L. C. (1991). Foundations of program evaluation. Newbury Park, CA:
Sage.
Smith, N. L. & Brandon, P. R. (Eds.). (2008). Fundamental issues in evaluation. New York, NY: Guilford.
Stake, R.E. (1980). Program Evaluation, Particularly Responsive Evaluation. Rethinking Educational Research
(pp.72-87). London: Hodder & Stoughton.
Stake, R.E. & Trumbull, D. J. (1982). Naturalistic generalizations. Review Journal of Philosophy & Social
Science, 7, 1-12.
Rossi, P. H., and Freeman, H. E. (1985). Evaluation: A Systematic Approach. (3rd ed.) Beverly Hills, Calif.:
Sage, 1985.
Weiss, C. H. (1978). Improving the linkage between social research and public policy. In Lynn, L.E. (Ed.).
Knowledge and policy: The uncertain connection (pp. 23-81). Washington, DC: National Academy of Sciences.
Weiss, C.H. (1988). "If Program Decisions Hinged Only on Information: A Response to Patton." Evaluation
Practice 9(3): 15-28.
Notes
Note 1. Readers gain new perspectives and ideas from reading the evaluation reports, which results in
accumulated knowledge and evidence that can be used by them in many ways including defining a problem,
suggesting a solution and making a policy. Differing from the instrumental use of evaluation results,
enlightenment emphasizes the long term effect on policy-making.

7
International Education Studies www.ccsenet.org/ies

Table 1. A summary of theorists' positions on the issue of evaluator's role

Scriven Campbell Stake Weiss Rossi
Program “everything” Preference for Preference for Programs whose Divide programs into 3
selection (p.84) pilot local programs evaluation kinds: innovative,
programs results are likely established and
(p.136) to be used fine-tuning
Criteria Prescribe value Describe Describe values Describe values Prefer criteria to come
selection through needs- values from from local from different from stakeholder
assessment political stakeholders stakeholders agreement but unclear
process (p.307) (p.210) how to avoid biases
(p.160)
Data collecting The outcome effects Outcome 13 kinds of data Not only the Collecting data
scope of the program consequence covering the immediate regarding the program
of the program outcome of the implementation,
program antecedents, program, but outcome as well as its
(p.161) transaction and also its input, efficiency
the outcome implementation
(p.282) and long-term
outcome (p.205)
Data collecting Not specific Randomized- Qualitative Both Accept both
method experiments methods, case quantitative and quantitative and
or strong studies qualitative qualitative methods
quasi-experim methods with a preference
ents (p.204-205) towards the former.
Evaluation One final evaluative Reports about Accurate Separate Include not only the
findings judgment regarding the causal portrayal of the summaries for program effects, but
the program effects inference of program, diverse different also its policy
and “need” program formats stakeholder relevancy such as
effect (p.282-283) groups, with cost-efficiency,
knowledge and alternative options, etc.
information that (p.409)
best interest
them.
Use of Instrumental use of Instrumental Provide Enlightenment Instrumental,
evaluation the results to choose use of the vicarious use of the results conceptual and
results the better evaluand results to experiences of to accumulate persuasive use of the
or improve one. solve a social the program knowledge and results. (p.410-411)
(p.109) problem. shape
policy-making
Dissemination Not mentioned Dissemination Not mentioned Actively “ a definite
of the is not the facilitate the responsibility of
evaluation concern of an dissemination of evaluation researchers”
results evaluator results (p.207) (p.410)
(p.162)

8
 International Education Studies Vol. 3, No. 2; May 2010

Table 2. A summary of my position on the issue of evaluator's role

Phase of Evaluation Evaluator’s responsibility
Program selection
 Determine the nature of the program
 Analyze the political influences that affect the program

Criteria selection
 Use values from different stakeholder groups
 Conduct needs-assessment to prioritize the values of different stakeholder groups

Data collecting scope

 Collect data that can answer stakeholders' inquiries and concerns
 Emphasis on the program outcome and look into program input and implementation

Data collecting methods

 Get to know clients' expectation and available resources
 Be familiar with both quantitative and qualitative methods

Evaluation findings
 Provide answers to the inquiry of different stakeholder groups
 Offer evaluative judgment regarding the program effect

Use of evaluation results

 Use the results as evidence to make decision regarding the program (promote, modify
or terminate the program)
 Increase people’s knowledge of certain phenomenon

Dissemination of evaluation
 Provide separate summaries of evaluation results to different stakeholder groups with
results
information about their interest and concern