IOP Concise Physics

Investigative Science Learning Environment

When learning physics mirrors doing physics
Eugenia Etkina, David T Brookes and Gorazd Planinsic

Chapter 4
Scientific abilities

Practicing reasoning in authentic contexts is a vital part of the ISLE approach. By

authentic contexts, we mean situations when the students need to employ authentic
reasoning similar to that physicists employ when solving problems. This aspect of
ISLE is best realized when students design their own experiments to answer the
questions posed for them or even pose their own questions. Student experimentation
in ISLE is drastically different from traditional approaches where students observe
demonstrations performed by a teacher or labs where students perform ‘verification’
experiments following step-by-step instructions. To help students learn how to solve
problems and how to proceed in the ‘design’ situations, we developed:
(a) a list of reasoning processes that physicists use and that we can help our
students develop (we coined the name of ‘scientific abilities’ for those);
(b) specific scaffolding questions that help students develop scientific
abilities; and
(c) a set of self-assessment rubrics that students can use during and after the
process of design to help them develop the scientific abilities. These rubrics
suggest what they should think about and not what they should do.

4.1 Defining scientific abilities

The work on scientific abilities was carried out by a large group of people and is
described in several publications (Etkina et al 2006, 2008). The list of scientific
abilities, rubrics and many ISLE-based resources including instructional labs are
posted online and are free to download at https://sites.google.com/site/scientific-
abilities/.
We use the term ‘scientific abilities’ to describe some of the most important
procedures, processes and methods that scientists use when constructing knowledge
and when solving experimental problems. We use the term ‘scientific abilities’

doi:10.1088/2053-2571/ab3ebdch4 4-1 ª Morgan & Claypool Publishers 2019

 Investigative Science Learning Environment

instead of ‘science process skills’ to underscore that these are not automatic skills,
but are instead processes that students need to use reflectively and critically
(Salomon and Perkins 1989). The list of scientific abilities developed by our physics
education research group is as follows:
(a) the ability to represent information in multiple ways;
(b) the ability to use scientific equipment to conduct experimental investigations
and to gather pertinent data to investigate phenomena, to test hypotheses,
or to solve practical problems;
(c) the ability to collect and represent data in order to find patterns, and to ask
questions;
(d) the ability to devise multiple explanations for the patterns and to modify
them in light of new data;
(e) the ability to evaluate the design and the results of an experiment or a
solution to a problem;
(f) the ability to communicate.

This list is based on the analysis of the history of practice of physics (Holton and
Brush 2001, Lawson 2000, Lawson 2003), the taxonomy of cognitive skills (Bloom
1956, Krathwohl 2002), and recommendations of science educators (Schunn and
Anderson 2001).
To help students develop these abilities, one needs to engage students in
appropriate activities, and to find ways to assess students’ performance on these
tasks, to provide timely feedback and to revise planned instruction based on student
work. Activities that incorporate feedback to the students and to the instructor are
called formative assessment activities. As defined by Black and Wiliam, formative
assessment activities are ‘all those activities undertaken by teachers, and by their
students in assessing themselves, that provide information to be used as feedback to
modify the teaching and learning activities’ in which they are engaged. (Black and
Wiliam 1998b). Black and Wiliam also found that self-assessment during formative
assessment is more powerful than instructor-provided feedback; meaning the
individual, small-group, and large-group feedback system enhances learning more
than instructor guided feedback. Sadler (1989) suggested three guiding principles,
stated in the form of questions, that students and instructors need to address in order
to make formative assessment successful.
1. Where are you trying to go? (Identify and communicate the learning and
performance goals.)
2. Where are you now? (Assess, or help the student to self-assess, current levels
of understanding.)
3. How can you get there? (Help the student with strategies and skills to reach
the goal.)

As noted above, students need to understand the target concept or ability that
they are expected to develop and the criteria for good work relative to that concept

4-2
 Investigative Science Learning Environment

or ability. They need to assess their own efforts in light of the criteria. Finally, they
need to share responsibility for taking action in light of the feedback. The quality of
the feedback rather than its existence or absence is a central point. The feedback
should be descriptive and criterion-based, as opposed to numerical scoring or letter
grades without clear criteria.
With all the constraints of modern teaching, including large-enrollment classes
and untrained teaching assistants (TAs), how can one make formative assessment
and self-assessment possible? One way to achieve this goal is to use scoring rubrics.
A scoring rubric is one of the ways to help students see the learning and performance
goals, self-assess their work, and modify it to achieve the goals (three guiding
principles as defined by Sadler above). The rubrics contain descriptions of different
levels of performance, including the target level. A student or a group of students
can use the rubric to self-assess their own work. An instructor can use the rubric to
evaluate students’ responses and to provide feedback.

4.2 Fine-tuning scientific abilities and devising rubrics to assess them

After making the list of scientific abilities that we wanted our students to develop, we
started devising assessment rubrics to guide their work. Rubrics are descriptive
scoring schemes that are developed by teachers or other evaluators to guide students’
efforts (Brookhart 1999). This activity led to a fine-tuning of the abilities, that is, to
break each ability into smaller sub-abilities that could be assessed. For example, for
the ability to collect and analyze data, we identified the following sub-abilities:
• the ability to identify sources of experimental uncertainty,
• the ability to evaluate how experimental uncertainties might affect the data,
• the ability to minimize experimental uncertainty,
• the ability to record and represent data in a meaningful way, and
• the ability to analyze data appropriately.

Each item in the rubrics that we developed corresponded to one of the sub-abilities.
We agreed on a scale of 0–3 in the scoring rubrics to describe student work
(0—missing, 1—inadequate, 2—needs some improvement, 3—adequate) and devised
descriptions of student work that could merit a particular score. For example, for the
sub-ability ‘to record and represent data in a meaningful way’ a score of 0 means that
the data are either missing or incomprehensible, a score of 1 means that some
important data are missing, a score of 2 means that all important data are recorded
but presented in a way that requires some effort to comprehend, and a score of
3 means that all important data are present, organized, and recorded clearly.
Simultaneously, while refining the list of abilities, we started devising activities
that students could perform in problem solving sessions and labs. Defining sub-
abilities and developing scoring rubrics to assess them informed the writing of these
activities. After we developed the rubrics, we started using them to score samples of
student work. Each person in a nine-person group assigned a score to a given sample

4-3
 Investigative Science Learning Environment

using a particular rubric; we then assembled all the scores in a table and discussed
the items in the rubrics where the discrepancy was large.
Based on these discussions, we revised the wording of the rubrics and tested them
by scoring another sample of student work. This process was iterated until we
achieved a nearly 100% agreement among our scores.
In the sections below, we list scientific abilities and corresponding sub-abilities
that we identified, provide examples of scoring rubrics that we devised and discuss
where in the instructional process we use the rubrics. For each scientific ability, we
provide examples of the tasks written for the students. In subsequent sections, we
will report how we used the rubrics to study students’ development of some of the
suggested abilities.
1. Ability to represent information in multiple ways
In introductory physics courses, students are often given a verbal description of a
physical process and a problem to solve relative to that process. They can start their
analysis by constructing a sketch to represent the process and include in the sketch
the known information provided in the problem statement. They construct more
physical representations that are still relatively easy to understand—for example,
motion diagrams, free-body diagrams, graphs, qualitative work-energy and impulse-
momentum bar charts, circuit diagrams, ray diagrams, field lines and more (see
table 4.1). Finally, they use these physical representations to help construct a
mathematical representation of the process.
What sub-abilities help to make this multiple representation strategy productive
for reasoning and problem solving?
• The ability to correctly extract information from a representation;
• The ability to construct a new representation from another type of
representation;
• The ability to evaluate the consistency of different representations and modify
them when necessary.

In addition to such sub-abilities that students need to master while using multiple
representations, there are specific sub-abilities needed for each type of representa-
tion. For example, to use force diagrams (or free-body diagrams FBDs) productively
for problem solving, students must learn to:
• Choose a system of interest before drawing the diagram.
• Use force arrows to represent the interactions of the external world (environ-
ment) with the system.
• Label the force arrows with two subscripts
 (for example, the force that Earth
exerts on the object is labeled as FE on O).
• Try to make the relative lengths of force arrows consistent with the problem
situation (the sum of the forces should point in the same direction as the
system object’s acceleration).
• Include labeled axes on the diagram.

4-4
 Investigative Science Learning Environment

Table 4.1. Examples of cognitive tools—multiple representations.

Name Representation

Motion diagram

Force diagram
(free body
diagram)

Graph

Work-energy bar
charts

Impulse-
momentum bar
charts

Circuit diagram

(Continued)

4-5
 Investigative Science Learning Environment

Table 4.1. (Continued )

Name Representation

Ray diagram

Field lines

Such diagrams if drawn correctly can be used to help write Newton’s second law in
component form—to represent the situation mathematically. Based on these
considerations, we constructed a rubric to help students self-assess themselves while
drawing force diagrams.
We also made a list of several types of multiple representation activities (a task
may consist of some combination of these activities). Some examples are
given below.
Provide students with one representation and have them create another.

Example: Draw motion and force diagrams for the process described with the
following equations:
x : ax = [0 + ( −(100 kg) × (10 N kg −1) × cos 20°)]/(100 kg)
y : 0 = [N + ( −(100 kg) × (10 N kg −1) × sin 20°)]/(100 kg)
0 − (16 m s−1) = axt .
Provide students with two or more representations and have them check for
consistency between them.

Example: Two forces exert impulses on a hockey puck, which can move with no
friction on an icy surface. The graphs on the left in figure below show the time
dependence of the x- and y-components of the sum of the forces exerted on the puck.
Which of the trajectories (a)–(d) of the puck’s motion in the x-y-plane shown on the
right cannot be the result of these forces? The numbers correspond to successive
clock readings that are marked on the force graphs.

4-6
 Investigative Science Learning Environment

Provide students with one representation and have them choose from a multiple-
choice list a consistent different type of representation (for example, provide a word
description of a process and have students select from a list a consistent graphical
description of the process).

Example: In the figure below, choose the correct approximate velocity-versus-time

graph for the following hypothetical motion: a car moves at constant velocity, and
then slows to a stop and without a pause moves in the opposite direction with the
same acceleration.

Have students use a representation while solving a problem.

Example: In a popular new hockey game, the players use small launchers with
springs to move the 0.0030 kg puck. Each spring has a 120 N m−1 spring constant
and can be compressed up to 0.020 m. Determine the maximum speed of the puck.

4-7
 Investigative Science Learning Environment

First, represent the process with a work-energy bar chart and then solve the
problem.

Below, we show the rubrics for self-assessment of the ability to represent

information in multiple ways (table 4.2).
2. The ability to use scientific equipment to conduct experimental investigations and
to gather pertinent data to investigate phenomena, to test hypotheses, or to solve
practical problems
As we discussed above, in ISLE, experiments play three distinctly different
epistemological goals: experiments that help students devise models/explanations/
relations, experiments that help them test those and experiments in which students
combine accepted models to solve a practical problem (Etkina et al 2002). In short,
an experiment can have one of the three roles: observational, testing or application
experiment.
As mentioned above, when conducting an observational experiment, a student
focuses on investigating a physical phenomenon without having expectations of its
outcomes. When conducting a testing experiment, a student has an expectation of its
outcome based on the idea under test. In an application experiment, a student uses
established concepts or relationships to address practical problems. However, in the
process of scientific research, the same experiment can fall into more than one of
these categories. For example, an experiment that was initially planned as testing
experiment can show some surprising results and thus becomes an observational
experiment that eventually (after several new testing experiments) leads to discovery
of a new idea.
What abilities do students need when designing experimental investigations? We
have identified the following steps that students should take to design, execute and
make sense out of a particular experimental investigation. We assigned a sub-ability
for each step and wrote corresponding descriptors in the rubrics. The results of these
discussions are presented in table 4.3.
For each of the identified sub-abilities, we devised a rubric item that describes
different levels of proficiency. The rubrics for the three types of experiments are
presented in tables 4.4, 4.5, and 4.6.
Students use these rubrics when they solve problems or conduct experiments.
Ideally, we want them to continuously refer to the rubrics while solving a problem or
designing and performing the experiment. In a lab environment, rubrics guide the
students as to what experimental aspects they should specifically pay attention to.
After they perform the experiment, the students write a lab report (in the lab).
During the process of writing, they use the descriptors in the rubrics to improve their
report. For example, in one case, students wrote that they used a thermometer to
measure the temperature of a hot rock. They recorded the rock’s temperature in their
report. However, what students actually measured was the temperature of the water
in which the rock was submerged. Using the rubrics, they self-assessed their writing
and revised their description of how they used available equipment to measure the
required physical quantity. In the revised report, students wrote that to determine

4-8
 Table 4.2. Ability to represent information in multiple ways.

RUBRIC A: Ability to represent information in multiple ways

Scientific ability Missing Inadequate Needs improvement Adequate

A1 Is able to extract the No visible attempt is Information that is Some of the information All necessary
information from made to extract extracted contains is extracted correctly, information has been
representation information from errors such as labeling but not all of the extracted correctly,
correctly the problem text. quantities incorrectly, information. For and written in a
mixing up initial and example, physical comprehensible way.
final states, choosing a quantities are Objects, systems,
wrong system, etc. represented with physical quantities,
Physical quantities numbers but there are initial and final states,
have no subscripts no units. Or directions etc, are identified

4-9
(when those are are missing. Subscripts correctly and units are
needed). for physical quantities correct. Physical
are either missing or quantities have
inconsistent. consistent subscripts.
A2 Is able to construct No attempt is made to Representations are Representations are Representations are
Investigative Science Learning Environment

new representations construct a attempted, but use created without constructed with all
from previous different incorrect information mistakes, but there is given (or understood)
representations representation. or the representation information missing, information and
does not agree with i.e. labels, variables. contain no major
the information used. flaws.

(Continued)
 Table 4.2. (Continued )

RUBRIC A: Ability to represent information in multiple ways

Scientific ability Missing Inadequate Needs improvement Adequate

A3 Is able to evaluate the No representation is At least one Representations created All representations, both
consistency of made to evaluate representation is made agreement with each created and given, are
different the consistency. but there are major other but may have in agreement with
representations and discrepancies between slight discrepancies each other and the
modify them when the constructed with the given explanations of the
necessary representation and the representation. Or consistency are
given one. There is no there is no explanation provided.
attempt to explain of the consistency.
consistency.
A4 Is able to use No attempt is made to The problem is solved The problem is solved The problem is solved
representations to solve the problem. correctly but no correctly but there are correctly with at least
solve problems representations other only two three different

4-10
than math were used. representations: math representations
and words explaining (sketch, physics
the solution. representation and
math or sketch, words
and math, or some
Investigative Science Learning Environment

other combination).
A5 Force diagram (FD) No representation is FD is constructed but FD contains no errors in The diagram contains no
constructed. contains major errors, vectors but lacks a key errors and each force
such as incorrect feature such as labels is labeled so that it is
mislabeled or not of forces with two clearly understood
labeled force vectors, subscripts or vectors what each force
length of vectors, are not drawn from represents.
wrong direction, extra single point, or axes
incorrect vectors are are missing.
added, or vectors are
missing.
 A6 Motion diagram No representation is Diagram does not show Diagram has correct The diagram contains no
constructed. proper motion: either spacing of the dots but errors and it clearly
lengths of arrows us missing velocity describes the motion
(both velocity and arrows or velocity of the object. Dots,
velocity change) are change arrows. velocity arrows and
incorrect or missing velocity change
and or spacing of dots arrows are correct.
are incorrect.
A7 Sketch No representation is Sketch is drawn but it is Sketch has no incorrect Sketch contains all key
constructed. incomplete with no information but has items with correct
physical quantities either no or very few labeling of all physical
labeled, or important labels of given quantities, which have
information is quantities. Subscripts consistent subscripts;
missing, or it contains are missing or axes are drawn and
wrong information, or inconsistent. The labeled correctly.

4-11
coordinate axes are majority of key items
missing. are drawn.
A8 Energy bar chart No representation is Bar chart is either Bar chart has the energy Bar chart is properly
constructed. missing energy values, bars drawn correctly labeled and has energy
the bars drawn do not (energy is conserved), bars of appropriate
Investigative Science Learning Environment

show the conservation but some labels are magnitudes. The

of energy or are drawn missing or the system/ system and initial and
in the wrong places. initial and final states final states are clearly
Bars could also be are not identified. The identified.
labeled incorrectly. bar chart matches the
The system and/or process described with
initial and final states some other
are not identified. representation.

(Continued)
 Table 4.2. (Continued )

RUBRIC A: Ability to represent information in multiple ways

Scientific ability Missing Inadequate Needs improvement Adequate

A9 Mathematical No representation is Mathematical No error is found in the Mathematical

constructed. representation lacks reasoning, however representation
the algebraic part (the they may not have contains no errors and
student plugged the fully completed steps it is easy to see
numbers right away), to solve the problem progression of the first
has the wrong or one needs effort to step to the last step in
concepts being comprehend the solving the equation.
applied, signs are progression. No The solver evaluated
incorrect, or evaluation of the math the mathematical
progression is unclear. in the problem is representation.
The first part should present.

4-12
be applied when it is
appropriate.
A10 Ray diagram No representation is The rays that are drawn Diagram is missing key Diagram has object and
constructed. in the representation features but contains image located in the
do not follow the no errors. One correct spot with the
correct paths. Object example could be the proper labels. Rays
Investigative Science Learning Environment

or image may be object is drawn with are correctly drawn

located at wrong the correct lens/mirror with arrows and
position. but rays are not drawn contain at least two
to show image. Or the rays. A ruler was used
rays are too far from to draw the images.
the main axis to have a
small-angle
approximation. Or the
diagram is drawn
without a ruler.
 A11 Graph No graph is present. A graph is present but The graph is present and The graph has correctly
the axes are not axes are labeled but labeled axes,
labeled. There is no the axes do not independent variable
scale on the axes. The correspond to the is along the horizontal
data points are independent and axis and the scale is
connected. dependent variable or accurate. The
the scale is not trendline is correct.
accurate. The data
points are not
connected but there is
no trendline.

4-13
Investigative Science Learning Environment
 Table 4.3. Sub-abilities involved in successfully carrying out three types of experiments.

Sub-abilities involved in the ability to design Sub-abilities involved in the ability to design Sub-abilities involved in the ability to design
and conduct an observational experiment and conduct a testing experiment and conduct an application experiment

Identifying the phenomenon to be Identifying the model/explanation/relation to Identifying the problem to be solved.
investigated. be tested.
Designing a reliable experiment that Designing a reliable experiment that allows Designing an experiment that solves the
investigates the phenomenon. one to compare the outcome to the problem.
prediction based on the model/explanation/
relation under test using the available
equipment.
Deciding what is to be measured, and Deciding what is to be measured, and Deciding what is to be measured.
identifying independent and dependent identifying independent and dependent
variables. variables.
Using available equipment to make Using available equipment to make Using available equipment to make
measurements. measurements. measurements.

4-14
Describing what is observed, both in words Deciding whether the outcome of the Making a judgment about the results of the
and by means of a picture of the experiment matches the prediction. experiment.
experimental set-up.
Describing a pattern or devising an Making a reasonable judgment about the Evaluating the results by means of an
explanation. model/explanation/relation under test. independent method.
Investigative Science Learning Environment

Identifying shortcomings in an experimental Identifying shortcomings in an experimental Identifying shortcomings in an experimental

design and suggest specific improvements. design and suggest specific improvements. design and suggest specific improvements.
 Table 4.4. Observational experiment rubric.

RUBRIC B: Ability to design and conduct an observational experiment

Scientific ability Missing Inadequate Needs improvement Adequate

B1 Is able to identify the No phenomenon is The description of the The description of the The phenomenon to be
phenomenon to be mentioned. phenomenon to be phenomenon is vague or investigated is clearly
investigated investigated is incomplete. stated.
confusing, or it is not
the phenomena of
interest.
B2 Is able to design a reliable The experiment does The experiment may not Some important aspects of The experiment might yield
experiment that not investigate the yield any interesting the phenomenon will not interesting patterns
investigates the phenomenon. patterns. be observable. relevant to the
phenomenon investigation of the

4-15
phenomenon.
B3 Is able to decide what The physical Only some of physical The physical quantities are The physical quantities are
physical quantities are quantities are quantities are relevant. relevant. However, relevant and independent
to be measured and irrelevant. independent and and dependent variables
identify independent dependent variables are are identified.
and dependent not identified.
Investigative Science Learning Environment

variables
B4 Is able to describe how to At least one of the All chosen measurements All chosen measurements All chosen measurements
use available chosen can be made, but no can be made, but the can be made and all
equipment to make measurements details are given about details of how it is done details of how it is done
measurements cannot be made how it is done. are vague or incomplete. are clearly provided.
with the available
equipment.

(Continued)
 Table 4.4. (Continued )

RUBRIC B: Ability to design and conduct an observational experiment

Scientific ability Missing Inadequate Needs improvement Adequate

B5 Is able to describe what is No description is A description is A description is complete, Clearly describes what
observed without trying mentioned. incomplete. No labeled but mixed up with happens in the
to explain, both in sketch is present, or explanations or pattern. experiments both verbally
words and by means of observations are The sketch is present but and with a sketch.
a picture of the adjusted to fit is difficult to understand. Provides other
experimental setup expectations. representations when
necessary (tables and
graphs).
B6 Is able to identify the No attempt is made to The shortcomings are Not all aspects of the design All major shortcomings of
shortcomings in an identify any described vaguely and are considered in terms of the experiment are
experimental and shortcomings of no suggestions for shortcomings or identified and reasonable
suggest improvements the experimental. improvements are improvements. suggestions for

4-16
made. improvement are made.
B7 Is able to identify a No attempt is made to The pattern described is The pattern has minor The pattern represents the
pattern in the data search for a irrelevant or errors or omissions. relevant trend in the data.
pattern. inconsistent with the Terms proportional are When possible, the trend
data. used without clarity, e.g. is described in words.
Investigative Science Learning Environment

is the proportionality
linear, quadratic, etc.
B8 Is able to represent a No attempt is made to The mathematical No analysis of how well the The expression represents
pattern mathematically represent a pattern expression does not expression agrees with the the trend completely and
(if applicable) mathematically. represent the trend. data is included, or some an analysis of how well it
features of the pattern are agrees with the data is
missing. included.
B9 Is able to devise an No attempt is made to The explanation is vague, The explanation contradicts A reasonable explanation is
explanation for an explain the not testable, or previous knowledge or made. It is testable and it
observed pattern observed pattern. contradicts the pattern. the reasoning is flawed. explains the observed
pattern.
 Table 4.5. Testing experiment rubric. Here, the term hypothesis stands for model/explanation/relation.

RUBRIC C: Ability to design and conduct an experiment to test an idea/hypothesis/explanation or mathematical relation

Scientific ability Missing Inadequate Needs improvement Adequate

C1 Is able to identify the No mention is made An attempt is made to The hypothesis to be tested The hypothesis is clearly
hypothesis to be of a hypothesis. identify the hypothesis to is described but there are stated.
tested be tested but is described minor omissions or vague
in a confusing manner. details.
C2 Is able to design a The experiment does The experiment tests the The experiment tests the The experiment tests the
reliable experiment not test the hypothesis, but due to the hypothesis, but due to the hypothesis and has a high
that tests the hypothesis. nature of the design, it is nature of the design there likelihood of producing
hypothesis likely the data will lead to is a moderate chance the data that will lead to a
an incorrect judgment. data will lead to an conclusive judgment.
inconclusive judgment.
C4 Is able to make a No prediction is A prediction is made but it is Prediction follows from A prediction is made that

4-17
reasonable made. The identical to the hypothesis but is flawed * follows from hypothesis,
prediction based on experiment is not hypothesis, a prediction is because * is distinct from the
a hypothesis treated as a testing made based on a source * relevant experimental hypothesis,
experiment. unrelated to hypothesis assumptions are not * accurately describes the
being tested, or is considered and/or expected outcome of the
Investigative Science Learning Environment

completely inconsistent * prediction is incomplete or designed experiment,

with hypothesis being somewhat inconsistent with * incorporates relevant
tested, or the prediction is hypothesis and/or assumptions if needed.
unrelated to the context * prediction is somewhat
of the designed inconsistent with the
experiment. experiment.

(Continued)
 Table 4.5. (Continued )

RUBRIC C: Ability to design and conduct an experiment to test an idea/hypothesis/explanation or mathematical relation

Scientific ability Missing Inadequate Needs improvement Adequate

C5 Is able to identify the No attempt is made to An attempt is made to Relevant assumptions are Sufficient assumptions are
assumptions made identify any identify assumptions, but identified but are not correctly identified, and
in making the assumptions. the assumptions are significant for making the are significant for the
prediction irrelevant or are confused prediction. prediction that is made.
with the hypothesis.
C6 Is able to determine No attempt is made to The effects of assumptions The effects of assumptions The effects of the
specifically the way determine the are mentioned but are are determined, but no assumptions are
in which effects of described vaguely. attempt is made to determined and the
assumptions might assumptions. validate them. assumptions are
affect the prediction validated.
C7 Is able to decide No mention of A decision about the A reasonable decision about A reasonable decision about

4-18
whether the whether the agreement/disagreement the agreement/ the agreement/
prediction and the prediction and is made but is not disagreement is made but disagreement is made and
outcome agree/ outcome agree/ consistent with the experimental uncertainty experimental uncertainty
disagree disagree. outcome of the is not taken into account. is taken into account.
experiment.
C8 Is able to make a No judgment is made A judgment is made but is A judgment is made, is A judgment is made,
Investigative Science Learning Environment

reasonable about the not consistent with the consistent with the consistent with the
judgment about the hypothesis. outcome of the outcome of the experimental outcome,
hypothesis experiment. experiment, but and assumptions are
assumptions are not taken into account.
taken into account.
 Table 4.6. Application experiment rubric.

RUBRIC D: Ability to design and conduct an application experiment

Scientific ability Missing Inadequate Needs improvement Adequate

D1 Is able to identify the No mention is made of the An attempt is made to The problem to be solved is The problem to be solved is
problem to be solved problem to be solved. identify the problem to described, but there are clearly stated.
be solved, but it is minor omissions or vague
described in a confusing details.
manner.
D2 Is able to design a The experiment does not The experiment attempts The experiment attempts to The experiment solves the
reliable experiment solve the problem. to solve the problem but solve the problem, but due problem and has a high
that solves the due to the nature of the to the nature of the design, likelihood of producing
problem design the data will not there is a moderate chance data that will lead to a
lead to a reliable the data will not lead to a reliable solution.
solution. reliable solution.

4-19
D3 Is able to use available At least one of the chosen All of the chosen All of the chosen All of the chosen
equipment to make measurements cannot measurements can be measurements can be measurements can be made
measurements be made with the made, but no details are made, but the details about and all details about how
available equipment. given about how it is how they are done are they are done are provided
done. vague or incomplete. and clear.
Investigative Science Learning Environment

D4 Is able to make a No discussion is presented A judgment is made about An acceptable judgment is An acceptable judgment is
judgment about the about the results of the the results, but it is not made about the result, but made about the result, with
results of the experiment. reasonable or coherent. the reasoning is flawed or clear reasoning. The effects
experiment incomplete, uncertainties of assumptions and
are not taken into account, experimental uncertainties
or assumptions are not are considered. The result
discussed. The result is is written as an interval.
written as a single number.

(Continued)
 Table 4.6. (Continued )

RUBRIC D: Ability to design and conduct an application experiment

Scientific ability Missing Inadequate Needs improvement Adequate

D5 Is able to evaluate the No attempt is made to A second independent A second independent method A second independent method
results by means of evaluate the consistency method is used to is used to evaluate the is used to evaluate the
an independent of the result using an evaluate the results. results. The results of the results and the evaluation
method independent method. However, there is little two methods are compared is correctly done with the
or no discussion about correctly using experimental uncertainties.
the differences in the experimental uncertainties. The discrepancy between
results due to the two But there is little or no the results of the two
methods. discussion of the possible methods, and possible
reasons for the differences reasons are discussed.
when the results are
different.
D7 Is able to choose a Mathematical procedure is A mathematical procedure Correct and complete Mathematical procedure is

4-20
productive either missing, or the is described, but is mathematical procedure is fully consistent with the
mathematical equations written down incorrect or incomplete, described but an error is design. All quantities are
procedure for solving are irrelevant to the due to which the final made in the calculations. calculated correctly with
the experimental design. answer cannot be All units are consistent. proper units. Final answer
problem calculated. Or units are is meaningful.
Investigative Science Learning Environment

inconsistent.
D8 Is able to identify the No attempt is made to An attempt is made to Relevant assumptions are All relevant assumptions are
assumptions made in identify any identify assumptions, identified, but are not correctly identified.
using the assumptions. but the assumptions are significant for solving the
mathematical irrelevant or incorrect problem.
procedure for the situation.
D9 Is able to determine No attempt is made to The effects of assumptions The effects of assumptions are The effects of the assumptions
specifically the way determine the effects of are mentioned, but are determined, but no attempt are determined and the
in which assumptions assumptions. described vaguely. is made to validate them. assumptions are validated.
might affect the
results
 Investigative Science Learning Environment

the temperature of the rock they measured the temperature of water in which the
rock was submerged and waited for a certain time before recording the temperature
so that the thermometer, rock and the water were in equilibrium.
3. Ability to record, represent and analyze data
Data collection and analysis are important in the practice of experimental science.
These abilities are independent of the type of experiment that is being performed,
and hence have been placed in a different category. We identified sub-abilities that
students need for successful data collection and analysis and devised rubrics for each
sub-ability. (The simplified list below is appropriate for students. Scientists do this at
much more sophisticated level.):
• Ability to identify sources of experimental uncertainty.
• Ability to evaluate of how experimental uncertainties might affect data.
• Ability to minimize experimental uncertainty.
• Ability to record and represent data in a meaningful way.
• Ability to analyze data appropriately.

The rubric for each sub-ability (table 4.7) has descriptors indicating what are typical
mistakes/difficulties that students have and what needs to be done for satisfactory
achievement.
Over the years, we have found that a traditional approach to uncertainties
prevents students from understanding the purpose of estimating the uncertainty,
which also agrees with research findings (Volkwyn et al 2008). Therefore, we use the
‘weakest link rule’ where the uncertainty in the final result is determined only by
the largest percent uncertainty (Good 1976). It can be the random uncertainty or the
uncertainty due to the instrument with the highest percent. A handout for the
students to learn about this approach can be found at https://drive.google.com/file/d/
0By53x8SYAF1lLWhORU5OTnlHbHc/view.
4. Ability to evaluate
We define evaluation as making judgments about information based on specific
standards and criteria (Anderson and Krathwohl 2001). More specifically, a given
statement is judged by determining whether it satisfies a criterion well enough to pass
a certain standard. Scientists constantly use evaluation to assess their own work and
the work of others when conducting their own research, serving as referees for peer-
reviewed journals, or serving on grant-review committees.
The ability to evaluate is crucial also for our students. During a physics course,
students are expected to identify, correct, and learn from their mistakes with the help
of an instructor. This aid may come in many forms, such as when an instructor
provides problem solutions to a class, or tutoring to an individual student. However,
in each case, the student relies upon an instructor (or sometimes a textbook) in order
to determine whether, and how, their work is incomplete. Since the students are not
given any other means with which to evaluate their work, the students come to see
evaluation by external authorities as the only way for them to identify and learn
from their mistakes. This dependence on external evaluation has several negative
effects on students, inhibiting their learning and desire to learn (Warren 2006).

4-21
 Table 4.7. Rubric for data collection and analysis.

RUBRIC G: Ability to collect and analyze experimental data

Scientific ability Missing Inadequate Needs improvement Adequate

G1 Is able to identify sources No attempt is made to An attempt is made to Most experimental All experimental
of experimental identify experimental identify experimental uncertainties are uncertainties are
uncertainty uncertainties. uncertainties, but most correctly identified. correctly identified.
are missing, described But there is no There is a distinction
vaguely, or incorrect. distinction between between experimental
random and uncertainty and
experimental random uncertainty.
uncertainty.
G2 Is able to evaluate No attempt is made to An attempt is made to The final result does take The experimental
specifically how evaluate experimental evaluate experimental the identified uncertainty of the final
identified experimental uncertainties. uncertainties, but most uncertainties into result is correctly

4-22
uncertainties may are missing, described account, but is not evaluated. The
affect the data vaguely, or incorrect, correctly evaluated. weakest link rule is
only absolute The weakest link rule used appropriately
uncertainties are is not used or is used and the choice of the
mentioned, or the final incorrectly. biggest source of
Investigative Science Learning Environment

result does not take uncertainty is justified.

the uncertainty into
the account.
G3 Is able to describe how to No attempt is made to A description of how to An attempt is made to The uncertainty is
minimize experimental describe how to minimize experimental minimize the minimized in an
uncertainty and minimize experimental uncertainty is present, uncertainty in the final effective way.
actually do it uncertainty and no but there is no attempt result is made, but the
attempt to minimize is to actually minimize it. method is not the most
present. effective.
 G4 Is able to record and Data are either absent or Some important data are All important data are All important data are
represent data in a incomprehensible. absent or present, but recorded present, organized,
meaningful way incomprehensible. in a way that requires and recorded clearly.
They are not some effort to The tables are labeled
organized in tables or comprehend. The and placed in a logical
the tables are not tables are labeled but order.
labeled properly. labels are confusing.
G5 Is able to analyze data No attempt is made to An attempt is made to The analysis is The analysis is
appropriately analyze the data. analyze the data, but it appropriate but it appropriate, complete,
is either seriously contains minor errors and correct.
flawed or or omissions.
inappropriate.

4-23
Investigative Science Learning Environment
 Investigative Science Learning Environment

There are several sets of criteria and strategies that are commonly used by
practicing physicists, and if we want physics students to engage in evaluation they
too must value and use these strategies. Each of these strategies relies upon
hypothetico-deductive reasoning (Lawson 2003), whereby the information is used
to create a hypothesis, which is then tested. The logical sequence for this testing can
be characterized as: if (hypothesis) and (auxiliary assumptions) then (expected result)
and/but (compare actual result to expected result), therefore (conclusion) (Warren
2010). For example, when a student derives an equation and needs to evaluate it
with dimensional analysis, the logical sequence is:

IF the equation is physically self-consistent,

AND I correctly remember the units for each quantity in the equation,
THEN I expect the units for each term in the equation to be identical,
AND/BUT the units for each term are/are not identical,
THEREFORE, the equation is/is not physically self-consistent.

The types of sub-abilities that students need to develop to be successful in evaluation

are numerous. Some of them are:
• ability to conduct a unit analysis to test the self-consistency of an equation;
• ability to analyze a relevant limiting/special case for a given model,
equation, claim;
• ability to identify the assumptions a model, equation, or claim relies upon;
• ability to make a judgment about the validity of assumptions;
• ability to use a unit analysis to correct an equation which is not self-
consistent;
• ability to use a special-case analysis to correct a model, equation, or claim;
• ability to judge whether an experimental result fails to match a prediction;
• ability to evaluate the results of an experiment by means of an independent
method.

Evaluation sub-abilities are integral components of multiple representation

abilities, design abilities and are represented in evaluation rubrics. Not only do we
want students to learn each of the evaluation strategies, we also want students to
value them and incorporate evaluation into their personal learning behavior. We
have developed two categories of tasks to help achieve this (see examples at https://
drive.google.com/file/d/0By53x8SYAF1leWNzOWtGdjBDZW8/view). One cate-
gory consists of supervisory evaluation tasks, wherein students act like a supervisor
by evaluating (and, if necessary, correcting) someone else’s work (usually the work
of an imaginary friend). The other category consists of integrated evaluation tasks,
which ask the students to evaluate, and, if necessary, to correct, their own work. For
both categories of task, the evaluated work may be a problem solution, experiment
design, experiment report, conceptual claim, or a proposed model. Supervisory
evaluation tasks are meant to help the students learn the goals, criteria, and method
of use for each evaluation strategy, while integrated evaluation tasks encourage

4-24
 Table 4.8. Evaluation rubric.

RUBRIC I: Ability to evaluate models, equations, solutions, and claims

Needs some
Scientific ability Missing Inadequate improvement Adequate

I1 Is able to conduct a unit No meaningful attempt An attempt is made to An attempt is made to The student correctly
analysis to test the self- is made to identify the identify the units of check the units of each conducts a unit
consistency of an units of each quantity each quantity, but the term in the equation, analysis to test the
equation in an equation. student does not but the student either self-consistency of the
compare the units of misremembered a equation.
each term to test for quantity’s unit, and/or
self-consistency of the made an algebraic
equation. error in the analysis.
I2 Is able to analyze a No meaningful attempt An attempt is made to An attempt is made to A relevant special case is
relevant special case is made to analyze a analyze a special case, analyze a relevant correctly analyzed and

4-25
for a given model, relevant special case. but the identified special case, but the a proper judgment is
equation, or claim. special case is not student’s analysis is made.
relevant, or major flawed, or the
steps are missing from student’s judgment is
the analysis (e.g. no inconsistent with their
conclusion is made). analysis.
Investigative Science Learning Environment

I3 Is able to identify the No assumptions are Some assumptions are All of the student’s All significant
assumptions a model, correctly identified. correctly identified by identified assumptions assumptions are
equation, or claim student, but some of are correct, but some correctly identified,
relies upon. = C8 the identified important and no identified
assumptions are assumptions are not assumptions are
incorrect. identified by student. incorrect.
I4 Is able to evaluate The student states his/her The student states their Student clearly states
another person’s own problem solution/ own solution/claim their own solution/

(Continued)
 Table 4.8. (Continued )

RUBRIC I: Ability to evaluate models, equations, solutions, and claims

Needs some
Scientific ability Missing Inadequate improvement Adequate

problem solution or No meaningful attempt conceptual claim, but and compares it with conceptual
conceptual claim by is made to evaluate by does not methodically the other person’s understanding, and
direct comparison with direct comparison. compare it with the solution/claim, but methodically
their own solution or other person’s does not make any compares it with the
conceptual solution/claim, and so concluding judgment other person’s work.
understanding does not state a based on this Based on this
judgment about the comparison. Or the comparison, the
validity of the other student does student makes a sound
person’s solution/ everything correctly, judgment about the
claim. Or a judgment but their presentation validity of the other

4-26
is made regarding the is incomplete (i.e. person’s work.
other person’s skipping logical steps).
solution/claim, but no
justification is given.
I5 Is able to use a unit No meaningful attempt Student proposes a Student proposes a Student proposes a
analysis to correct an is made to correct the corrected equation, corrected equation corrected equation
Investigative Science Learning Environment

equation which is not equation, even though but their proposal still which passes unit which is correct, at
self-consistent it failed a unit does not pass a unit analysis, but their least up to unit-less
analysis. analysis. proposal is incorrect constants.
(i.e. the student failed
to remember the
proper equation, and
therefore proposed an
equation which is not
physical).
 I6 Is able to use a special- No meaningful attempt An attempt is made to An attempt is made to The model, equation, or
case analysis to correct is made to correct the modify the model, modify the model, claim is correctly
a model, equation, or model, equation, or equation, or claim, but equation, or claim modified in
claim claim even though it the modifications have based on the special- accordance with the
failed a special-case nothing to do with the case analysis, but special-case that was
analysis. special-case that was some mistakes are analyzed.
analyzed. made in the
modification.

4-27
Investigative Science Learning Environment
 Table 4.9. Communication rubric.

RUBRIC F: Ability to communicate scientific ideas

Scientific ability Missing Inadequate Needs improvement Adequate

F1 Is able to communicate Diagrams are missing Diagrams are present but Diagrams and/or Diagrams and/or
the details of an and/or experimental unclear and/or experimental experimental
experimental procedure is missing experimental procedure are present procedure are clear
procedure clearly and or extremely vague. procedure is present and clearly labeled but and complete. It takes
completely but important details with minor omissions no effort to
are missing. It takes a or vague details. The comprehend.
lot of effort to procedure takes some
comprehend. effort to comprehend.
F2 Is able to communicate No discussion of the The experiment and The experiment and The experiment and
the point of the point of the findings are discussed findings are findings are discussed
experiment clearly and experiment is present. but vaguely. There is communicated but the clearly. There is deep

4-28
completely no reflection on the reflection on their reflection on the
quality and importance and quality and
importance of the quality is not present. importance of the
findings. findings.
Investigative Science Learning Environment
 Investigative Science Learning Environment

students to incorporate evaluation into their learning behavior. During a semester,

we tend to use mostly supervisory tasks for the first few weeks so that the students
can get acquainted with each strategy, and then transition to integrated tasks so that
they gain experience at using the strategies to evaluate and correct their own work.
Evaluation rubric is shown in table 4.8.
7. Ability to communicate
An important ability in the work of scientists is their oral and written commu-
nication, an ability that can be fostered in a physics course. For example, the quality
of a lab report can be judged for its completeness and clarity. A communication
ability rubric can help students know what is expected in communications in the
scientific world. The communication rubric is shown in table 4.9.

References
Anderson L W and Krathwohl D R 2001 A Taxonomy for Learning, Teaching, and Assessing: A
Revision of Bloom’s Taxonomy of Educational Objectives (New York: Longman)
Black P and Wiliam D 1998b Inside the black box: raising standards through classroom
assessment Phi Delta Kappan 80 139–48
Bloom B S 1956 Taxonomy of Educational Objectives: The Classification of Educational Goals,
Handbook I Cognitive Domain (New York: David McKay)
Brookhart S M 1999 The Art and Science of Classroom Assessment: The Missing Part of Pedagogy
(Washington, DC: The George Washington University, Graduate School of Education and
Human Development)
Etkina E, Van Heuvelen A, Brookes D T and Mills D 2002 Role of experiments in physics
instruction—a process approach Phys Teach. 40 351–5
Etkina E, Murthy S and Zou X 2006 Using introductory labs to engage students in experimental
design Am. J. Phys. 74 979–86
Etkina E, Van Heuvelen A, White-Brahmia S, Brookes D T, Gentile M, Murthy S and Warren A
2006 Scientific abilities and their assessment Phys. Rev. Sp. Top. Phys. Educ. Res. 2 020103
Etkina E, Karelina A and Ruibal-Villasenor M 2008 How long does it take? A study of student
acquisition of scientific abilities Phys. Rev. Sp. Top. Phys. Educ. Res 4 020108
Good R H 1976 Approximate treatment of experimental error Am. J. Phys. 44 1011–2
Holton S and Brush S 2001 Physics, The Human Adventure (New Brunswick, NJ: Rutgers
University Press)
Krathwohl D R 2002 A revision of Bloom’s taxonomy: an overview Theor. into Pract. 41 212–8
Lawson A E 2000 The generality of hypothetico-deductive reasoning: making scientific thinking
explicit Am. Biol. Teach. 62 482–95
Lawson A E 2003 The nature and development of hypothetico-predictive argumentation with
implications for science teaching Int. J. Sci. Educ. 25 1387–408
Sadler D R 1989 Formative assessment and the design of instructional system Instr. Sci. 18 119–44
Salomon G and Perkins D N 1989 Rocky road to transfer: rethinking mechanisms of a neglected
phenomenon Educ. Psychol. 24 113–42
Schunn C D and Anderson J 2001 Acquiring expertise in science: Exploration of what, when and
how Designing for Science: Implications from Everyday, Classroom and Professional Settings
ed K Crowley, C D Schunn and T Okada (Mahwah, NJ: Erlbaum), pp 351–92

4-29
 Investigative Science Learning Environment

Volkwyn T S, Allie S, Buffler A and Lubben F 2008 Impact of a conventional introductory

laboratory course on the understanding of measurement Phys. Rev. Sp. Top. Phys. Educ. Res
4 010108
Warren A R 2006 Evaluation strategies as a means for learning physics PhD Thesis The State
University of New Jersey
Warren A 2010 Impact of teaching students to use evaluation strategies Phys. Rev. Sp. Top. Phys.
Educ. Res 6 020103

4-30