Hong Kong Teachers Centre Journal, Vol.

Hong Kong Teachers Centre 2009

The beneficial washback of the school-based

assessment component on the speaking
performance of students

 !"#$%&'()*+,-./
LEE Wong Wai, Christina
Hong Kong Examinations and Assessment Authority

Abstract
This paper aims to show that the implementation of school-based assessment (SBA) has proved to have positive
impact on the performance of students in the public oral examination. An SBA component was introduced to the Hong
Kong Certificate of Education Examination English Language Examination in 2007. This consists of a reading /
viewing programme where students need to read / view texts, write up some comments and personal reflections, and
then take part in a discussion with classmates on the texts they have read / viewed, or make an individual presentation
and respond to questions. The assessment is based on the students oral performance. The 2007 experience has shown
that the speaking ability of students can be reliably assessed in school by their own teachers. Statistically, the SBA
component has proved to be as reliable as the speaking examination. The beneficial washback of SBA can be seen in
the results of the speaking examination. Candidates from schools that submitted SBA marks had a lower absentee rate
than candidates from schools not submitting SBA marks. They also performed better in the speaking examination.

Keywords
school-based assessment; Hong Kong Certificate of Education Examination (HKCEE) English Language; speaking
examination; beneficial washback / backwash; speaking performance

18

The beneficial washback of the school-based assessment component on the speaking performance of students


2007

2007



assessment more closely with the English Language

Background
The Hong Kong Certif icate of Education

teaching syllabus published by the Curriculum

Examination (HKCEE) is taken by students in Hong

Development Council in 1999 as well as the new Senior

Kong at the end of five years of secondary education.

Secondary curriculum to be implemented in September

Examinations are offered in 39 subjects, mostly with

2009. The SBA component seeks to provide a more

equivalent English and Chinese versions, to around

comprehensive appraisal of learners achievement by

100,000 candidates each year. The examinations

assessing those learning objectives which cannot be

assess candidates achievement of the learning targets

easily assessed in public examinations while at the

and objectives of the teaching syllabus promulgated

same time enhancing the capacity for student self-

by the Curriculum Development Council. The

evaluation and life-long learning. The SBA, like the

examinations are taken after a two-year course

rest of the HKCE English Language Examination,

comprising Secondary 4 and Secondary 5 (S4 and S5).

adopts a standards-referenced assessment system

A new HKCEE English Language syllabus

which seeks to recognise and report on the full range

including a school-based assessment (SBA)

of educational achievement in Hong Kong schools.

component was introduced in 2007 in order to align

Table 1 outlines the examination syllabus.

Table 1: 2007 HKCEE English Language Examination

Component

Public exam

Paper 1A - Reading
Paper 1B - Writing
Paper 2 - Listening & Integrated
Skills
Paper 3 - Speaking

School-based assessment

Weighting

Duration

20%
20%
30%

1 hour
1 hour 30 minutes
2 hours

15%

12 minutes

15%
19

The SBA component consists of a reading /

own assessments and regularly review their progress.

viewing programme where students read / view three

Teachers should use the assessment activities not only

texts (texts encompass print, non-print, fiction and

to make judgments about student standards (a snapshot

non-fiction material) over the course of two years, keep

of students achievement to date), but also to give

a log book of comments / personal reflections, and

feedback to students about specific aspects of their

then take part in a discussion with classmates or make

oral language skills so that they can improve for the

an individual presentation on the books / videos / films

next assessment. The SBA component can be valuable

that they have read / viewed, and respond to questions

preparation for students for their external HKCE

from their teacher, which will be derived from the

examination, especially for the reading and speaking

students written notes / personal responses / comments

papers, as many of the skills required are the same.

in their logbook. The assessment is based on the

The SBA component is worth 15% of the total

students speaking performance, that is, the reading /

English subject mark. In S4, teachers need to

viewing / writing will only serve as the means to this

undertake at least one assessment of studentsgroup

end and the specific content of the texts (i.e. names

interaction or individual presentation skills and

and places, story lines, other factual information etc.)

report one mark at the end of the school year. In

is not directly assessed as such.

S5, they need to again undertake at least one

Teachers are advised to develop the SBA

assessment of students group interaction or

component as an integrated part of the curriculum, not

individual presentation skills, and report one mark

as a separate examination paper. Students should

at the end of S5. These requirements are summarised

be encouraged to keep copies of the records of their

in Table 2.

Table 2: SBA Requirements

Requirements
Number and type of texts

S4

S5

One or two texts

One or two texts

to be read / viewed

Total
Three texts, one each from
three of the following four
categories (print fiction, print
non-fiction, non-print fiction,
non-print non-fiction)

Number and timing of

One task, group interaction

One task, group interaction

Two tasks, each on a text

assessment tasks to be

or individual presentation,

or individual presentation,

from a different category

undertaken

to be undertaken during the

to be undertaken anytime

second term of S4

during S5

Number, % and timing of

One mark reported at the

One mark reported at end

Two marks, 15% of total

marks to be reported

end of S4

of S5

English subject mark

20

The beneficial washback of the school-based assessment component on the speaking performance of students

An SBA handbook is published and distributed

2005, a number of concerns were raised regarding the

to schools to help teachers understand the rationale

SBA, in particular concerns about workload, fairness,

behind the introduction of SBA, and to provide

authentication of student work and teacher readiness.

guidelines regarding possible assessment tasks,

In April 2006, after a series of consultation seminars

assessment criteria and administrative arrangements.

and a comprehensive survey of all schools, school

councils and professional bodies, modifications were
made to the design of the SBA component and the

SBA implementation issues

implementation schedule.

Original proposals regarding the introduction of

the new language syllabus, including the details of the

A three-year phase in period was introduced to

SBA component, were favourably received by schools

accommodate variations between schools with respect

and teachers when they were consulted in 2003 and

to the optimum time to implement SBA. Schools can

2004. However, as the schools started implementing

choose among three options. Details of the

the new syllabus with their S4 students in September

implementation schedule are shown in Table 3.

Table 3: Three-year Phase-in Implementation Schedule

Year

Options for schools

2007

1. Submit SBA marks for feedback and to contribute 15% of final subject result; or
2. Submit SBA marks for feedback only; exam results to contribute 100% of final subject result; or
3. Not submit SBA marks; exam results to contribute 100% of final subject result

2008

1. Submit SBA marks for feedback and to contribute 15% of final subject result; or
2. Submit SBA marks for feedback only; exam results to contribute 100% of final subject result

2009

1. Submit SBA marks for feedback and to contribute 15% of final subject result

the 2007 public examination. Approximately one-third

SBA in the 2007 examination

of the schools Option 1. Table 4 shows the number of

Schools were asked to indicate their choice in

schools and candidates involved.

October 2006 when they registered their students for

Table 4: Number of schools and candidates choosing each option

Choice

No. of schools

Percentage (%)

No. of candidates

Percentage (%)

Option 1 (Yes)

199

34

31,875

43

Option 2 (Trial)

125

22

20,945

28

Option 3 (No)

254

44

21,388

29

Total

578

100

74,208

100

21

The 2007 examinations were conducted in May

SBA scores of students in a given school is made to

and June and schools choosing Options 1 and 2

resemble the distribution of scores of the same group

submitted their SBA scores at the end of the two-year

of students on the public examination. The method

course, in April 2007.

adjusts the mean and the standard deviation of SBA

scores, but the rank order of the SBA scores is not
changed.

Statistical moderation of SBA scores

One of the major concerns expressed by

Results of statistical moderation

stakeholders, in particular parents and school teachers,

is that SBA may not be a fair way of assessing student

In the 2007 examination, 199 schools opted to

performance because teachers will conduct different

submit SBA marks for feedback and to include the

teaching and assessment activities and schools will

marks in the subject result, while 125 chose to submit

have different assessment plans to cater to the needs

SBA marks for feedback only. The mean and standard

of their students. In order to ensure the fairness of

deviation of the SBA marks submitted by the majority

SBA, the HKEAA uses statistical methods to moderate

of schools fell within the expected range.

the SBA marks submitted by schools.

Schools were given feedback in the form of

Teachers know their students very well and thus

an SBA Moderation Report in October 2007. In

are best placed to judge their performance. In

the report, two comments were given in addition to

consultation with their colleagues, they can reliably

the mean and standard deviation of the SBA scores

judge the performance of all students within the school

before and after moderation. The f irst comment

in a given subject. However, when making these

related to the mean of the SBA scores awarded by

judgments, they are not necessarily aware of the

teachers as a whole. If the schools SBA scores were

standards of performance across all other schools.

within the expected range, only minimal adjustments

Despite training in carrying out SBA and even though

were made. More adjustments were necessary for

teachers are assessing students on similar tasks and using

schools with means that were higher or lower than

the same assessment criteria, teachers in one school may

expected. The second comment was about the

be harsher or more lenient in their judgments than

distribution of the SBA scores submitted by the

teachers in other schools. They may also tend to use a

school. If the standard deviation of the SBA scores

narrower or wider range of marks. Statistical

was within the expected range, only slight

moderation seeks to adjust for any arbitrary differences

adjustments were needed, while more adjustments

between schools in the standards of marking.

were made to school scores with wider or narrower

The method that the HKEAA uses to carry out

spreads than expected. A summar y of the

statistical moderation follows well established

moderation results of Option 1 schools are given in

international practice. In essence, the distribution of

Tables 5 and 6.

22

The beneficial washback of the school-based assessment component on the speaking performance of students

Table 5: Moderation results of the mean of SBA scores submitted by Option 1 schools
The mean of the SBA scores is ...

No. of Schools

Percentage (%)

within the expected range

144

72.4

slightly higher than expected

29

14.6

higher than expected

1.0

much higher than expected

slightly lower than expected

21

10.6

lower than expected

1.5

much lower than expected

Table 6: Moderation results of the S.D. of SBA scores submitted by Option 1 schools
The standard deviation of the SBA scores is ...

No. of Schools

Percentage (%)

179

89.9

slightly wider than expected

wider than expected

slightly narrower than expected

10

5.9

narrower than expected

4.5

much wider than expected

much narrower than expected

0.5

as expected

Since the SBA component carries a weighting

SBA, an adjustment of 3 marks means a change of

of 15% of the public assessment, any upward or

less than 1% to the subject total. Table 7 shows the

downward adjustment of the SBA marks has minimal

impact of statistical moderation on the actual scores

impact on the overall subject result. For example, with

of the candidates.

a maximum of 48 marks for the English Language

Table 7: Moderation effect on candidates

Mark adjustment (% of subject mark)

No. of Schools

Percentage (%)

0 (0)

5365

17

1-3 (<1)

19881

62

4-6 (<2)

6237

20

7-9 (<3)

392

31,875

100

23

The moderation results show that most teachers

have a good understanding of the assessment criteria

whether the different components of the exam

measure a single underlying dimension;

and can assess their students reliably. This is reassuring

the reliability of each of the components; and

and indicates that the initial concerns about teacher

the reliability of the composite score assuming

readiness and fairness might have been exaggerated.

1) equal weights, 2) weights as set by HKEAA

as a matter of policy, 3) weights that maximize
the reliability of the composite score.

Analysis of 2007 examination and

SBA data

These questions were addressed by using

Following the release of the 2007 HKCEE

structural equation modelling to f it a one-factor

results, analyses of examination data for English

congeneric measures model to the data (Hill, 2007).

Language were undertaken to determine:

The inter-paper correlations are given in Table 8.

Table 8: Inter-paper correlations (by listwise case exclusion, N=28,253)

Reading

Writing

L & IS

Speaking

SBA

Reading

1.000

0.858

0.887

0.776

0.803

Writing

0.858

1.000

0.852

0.767

0.797

Listening & Integrated Skills

0.887

0.852

1.000

0.764

0.796

Speaking

0.776

0.767

0.764

1.000

0.787

SBA

0.803

0.797

0.796

0.787

1.000

The results of the analysis are summarized in

namely English Language. The table gives three

Tables 9, 10 and 11 below. Table 9 indicates the extent

goodness-of-fit indices obtained from fitting a one-

to which the scores on the various parts of the

factor congeneric measures model to the data.

examination measure a single underlying ability,

Table 9: Goodness - of - fit indices

Goodness of fit index

Adjusted goodness of fit index

Root mean square residual

0.969

0.908

0.017

The values indicate strong support for the

SBA. This justifies the statistical moderation of SBA

existence of a single underlying ability for the various

marks on the basis of the public examination scores.

components of the English examination, including

24

The beneficial washback of the school-based assessment component on the speaking performance of students

Table 10: Reliability of the total scores weighted in different ways

Equal weights

Policy weights

Maximum reliability weights

0.957

0.959

0.961

The first column of Table 10 shows the reliability

weights. This indicates that the weighting given to

of the examination if equal weights are given to the

individual papers is in fact appropriate.

various papers and to SBA. The second column shows

Table 11 provides information at the component

the reliabilities of English Language given the weights

level. In the first column are the relevant weights. In

that were actually assigned to the various components

the second column are the factor loadings and in the

(e.g., Reading = 20%, Writing = 20%, etc.). It can be

third column, the variances of the residuals. Because

seen that with a reliability of 0.959, the English

correlation matrices were analyzed and variables were

Language examination is highly reliable. The third

standardized, the reliability of each component, shown

column shows what the reliability would be if the various

in the fourth column, is simply the square of the factor

components were weighted in such a way as to maximize

loadings. The factor score regressions in column five

the reliability of the examination. It can be seen that

indicate the weights that one would use to maximize

the increase is very small relative to the actual policy

the reliability of the component.

Table 11: Reliability of the different components of the English Language examination
Component

Weight

Reliability of

Factor score

component

regressions

Reading

0.20

0.939

0.117

0.882

0.312

Writing

0.20

0.915

0.163

0.837

0.218

Listening and Integrated Skills

0.30

0.932

0.132

0.869

0.276

Speaking

0.15

0.838

0.297

0.702

0.110

SBA

0.15

0.868

0.247

0.753

0.137

It can be seen that all components were reliably

own teachers should be more reliable than a one-off

measured and that the reliability of the SBA (which

12-minute speaking examination taken under high-

measures speaking) was higher than that for the

stress conditions.

speaking examination. This is contrary to most

From the above, it can be concluded that the CE

teachers expectations, but should not come as a

English Language examination, including the SBA

surprise. It is reasonable that multiple assessments

component, measured a single underlying ability and

conducted over the course of two years by students

provided a highly reliable total score for each candidate

25

as well as reliable scores for each of the components of

When a new examination syllabus is introduced,

the examination. The initial doubts about the reliability

the absentee rate also tends to increase, possibly due to

of the SBA component can therefore be dispelled.

a lack of confidence on the part of candidates who may

be unfamiliar with the new requirements. For example,

Effect of SBA on the speaking

in 1996, the last time when a major syllabus change

examination

was introduced to HKCEE English Language, the

absentee rate in the speaking examination was 19.0%,

Because of the SBA phase-in options offered to

up from 14.9% in 2006 and representing an increase

schools, the 2007 HKCE English Language

of about 4%. The absentee rate eventually dropped

Examination offers an opportunity for studying the

to about 12.3% in 2006. In 2007, with the

effect of SBA implementation on the performance of

introduction of a new examination syllabus, the

candidates, in particular their speaking performance,

absentee rate for all candidates was 13.3%. There

since the SBA component also focuses on the

was an unexpectedly small increase of 1% as

assessment of speaking ability.

compared to the 2006 figures.

While the absentee rate of private candidates has

Absentee rate

remained fairly stable, at around 13%, the absentee

The written papers for English Language are

rate of school candidates has fluctuated more

scheduled in early May each year while the speaking

markedly. Further analysis was done by dividing the

examination is conducted over a ten-day period in

schools into three groups based on their choice of SBA

June, after the written papers for all other subjects

implementation option. A comparison of the absentee

have been sat. Candidates who have not done well

rates of different groups of school candidates is shown

in the written papers tend to give up on the speaking

in Table 12.

examination. Therefore, the absentee rate of the

speaking examination has always been the highest
among all English papers.

Table 12: Absentee rate of candidates from different school groups

School choice

No. sat

No. of absentees

Absentee rate (%)

Option 1 (Yes)

27,935

3,398

10.8

Option 2 (Trial)

19,307

1,466

7.1

Options 1 & 2

47,242

4,864

9.3

Option 3 (No)

19,298

3,681

16.0

All Schools

66,540

8,545

11.4

(Yes + Trial)

26

The beneficial washback of the school-based assessment component on the speaking performance of students

The above figures reveal that the absentee rates

candidates from Option 3 schools are generally weaker

of Option 1 and Option 2 schools are lower than that

and therefore more prone to skip the speaking

for Option 3 schools, which have deferred the

examination.

implementation of SBA. A possible explanation for

this difference is that candidates from Option 1 and

Speaking examination scores

Option 2 schools had more speaking practice in school

A breakdown of the speaking examination scores

because of the SBA. They were therefore more

of the candidates from different school groups reveals

conf ident in taking part in the public speaking

an interesting pattern, as shown in Table 13.

examination, which involves similar speaking skills

required for the SBA tasks. It is also possible that the

Table 13: Speaking examination scores of candidates from different school groups
School choice

No. sat

Speaking exam mean (%)

Option 1 (Yes)

27,804

25.38 (53)

Option 2 (Trial)

19,381

26.00 (54)

Option 3 (No)

21,293

23.51 (49)

All Schools

68,478

24.97 (52)

It can be seen that the mean speaking

based on their scores in other English examination

examination scores of candidates from Option 1 and

papers, which is taken as an indication of their general

Option 2 schools were higher than that of Option 3

English ability. The actual and predicted scores were

schools. However, this cannot prove that the SBA

compared to see if there were any signif icant

component has a positive effect on the candidates

differences in the residuals (actual mean minus

speaking performance. It could be argued that the

predicted mean). A positive residual would indicate

schools opting for SBA implementation in 2007 were

that the group of candidates did better in the speaking

actually better schools with better students to begin

examination than predicted based on their performance

with than those choosing not to implement SBA at all.

in other papers, which indicates their general English

To further analyse the data, regression analysis

ability. Table 14 shows the differences between the

was carried out to predict the speaking examination

actual and predicted mean scores.

scores of candidates from different school groups

27

Table 14: Actual and predicted speaking scores of candidates from different school groups
School choice

Actual Mean

Predicted Mean

Residual

t-value

Option 1 (Yes)

25.38

25.13

0.25

-1.97

Option 2 (Trial)

26.00

25.97

0.03

-0.21

Option 3 (No)

23.51

23.86

- 0.35

2.34 *

On average, candidates from Option 1 and

This years oral exam constituted a big change in

Option 2 schools performed better in the speaking

format, but candidates were quite well prepared

examination than expected, achieving higher mean

for this change. Also, thanks to continuous SBA

scores than predicted although the differences are not

practice, more students were more willing to

statistically significant. It should also be noted that

contribute in both Parts A and B.

the residual of Option 1 schools, where the SBA scores

were submitted and actually included in the subject

During the oral examination this year, the

results, is more positive than Option 2 schools, where

number of candidates who did not say anything

SBA was implemented on a trial basis. However,

at all dropped significantly. It was noticed that

candidates from Option 3 schools got statistically

candidates were more confident and willing to

significant lower scores than expected, which means

talk... ...

that they performed worse in the speaking examination

relative to other papers.

These observations are consistent with the

It can be concluded that the implementation of

statistical evidence. However, as it was not possible

SBA did have an effect on the performance of the

to distinguish between school candidates and private

candidates in the speaking examination. Candidates

candidates in the examination room, or candidates who

without SBA practice in school did significantly worse

had or had not participated in SBA, these comments

than expected, while those who participated in SBA

apply to all candidates who took the speaking

did better than expected, regardless of their general

examination. It would still be fair to say that there is

English ability.

anecdotal evidence that candidates speaking

examination performance has improved in general after
the introduction of the SBA.

Chief Examiners comments

After each examination, the Chief Examiner of

Conclusion

each paper submits a report which includes comments

on the examination questions as well as on candidates

The 2007 experience indicates that most

performance. The following excerpts from the report

teachers have a good understanding of the assessment

on the speaking paper give an indication of examiners

criteria and can assess their students reliably in school.

views on the effectiveness of the SBA:

In the 2007 exam, 199 schools (34% of schools) chose

28

The beneficial washback of the school-based assessment component on the speaking performance of students

Option 1: to submit SBA marks for feedback and to

schools that chose not to submit SBA marks (Option

include the marks in the subject result, while 125 (22%)

3 schools). Candidates who did SBA in school also

of schools chose Option 2: to submit SBA marks for

performed better in the speaking examination than

feedback only. The mean of the SBA marks submitted

those from schools not submitting SBA marks.

by 72% of the Option 1 schools and 65% of the Option

Statistical evidence is supported by the Chief

2 schools fell within the expected range, while 90%

Examiners comments and observations.

of the Option 1 schools and 94% of the Option 2

The 2007 experience has shown that the speaking

schools submitted marks with a spread within the

ability of students can be reliably assessed in school by

expected range.

their own teachers and that the SBA is a valid and viable

Statistically, the SBA component has proved to

alternative to the speaking examination.

be as reliable as the public speaking examination. In

Further research is required to ascertain the

fact, the moderated SBA marks correlated slightly better

validity and reliability of the SBA and its effect on the

with the rest of the public examination papers than the

performance of candidates in the public examination.

speaking examination. This indicates that teachers are

A four-year longitudinal study is being carried out to

able to reliably assess the speaking abilities of their

monitor the setting, conduct, marking and moderation

students given statistical moderation to remove arbitrary

of the SBA component of the HKCE English Language

differences between schools in interpreting standards.

Examination over four school years (2005/06 to 2008/

The beneficial washback effects of SBA can

09). Analysis of the 2008 examination data is also

be seen in the results of the speaking examination.

underway and will provide more information on the

Candidates from schools that submitted SBA marks

effect of SBA on the performance of candidates as

(Option 1 and Option 2 schools) had a lower absentee

over 50% of schools have now chosen to implement

rate of about 9% compared to 16% for candidates from

SBA.

References
Hill, P.W. (2007). Reliability of 2007 HKCEE English Language. Unpublished study.
Hong Kong Examinations and Assessment Authority (2005). 2007 Hong Kong Certificate of Education
Examination, English Language, Handbook for the School-based Assessment Component.
Hong Kong Examinations and Assessment Authority (2005). Hong Kong Certif icate of Education
Examination, Regulations and Syllabuses 2007.
Hong Kong Examinations and Assessment Authority (2007). 2007 Hong Kong Certificate of Education
Examination, English Language, Examination Report and Question Papers.
Hong Kong Examinations and Assessment Authority (2007). Statistical Moderation of School-based
Assessment Scores.
29