VAHID NIMEHCHISALEM

1. WH QUESTIONS IN BOOK EVALUATION

Why, How, When, and Who?

INTRODUCTION

Among other English Language Teaching (ELT) materials, which cover an

extensive range from English language learning software to videos, books have
been traditionally of particular significance. The term book, in this chapter, is
interchangeably used for both the ELT coursebook (which covers the whole package
including a student book, workbook, and teacher’s guide) and the ELT textbook
(which refers to the core book covering all or most of the topics in a course). Books
are materials that have been used in English language classrooms for ages since
they help the teacher equally emphasize the syllabus to be covered during a course.
Using books for teaching languages has been harshly criticized by Sheldon (1988),
who refers to them as necessary evil, and Brumfit (1980), who argues that they are
skillfully marketed rubbish. Albeit with such criticisms, the book seems to be an
inseparable component of most language classes. This is particularly true for English
as a Foreign Language (EFL) situation. EFL learners, unlike those in English as a
Second Language (ESL) setting, are less likely to be exposed to authentic examples
of the English language. This underlines the significance of books in EFL settings,
and in turn accentuates the importance of their evaluation.
The present chapter will address some fundamental questions that have engaged
experts in the area of ELT learning-teaching material evaluation for years. More
specifically, these questions will be:
1. Why should anyone bother about book evaluation?
2. How is it possible to evaluate books?
3. When should books be evaluated?
4. Who should be in charge of book evaluation?
The chapter aims to answer each of these questions, respectively.

WHY BOOK EVALUATION?

The primary question that comes to mind is the rationale behind evaluating
ELT books. Based on Sheldon (1988), there are two main reasons for ELT book
evaluation. First, it can help program developers, syllabus designers, and language

© KONINKLIJKE BRILL NV, LEIDEN, 2018 | DOI:10.1163/9789004387379_001

V. NIMEHCHISALEM

instructors select the best possible book for the learners in their present educational
setting. A book selected in ad hoc manner may not match with the learners’ needs
and interests. Such practice will certainly have negative impacts on their learning
outcome. By contrast, informed decisions that are made based on systematic and
objective evaluation of the book will secure the match between the students’ needs
and the objectives of the book.
In addition to the fact that evaluation helps in selection of suitable books, it also
provides useful information for curriculum writers who wish to make adaptations to
less successful parts of the selected book. It is almost impossible to find a book that
perfectly suits the learners’ needs. Therefore, curriculum designers will often find
themselves obliged to make changes to its content or activities. Book evaluation results
help them diagnose any problematic section in the book while it is being taught. Once
these parts have been detected, they are discarded from the course outline. Similarly,
course instructors benefit from evaluating books, which shows the merits and demerits
of the books they are using. An awareness of the possible strengths and weaknesses
of the books enables teachers to replace the less successful parts with more useful and
suitable content and tasks. This creates an opportunity for instructors to improve the
teaching quality and learning outcome of their future instruction.
Failing to evaluate or misevaluate books will have serious but silent consequences
which may be why we normally choose to turn a blind eye to book evaluation
reports. In traditional educational systems, ELT books are developed by the Ministry
of Education and mandated to be used at schools. In such systems, even when the
books do not prove useful, most teachers will simply continue teaching them because
they are given no other choice than covering the entire syllabus and preparing their
class for an examination. In less radical settings, the teacher may be allowed to
use alternative materials to replace the book prescribed by the Ministry. In these
systems, few people usually complain since the general assumption is that the book
is provided for ‘free’. However, it should be noted that any state-sponsored textbook
has its own hidden costs (Mukundan, 2003).

HOW TO EVALUATE BOOKS?

Methods of book evaluation vary. Five different ways will be discussed here,
including impressionistic evaluation, scale-based evaluation, teacher logs, software
packages, and composite frameworks.

Impressionistic Evaluation

This form of evaluation relies on evaluators’ intuitional decision about the usefulness
of the book. The evaluator looks into some pages while flipping through others and
making holistic judgments on the usefulness of the book. This form of evaluation
extensively relies on the evaluator’s experience and expertise. Novice practitioners
find it challenging to make reliable judgments impressionistically. The reason is

2
 WH QUESTIONS IN BOOK EVALUATION

that they still need some time to gain experience and develop the implicit set of
criteria based on which they can make good decisions. Impressionistic evaluation is
cost-effective and takes a relatively short time, but its main shortcoming is its high
subjectivity which puts the reliability of its outcome at risk.

Scale-Based Evaluation
Scale-based evaluation provides a set of evaluative criteria in the form of a checklist,
based on which the evaluator examines the most crucial features of a book. The result
of such evaluation will often be valid if the checklist is developed and rigorously
tested for its validity. It will also result in a reliable decision if the evaluator has
sufficient experience. The outcome of scale-based book evaluation is commonly more
explicit, sophisticated, and comprehensive than that of impressionistic evaluation
that is implicit and intuitive. Checklists commonly focus on the most significant
domains to be considered in evaluating ELT books. The outcome therefore usually
covers most of the constructs related to the issue of ELT book evaluation. A lot of
ELT book evaluation checklists are available in the literature (e.g., Cunningsworth,
1995; Harmer, 1991; Ur, 1996). These checklists help language instructors and
evaluators examine important features of the selected book, features like physical
attributes, aims, layout, methodology, organization, presentation of language skills
(i.e., speaking, listening, reading, and writing), language sub-skills (grammar,
vocabulary, etc.), and language functions, among others.
Depending on the data its items elicit, a checklist can be placed in either side of a
continuum with qualitative checklists on one side and quantitative checklists on the
other. Qualitative checklists have open-ended items. Richards (2001) developed a
checklist that can be considered a qualitative checklist since the evaluator using it will
have to provide subjective responses to the questions that it puts forth. In contrast,
Skierso’s (1991) checklist is quantitative since it gets the evaluator to assign 0 to 4,
signifying ‘totally lacking’ to ‘excellent’ for the book being examined. The Likert
style scale of a quantitative scale allows the evaluator to measure the suitability of
the book under evaluation. It is also possible to have a checklist that represents a
combination of qualitative and quantitative items. For instance, Sheldon’s (1988)
checklist begins with a part on ‘Factual details’ of the book under examination. The
part is followed by a section on ‘Assessment’ of the quality of different features of
the book (such as ‘authenticity’ and ‘flexibility’) that is rated, ‘poor’, ‘fair’, ‘good’,
or ‘excellent’. Finally, the checklist ends with a section with questions like ‘To what
extent has the book realized its stated objectives?’
Each type of book evaluation checklist has its own merits or demerits. Qualitative
checklists will naturally result in more in-depth examination of the book while
quantitative checklists may turn out to be more convenient and reliable. Mukundan
and Ahour (2010) present an overview of textbook evaluation checklists developed

quantitative categories.

3
V. NIMEHCHISALEM

Checklists are useful instruments, but it is easy to develop an unreliable and

invalid checklist. Many checklists are available in the literature without any
empirical proof of their construct validity. Even when they are validated, some of
them lack practicality. For example, there are items in some checklists that seek
to elicit information on the load and distribution of the new vocabulary items.
Obviously, it is very difficult to provide accurate information on how many times
a word appears throughout a book unless the book comes with an accurate index
or unless some specialized software is used. Related to the issue of practicality is
economy. Some checklists are too long. Skierso’s (1991) checklist, for example,
exceeds 4000 running words and contains many domains. This may contribute to
its construct validity but undermines its economy. Another disadvantage of some
checklists is the use of technical jargon which may sound vague for the novice
evaluator. To offer an example, in the vocabulary section of some of the available
checklists (such as Mukundan & Nimehchisalem, 2012), the term ‘recycle’ may
confuse some evaluators. Rephrasing the term in a way that inexperienced evaluators
would find it more comprehensible (i.e., ‘repeat’instead of ‘recycle’) would facilitate
working with the instrument. A final noteworthy point which is often ignored in
the development of some checklists is the methodological principles of instrument
development. As an example, when a single item measures more than one dimension
and violates the principle of unidimensionality, it can confuse the rater and reduce
reliability of its outcome.

Reflective Logs
Reflective or teaching logs provide another option for evaluating ELT books.
Teachers may keep logs either in a structured or an unstructured way. Structured
logs are easier to work with since they provide a list of questions for the teacher to
reflect upon, questions like:
1. Which part(s) of the book was effective?
2. What part(s) can be adapted and improved in future classes?
3. What were the students’ reactions to the lesson presented by the book?
On the other hand, teachers may keep a journal-like unstructured log on how the
book works in the real class situation. Unstructured logs do not limit the scope of
evaluation, leaving the teacher free to evaluate a book as it works. This sometimes
results in more creative and natural evaluation on the part of the teacher, and if
shared with the book authors or publishers, it provides them with more original
feedback on what needs to be changed in the next editions.
Despite the useful information they can provide, teacher logs are not very
popular among teachers. Keeping a log is an added task to language teachers’
heavy schedule. Therefore, some teachers may refrain from keeping a log, and in
cases it is mandatory, some teachers do not take it seriously and just dumb it down.
In addition, some teachers may not find it easy to write about the way a book

4
 WH QUESTIONS IN BOOK EVALUATION

works. There may also be a lot of redundant information in a log which makes
its analysis taxing. Finally, teacher logs are highly subjective which reduces the
generalizability of their outcome. Therefore, some curriculum evaluators may find
field notes compiled by an expert who impartially observed the class while the
book was being taught more reliable sources of feedback and a better alternative
for teacher logs.

Software Packages
In recent years, computers have enabled book evaluators to examine the load and
distribution of vocabulary items presented by ELT books quickly and accurately.
Software packages like Concordance, EZText, HAMLET, HyperResearch,
MicroConcrd, and MonoConc Pro, TextQuest, WordNet, and WordSmith can be
used to find useful information on the words used in a book, such as:
1. Total number of running words (tokens),
2. Total number of different words (types),
3. List of all the words in alphabetical order or ordered based on their frequency, and
4. Concordance listing of a word, which shows a selected word as it was used in the
book with five words before and five words after it.
Research on ELT book evaluation using computer software prevails. These studies
focus on the presentation of varying vocabulary or grammatical items presented
in ELT books, to name but a few, articles (Mukundan, Leong, & Nimehchisalem,
2012), modal auxiliary verbs (Khojasteh, 2012), prepositions (Norwati, 2013), and
phrasal verbs (Zarifi, 2013).
Software like WordSmith offers a unique method to investigate how well a word
has been presented and repeated in a book, but this provides a narrow worm-view
of the suitability of a book. The overall suitability of a book cannot be based on
the outcome of such studies. Another shortcoming of this method of ELT book
evaluation is that it fails to study the book as it works in the real classroom setting.
Finally, in some cases, analysis may be demanding. Software can help only when
the book is available in text-file format. Otherwise, the book has to be scanned,
converted to text-file format, and edited before analysis.

Composite Frameworks

A composite framework of ELT book evaluation integrates two or more methods

to evaluate the suitability of an ELT book (Mukundan, 2009). A good example is
Retrotext-E (Mukundan, 2010), which enables an evaluation of:
1. Word loading and distribution patterns,
2. Checklist evaluation, and
3. Teacher log

5
V. NIMEHCHISALEM

The first function of the software enables the evaluator to obtain an overall
frequency count and distribution patterns of all the words in the book much in the
same way as WordSmith. It can also show to what extent the words in the book and/
or each of its chapters corresponds with those in a reference word list such as the
General Service List (West, 1953) or Academic Word List (Coxhead, 2000). In this
way, the evaluator can tell how adequately the book presents the essential vocabulary
to the target audience. The software also comes with a Likert style quantitative
checklist that the teacher can use after each lesson to evaluate the usefulness of that
particular lesson in a more systematic and objective manner. Finally, the Teacher log
function allows teachers to type and save their reflective logs and view other teachers’
logs who have also used Retrotext-E to keep their own logs and who wish to share
them with their colleagues. Thus, apart from its composite ELT book evaluation
framework, a unique feature of Retrotext-E is that it can save and compile all the
evaluations made by different teachers. These evaluations can all be reviewed later
by a program evaluator, publisher, author, or researcher who is interested in more
than one teachers’ evaluation of the same ELT book as it worked in their classes. The
shortcoming of the software is that it cannot be downloaded online yet, a feature that
its future versions are expected to have.

WHEN TO EVALUATE BOOKS?

It is possible to evaluate books before they are used, while they are being used,
as well as after they have been used, depending on the purpose of their evaluation

purposes. When a syllabus designer is looking for a book that best suits the needs of
the students in their target teaching situation, s/he needs to evaluate several books
before they have been actually used. A systematic prognostic evaluation of the
available books results in more informed decisions particularly when it is conducted
by a team of experienced evaluators who are aware of the target students’ needs
and interests as well as the target teachers’ preferences. Unfortunately, predictive
evaluation is totally ignored in traditional top-down educational systems where the
book is developed by government-sponsored publishers and prescribed to be used
all over the country. This invariably results in poor learning quality on the part of the
student whose needs and interests are often neglected while developing the book.
Sometimes program evaluators need to assess the usefulness of a book that has
been recently selected to be used in their present teaching situation. This type of
assessment, which is called in-use or progressive evaluation, will help the program
evaluator observe the way in which the book is used by a teacher in real language
classroom setting and find out the strengths and weaknesses of the book. This form of
evaluation often results in valuable findings that can be used for making adaptations
for certain parts of the book which were not really suitable for the students. These
changes are observed in future classes and may be revised or may replace the original
part in the book. This type of evaluation is formative in nature and allows an on-going

6
 WH QUESTIONS IN BOOK EVALUATION

assessment of each lesson and activity as they are taught throughout the course. In-
use evaluation is very valuable but it is also time-consuming and can be costly.
This type of evaluation is common in developed countries in which built-in syllabus
design procedures are followed (Ellis, 1993). In such learning-teaching contexts
syllabus designers adapt the syllabus to the way in which students tend to learn the
language (Brumfit, 1981). Hence, in order to fulfill the syllabus objectives, teachers
are free to select from a bank of books or materials suggested by the Ministry or to
develop their own activities and materials. This makes it crucial to evaluate teacher-
developed materials progressively in order to diagnose any probable shortcoming
and to ensure that their course is in line with the determined syllabus objectives.
Finally, at times ELT books may be evaluated after they have been used for
retrospective purposes. Post-use evaluation gives the evaluator an overall picture
of the way a book worked or failed to work in a given learning-teaching situation.
Based on the findings of such evaluation, ground rules are set for developing similar
materials in the future.

WHO SHOULD EVALUATE BOOKS?

Book evaluation can be a very complicated and demanding task with long-lasting
effects. ELT books have many stakeholders who are usually denied the right of being
able to voice their needs and interests. The person that seems to be the best candidate
to evaluate an ELT book is the teacher who is using it. However, it should be noted
that the teacher should have the adequate expertise and experience to provide a
valid and reliable evaluation of the book. Apart from these, it should be noted that
teaching is a low-paid and highly demanding job with heavy workload. Experienced
teachers are usually busier and having to evaluate a book may be the last straw.
Therefore, it is equally important to consider that the teacher is given sufficient time
with extendable deadlines to evaluate a book.
Determining who should evaluate books is a very tricky decision to make
particularly in situations where textbooks are prescribed by the State. If the same
state-sponsored publisher that developed the book is demanded to evaluate it, the
results will definitely be biased. Even if the best instrument is used, the evaluation
outcome will be far from reliable. Therefore, it may be argued that the best person
who can evaluate ELT books is an impartial expert that is also an experienced
teacher trained for this purpose. However, the question in such situations is whether
this individual is familiar with all the students’ needs and interests as well as all the
teachers’ preferences.

CONCLUSION

This chapter has discussed four basic questions in ELT book evaluation. First the
reason behind ELT book evaluation was discussed. As it can be concluded, book
evaluation is essential because it (1) helps curriculum developers select the right
V. NIMEHCHISALEM

book for a given group of students, (2) aids them to eliminate and adapt the unsuitable
parts in the book, which consequently (3) saves a lot of unnecessary hidden costs.
In addition to these, the most remarkable reason for book evaluation is that it results
in improved learner engagement and learning outcome. When a book is chosen
meticulously as a result of rigorous evaluation and when it is progressively assessed
for its usefulness, the instructor will take the course more seriously and the student
will find it more engaging and learn better.
Next, the different possible ways for book evaluation were discussed. Books are
frequently evaluated impressionistically based on the evaluator’s implicit criteria.
The outcome is often something very subjective. Even when the judgment is based
on an instrument, one cannot claim that one’s evaluation is completely objective.
This brings about issues of validity and reliability. There are methods to improve
the relevance (validity) and consistency (reliability) of the evaluation. However,
whether in reality these methods are followed to enhance the outcome makes an
enormous difference.
It was also noted that evaluation works best when it starts before, while, and
after its use. This means that book evaluation is often not the one-shot summative
type as it is commonly perceived. Rather, it has to be on-going and formative if it
is meant to yield some concrete dependable outcome. This sounds very easy, but in
practice, it is usually avoided since it costs time, money, and energy to evaluate a
book in this way.
Finally, the question of who should evaluate books was discussed. As mentioned,
numerous stakeholders may be involved in ELT book evaluation. The most important
stakeholders would be the learners whose needs must not be neglected; otherwise,
the selected book will not be learner-centered. As another group of stakeholders,
teachers usually look for books that are comprehensive, including a teacher’s guide
among other support materials, such as online test banks with answer keys. The
other group would be those in charge of covering the costs that could be the students
themselves, their guardians, their school, or the government. Publishers should also
be in the picture; otherwise, there will be no material available at all to be evaluated.
As it was concluded, the best person to evaluate a book would be an experienced
impartial teacher who has been trained for the task.
Those who are involved in evaluating ELT materials usually find themselves
tangled in a web of issues in their endeavor to make the right decision. In practice,
book evaluation is not an easy and smooth task to undertake. It may bring about
bitter consequences for some stakeholders who will choose to resist against
change. In such situations, one may think ignorance is bliss. However, it will
cause chronic irreparable damages to the educational system and ultimately to the
society. When a book is selected based on invalid criteria, misperception of learners’
needs, and incorrect assumption of the teacher’s preferences, it will negatively
influence the entire course. If managed efficiently, evaluation as it has been depicted
in this chapter can be a turning point in the history of an unsuccessful educational
system.

8
 WH QUESTIONS IN BOOK EVALUATION

REFERENCES
Brumfit, C. J. (1980). Seven last slogans. Modern Language Journal, 7(1), 30–31.
Brumfit, C. J. (1981). The notional syllabus revisited. Applied Linguistics, 2(1), 83–89.
Coxhead, A. (2000). A new academic word list. TESOL Quarterly, 34(2), 213–238.
Cunningsworth, A. (1995). Choosing your coursebook. Oxford: Heinemann.
SLA research and language teaching. Oxford: Oxford University Press.
Harmer, J. (1991). The practice of English language teaching. Harlow: Longman.
Khojasteh, L. (2011). A corpus-based study of modal auxiliary verbs used in the Malaysian English language
textbooks (Unpublished doctoral dissertation). Universiti Putra Malaysia (UPM), Serdang, Malaysia.
Mukundan, J. (2003). State-sponsored textbooks: Are there hidden costs in these “free” books? The English
Teacher: An International Journal of the Assumption University, Bangkok, Thailand, 6(2), 133–143.
Mukundan, J. (2009). ESL textbook evaluation: A composite framework. Köln: LambertAcademic Publishing.
Mukundan, J. (2010). Retrotext-E 1.0: The beginnings of computer-based ELT textbook evaluation.
Advances in Language and Literary Studies, 1
Mukundan, J., & Ahour, T. (2010). A review of textbook evaluation checklists across four decades
Research for materials development in
language learning (pp. 336–352). London: Continuum International Publishing Group.
Mukundan, J., Leong, A., & Nimehchisalem, V. (2012). Distribution of articles in Malaysian secondary
school English language textbooks. English Language and Literature Studies, 2
Mukundan, J., & Nimehchisalem, V. (2012). Evaluative criteria of an English language textbook
evaluation checklist. Journal of Language Teaching Research, 3(6), 1128–1134.
Richards, J. C. (2001). Curriculum development in language teaching. Cambridge: Cambridge University
Press.
Roslim, N. (2013). A corpus-based analysis of prepositions used in the English language textbooks
(Unpublished doctoral dissertation). Universiti Putra Malaysia (UPM), Serdang, Malaysia.
Sheldon, L. E. (1988). Evaluating ELT textbooks and materials. ELT Journal, 42
Skierso, A. (1991). Textbook selection and evaluation. In M. Celce-Murcia (Ed.), Teaching English as a
second or foreign language (2nd ed., pp. 432–453). Boston, MA: Heinle & Heinle Publishers.
Ur, P. (1996). A course in language teaching: Practice and theory. Cambridge: Cambridge University
Press.
West, M. (1953). A general service list of English words. London: Longman, Green and Co.
Zarifi, V. (2013). A corpus-based study of phrasal verbs in Malaysian ESL secondary school textbooks.
(Unpublished doctoral dissertation). Universiti Putra Malaysia (UPM), Serdang, Malaysia.

Vahid Nimehchisalem
Faculty of Modern Languages and Communication
Universiti Putra Malaysia
Malaysia

9
 SALEH AL-BUSAIDI

2. LEARNERS’ NEEDS IN
MATERIALS EVALUATION

INTRODUCTION

Materials and especially commercial textbooks often form the basis for learning and
teaching. In EFL settings, textbooks are the main source of contact with the foreign
language (Hamid et al., 2016). Materials evaluation can be a complex and sometimes
daunting task. Evaluators often develop or use criteria in the form of checklists to
guide their judgment. These checklists vary in their coverage from physical features
of the textbook to its underlying principles of teaching and learning. The intended
outcome from such an exercise is to identify the material that closely matches the
requirements of the target situation. The literature is replete with checklists (see

Nimehchisalem, 2012; Skiero, 1991). However, what is noticed about such checklists
is that they are differently informed and motivated (Roberts, 1996). Also, they tend
to be generic and program centered. They do not take into account the specific needs
of the learners who are the end users of such materials and their perspective should
therefore be central to the selection process. Learners become more motivated to
learn the language if it is presented to them in material that is relevant to their needs
and culture (Tajeddin & Teimournezhad, 2015). As is widely known, commercial
materials are not written to serve a particular group of learners. Their needs have to
be clearly defined. Needs generally refer to the gap between the current state and the
desired state. They can take the form of knowledge, skills or attitudes to be acquired.
Needs analysis is the process of identifying the gap between the existing skills and
knowledge and those that are needed for the target language use. It seeks information
on different types of information. First, it identifies the situations in which the
language will be used. Second, it describes the purposes for which the language is
needed. It also specifies the type and register of communication where the target
language will be used, that is, written, spoken, formal, informal, online, and offline.
Finally, needs analysis determines the level of proficiency that will be required.
Materials evaluators should remember that the assessment of learners’ needs is not
straightforward and therefore it requires eliciting information from multiple sources.
Information from needs analysis is used to determine the course goals and content.
It is crucial that needs analysis is conducted efficiently and carefully.
This chapter describes the needs analysis phase, which the author strongly argues
should precede or at least accompany the materials evaluation phase. It is only after

© KONINKLIJKE BRILL NV, LEIDEN, 2018 | DOI:10.1163/9789004387379_002

S. AL-BUSAIDI

we have established a firm understanding of the needs of the target learners that we
are able to find the most suitable materials. In many cases of materials evaluation,
learners’ needs are often oversimplified, and learners’ perspectives are often
overlooked. Materials are normally selected in relation to program objectives which
may and may not necessarily reflect the true learners’ needs, and other logistical
considerations. Alternatively, materials are evaluated based on readymade generic
checklist which are not designed for a particular group of learners (Sheldon, 1988).

WHAT ARE NEEDS?

The term ‘need’ refers to the gap between ‘what is’ and ‘what should be’ (Witkin
et al., 1995), or the gap between ‘real’ and ‘ideal’ (Reviere et al., 1996). In other
words, needs are concerned with the future, or what should happen, rather than with
what used to happen or what is happening at present (Titcomb, 2000). Researchers
have drawn distinctions between the term ‘need’ and other related terms. Hutchinson

the target situation) and learning needs (what learners need to do in order to learn).
Nation and Macalister (2010) divide needs into three types: necessities (what is
necessary in the learners’ use of the target language), lacks (what learners lack from
previous training or education), and wants (what learners wish to learn). A distinction
is also made between ‘subjective’ needs versus ‘objective’ needs. The former refer
to the needs as perceived by learners themselves, whereas the latter refer to needs as
determined by objective means such as tests. Subjective needs are identified through
self-assessment tools where learners decide about their current state and where
they want to be. This division of needs into different types helps understand them
and helps materials evaluators in their review work. It helps materials evaluators
prioritize the course focus and fit it within the time limit.

WHAT IS NEEDS ANALYSIS?

The process that is followed to identify learner needs is often referred to as ‘needs
analysis’. Other terms used are ‘needs assessment’ and ‘situation analysis’. These
terms are often used interchangeably. Nation and Macalister (2010) differentiate
between needs analysis and environment analysis. Needs analysis refers to the
learners’ needs of the target language, whereas environment analysis refers to the
situational factors that affect decisions about the course. Examples of the situational
factors include time factor, teachers’ background and education and resources. In
this chapter, I will use the term ‘needs analysis’ for the sake of consistency to cover
all elements related to learners’ needs and situational constraints.
Needs analysis determines the gaps between the current ability and the desired
ability. The rationale behind needs analysis is that each learner and learning situation
is unique and has their own purposes for learning a particular language. The concept
‘needs analysis’ is also used in industry and business. This stage is considered the

12
 LEARNERS’ NEEDS IN MATERIALS EVALUATION

most laborious, yet most important, aspect of materials evaluation. However, this is
one of the fundamental phases as it lays the ground for all subsequent work in the
process.
There are benefits in conducting needs analysis. First, it shows learners the
relevance of what they are learning to their needs and interests. Learners, especially
adults, are likely to be more engaged when they see the importance and relevance of
what they are learning. Second, needs analysis helps learners identify where they are
in terms of knowledge, skills and competencies and where they wish to be. Needs
analysis is used to discover leaner needs and present them into meaningful units
that can be used to create learning goals and objectives and subsequently develop or
evaluate materials. There is no best way to identify learners’ needs. People follow
different procedures and use different means of establishing needs. The decision
about which way to use depends on the scope, resources and constraints. However,
in many cases, materials evaluators and teachers inevitably rely, at least partially,
on their prior knowledge and experience about what learners need. While learning
situations share many characteristics, each situation has its own requirements,
constraints and resources.
Answers to the questions in the needs analysis should lead to decisions about
the different aspects materials evaluators should pay attention to when reviewing
materials. Depending on how extensive and detailed the needs analysis is, the
information gleaned from this procedure will vary. The next section discusses some
fundamental considerations in conducting needs analysis.

KEY CONSIDERATIONS IN NEEDS ANALYSIS

There are some important points that materials evaluators should consider when
conducting needs analysis. One important fact to realize about needs analysis is
that it is an ongoing process. There are two aspects to this. First, it is not always
possible to conduct needs analysis before the course takes place. Teachers may find
themselves in situations where the course is already underway. In this case, teachers
can find out about learners’ needs. One way to do this is to ask learners to rate a list
of statements that describe their learning preferences, course content, etc. (Nation &
Macalister, 2010). Second, needs are not straightforward. Their identification and
interpretation is based on the values and judgments of the different ‘stakeholders’.
It is therefore crucial that information about learner needs is elicited from various
sources. Third, needs are dynamic in nature and they are likely to change after the
initial needs analysis, depending on learner variables and environmental constraints.
If the same course is offered to a different group of learners, there should ideally be
another round of needs analysis conducted in order to identify the needs of the new
students and determine the course elements that need to be modified. This will ensure
that the course responds to the students’ needs and interests. It is therefore necessary
to repeat the needs analysis exercise and make it part of the material evaluation plan.
This needs analysis does not have to be extensive.

13
S. AL-BUSAIDI

Another important consideration is the relevance of the data gathered. This will
be determined by the way information is gathered and the source of that information.
Each learning situation consists of a number of stakeholders (i.e., individuals who
benefit or are affected by the course). The materials evaluator should identify the ‘key’
stakeholders who should be targeted at the needs analysis stage. These data sources
should be the ones who will give objective information about what is required. When
learners realize that the course is arranged in a way that responds to their needs and
interests they feel motivated to exert more effort in meeting its requirements.
A further important consideration is reliability and validity of the needs analysis
tools. Needs are varied and they constantly change. They are also not straightforward.
Therefore, it is important that needs analysis covers a wide range of perspectives and
situations where the target language is used. It is important that learners’ perspectives
are sought and considered when new materials are evaluated. However, the reliability
and validity of self-perceived student data should be checked. The complexity
of needs analysis requires that tools used for data collection be reliable and valid
(Nation & Macalister, 2010). Reliability and validity can be maximized through the
use of varied and standardized instruments. Examples of standardized instruments
include tests and observation checklists. The more sources of information are used
the more reliable the data is likely to be. As for validity, the materials evaluator
should ensure that the tools are relevant to the task and to the nature of the course.
Again, this should be judged by experts and professionals other than those involved
in the course. Nation and Macalister (2010) point out that in many cases logistics
and practicality overrule validity and reliability but they stress that priority should
always be given to validity.
When gathering information about learners’ needs, we should distinguish
between what learners ‘want’ and what they actually ‘need’. It is the needs that
should be given priority for instruction to have impact on learning. In some situations,
learners may not be aware of what will make the best impact. For example, for a
group of adult learners who need to learn English to write technical communication,
they may want to be given ready-made formats and expressions instead of acquiring
the skills of expressing meaning and communicating ideas using a variety of
linguistic tools.
We should remember that if the new materials are intended to bring change to the
learner either in terms of new knowledge, behavior or attitude, then the evaluation
should be based on the learner’s needs and interests. Judgments about course goals,
objectives and content should not be based on the material evaluator’s assumptions
but rather on rigorous and systematic analysis of the learner’s needs. Proper needs
analysis leads to informed decisions about instructional intervention. Finally,
information that is obtained from needs analysis must be relevant to and useful for
the development of the course as a whole. Therefore, it is advisable to pilot the
instrument first to measure its effectiveness and identify any potential factors that
might affect the process of obtaining information about learn needs. The usefulness
of a needs analysis depends on the type of data gathering techniques used.

14
 LEARNERS’ NEEDS IN MATERIALS EVALUATION

HOW ARE NEEDS IDENTIFIED?

There is no one standardized, agreed upon procedure for conducting needs analysis,
as it depends on the nature of the course, scope of work and the resources and
constraints in each situation. However, researchers have offered some guiding steps
that can help materials evaluators in their mission of investigating learners’ needs.
Table 2.1 summarizes these steps. This procedure assumes that needs analysis is a
recursive process.

Table 2.1. Steps of needs analysis

No. Step Explanation

1 Identify the purpose and audience of the Decide about who the stakeholders
needs analysis. are.
2 Decide on the information to be gathered. List the type of information you need
to discover. Are you interested in the
situations, tasks, topics, or skills?
3 Decide on the best way to gather There are various instruments but the
information. materials evaluator should choose
the most suitable one(s).
4 Gather data. Data can be gathered in notes, forms,
audio or video recording.
5 Interpret data. Find ways to make sense of the
data. Data needs to be classified
into meaningful chunks.
6 Act on data. Use findings to make decisions about
different aspects of the course that
the materials should meet.
Evaluate the impact and effectiveness Once the materials have been adopted
of the action. and used, they should be evaluated in
terms of relevance and impact.
8 Revise the course as needed or gather Based on feedback, the materials are
further information. adapted or replaced; or further data
is gathered.

There are many ways materials evaluators can use to arrive at learners’ needs.
The decision about the best way to use depends primarily on the nature of the course
and the context, and resources available. The different types of needs analysis tools
include:
1. Interviews (individual or group)
2. Observations
3. Questionnaires

15
S. AL-BUSAIDI

4. Tests (placement, diagnostic, achievement, proficiency)

5. Case studies (individual or group)
6. Diary studies
Records analysis
8. Text analysis (e.g. analysis of materials learners will need to comprehend)
9. Meetings/discussions
10. Systems analysis (analysis of organizational structures and decision making
process)
11. Literature review (review of research findings on similar situations or needs)
Discovering needs involves identifying the situations where the target language
will be used, the tasks that the learners need to perform and the linguistic knowledge
required to function in these situations and tasks. Based on the context, the materials
evaluator chooses the most suitable instrument that elicits spoken or written input
from the learners. For example, learners may be asked to respond to a prompt or a set
of questions. Alternatively, the materials evaluator may decide that it is important to
observe learner behavior in action. Good needs analysis utilizes more than one tool
for the sake of achieving maximum reliability of data.
Once data is gathered, it needs to be interpreted. Very little is said in the literature
about the interpretation of data from needs analysis. Researchers seem to assume
that the materials evaluator has the skills of analyzing and interpreting data. From
my experience, data collection and data analysis or interpretation are two completely
different things. The value of any data lies in the way it is interpreted. Therefore, data
interpretation is the most critical stages as needs analysts make sense of the data they
have gathered and use that data to take decisions about learning and teaching. I shall
try to give the reader some tips that will hopefully be useful in tackling this complex
part of needs analysis. The type of information that needs analysis can yield will
depend on the nature and extent of the procedure undertaken. Some needs analysis
exercises are more extensive and tap on more information sources than others and
therefore result in more extensive data. Below are examples of the information that
needs analysis can produce.
Who the learners are
The learners’ current level of the language
Their interests
Their learning preferences
Their attitudes
Their goals and expectations
Target contexts when the target language is used
When data is gathered in its raw format, it is likely to be in a mass. The data
might be qualitative or quantitative. The data analyst’s role will be to give meaning
to the information gathered. Analyzing data means determining how to organize,
synthesize and interrelate the different issues present in it. Since data can be

16
 LEARNERS’ NEEDS IN MATERIALS EVALUATION

analyzed and interpreted in many different ways, it is advisable to involve other

people in the process. This would help avoid subjectivity in the interpretation and
will result in greater understanding of the data. In some cases, the data will need to
be compared with other criteria, standards or benchmarks. For example, if learners
are given a test as a way to measure their language proficiency, their performance
should ideally be measured against standardized language proficiency levels. The
following task will demonstrate the different instruments that are popular in needs
analysis.

Task
Look at the list of situations below and decide about the best instrument(s) you would
use for needs analysis based on the nature of the course. You need to hypothesize
about the different aspects of the course, such as level of the students, their workplace,
etc. Once you have decided about the best instrument for each situation, compare
your work with other students. Discuss your choices.
In order to help you decide about the best instrument for your course/situation, it
would be good to think of a context, an imaginary context, where the course would
be taught. Once you have defined your context, choose one or two instruments that
you would use to collect information about the learners’ needs. Then write a brief
description for each instrument.

Table 2.2. Needs analysis instruments and audience in different situations

No. Situation Instrument Target audience

1 An English course for taxi drivers

2 An English course for mechanics
3 An English course for office secretaries
4 An English course for hospital housekeepers
5 An English course for children (3–5 years old)
6 An English course for pilots
An English course for housemaids

Now look at the third column in Table 2.2. Think about the people you would
need to elicit information from about each situation. Again, compare notes with
another student and discuss your choices.
The outcome from needs analysis is a list of the literacy requirements
that learners need in their use of the target language. This should include the
competencies that enable learners to perform the tasks. One important point to
consider at this stage is that the materials evaluator’s beliefs about language
learning and teaching will inevitably affect their interpretation of the data.
S. AL-BUSAIDI

However, one way to reduce the effect of beliefs is to get other perspectives on
the analysis and the interpretation.
Once needs are determined, decisions about the most suitable materials should
be made. It would be difficult to find a single commercial textbook that caters for
the precise needs of a particular group of learners. It is therefore advisable that more
than one textbook be selected and piloted before any of them is finally adopted for
use. Many commercial textbooks claim that they have been designed ‘with learners’
needs in mind’. Like any business, publishing is also concerned with maximizing
profit. As a result, textbooks are sometimes seen as a ‘commodity’ that must sell. To
achieve this, publishers try all means to market their materials. They ‘cast their net’
quite widely in order to capture as many users as possible. This invariably comes at
a cost as it would be quite impossible to write materials to respond to specific needs
while at the same time target a very wide audience.
When reviewing materials, evaluators should consider the pedagogical
aspects of the materials, in terms of presentation, methodology, etc. Information
about such aspects may not be easily evident in the needs analysis data or in
the materials themselves. The appropriateness of the methodology adopted
by a certain textbook to the learning situation also depends on the teachers’
preferences and beliefs which should be sought in the needs analysis stage. In
fact, for more successful and effective learning and teaching, teachers should be
involved throughout the stages of the review. “If teachers are not enthused by
materials their dissatisfaction is always apparent to the learners, the materials
lose credibility and the learners motivation and investment of energy are reduced”

However, materials evaluators and teachers should first identify the learners’
needs and then match them with the closest materials. Material adaption is inevitable
in the field of English language teachers. Teachers almost always find themselves
having to supplement the materials assigned to the course. Many teachers actually
welcome that and see this as an opportunity to use their creativity and connect the
course more closely to the learners’ needs.
It is hoped such an extensive needs analysis will yield important data that materials
evaluators could then confidently use as criteria for vetting any teaching material.

CONCLUSION

Needs analysis is the forgotten basis for the materials evaluation. Materials
evaluators often rely on readymade checklists that have been generically designed
for a wide range of materials. Such checklists do not normally take into account
the specific needs of the learners. Also, in such situations, learners do not normally
have the chance to express their views about the new materials. Needs analysis gives
learners the opportunity to be part of the decision making process rather than wait
until the material has been adopted and used. However, materials evaluators should
remember that needs are not normally clear and straightforward. Identifying needs

18
 LEARNERS’ NEEDS IN MATERIALS EVALUATION

requires the selection of suitable instruments, identification of the right informants

and the right analysis and interpretation of the data. Once needs are established, it
will be much easier for the materials evaluator to select the teaching materials that
best fit the learners’ needs. There are certain guidelines that the materials evaluators
can consult in carrying their needs analysis.

REFERENCES
Cunningsworth, A. (1995). Choosing your coursebook. Oxford: Heinemann.

L. McIntosh (Eds.), Teaching English as a second or foreign language

MA: Newbury House Publishers.
Hamidi, H., Bagheri, M., Sarinavaee, M., & Seyyedpour, A. (2016). Evaluation of two general English
textbooks: New interchange 2 vs. four corners 3. Journal of Language Teaching and Research, 7(2),
345–351.
English for specific purposes. Cambridge: Cambridge University
Press.
Mukundan, J., & Nimehchisalem, V. (2012). Evaluative criteria of an English language textbook
evaluation checklist. Journal of Language Teaching & Research, 3(6), 1128–1134.
Nation, I. S., & Macalister, J. (2009). Language curriculum design. New York, NY & London: Routledge.
Reviere, R. (1996). Needs assessment: A creative and practical guide for social scientists. Washington,
DC: Taylor & Francis.
Roberts, J. (1996). Demystifying materials evaluation. System, 24
Sheldon, L. E. (1988). Evaluating ELT textbooks and materials. ELT Journal, 42
Skierso, A. (1991). Textbook selection and evaluation. In M. Celce-Murcia (Ed.), Teaching English as a
second or foreign language (pp. 432–453). Boston, MA: Heinle and Heinle Publishers.
Tajeddin, Z., & Teimournezhad, S. (2015). Exploring the hidden agenda in the representation of culture
in international and localised ELT textbooks. The Language Learning Journal, 43(2), 180–193.
Titcomb, A. (2000). Needs analysis: ICYF evaluation concept sheet. Tucson, AZ: University of Arizona.
Tomlinson, B. (1998). Comments on part B. In B. Tomlinson (Ed.), Materials development in language
teaching (pp. 146–148). Cambridge: Cambridge University Press.
Witkin, B. R., & Altschuld, J. W. (1995). Planning and conducting needs assessments: A practical guide.
Thousand Oaks, CA: Sage Publications.

Saleh Al-Busaidi
College of Education
Sultan Qaboos University
Sultanate of Oman

19
 DARÍO LUIS BANEGAS

3. EVALUATING LANGUAGE AND

CONTENT IN COURSEBOOKS

INTRODUCTION

English language teaching coursebooks, as cultural artifacts in formal education

(Gray, 2010; on cultural representations see Tajeddind & Teimournezhadb, 2014),
are not simply an end-product. They are developed through a complex circuit which
includes market analysis, design, piloting, distribution, and Ministry approval in
some contexts, implementation and use (McGrath, 2013). Such a circuit is inevitably
permeated by contextual forces and affordances which enter in a dialogic relationship
with the ELT curriculum, planning, and practice as illustrated in the literature (Atai

Haqverdi, 2013; Sabet & Sadeh, 2012).

The role which teachers play in this circuit is paramount. They are in charge
of implementing, adopting, and, probably or expectedly, adapting coursebooks.
In other words, teachers bring coursebooks to life and not necessarily in ways
naively expected by their authors. Teachers are critical and active agents and not
robots following a teacher’s guide (Kiai, 2013). They need to evaluate coursebooks
according to their context, learners’ needs, institutional needs, and the approach
which they seek to explore.
In this chapter, I focus on evaluating language and content in coursebooks
following CLIL (Content and Language Integrated Learning), where the
language is English as a foreign language (EFL). I first provide reasons why
teachers should evaluate coursebooks whatever their approach. Secondly, I offer a
brief outline of CLIL and how it may be found in coursebooks. Thirdly, I suggest
ways in which teachers can decide if a coursebook is worth adopting under a CLIL
framework.

TEACHERS AS COURSEBOOK EVALUATORS

In a thorough review of materials development in ELT, Tomlinson (2012) addresses

key aspects such as evaluation criteria, writers’ perspectives, and the value of
coursebooks among others. In relation to this last aspect, Tomlinson (2012, p. 158)
raises a crucial issue: teachers’ empowerment:
Opponents of coursebooks argue that they can disempower both teacher and
learners, cannot cater for the needs and wants of their actual users, are used

© KONINKLIJKE BRILL NV, LEIDEN, 2018 | DOI:10.1163/9789004387379_003

D. L. BANEGAS

mainly to impose control and order (e.g., ‘OK, class, turn to page 46 of your
coursebook’, Mukundan, 2009b, p. 99), and provide only an illusion of system
and progress.
When the perception that coursebook disempower teachers is inscribed in
our practices, teachers can challenge this view by turning into evaluators of the
coursebooks they use or wish to use. McGrath (2013, p. 105) notes that “coursebooks
are normally an investment, in more ways than one”, and this entails that teachers
need to become critical users and consumers of marketed materials in order to match
their own aims with those in a coursebook. However, this does not mean regarding a
coursebook as a given, an inherent element in our practices, or the only resource we
should benefit from. There are other types of materials (see Stanley, 2013) and there
could also be lessons with almost no materials (see Thornbury, 2013).
Even when coursebook choice is not in the hands of teachers, they can still
evaluate them in order to enrich them with context-responsive teacher-made
materials and become authors of their practices, pedagogies and materials. Taking a
coursebook as an exercise is another way of professional development individually
or collaboratively. However, McGrath (2013) observes that teachers who are not
usually in a position to make decisions on coursebooks cannot be asked to evaluate
them without any guidance or support from coordinators or department heads.
Coursebook evaluation needs to be a systematic activity, and by systematic I mean
an activity based on informed and supported views that make room for inter-
subjectivity rather than personal feelings or random perceptions.
Teachers should approach coursebook evaluation from a constructive and positive
perspective. In other words, they need to see the extent to which a coursebook
matches their aims and context and in what ways they can enrich it with other
materials. That said, we may agree that there is no perfect coursebook and that we
will always find something lacking or missing. In this regards, we need to remind
ourselves that a coursebook is not the solution to all our problems, but one tool to
help us create our own responses.

CLIL AND COURSEBOOKS

As I mentioned above, teachers can be guided and empowered through coursebook

evaluation by providing them with the rationale behind a coursebook and the
approach their practices and coursebooks should respond to. Although we believe
that the rationale and approach behind a coursebook are the result of connections
between theory and practice which usually appear in the teacher’s book, Tomlinson
(2013) still urges coursebook writers to produce principled materials, that is, ELT
materials based on second language acquisition theory and research.
That said, I offer below a brief summary of CLIL before engaging in suggestions
based on CLIL-driven coursebooks.

22
 EVALUATING LANGUAGE AND CONTENT IN COURSEBOOKS

CLIL is an approach which condenses practices through which learners develop

foreign language learning together with curriculum content. It is underpinned by
sociocultural concepts such as scaffolding and mediation, and contributions from the

According to Cenoz (2013), CLIL could be conceptualized and implemented as

an educational approach or a language teaching approach. Such a differentiation
seems to respond to the fact that CLIL could be adopted through two macro models:
content-driven (CLIL as an educational approach, see also Dalton-Puffer & Nikula,
2014) or language-driven (CLIL as a language teaching approach). On the other hand,
it also seems to respond to what I call the geographies of CLIL, i.e. the sociocultural
settings where it is implemented and the resources available. CLIL in Europe is
becoming content-driven, but in other places, CLIL is seen as an opportunity to
revitalise the ELT curriculum only. In line with this last view, Hall (2011, p. 195)
wonders: “what is the relationship between CLIL and ‘strong’ forms of CLT?” In the
paragraphs below, I discuss both content- and language-driven CLIL in more detail.
Content-driven CLIL refers to the teaching of a subject, for example History
or Science, through a language other than the usual language of instruction, for
example English in Spain (see Morton, 2013). In this model, the focus is on learning
the subject matter or curriculum content and another language at the same time. In
this model, learners learn for example, Science, and the specific genre, grammar, and
lexis, to talk about different Science contents (see Llinares, Morton, & Whittaker,
2012). Activities and resources scaffold both content and language learning (see

content over language as, in most cases, subject-matter or non-language teachers

proficient in the target language, are in charge of the lessons. Those familiar with
bilingual education (see Abello-Contesse et al., 2013) or immersion programs (see
Somers & Surmont, 2012) may see that CLIL is not truly innovative or original.
On the other hand, language-driven CLIL refers to the incorporation of curricular
content into the EFL lessons. These lessons remain in the ELT teachers’ hands and
they may seek support from their non-language colleagues. While the aim is to
learn content and language together, activities, materials, and assessment prioritize
language over content. In relation to the curricular content, this needs to be new to
the learners or familiar to them to some extent. For example, learners may know
about poetry and stylistics in their L1 and through CLIL they may learn about poetry
and literary movements in the UK.
In relation to content, two clarifications are needed. First, the content should be
curriculum-related. If the lessons integrate language with any content, then it may
be seen as a topic-based lesson, something usually found in task-based learning or
communicative language teaching. Secondly, the curricular content needs to provide
a cognitive challenge and be original. This clarification is necessary because some
teachers and administrators may wrongfully believe that CLIL means teaching the
same content twice, that is, through two different languages (e.g. Persian and English).
CLIL becomes a motivating approach when the contents are a novelty, there is an

23
D. L. BANEGAS

element of authenticity of purpose (Pinner, 2013), and learners’ voices are sought
to encourage involvement, discovery and awareness (Banegas, 2013; Coyle, 2013).
Language also deserves some clarifications. Language learning should be
presented through all skills (speaking, listening, reading, and writing). Furthermore,
language teaching in CLIL should help learners not only expand their content-
specific vocabulary but also the language needed to talk about the content and
solve the different tasks proposed by the teacher. These recommendations have
been skillfully integrated into the language triptych (Coyle, Hood, & Marsh, 2010).
The triptych includes: (1) language of learning, i.e. content-specific terminology,
(2) language for learning, i.e. language to, for example, compare two rivers
or two historical events (comparative adjectives, linkers, etc.) and (3) language
through learning, i.e. language needs which emerge from the teaching and learning
processes.
Overall, CLIL is based on learning curricular content and another language with
others and scaffolded through materials which offer opportunities for language and
cognitive development. In the section which follows, I discuss the extent to which
these features are found in the language-driven CLIL market.

CLIL IN COURSEBOOKS

There is growing interest in producing guidelines for developing CLIL materials

aimed at authors and teachers (see Evans, James Hartshorn, & Anderson, 2010;
Mehisto, 2012; Meyer, 2010). In general, such guidelines stress the aspects which I
list below:
Cohesion
Stability
Flexibility
Responsiveness connected to the curriculum
Contextualized sources of input and activities
Cognitive-rich and language-rich texts
Inclusion of authentic sources and tasks
However, there are only a few publications which examine CLIL materials. In
Argentina, where CLIL can be found as language-driven in state schools but also as
content-driven in private bilingual schools, Banegas (2014) investigated how CLIL
is represented in general English coursebooks. To this effect he analyzed four series
adopted by teachers in Argentina and found:
(1) little correlation between featured subject specific content and school
curricula in L1, (2) oversimplification of contents and (3) dominance of reading
skills development and lower-order thinking tasks. (Banegas, 2014, p. 345)
Towards the content end of CLIL, Morton (2013) investigated teachers’ use
of CLIL materials in four European countries (Austria, Finland, Spain, and the

24
 EVALUATING LANGUAGE AND CONTENT IN COURSEBOOKS

Netherlands). Through an online survey and data from a multi-case study, the author
found that:
Teachers did not tend to use materials designed for native speakers.
Teachers adapted authentic materials in line with their teaching aims.
Teachers developed their own materials from scratch.
Teachers were mostly concerned with appropriateness of materials for learners in
terms of content, language, and context.
Both Banegas (2014) and Morton (2013) suggest that the market cannot cater for
all teachers’ needs and contexts, and that teachers may create their own materials.
Nonetheless, they suggest that coursebooks are still a useful organizing tool for
teachers.
Unfortunately, this section cannot go any further for an interest in published
CLIL materials is incipient. While we can find CLIL books mainly produced for
the European market particularly aimed at secondary education schools which adopt
CLIL as an educational approach, the CLIL-coursebook market for ELT, CLIL as a
language learning approach, is still at an embryonic stage. At the same time, I do not
wish to address the possibility of teachers as CLIL material developers. That deserves
a self-contained publication elsewhere. However, teachers can refer to different
publications (Mehisto, 2012; Meyer, 2010; Mehisto, Marsh, & Frigols, 2008).

EVALUATING CLIL IN COURSEBOOKS: WHAT TO LOOK FOR

English language teachers wishing to explore CLIL through marketed coursebooks

can choose:
a. Coursebooks written in English as the L1 for students in English-speaking
countries.
b. General English language learning coursebooks internationally commercialised.
c. Specially written CLIL coursebooks, which have a European audience in mind.
I am confident that teachers can find and create other alternatives, but below I
discuss the options above. Through questions, I provide suggestions on what to look
for based on my experience as a teacher. I must clarify that the bullet-point questions
which follow are merely a guide of what teachers may look for. I hope that my own
informed decisions and explorations serve as the basis for coursebook evaluation in
terms of language and content.

Coursebook Written in English as the L1

In my identity as a teacher I usually enrich my lessons with coursebooks targeted
at learners in mainstream education in the UK. For example, I use A-level study
guides for Geography, History, and Literature to expand on a reading text in an ELT
coursebook.

25
D. L. BANEGAS

Above all, I consider my learners’ motivations, needs, and development and the
curriculum which I am expected to respond to. With these factors in mind, I take a
Geography study guide for example and I ask myself:
Does the coursebook contain contents and ideologies which are similar to my
own curriculum? If it doesn’t, how can I make links or provide learners with
different views?
To what extent is it UK-driven? Are there ‘international’ topics?
Is it easy to purchase and affordable? Are there any restrictions?
Is the language accessible?
Is the book content-rich and language-rich?
Does it encourage lower- as well as higher-order thinking skills?
Are texts broken down into manageable units? If not, how can I achieve that?
Does it contain summaries?
Does it contain graphics and useful visual support to scaffold learning?
Does it include a companion website with free access activities?
Does it come with activities, boxes with key words, or a glossary? If it doesn’t,
how can I create activities based on the texts offered?
If there are activities, does it include answer keys for self-study or peer correction
and reflection?
Does it offer flexibility in terms of content organization? Can I start with Unit 3
and then cover Unit 1?
Even if I choose one among several options, I know that I do not need to follow
it strictly as if it were a script. More importantly, I need to ensure that even learners
with a low level of English, whatever this means, can still profit from more complex
texts. I always have in mind Tomlinson’s (2008) position of approaching materials
as language learning materials and that we should not treat linguistically low level
learners as intellectually low level learners.

General English Language Learning Coursebooks

I must admit that I am more critical with ELT coursebooks for a number of
reasons: (1) I am a teacher of English, (2) coursebooks writers are ELT experts or
knowledgeable of our field, and (3) coursebooks still dominate the ELT scene. When
I wish to adopt a general English coursebook with a CLIL component, I ask myself
the following questions:
What target audience does the book aim at?
Is CLIL included as a regular or add-on component?
Is a given curricular content the core of a unit? Does it appear as decorative/
illustrative or disjointed from the rest of the unit?
Is the content relevant and cognitively challenging, yet, scaffolded?
Is the content related to my school curriculum or is it just ‘any content or topic’?
Is content presented through different oral, audiovisual, and written formats?

26
 EVALUATING LANGUAGE AND CONTENT IN COURSEBOOKS

If it is presented only through reading, do the activities promote higher-order

thinking skills and content appropriation? Or are they simply superficial reading
comprehension questions?
Is the language accessible and scaffolded?
Is the language found in the content section part of the unit in which it is included?
Is there evidence of recycling and sequencing?
Are there activities which integrate content and language through activities which
focus on vocabulary, syntax, and (subject-specific) discourse?
Are there activities to promote language of/for/through learning?
Do activities and texts integrate a focus on forms as well as meaning?
As teachers we need to look for general English coursebooks where CLIL is
systematic and responsibly included rather than CLIL as a watered down alternative.
In addition, the content needs to be cognitively stimulating and differ from contents
usually found in ELT coursebooks (see Banegas, 2014). Otherwise we may run the
risk of naming CLIL something that may as well be named topic-based learning or
simply communicative learning.

CLIL Coursebooks
Although CLIL coursebooks are relatively new and tend to be European-focused,
teachers may wish to consider this option because the coursebook approach and
rationale are (or should be) based on CLIL research and theoretical underpinnings.
We may agree that the questions I posited for general English coursebooks with a
CLIL component may also be applicable for CLIL coursebooks. However, I should
remark that the activities must take learners from lower to higher-order thinking
skills and that the content should be complex enough in order to ensure cognitive
development and motivation. Furthermore, the coursebook should aim at providing
learners with language of/for/through learning and features of subject-specific
discourse through awareness raising and explicit notes.
I include below some guiding questions to help teachers decide whether they wish
to adopt a CLIL coursebook:
Do the coursebook contents reflect my curriculum and learners’ interests as well
as needs?
Is there variety in terms of sources of input and activities?
Is the language too easy or too difficult?
Are there authentic texts or modified texts?
Is the content too easy or too difficult?
Are contents related and sequenced?
Is the coursebook internally coherent and cohesive?
Is it flexible? Can I skip parts? Can I change the order in which units are
presented?
D. L. BANEGAS

CONCLUSION

Whatever the choices and informed decisions made by teachers, it is essential that
learners and their context are assessed against coursebooks and our CLIL aspirations.
By context I do not only mean the geographical and institutional location but also
the curriculum, educational policies, educational practices, and teachers’ views. The
coursebook needs to be taken as a tool, not a corset. It should open windows rather
than restrict practices.
We may agree that evaluating a coursebook is not an easy task because our decision
will have an impact on our teaching and learning practices. Nonetheless, coursebook
evaluation is also an ongoing process that we carry as we use a coursebook. Along
these lines, coursebook evaluation is a process that starts before we teach a course
and continues even after we finish with the course.
Last, evaluating coursebooks within a CLIL approach should be seen as a
professional development opportunity to be carried out collaboratively. Teachers
need to ensure that content and language are treated holistically but with instances to
practice both separately and together through different activities.

REFERENCES
Abello-Contesse, C., Chandler, P. M., López-Jiménez, M. D., & Chacón-Beltrán, R. (Eds.). (2013).
Bilingual and multilingual education in the 21st century: Building on experience. Bristol: Multilingual
Matters.
Atai, M. R., & Mazlum, F. (2013). English language teaching curriculum in Iran: Planning and practice.
Curriculum Journal, 24(3), 389–411.
Banegas, D. L. (2013). The integration of content and language as a driving force in the EFL lesson.
In E. Ushioda (Ed.), International perspectives on motivation: Language learning and professional
challenges
Banegas, D. L. (2014). An investigation into CLIL-related sections of EFL coursebooks: Issues of CLIL
inclusion in the publishing market. International Journal of Bilingual Education and Bilingualism,
17(3), 345–359.
Cenoz, J. (2013). Discussion: Towards an educational perspective in CLIL language policy and
pedagogical practice. International Journal of Bilingual Education and Bilingualism, 16(3), 389–394.
Chadwick, T. (2012). Language awareness in teaching: A toolkit for content and language teachers.
Cambridge: Cambridge University Press.
Coyle, D. (2013). Listening to learners: An investigation into ‘successful learning’ across CLIL contexts.
International Journal of Bilingual Education and Bilingualism, 16(3), 244–266.
Coyle, D., Hood, P., & Marsh, D. (2010). Content and language integrated learning. Cambridge:
Cambridge University Press.
Dalton-Puffer, C., & Nikula, T. (2014). Content and language integrated learning. Language Learning
Journal, 42
Teaching other subjects through English. Oxford: Oxford University Press.
Evans, N. W., James Hartshorn, K., & Anderson, N. J. (2010). A principled approach to content-based
materials development for reading. In N. Harwood (Ed.), English language teaching materials:
Theory and practice (pp. 131–156). Cambridge: Cambridge University Press.
Ghorbani, M. R. (2009). ELT in Iranian high schools in Iran, Malaysia and Japan: Reflections on how tests
influence use of prescribed coursebooks. Reflections on English Language Teaching, 8(2), 131–139.
Gray, J. (2010). The branding of English and the culture of the new capitalism: Representations of the
world of work in the English language coursebooks. Applied Linguistics, 31

28
 EVALUATING LANGUAGE AND CONTENT IN COURSEBOOKS

Hall, G. (2011). Exploring English language teaching. Abingdon: Routledge.

Asian EFL
Journal, 9(2), 130–150.
Kiai, A. W. (2013). Am I a robot? English language teachers on teachers’ guides. Argentinian Journal of
Applied Linguistics, 2(1), 23–46.
Llinares, A., Morton, T., & Whittaker, R. (2012). The roles of language in CLIL. Cambridge: Cambridge
University Press.
McGrath, I. (2013). Teaching materials and the roles of EFL/ESL teachers: Theory and practice.
London & New York, NY: Bloomsbury.
Mehisto, P. (2012). Criteria for producing CLIL learning material. Encuentro, 21, 15–33.
Mehisto, P., Marsh, D., & Frigols, M. J. (2008). Uncovering CLIL: Content and language integrated
learning in bilingual and multilingual education. Oxford: Macmillan.
Meyer, O. (2010). Towards quality-CLIL: Successful planning and teaching strategies. Pulso, 33, 11–29.
Morton, T. (2013). Critically evaluating materials for CLIL: Practitioners’ practices and perspectives.
In J. Gray (Ed.), Critical perspectives on language teaching materials (pp. 111–136). Basingstoke:
Palgrave Macmillan.
Revista Española de Lingüística
Aplicada, 1(Volumen Extraordinario)
Pinner, R. (2013). Authenticity of purpose: CLIL as a way to bring meaning and motivation into EFL
contexts. Asian EFL Journal, 15(4), 138–159.
Sabet, M. S., & Sadeh, N. (2012). CLIL European-led projects and their implications for Iranian EFL
context. English Language Teaching, 5(9), 88–84.
Somers, T., & Surmont, J. (2012). CLIL and immersion: How clear-cut are they? ELT Journal, 66(1),
113–116.
Stanley, G. (2013). Language learning with technology: Ideas for integrating technology in the classroom.
Cambridge: Cambridge University Press.
Tahmasebi, G.-A., Ghaedrahmat, M., & Haqverdi, H. (2013). The relationship between language
proficiency and Iranian EFL learners’ knowledge of vocabulary depth versus vocabulary breadth.
Latin American Journal of Content and Language Integrated Learning, 6(2), 96–111.
Tajeddin, Z., & Teimournezhad, S. (2015). Exploring the hidden agenda in the representation of culture in
international and localised ELT textbooks. The Language Learning Journal, 43(2), 180–193.
Thornbury, S. (2013). Resisting coursebooks. In J. Gray (Ed.), Critical perspectives on language teaching
materials (pp. 204–223). Basingstoke: Palgrave Macmillan.
Tomlinson, B. (2008). Language acquisition and language learning materials. In B. Tomlinson (Ed.),
English language learning materials: A critical review (pp. 3–13). London & New York, NY:
Continuum.
Tomlinson, B. (2012). Materials development for language learning and teaching. Language Teaching,
45
Tomlinson, B. (2013). Second language acquisition and materials development. In B. Tomlinson (Ed.),
Applied linguistics and materials development (pp. 11–31). London & New York, NY: Bloomsbury.

Darío Luis Banegas

University of Warwick
Coventry, UK

29