Practical Problem-Based Learning in Computing Education

MICHAEL J. O’GRADY, University College Dublin

Computer Science (CS) is a relatively new disciple and how best to introduce it to new students remains an
open question. Likewise, the identification of appropriate instructional strategies for the diverse topics that
constitute the average curriculum remains open to debate. One approach considered by a number of practi-
tioners in CS education involves Problem-Based Learning (PBL), a radical departure from the conventional
lecturing format. PBL has been adopted in other domains with success, but whether these positive experi-
ences will be replicated in CS remains to be seen. In this article, a systematic review of PBL initiatives in
undergraduate and postgraduate CS is presented from a Computing Education Research (CER) perspective.
This includes analyses of a range of practical didactic issues, including the degree to which PBL has been
systematically evaluated, practical problem description in the literature, as well as a survey of topics for
which a PBL approach has been adopted.
Categories and Subject Descriptors: K.3.2 [Computer and Education]: Computer and Information
Science Education—Computer Science Education
General Terms: Human Factors
Additional Key Words and Phrases: Problem-Based Learning (PBL), Computing Education Research (CER)
ACM Reference Format:
O’Grady, M. J. 2012. Practical problem-based learning in computing education. ACM Trans. Comput.
Educ. 12, 3, Article 10 (July 2012), 16 pages.
DOI = 10.1145/2275597.2275599 http://doi.acm.org/10.1145/2275597.2275599

1. INTRODUCTION
Practitioners in Computer Science (CS) education face many of the same issues as
their peers in other disciplines. These include how best to enable deep learning in the
subject matter, as well as standard issues such as retention rate and so on. CS is a
very young discipline relatively speaking, only coming into university establishments
in the last 50 years. Thus the subject matter remains quite dynamic in many respects.
More importantly, there is an ongoing debate concerning the core nature of CS and
computing [Denning 2005; Müller 2008]. This issue of definition is aggravated by the
interdisciplinary nature of CS, and its need to respond to ongoing technical develop-
ments, which have been remarkable in the last few decades, and continue unabated.
All of this combines to raise particular difficulties for CS education practitioners who
must accommodate the notions of CS as a discipline in its own right, as well as manage
the needs of other constituencies who may have a different perspective, for example,
one that may regard computing as tool for aiding in the fulfilment of a multiplicity of
purposes.

This work is partially supported by Science Foundation Ireland under grant 07/CE/I1147.
Author’s address: M. J. O’Grady, School of Computer Science & Informatics, University College Dublin,
Belfield, Dublin 4, Ireland; email: [email protected].
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted
without fee provided that copies are not made or distributed for profit or commercial advantage and that
copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights
for components of this work owned by others than ACM must be honored. Abstracting with credit is per-
mitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component
of this work in other works requires prior specific permission and/or a fee. Permission may be requested
from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701, USA, fax +1 (212)
869-0481, or [email protected].
c 2012 ACM 1946-6226/2012/07-ART10 $10.00
DOI 10.1145/2275597.2275599 http://doi.acm.org/10.1145/2275597.2275599

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:2 M. J. O’Grady

As with many disciplines, those charged with computing education have acted pri-
marily as knowledge transmitters. However, this method of instruction, though essen-
tial in some instances, is perceived as limited in that it does not promote higher order
thinking and advanced reasoning skills among others. Both skills are essential for
students seeking to make careers in the computing industry. However, how to do this
in CS courses remains an open question; thus many CS education practitioners have
explored alternative approaches to course delivery. For example, Project-Based Learn-
ing [Krajcik and Blumenfeld 2006] and Inquiry-Based Learning [Edelson et al. 1999]
are just two approaches that have been considered. For the purposes of this discussion,
Problem Based Learning (PBL) is considered in the context of CS undergraduate and
postgraduate education. PBL has been adopted successfully in medical education and
its potential in other disciplines is being actively explored. Indeed, Ellis et al. [1998]
consider that the CS discipline lends itself to the PBL approach as it is for the most
part problem driven. Furthermore, the project group is the dominant modus operandi
of the computer industry—an industry that is highly dynamic, making lifelong learn-
ing a necessity. Since the inception of PBL, a number of CS educators have explored
its potential by designing and delivering individual modules around this format and
published their experiences.
In this article, an analysis of the existing literature in PBL CS education is provided.
This complements and extends an earlier review described by Beaumont et al. [2004].
The paper is structured as follows: Section 2 provides a brief introduction to PBL.
The methodology followed for this review is described in Section 3. Section 4 presents
the results of the analyses in a number of didactic dimensions, after which the article
concludes.
2. REVIEW OF PROBLEM-BASED LEARNING
Problem-Based Learning (PBL) [Boud and Feletti 1997; Dolmans et al. 2005] is an in-
structional approach that makes the student the focus of the learning process, seeking
to empower them such that they take responsibility themselves for their own learn-
ing. This differs from the traditional approach where the lecturer or instructor, acting
as a transmitter of knowledge, drives the learning process. PBL is grounded in the
philosophy of John Dewey, who believed that learning is based on discovery and that
it is guided by mentoring rather than the transmission of knowledge. In this way the
curiosity of the learner is aroused, resulting in them questioning, critically thinking
about problems, and, hopefully, creatively solving problems.
PBL has its origins in medical education and was first implemented at
McMaster University in the 1960s. It was recognized by Barrows [1996] that a com-
bination of hypothetical-deductive reasoning augmented with specific knowledge in a
range of domains was essential to the diagnostic process. The traditional lecture ap-
proach isolated most of the relevant content from its context or its clinical application.
PBL was seen as a remedy to this deficiency. In the intervening years, PBL has been
widely implemented by many medical faculties, and is gaining momentum in other
domains such as business and engineering. It must be stated unequivocally that there
are a number of interpretations of PBL amongst its practitioners and that as yet, a
consensus has not emerged. Indeed, a rigorous definition may never emerge as PBL
methodologies may need to be modified according to the characteristics of the learning
domain in question.
In practice, following the methodology outlined by Barrett [2005], students are pre-
sented with a problem. In small groups, this problem is discussed and refined, result-
ing in students identifying what it is that they need to know to solve this problem.
Outside of the tutorial, students engage in independent study on the learning issues.
They then report back on what they have learnt, share information, debate and argue

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
Practical Problem-Based Learning in Computing Education 10:3

if need be, and come to a consensus about a solution to the problem. Depending on
the scale of the problem, and whether the lecturer is adopting an approach of full or
partial problem disclosure, this cycle may be repetitive. Ultimately, students review
what it is that they have learnt, engaging in a process of self, peer, and tutor review.
Finally, it is instructive to reflect on the effectiveness of PBL. The wide-spread
implementation of PBL in medical schools suggest faculty view it as a positive de-
velopment. For example, Tiwari et al. [2006] in a study of undergraduate nursing
students identified significant differences in students’ disposition to critical thinking
with those exposed to PBL showing a higher score that those who attended a tradi-
tional lecture format. In a meta-analysis of studies conducted between 1976 and 2007,
Walker and Leary [2009] claimed that on standardized tests of basic concepts, PBL can
hold its own in comparison with lecture-based approaches. Furthermore their study
found that when assessments measured application of knowledge and principles, the
results clearly favored PBL, a conclusion also reached by Strobel and van Barneveld
[2009]. However some PBL researchers have observed that empirical evidence sup-
porting the superiority of PBL over traditional approaches is almost non-existent and
that due caution should be exercised [Carrero et al. 2007; Sanson-Fisher and Lynagh
2005]. Kirschner et al. [2006] claim that minimal guidance during instruction does not
work, concluding that PBL and other constructivist approaches do not take the human
cognitive architecture into consideration. Schmidt et al. [2007] take issue with this,
agreeing that novices are a particular case in point, but that the use of PBL is not
equivalent with minimally guided instruction. This debate will undoubtedly continue.
For the purposes of this discussion, the observation of Boud and Feletti [1997] that "it
is rarely possible to translate a given approach from one context to another without
considerable modification" is noted; whether the success of PBL in other disciplines
can be replicated in CS remains to be seen.

3. REVIEW METHODOLOGY
The overall objective of this study is to investigate how PBL is being harnessed in cur-
ricula oriented toward the teaching of CS. Keeping in mind that in the ACM Comput-
ing Curricula [ACM 2005], CS is perceived as one sub-discipline, along with Computer
Engineering, Information Systems, Information Technology, and Software Engineer-
ing, there are a wide variety of courses that could be perceived as common to each
sub-discipline. While acknowledging the distinction, in practice, many course mod-
ules are being developed that could be seamlessly integrated into a course in each of
these sub-disciplines. Thus any PBL case study that contributes to any of these sub-
disciplines is of interest. The domain is undergraduate and postgraduate curricula.
Professional and adult education courses, as well as K-12, are outside the scope of this
study.

3.1. Methodology Specification

A number of categorization methodologies for Computing Education Research (CER)
have been documented in the literature, for example, see Joy et al. [2009]. These
seek to identify a range of characterises of the published research that would serve as
suitable criteria for categorisation or taxonomic purposes. This is a subjective process,
and additional dimensions could be required, according to the requirements of the
research agenda in question. Thus, a generic categorisation methodology that meets
the needs of all research constituencies does not exist, as of now. In the meantime, a
variety of schemes have been proposed, amended, and adopted.
Fincher and Petre [2004] identified 10 research area categories in computer science
education research. Vessey et al. [2005] proposed a unified classification scheme for

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:4 M. J. O’Grady

computing research that incorporates five dimensions: topic, research approach, re-
search method, unit/level of analysis, and reference discipline. Simon [2007] proposes
four dimensions: topic, context nature, and scope. Sheard et al. [2009] used five cri-
teria for classifying research: type of research, data gathering techniques, analysis
techniques, aims of the research, and research outcomes.
Recently, the didactic triangle [Kansanen and Meri 1999] has been considered as
a basis for CER. Berglund and Lister [2010] have harnessed this construct as a ba-
sis of analyzing the teaching and learning of programming. Kinnunen et al. [2010]
have defined a methodology based on the didactic triangle but augmented it with two
additional levels. The basic triangle, a triad of teacher-content-student, represents
the course perspective. The second level represents the perspective of an organization
while the third level represents that of the society. The net result is a three-layered di-
dactic structure where the nodes co-exist and interact, giving rise to eight classification
categories. This classification scheme supports meta analyses of published research,
and, according to the authors, seeks to identify missing types of research.
Any of the schemes described here provides a basis for an analysis of the PBL litera-
ture in the CS domain. However, how any approach in and of itself explicitly supports
the extraction of details pertaining to practice is open to question. It is not the inten-
tion here to develop a new methodology that specifically addresses the needs of this
survey. Thus it is proposed to adopt a two-stage approach.

(1) In the first instance, all articles will be categorized using the approach proposed
by Kinnunen et al. [2010]. This offers an intuitive classification scheme that incor-
porates the key actors (students, teachers, and so on) as well as the content and
objectives of educational programs and their interrelationships.
(2) Second, a number of additional questions will be formulated that seek to identify
what motivated the adoption of PBL and how the success or otherwise of the ini-
tiative was quantified. Given the importance of problems, are any of the problems
used in the case studies documented and discussed. Finally, where in the CS cur-
riculum is PBL being harnessed.

Formally, the four additional questions being addressed in this analysis are as
follows.

(1) What has motivated the adoption of PBL?

What factors have influenced lecturers to adopt PBL? For example, was there an
institutional or departmental policy? Or was it an initiative on behalf of the lec-
turer who was dissatisfied with traditional teaching methods and believed students
would respond better to a different approach?
(2) Has PBL been systematically evaluated in CS courses?
Has there been any efforts made to evaluate the effectiveness of PBL in comparison
to other approaches? Rather than a generic questionnaire canvassing students’
opinions on select aspects of an arbitrary course, has there been some effort made
to ascertain what aspects of the PBL process were successful or otherwise?
(3) Are the problems used in PBL courses documented?
Suitably scoped problems are essential to PBL, and will not be found in typical
course textbooks. What the key features of such problems might be has been ex-
plored, for example Duch [2001] and Weiss [2003] suggest a number of desirable
characteristics. For those new to PBL, examples of problems that have been ap-
plied in the classroom would inform their own problem design efforts. Likewise,
experienced practitioners may find a reflection on practitioners’ experiences with
documented problems instructive.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
Practical Problem-Based Learning in Computing Education 10:5

(4) Where in the CS curriculum has PBL been harnessed?

This addresses the practical question of what levels has PBL been harnessed in a
typical computing curriculum and what topics have been presented through PBL.

3.2. Research Paper Selection and Refinement

In the first instance, a number of relevant journals were identified and their online
repositories searched. Second, a number of digital libraries were searched including
those of the Association for Computing Machinery (ACM) and the Institute of Electri-
cal and Electronic Engineers (IEEE). Finally, Academic Search Premier was queried.
Generic searches using appropriate combinations (including spelling variations) of the
terms “Problem Based Learning” and “computing” or “software” resulting in a signifi-
cant number of potentially relevant publications being returned. To make this process
more manageable, only those publications that had some combination of these search
terms in the title, abstract, or keywords were considered. The acronym PBL was not
used as this returned a significant quantity of articles on Project Based Learning. The
abstract was read initially to determine the degree to which the article was concerned
with the practice of PBL. Only articles whose content was influenced by practice were
retained. The text and references of all these articles were examined, leading to the
identification of additional studies which were subsequently found in Google Scholar.
Finally, a number of colleagues provided references to articles that they considered
relevant. For this study, the cut-off date was November 2011.
Ultimately, 107 publications that merited a more detailed analysis were identified.
Of these, 44 were rejected for a variety of reasons. Some proposed modified PBL prac-
tices; others were courses that were in preparation but had not been delivered in prac-
tice. A number of authors described blended learning approaches with PBL being just
one contributory approach. Articles that did not make reference to the PBL literature
were also excluded—using problems to aid learning is not the same as adopting a PBL
approach. Where a number of case studies referred to the same course over a num-
ber of years, the most recent treatment was analyzed as this tended to be the most
relevant. Studies that were not part of a broad computing curriculum were excluded.
In some instances, the classic seven-step model for PBL [Schmidt 1983] was not fully
adopted due to a variety of constraints. However, this did not serve as justification
for excluding a study. Finally, 63 articles were selected for inclusion in this study.
Figure 1 illustrates the spread of these articles by date of publication. Articles ap-
peared in print from the mid-1990s onwards. While the number increased in the last
decade, it is too early to say if there is trend that might indicate whether PBL will
flourish in CS education, or otherwise.

4. ANALYSIS
All articles were initially categorized using the scheme proposed by Kinnunen et al.
[2010]. Each was then further scrutinized to ascertain how PBL was adopted in
practice, thus providing answers to the additional research questions outlined in
Section 3.1.

4.1. Categorization of PBL Publications

The results of the categorization may be seen in Table I. While all articles have been
assigned a primary category, it sometimes occurs that an article may have a strong
claim for inclusion in another category. Four such articles were identified, and these
are indicated in parenthesis. Overall, it can be seen that all bar four of the publications
were focused at the course level.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:6 M. J. O’Grady

Fig. 1. Publications concerning PBL in CS by year.

To elucidate further on the classification process: many studies focused on the issue
of pedagogy. In many cases, where authors reported on course design, augmented with
some brief observations or anecdotal comments concerning either their own experience
or that of their students, for example Fabiani [2009] and Matzko and Davis [2008],
such studies were classified under Category 7.3, the pedagogical activities of teach-
ers. Other studies considered the issue of pedagogy but proceeded to provide a more
in-depth analysis of the results, for example, those studies by Qiu and Chen [2010],
Pereira et al. [2010], and Wang et al. [2010]. Such studies were classified as Category
8 which focuses on how students relate to pedagogical actions. Indeed, two studies
in this category, those of Hämäläinen [2004] and Clarke et al. [2005], compared the
outcomes of both the traditional and PBL approach across a number of dimensions.
Richardson and Delaney [2010] describe a PBL case study and proceed to provide a
detailed reflection on their experience; this study is classified as Category 7.1, as it
primarily reports on their conceptions of students’ experiences. Mitchell and Delaney
[2004] focuses on the issue of assessment to determine learning outcomes; this study
is classified as Category 6, the relation of the teacher to the goals of the instructional
process. All of these studies are placed at the course level.
Three studies were placed at the organizational level. Fee and Holland-Minkley
[2010] consider PBL across the entire curriculum of their school whereas Gibson [2009]
considers how the teaching of formal methods can be integrated across a curriculum.
Bunch [2009] presents the design of a curriculum for a four-year degree in web devel-
opment. The study described by Hogan and Thomas [2005] is of particular interest in
that it is primarily motivated to equip students with those soft skills that they will
need to progress in their working lives, in this case, to work as professional software
engineers. Thus the study is placed at the society level.
Most of the studies cluster around two aspects: pedagogical interventions and stu-
dents’ reactions to these innovations. Teachers’ perceptions and experiences of the PBL
process and students’ experiences were also documented. However, no systematic stud-
ies focusing exclusively on teachers’ or organizations’ views were found. Likewise, no
study explored students’ perception of their lecturers or facilitators in the PBL process.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
Practical Problem-Based Learning in Computing Education 10:7

Table I. Studies Classified According to their Primary Didactic Focus and Secondary Focus

Studies of students actual engagement with the PBL process remains an open area for
research in CS contexts. Likewise, issues pertaining to the overall objectives or goals
of a CS PBL course, and the degree to which these were met, both from a CS learn-
ing outcomes perspective, but particularly in terms of those student attributes, for
example, initiative, that PBL promises to engender, remain to be addressed. Attitudes
and experiences of teaching faculties as a group, and the perceptions of management,
remains to be seen.
Given the relative newness of PBL and the dynamic nature of the computing tech-
nology with its inherent effects on computing curricula, the emphasis on the course
level is understandable. From an organization level perspective, it would be useful to
see some studies exploring the pertinent aspects of students’ and teachers’ experience
in curricular where PBL is the norm. At the society level, it would be useful to as-
certain how students who have completed their computing studies through PBL are
perceived by key stakeholders including employers and research supervisors.

4.2. What Has Motivated the Adoption of PBL?

Motivation for introducing PBL can be considered under two headings: Who inspired
the adoption of PBL in an arbitrary institution? and What motivated this? Disappoint-
ingly, a significant number of authors did not explicitly address these issues (Figure 2).
Of those that did, the lecturer was found to be the dominant driving force for introduc-
ing PBL, followed by faculty (19%) and the institution (10%). The reasons for the intro-
duction of PBL were mainly pedagogical (27%) and included enabling student-centered
learning, encouraging students to learn by doing, and to foster independence, among
others. The other dominant reason was the perception that PBL would aid students
to acquire skills that would serve them better in the workplace (25%). Finally, cer-
tain courses had been experiencing difficulties over a period of time, for example, poor
grades and high dropout rates. In these cases (16%), it was hoped that the introduction
of PBL would help address these difficulties. All authors reported that this was indeed

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:8 M. J. O’Grady

Fig. 2. (a) Who championed the introduction of PBL? (b) Why was PBL adopted?

the case. Some used exam results and student evidence as evidence; however, in many
cases, it was anecdotal.

4.3. Has PBL Been Systematically Evaluated in CS Courses?

It is good practice to acquire feedback on all aspects of a course, and practically all in-
stitutions demand this. Thus all research articles reported feedback to varying degrees.
As far as could be ascertained, many of the instruments used were those approved by
the institution in question. This would have had the advantage of continuity, enabling
comparison along a consistent range of dimensions, including final grades. However,
this does nothing to isolate factors that might have influenced student perceptions
and experiences of PBL, either positively or negatively. Indeed, only 37% of the stud-
ies reported on attempts to systematically evaluate the PBL approach. While feedback
tends to be positive, it is invariably mixed as expected. Some students respond and
flourish with PBL; others prefer a familiar didactic approach. In most cases, however,
this represented a student’s first exposure to a PBL environment. In many cases, it
was also the instructor’s first attempt at delivering a PBL course. In combination, this
would increase the probability of a student’s experience being a negative one.
It is concluded that a more systematic approach to the evaluation of the PBL ap-
proach in computing courses is required. This would fulfil two functions. In the first
instance, it would address the major question as to whether PBL is a particularly apt
paradigm for the teaching of computing in general. Secondly, it would enable greater
insights into its applicability in a sub-discipline or individual topics, for example, soft-
ware engineering.

4.4. Were the Problems Used in PBL Courses Documented?

Only 19% of the articles included some sample problems and discussed some of the
issues in their design and implementation. While disappointing, it must be acknowl-
edged that space limitations in printed proceedings may be a factor here; nevertheless,
alternative methods of presenting individual problems must be identified, possibly us-
ing supplements or appendices accessible via appropriate digital repositories.
Good problems are a prerequisite to successful PBL. Constructing such problems is
time-consuming and a non-trivial task, demanding significant reflection on the part of
the PBL practitioner, and can only be validated in the class room. Experience can make

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
Practical Problem-Based Learning in Computing Education 10:9

Table II. How PBL Has Been Adopted from a Topic and Level Perspective, for the 59 Course-Level
Studies Identified in the Category Analysis

this process easier; however, for those new to PBL, defining suitable scoped problems
can be a daunting task. Thus the availability of sample problems, illuminated with
some of the factors influencing their design, would be an invaluable resource.

4.5. At What Levels Has PBL Been Harnessed?

PBL has been adopted at all stages throughout computing undergraduate and post-
graduate curricula (Table II). More than 37% of the described case studies involve
first-year students. In other levels, the adoption of PBL is far lower. It might be ex-
pected that at postgraduate level, the use of PBL would be greater, given the relative
maturity of the students; however, this does not seem to be the case.
Why PBL dominates at Level 1 may be guessed at. Keeping in mind the motivations
for adopting PBL discussed in Section 4.2, in the case of Level 1, studies specified
performance issues, pedagogy, and acquiring a better skill sets. It is likely that at the
other levels, similar factors contributed. A further explanation may be that as students
are making the significant transition from high school education to university, this may
be perceived as a favorable opportunity to change their mindset about what university
education is about and expose them to alternative approaches to learning.

4.6. What Topics Are Being Covered by a PBL Approach?

PBL has been harnessed to teach a variety of topics pertinent to the average comput-
ing curriculum. Software engineering and programming courses have been the most
popular; however, a range of other topics including some that may be viewed as ab-
stract in nature have been covered via PBL. If the ACM Computer Science Curriculum
[ACM 2008] is perceived as archetypical, 10 of the 14 specified knowledge areas have
been addressed to some degree. Whether these are covered to the required depth and
breadth is open to question. The four outstanding knowledge areas where the harness-
ing of PBL is still outstanding includes Computational Science, Social and Professional
Issues, Information Management, and Graphics and Visual Computing.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:10 M. J. O’Grady

5. CONCLUSION
The penetration of PBL into the average computing curricula is shallow. In many
cases, faculty members are working in isolation, and face the formidable challenge
of introducing PBL into curricula that remain essentially didactic. In some cases, it
is a measure of last resort in an effort to improve retention or improve performance.
Nevertheless, lecturers’ and students’ experiences are broadly positive. It is probable
that the adoption of PBL in computing will continue in an ad-hoc and random fashion.
This situation is unlikely to change unless the key actors—students, teachers, and
key stakeholders in society—perceive that PBL offers quantifiable benefits that other
approaches do not. Experience of employers with graduates who have experience of
PBL would be of particular interest.
If PBL is to succeed in the broad computing education spectrum, practitioners must
adopt a more systematic approach towards its adoption and validation. Motivations,
objectives, learning outcomes, and graduate attributes must be clearly defined. An
agreed approach to measuring and validating the effectiveness of PBL must be identi-
fied. Innovative and challenging problems must be conceived, and validated. Finally,
practitioners must be prepared to share their experiences in appropriate fora.
Overtime, there continues to be an increasing incidence of institutions where entire
curricula are delivered through PBL. This offers a significant opportunity for CS PBL
practitioners to evaluate PBL in contexts where PBL is the norm.

APPENDIX A
The corpus assembled for this analysis is listed here; articles are subdivided by level
and then listed alphabetically.

A. LEVEL 1 TOPICS
A MBR ÓSIO , A. P. AND C OSTA , F. M. 2010. Evaluating the impact of PBL and tablet PCs in
an algorithms and computer programming course. In Proceedings of the 41st ACM Technical
Symposium on Computer Science Education (SIGCSE’10). 495–499.
B AKAR , M. S. AND S HAIKH A B R AHMAN, S. N. 2005. A kick start in implementation of PBL in
computer programming. In Proceedings of the 2005 Regional Conference on Engineering Educa-
tion (RCEE’05).
B ARG , M., F EKETE , A., G REENING , T., H OLLANDS, O., K AY, J., K INGSTON, J. H., AND
C RAWFORD , K. 2000. Problem-based learning for foundations computer science courses. Comp.
Sci. Educ. 10, 2, 1–20.
B EAMONT, C. AND F OX , C. 2003. Learning programming: Enhancing quality through problem-
based learning. In Proceedings of the 4th Annual LTSN-ICS Conference (LTSN-ICS’03). 241–
242.
B ENTLEY, J. 2002. Introductory systems analysis and design: A problem-based learning ap-
proach. In Proceedings of the 13th Australasian Conference on Information Systems (ACIS’02).
Paper 47.
C LARKE , S., T HOMAS , R., AND A DAMS , M. 2005. Developing case studies to enhance student
learning. In Proceedings of the 7th Australasian Computing Education Conference (ACE’05).
101–108.
D UKE , R., S ALZMAN, E., B URMEISTER , J., P OON, J., AND M URRAY, L. 2000. Teaching pro-
gramming to beginners - choosing the language is just the first step. In Proceedings of the
Australasian Conference on Computing Education (ACE’00). 79–86.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
Practical Problem-Based Learning in Computing Education 10:11

F EKETE , A., G REENING , T., AND K INGSTON, J. 1998. Conveying technical content in a
curriculum using problem based learning. In Proceedings of the 3rd Australasian Conference on
Computer Science Education (ACSE’98). 198–202.
G ARC ÍA -FAMOSO , M. 2005. Problem-based learning: A case study in computer science. In Pro-
ceedings of the International Conference on Multimedia and Information and Communication
Technologies in Education (m-ICTE’05). 817–821.
G REENING , T., K AY, J., K INGSTON, J. H., AND C RAWFORD , K. 1996. Results of a PBL trial
in first-year computer science. In Proceedings of the 2nd Australasian Conference on Computer
Science Education (ACSE’97). 201–206.
H ELIOTIS , J. AND Z ANIBBI , R. 2011. Moving away from programming and towards computer
science in the CS first year. J. Comput. Small Coll. 26, 3, 115–125.
M ASKELL , D. L. AND G RABAU, P. J. 1998. A multidisciplinary cooperative problem-based
learning approach to embedded systems design. IEEE Trans. Educ. 41, 2, 101–103.
M YKYTYN, K., P EARSON, A., S OUREN, P., AND M YKYTYN, P. 2008. The use of problem-based
learning to enhance MIS education. Decis. Sci. J. Innov. Educ. 6, 1, 89–113.
N UUTILA , E., T ÖRM Ä , S., K INNUNEN, P., AND M ALMI , L. 2008. Learning programming with
the PBL method - Experiences on PBL Cases and Tutoring. In Reflections on the Teaching of
Programming. Lecture Notes in Computer Science, vol. 4821, 47–67.
O’K ELLY, J. 2005. Designing a hybrid PBL course: A case study of first year computer science
in NUI Maynooth. Handbook Enq. Prob. Based Learn. 43–53.
P ENG , W. 2010. Practice and experience in the application of problem-based learning in
computer programming course. In Proceedings of the International Conference on Educational
and Information Technology (ICEIT’10). 170–172.
P EREIRA , H. B. DE B., Z EBENDE , G. F., AND M ORET, M. A. 2010. Learning computer
programming: Implementing a fractal in a turing machine. Comput. Educ. 55, 2, 767–776.
Q IU, M. AND C HEN, L. 2010. A problem based learning approach to teaching an advanced
software engineering course. In Proceedings of the 2nd International Workshop on Education
Technology and Computer Science (ETCS’10). 252–255.
R ICHARDSON, I. AND D ELANEY, Y. 2009. Problem based learning in the software engineering
classroom. In Proceedings of the 22nd Conference on Software Engineering Education and
Training (CSEET’09). 174–181.
S HEN, V. R. L. AND J UANG , T.-Y. 2006. A web and problem-based learning system in artificial
intelligence. In Proceedings of the 5th IEEE/ACIS International Conference on Computer and
Information Science and 1st IEEE/ACIS International Workshop on Component-Based Software
Engineering, Software Architecture and Reuse (ICIS-COMSAR06). 299–304.

B. LEVEL 2
B EDNARIK , R. 2004. Problem-based learning in teaching theoretical computer science. In Pro-
ceedings of the International Conference on Engineering Education and Research Conference
(iCEER’04). 801–807.
G ÜLSEYEN, S. 2004. Problem-based learning application to teach IT skills to prospective teach-
ers in a Faculty of Education. In Proceedings of the 5th International Conference on Information
Technology Based Higher Education and Training (ITHET’04). 187–191.
H AMILTON, M., H ARLAND , J., AND PADGHAM , L. 2003. Experiences in teaching computing
theory via aspects of problem-based learning. In Proceedings of the 5th Australasian Conference
on Computing Education (ACSE’03). 207–211.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:12 M. J. O’Grady

L AMBRIX , P. AND K AMKAR , M. 1998. Computer science as an integrated part of engineering

education. In Proceedings of the 6th Annual Conference on the Teaching of Computing and the
3rd Annual Conference on Integrating Technology into Computer Science Education (ITiCSE’98).
153–156.
L ENNARTSSON, B. AND S UNDIN, E. 2002. Experience from a course aiming at understanding
system development with focus on system design and integration. In Proceedings of the 32nd
Annual Conference on Frontiers in Education (FIE’02). 1–6.
M ARTINS , C. A. P. S., C ORR ÊA , J. B. T., G ÓES , L. F. W., R AMOS , L. E. S., AND M EDEIROS ,
T. H. 2002. A new learning method of microprocessor architecture. In Proceedings of the 32nd
Annual Frontiers in Education Conference (FIE’02). 16–21.
M ATZKO , S. AND D AVIS , T. A. 2008. A graphics-based approach to data structures. In Proceed-
ings of the 13th Annual Conference on Innovation and Technology in Computer Science Educa-
tion (ITiCSE’08). 109–113.
R YOO , J., F ONSECA , F., AND J ANZEN, D. S. 2008. Teaching object-oriented software engineer-
ing through problem-based learning in the context of game design. In Proceedings of the 21st
Conference on Software Engineering Education and Training (CSEET’08). 137–144.

C. LEVEL 3
B RODIE , L., Z HOU, H., AND G IBBONS , A. 2008. Steps in developing an advanced software en-
gineering course using problem based learning. Eng. Educ.: J. High. Educ. Acad. Eng. Subj.
Cent. 3, 1, 2–12.
H UANG , H. AND YANG , D. 2008. Teaching design patterns: A modified PBL approach. In Pro-
ceedings of the 9th International Conference For Young Computer Scientists (ICYCS’08). 2422–
2426.
M ISHRA , V. K., M ISHRA , M., AND S HARMA , H. R. 2010. Active learning environment: Gain the
most from PBL for teaching virtual machine module on Operating System. In Proceedings of the
1st International Conference on Parallel Distributed and Grid Computing (PDGC’10). 378–382.
M ITCHELL G. G, AND D ELANEY J. D. 2004. An assessment strategy to determine learning
outcomes in a software engineering problem-based learning course. Int. J. Eng. Educ. 20, 3,
494–502.
N ADJM -T EHRANI , S. 1997. Towards real-time systems education with PBL. In Proceedings of
the 2nd International Workshop on Real-Time Systems Education (RTEW’97). 39–48.
N OOR , R. M. AND H USSIN, N. 2004. First experience in implementing PBL for network design
and management course. J. Prob.-Based Learn. 2, 1, 11–18.
VAT, K. H. 2001. Teaching HCI with scenario-based design: the constructivist’s synthesis. In
Proceedings of the 6th Annual Conference on Innovation and Technology in Computer Science
Education (ITiCSE’01). 9–12.
VAT, K. H. 2004. Toward a learning organization model for student empowerment: A teacher-
designer’s experience as a coach by the side. In Proceedings of the IADIS International Confer-
ence on Cognition and Exploratory Learning in Digital Age (CELDA’04). 131–140.
WANG , Y., D U, H., AND H AO , Y. 2010. A case study of problem based learning instruction
design under web-based environment. In Proceedings of the 2nd International Conference on
Education Technology and Computer (ICETC’10). 586–591.
Y I -R AN, H., C HENG , Z., F ENG , Y., AND M ENG -X IAO , Y. 2010. Research on teaching operating
systems course using problem-based learning. In Proceedings of the 5th International Conference
on Computer Science and Education (ICCSE’10). 691–694.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
Practical Problem-Based Learning in Computing Education 10:13

D. LEVEL 4
A RMAREGO , J. 2002. Advanced software design: A case in problem-based learning. In Pro-
ceedings of the 15th Conference on Software Engineering Education and Training (CSEET’02).
44–54.
D UNLAP, J. C. 2005. Problem-based learning and self-efficacy: How a capstone course prepares
students for a profession. Educ. Tech. Res. Dev. 53, 1, 66–85.
F ITZPATRICK , C. 2008. A Problem-Based Learning (PBL) module on electronics and the
environment. In Proceedings of the IEEE International Symposium on Electronics and the
Environment (ISEE’08). 1–6.
G ARCIA -R OBLES , R., D IAZ - DEL -R IO , F., V ICENTE -D IAZ , S., AND L INARES -B ARRANCO , A.
2009. An eLearning standard approach for supporting PBL in computer engineering. IEEE
Trans. Educ. 52, 3, 328–339.
H ÄM ÄL ÄINEN, W. 2004. Problem-based learning of theoretical computer science.In Proceedings
of the 34th Annual Conference on Frontiers in Education (FIE’04). 20–23.
K AY, J. AND K UMMERFELD , B. 1998. A problem-based interface design and programming
course. In Proceedings of the 29th Technical Symposium on Computer Science Education
(SIGCSE’98). 194–197.
K IRSCH , R. P. 1996. Teaching OLE automation: A problem-based learning approach. In
Proceedings of the 27th Technical Symposium on Computer Science Education (SIGCSE’96).
68–72.
M OEN, P. 2006. Problem-based learning in data mining. In Proceedings of the 4th International
Workshop on Teaching, Learning and Assessment of Databases (TLAD’06). 1–5.
Ó C INN ÉIDE , M. AND T YNAN, R. 2004. A problem-based approach to teaching design patterns.
SIGCSE Bull. 36, 4, 80–82.
O RIOGUN, P., F ERGUSON, A., L UDMANN, M., M AISURIA , N. AND S WHE , Y U. M. 2001. Using
the problem-based learning grid in the management of a software engineering module: Towards
an effective learning intervention for final year undergraduates. In Proceedings of the 3rd Asia
Pacific Conference on Problem Based Learning (PBL’01). 99–110.
R OVER , D. T., M ERCADO , R. A., Z HANG , Z., S HELLEY, M. C. AND H ELVICK , D. S. 2008.
Reflections on teaching and learning in an advanced undergraduate course in embedded
systems. IEEE Trans. Educ. 51, 3, 400–412.
YANG , J., X IE , Y., G ENG , Q., WANG , J., AND B AO , N. 2008. Using cP2BL in teaching
multi-core related contents. In Proceedings of the 9th international Conference for Young
Computer Scientists (ICYCS’08). 2449–2453.

E. LEVEL 5
C AVEDON, L., H ARLAND , J., AND PADGHAM , L. 1996. Problem-based learning with technologi-
cal support in an AI subject: Description and evaluation. In Proceedings of the 2nd Australasian
Conference on Computer Science Education (ACS’97). 191–200.
C HEONG , F. 2008. Using a problem-based learning approach to teach an intelligent systems
course. J. Inf. Tech. Educ. 7, 47–60.
D OS S ANTOS , S. C., DA C ONCEIÇÃO M ORAES B ATISTA , M., C AVALCANTI , A. P. C., A LBU -
QUERQUE , J. O. AND M EIRA , S. R. L. 2009. Applying PBL in software engineering educa-
tion. In Proceedings of the 22nd Conference on Software Engineering Education and Training
(CSEET’09). 182–189.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:14 M. J. O’Grady

FABIANI , E. 2009. Experiencing a problem-based learning approach for teaching reconfigurable

architecture design. Int. J. Reconfig. Comput., Article ID 923415.
L AI , Y. 2005. Teaching computer applications to pre-school teachers through problem-based
learning approach. SIGCSE Bull. 37, 4, 89–92.
L INGE , N. AND PARSONS , D. 2006. Problem-based learning as an effective tool for teaching
computer network design. IEEE Trans. Educ. 49, 1, 5–10
P URAO , S., VAISHNAVI , V., B AGBY, J., B ORTHICK , F., C AMERON, B., L ENZE , L., S AWYER ,
S., S UEN, H. AND W ELKE , R. 2008. Using problems to learn service-oriented computing. In
Proceedings of the 2008 IEEE Congress on Services - Part I (SERVICES’08). 33–36.
R ICHARDSON, I. AND D ELANEY, Y. 2010. Software quality: From theory to practice. In
Proceedings of the 7th International Conference on the Quality of Information and Communica-
tions Technology (QUATIC’10). 150–155.
S CHULTZ , N. AND C HRISTENSEN, H. 2004. Seven-step problem-based learning in an interac-
tion design course. Euro. J. Eng. Educ. 29, 4, 533–541.

F. THE CURRICULUM AND SOCIETY PERSPECTIVE

B UNCH , J. 2009. A Constructivist approach to teaching web development in post-secondary
vocational settings. J. Inf. Tech. Educ. 8, 257–271.
F EE , S. B. AND H OLLAND -M INKLEY, A. M. 2010. Teaching computer science through prob-
lems, not solutions. Comp. Sci. Educ. 20, 2, 129–144.
G IBSON, J. P. 2009. Weaving a formal methods education with problem based learning. In
Leveraging Applications of Formal Methods Verification and Validation, Communications in
Computer and Information Science 17, 460–472.
H OGAN, J. M. AND T HOMAS , R. 2005. Developing the software engineering team. In Proceed-
ings of the 7th Australasian Conference on Computing Education (ACCE’05). 203–210.

ACKNOWLEDGMENTS
The support of Dr Terry Barrett, Centre for Teaching and Learning, University College Dublin, and CLAR-
ITY: Centre for Sensor Web Technologies is greatly appreciated. The insightful comments of the anonymous
reviewers are gratefully acknowledged.

REFERENCES
ACM 2005. Computing curricula 2005: The overview report.
http://www.acm.org/education/curricula-recommendations.
ACM. 2008. Computer science curriculum 2008: An interim revision of CS 2001.
http://www.acm.org/education/curricula-recommendations.
B ARRETT, T. 2005. Understanding problem-based learning. In Handbook of Enquiry & Problem Based
Learning, T. Barrett, I. MacLabhrainn, and H. Fallon Eds., AISHE, Dublin, 14–25.
B ARROWS, H. S. 1996. Problem-based learning in medicine and beyond: A brief overview. New Direct. Teach.
Learn. 68, 3–12.
B EAUMONT, C., S ACKVILLE , A., AND C HENG, C. S. 2004. Identifying Good Practice in the use of PBL to
teach computing. ITALICS e-journal 3, 1, 1–19.
B ERGLUND, A. AND L ISTER , R. 2010. Introductory programming and the didactic triangle. In Proceedings
of the 12th Australasian Conference on Computing Education (ACE’10). 35–44.
B OUD, D. AND F ELETTI , G. 1997. The Challenge of Problem-Based Learning. Kogan Press, London.
B UNCH , J. M. 2009. A constructivist approach to teaching web development in post-secondary vocational
settings. J. Inf. Tech. Educ. 8, 257–271.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
Practical Problem-Based Learning in Computing Education 10:15

C ARRERO, E., G OMAR , C., P ENZO, W., AND R ULL , M. 2007. Comparison between lecture-based approach
and case/problem-based learning discussion for teaching pre-anaesthetic assessment. Euro. J. Anesthe-
siol. 24, 12, 1008–1015.
C LARKE , S., T HOMAS, R., AND A DAMS, M. 2005. Developing case studies to enhance student learning. In
Proceedings of the 7th Australasian Conference on Computing Education (ACE’05). 101–108.
D ENNING, P. J. 2005. Is computer science science? Comm. ACM 48, 27–31.
D OLMANS, D. H. J. M., D E G RAVE , W., W OLFHAGEN, I. H. A. P., AND VAN D ER V LEUTEN, C. P. M. 2005.
Problem-based learning: Future challenges for educational practice and research. Med. Educ. 39, 7,
732–741.
D UCH , B. 2001. Writing problems for deeper understanding. In The Power of Problem-Based Learning: A
Practical How to For Teaching Undergraduate Courses in Any Discipline, B. Duch, S. Groh, and D. Allen
Eds., Stylus Publishing, Sterling, VA, 47–53.
E DELSON, D. C., G ORDIN, D. N., AND P EA , R. D. 1999. Addressing the challenges of inquiry-based learning
through technology and curriculum design. J. Learn. Sci. 8, 391–450.
E LLIS, A., C ARSWELL , L., B ERNAT, A., D EVEAUX , D., F RISON, P., M EISALO, V., M EYER , J., N ULDEN,
U., R UGELJ, J., AND T ARHIO, J. 1998. Resources, tools, and techniques for problem based learning in
computing. SIGCUE Outlook 26, 41–56.
FABIANI , E. 2009. Experiencing a problem-based learning approach for teaching reconfigurable architecture
design. Int. J. Reconfig. Comput.
F EE , S. B. AND H OLLAND -M INKLEY, A. M. 2010. Teaching computer science through problems, not solu-
tions. Comp. Sci. Educ. 20, 2, 129–144.
F INCHER , S. AND P ETRE , M. 2004. Computer Science Education Research. Taylor & Francis, London.
G IBSON, J. P. 2009. Weaving a formal methods education with problem-based learning. In Leveraging Ap-
plications of Formal Methods, Verification and Validation, T. Margaria and B. Steffen, Eds. Communi-
cations in Computer and Information Science Series, vol. 17, Springer, Berlin, 460–472.
H ÄM ÄL ÄINEN, W. 2004. Problem-based learning of theoretical computer science. In Proceedings of the 34th
Annual Conference on Frontiers in Education (FIE’04). 20–303.
H OGAN, J. M. AND T HOMAS, R. 2005. Developing the software engineering team. In Proceedings of the 7th
Australasian Conference on Computing Education (ACE’05). 203–210.
J OY, M., S INCLAIR , J., S UN, S., S ITTHIWORACHART, J., AND L ÓPEZ -G ONZ ÁLEZ , J. 2009. Categorizing
computer science education research. Educ. Inf. Technol. 14, 105–126.
K ANSANEN, P. AND M ERI , M. 1999. The didactic relation in the teaching-studying-learning process. TNTEE
Pub. 2, 1, 107–116.
K INNUNEN, P., M EISALO, V., AND M ALMI , L. 2010. Have we missed something?: Identifying missing types
of research in computing education. In Proceedings of the 6th International Workshop on Computing
Education Research (ICER’10). 13–22.
K IRSCHNER , P. A., S WELLER , J., AND C LARK , R. E. 2006. Why minimal guidance during instruction
does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and
inquiry-based teaching. Educ. Psychol. 41, 2, 75–86.
K RAJCIK , J. S. AND B LUMENFELD, P. C. 2006. Project-based learning. In The Cambridge Handbook of the
Learning Sciences, R. K. Sawyer Ed., Cambridge University Press, Cambridge, UK, 317–334.
M ATZKO, S. AND D AVIS, T. A. 2008. A graphics-based approach to data structures. SIGCSE Bull. 40,
109–113.
M ITCHELL , G. AND D ELANEY, J. 2004. An assessment strategy to determine learning outcomes in a soft-
ware engineering problem-based learning course. Int. J. Eng. Educ. 20, 3, 494–502.
M ÜLLER , V. C. 2008. What a course on philosophy of computing is not. APA Newsl. Phil. Comp. 8, 1, 36–38.
P EREIRA , H. B. D. B., Z EBENDE , G. F., AND M ORET, M. A. 2010. Learning computer programming:
Implementing a fractal in a turing machine. Comp. Educ. 55, 2, 767–776.
Q IU, M. AND C HEN, L. 2010. A problem-based learning approach to teaching an advanced software
engineering course. In Proceedings of the 2nd International Workshop on Education Technology and
Computer Science (ETCS’10). 252–255.
R ICHARDSON, I. AND D ELANEY, Y. 2010. Software quality: From theory to practice. In Proceedings of the 7th
International Conference on the Quality of Information and Communications Technology (QUATIC’10).
150–155.
S ANSON -F ISHER , R. AND LYNAGH , M. 2005. Problem-based learning: A dissemination success story. Med.
J. Aust. 183, 5, 258–260.

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.
10:16 M. J. O’Grady

S CHMIDT, H., L OYENS, S. M., VAN G OG, T., AND PAAS, F. 2007. Problem-based learning is compatible with
human cognitive architecture: Commentary on Kirschner, Sweller, and Clark. 2006. Educ. Psychol. 42,
2, 91–97.
S CHMIDT, H. G. 1983. Problem-based learning: Rationale and description. Med. Educ. 17, 1, 11–16.
S HEARD, J., S IMON, S., H AMILTON, M., AND L ÖNNBERG, J. 2009. Analysis of research into the teaching and
learning of programming. In Proceedings of the 5th International Workshop on Computing Education
Research Workshop (ICER’09). 93–104.
S IMON. 2007. A classification of recent Australasian computing education publications. Comp. Sci. Educ. 17,
3, 155–169.
S TROBEL , J. AND VAN B ARNEVELD, A. 2009. When is PBL more effective? A meta-synthesis of meta-
analyses comparing PBL to conventional classrooms. Interdiscipl. J. Prob.-Based Learn. 3, 1, 44–58.
T IWARI , A., L AI , P., S O, M., AND Y UEN, K. 2006. A comparison of the effects of problem-based learning and
lecturing on the development of students’ critical thinking. Med. Educ. 40, 6, 547–554.
V ESSEY, I., R AMESH , V., AND G LASS, R. L. 2005. A unified classification system for research in the com-
puting disciplines. Inf. Softw. Technol. 47, 245–255.
WALKER , A. AND L EARY, H. 2009. A problem based learning meta analysis: Differences across problem
types, implementation types, disciplines, and assessment levels. Interdiscipl. J. Prob.-Based Learn. 3, 1,
12–43.
WANG, Y., D U, H., AND H AO, Y. 2010. A case study of problem-based learning instruction design under
web-based environment. In Proceedings of the 2nd International Conference on Education Technology
and Computer (ICETC’10). 586–591.
W EISS, R. E. 2003. Designing problems to promote higher-order thinking. New Direct. Teach. Learn. 2003
95, 25–31.

Received June 2010; revised August 2011, January 2012; accepted March 2012

ACM Transactions on Computing Education, Vol. 12, No. 3, Article 10, Publication date: July 2012.