Whisper for L2 speech scoring

Abstract

In this paper, we examine whether confidence scores produced by the C++ re-implementation of Whisper (Radford et al., in: International conference on machine learning, 2023) can be used to score L2 learners of English and classify them. We test whether the language prediction and its probability can be used to classify French learners of English using a specifically collected dataset for read speech and a graded corpus, the ANGLISH corpus (Tortel and Hirst, in: Proceedings of speech prosody 2010, 2010. https://doi.org/10.21437/SpeechProsody.2010-49). We show that probability scores associated with the Whisper subtokens can be used to classify learners into levels using the knn algorithm. We show the limitations of the language detection probability beyond an initial threshold where the native language L1 of the learner can actually be predicted by the speaker. We have also used the ISLE corpus (Menzel et al., in: Proceedings of LREC 2000: Language resources and evaluation conference, European Language Resources Association, 2000) to test the prediction of the levels of Italian and German learners of English (Atwell et al., in: ICAME Jurnal, 27:5–18, 2003). We show how language detection for Whisper’s multilingual larger models can be used to detect less advanced learners’ first language but cannot be used for learner level classification with advanced learners. Using a greedy alignment algorithm, we also discuss the confidence score assigned to Whisper output subtokens and how this may be used for speaker scoring, prediction of learner levels, and learner feedback. We show that low confidence scores and alternative transcriptions can be used as potential cues for learner pronunciation errors.

Appendices

Appendix 1: Reference text of the reading task

Observing the steady fall of the barometer, Captain MacWhirr thought, “There’s some dirty weather knocking about.” This is precisely what he thought. He had had an experience of moderately dirty weather-the term dirty as applied to the weather implying only moderate discomfort to the seaman. Had he been informed by an indisputable authority that the end of the world was to be finally accomplished by a catastrophic disturbance of the atmosphere, he would have assimilated the information under the simple idea of dirty weather, and no other, because he had no experience of cataclysms, and belief does not necessarily imply comprehension. The wisdom of his county had pronounced by means of an Act of Parliament that before he could be considered as fit to take charge of a ship he should be able to answer certain simple questions on the subject of circular storms such as hurricanes, cyclones, typhoons; and apparently he had answered them, since he was now in command of the Nan-Shan in the China seas during the season of typhoons. But if he had answered he remembered nothing of it. He was, however, conscious of being made uncomfortable by the clammy heat. He came out on the bridge, and found no relief to this oppression. The air seemed thick. He gasped like a fish, and began to believe himself greatly out of sorts.

The Nan-Shan was ploughing a vanishing furrow upon the circle of the sea that had the surface and the shimmer of an undulating piece of gray silk. The sun, pale and without rays, poured down leaden heat in a strangely indecisive light, and the Chinamen were lying prostrate about the decks. [...] Captain MacWhirr noticed two of them especially, stretched out on their backs below the bridge. As soon as they had closed their eyes they seemed dead. Three others, however, were quarrelling barbarously away forward; and one big fellow, half naked, with herculean shoulders, was hanging limply over a winch; another, sitting on the deck, his knees up and his head drooping sideways in a girlish attitude, was plaiting his pigtail with infinite languor depicted in his whole person and in the very movement of his fingers. The smoke struggled with difficulty out of the funnel, and instead of streaming away spread itself out like an infernal sort of cloud, smelling of sulphur and raining soot all over the decks.

1.1 Appendix B: Examples of Whisper mistranscriptions of uncomfortable

Ref_text	Model	Gap_text	Count
uncomfortable	medium_en	and comfortable	1
uncomfortable	medium_en	and comforted	1
uncomfortable	medium_en	comfortable	1
uncomfortable	medium_en	incompatible	1
uncomfortable	medium	“uncomfortable”	1
uncomfortable	medium	comfortable	1
uncomfortable	tiny.en	a comfortable	1
uncomfortable	tiny.en	and comfortable	4
uncomfortable	tiny.en	in campatible	1
uncomfortable	tiny.en	meant and capable	1
uncomfortable	tiny.en	of incontivable	1
uncomfortable	tiny.en	uncountable	1
uncomfortable	tiny.en	ungothable	1
uncomfortable	tiny	and comfortable	2
uncomfortable	tiny	comfortable	1
uncomfortable	tiny	incomparable	1
uncomfortable	tiny	incompatible	1
uncomfortable	tiny	main and comfortable	1
Uncomfortable	tiny	meant and comfortable	1
uncomfortable	tiny	uncorruptable	1
uncomfortable	tiny	ungovernable	1

1.2 Appendix C: Examples of Whisper mistranscriptions of leaden

Ref_text	Model	Gap_text	Count
leaden	medium_en	, leading it	1
leaden	medium_en	laden	1
leaden	medium_en	lead and	2
leaden	medium_en	leading	4
leaden	medium_en	leading it	1
leaden	medium_en	linen and	1
leaden	medium_en	Unleaded	1
leaden	medium	, leading	2
leaden	medium	, leading heads	1
leaden	medium	, leading it	1
leaden	medium	, let on	1
leaden	medium	, letting it	1
leaden	medium	-laden heats	1
leaden	medium	a laden	1
leaden	medium	laden	4
leaden	medium	laden , ate	1
leaden	medium	laden heats	2
leaden	medium	laden hits	1
leaden	medium	lead and	1
leaden	medium	lid and	3
leaden	medium	lid and hip	1
leaden	medium	lid and hit	1
leaden	tiny.en	, leading	1
leaden	tiny.en	-leading	1
leaden	tiny.en	a little	1
leaden	tiny.en	laddened	1
leaden	tiny.en	lead and	15
leaden	tiny.en	lead and hits	1
leaden	tiny.en	lead in	3
leaden	tiny.en	leading	2
leaden	tiny.en	leading heads	1
leaden	tiny.en	leading it	1
leaden	tiny.en	lid and hit	1
leaden	tiny.en	little	1
leaden	tiny	, leading	2
leaden	tiny	, leading heads	1
leaden	tiny	, leading hits	1
leaden	tiny	, leading it	1
leaden	tiny	, led	1
leaden	tiny	, lit	1
leaden	tiny	foredown led	1
leaden	tiny	laden	3
leaden	tiny	laden hits	1
leaden	tiny	lead and	11
leaden	tiny	lead in	2
leaden	tiny	leading hits	1
leaden	tiny	linen , hit	1
leaden	tiny	on lead and	1
leaden	tiny	powered and ledon	1
leaden	tiny	the hidden hits	1
leaden	tiny	the lead in	1
leaden	tiny	the unleading hits	1

Appendix D: Examples of Whisper mistranscriptions of Herculean

Ref_text	Model	Gap_text	Count
herculean	medium_en	a Korean	1
herculean	medium_en	a cholera	1
herculean	medium_en	a kulean	1
herculean	medium_en	achilles	1
herculean	medium_en	arkadian	1
herculean	medium_en	her Korean	1
herculean	medium_en	her chilean	1
herculean	medium_en	her clean	3
herculean	medium_en	her culean	1
herculean	medium_en	her curling	1
herculean	medium_en	hickory	1
herculean	medium	Herculean	1
herculean	medium	a Cullian	1
herculean	medium	aculure on	1
herculean	medium	arcane	1
herculean	medium	arculean	1
herculean	medium	curly	1
herculean	medium	her Acheulean	1
herculean	medium	her clean	3
herculean	medium	her curly on	1
herculean	medium	her killian	1
herculean	medium	heroclone	1
herculean	medium	oculial	1
herculean	tiny.en	Arkalian	1
herculean	tiny.en	a crayon	1
herculean	tiny.en	a curian	1
herculean	tiny.en	a curling	1
herculean	tiny.en	a hate -culline	1
herculean	tiny.en	aculean	1
herculean	tiny.en	aculian	1
herculean	tiny.en	air -culion	1
herculean	tiny.en	ekulean	1
herculean	tiny.en	her cally and	1
herculean	tiny.en	her clean	2
herculean	tiny.en	her curling	1
herculean	tiny.en	her curly and	1
herculean	tiny.en	her gluing	1
herculean	tiny.en	her kiln	1
herculean	tiny.en	herculine	1
herculean	tiny.en	oculi and	1
herculean	tiny.en	our culian	1
herculean	tiny.en	the hakiran ,	1
herculean	tiny	a chulian	1
herculean	tiny	a chulian	1
herculean	tiny	a clean	1
herculean	tiny	a crayon	1
herculean	tiny	a heckling	1
herculean	tiny	aircolion	1
herculean	tiny	echelion	1
herculean	tiny	her Qulian	1
herculean	tiny	her chulian	2
herculean	tiny	her chuling	1
herculean	tiny	her clean	2
herculean	tiny	her collier and	1
herculean	tiny	her culean	1
herculean	tiny	her culey and	1
herculean	tiny	her curly and	2
herculean	tiny	her cutely and	1
herculean	tiny	her killer and	1
herculean	tiny	her killing	1
herculean	tiny	herculey and	1
herculean	tiny	herculian	1
herculean	tiny	herkali	1
herculean	tiny	herkilling	1
herculean	tiny	herkilly and	1
herculean	tiny	herkul and	1
herculean	tiny	hurtly enchilers	1
herculean	tiny	our Qliian	1
herculean	tiny	percolian	1
herculean	tiny	to be covered	1
herculean	tiny	were her killion	1