3
2 f
m
frroom Device Action
2 , 22002}3byareh
beerr 4D, DRar Z T
a y
e
ovvem
m b _A OmE}
, N O .9E bNyAM
,
N o T Task Guide
Sahur
tu sd
IP IPrdaayy, N
u rsrdda{yRE6M . 25E4 R_ 72 {RENoove
.16 M ve m
1 ,
TShatu IP72.1{US {U .25 OTmbber
1 SE 4.9 E_er 2,
Task Instructions:
IP R_ by AD4, 2202
NA O D 02 3
MEmarR} b3 frfrom
In this task, you will be provided with a Prompt from a user (e.g., o question, or even a command)
reah m
Usefulness on the Needs Met rating } Za yscale
and potential results.
The Relationship Between Needs Met
Your job is to assign Needs Met rating to the results offered, reflecting how useful it would be to
and
the Freshness
user based on the Prompt.
Let’s have a look at the instructions for this kind of task:
As we can see, there’s nothing very specific here, but do not forget about the “contextual
headings”; at first glance, they can look a little strange, but they are sometimes included to
provide more information and context for your judgements.
Now, let’s have a look into the results offered for this query: m
T
Sahur 3 frroom
tu sd 2 f
0 3by eh
IP IPrdaayy, N 2,,220R2} Z r
17 {R , N ov b e r 4 D ar a
r
2.1 EMov em m be_AD m }
em b
{U6.25 OT ovveem O E
SE 4.9 Eb_erer 2, ay , No OT.9E bNyAM
N
R_ by AD4, 2202 d y, M 54 _
NA O D 02 3 h utrusrda{R.1E6.2SER
MEmarR} b3 frfrom TSa IP72 {U
} Za y om 1
reh IP
R1 shows a number of clickable icons, or ‘buttons’.
It is very important to access links and icons provided by the task, as our goal is to assess the
user experience - therefore we must have a clear understanding of the options thereof available.
The first icon above, for example, leads users to the official Aerosmith YouTube Channel:
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
m
3 frroom
2 f
2 , 22002}3byareh
m beerr 4D, DRar Z T
Sahur
e b _A OmE}
ovvem tu sd
a y , N O .9E bNyAM
N o T IP IPrdaayy, N
,
u rsrdda{yRE6M . 25E4 R_ 17 {R , N ov
2.1 EMov em
TShatu IP72.1{US As the user Oem b
{U6.25wants
1 SE 4.9TEb_erer their
2 music played, but did not
IP R_ by AD4, ,2202
specify the song NA OorD album,
0 3 this would be a broad
MEmarR}2b3 frfrom
query. } Za y om
reh
HM to HM+ applies to this scenario as it is
authoritative and has the desired music, however it
does not play anything and therefore cannot be FullyM.
R2 presents options to see results related to an Overview of the band, Songs, Members, etc.
T When icons on the result block don’t work, please take into consideration m
frroom the following from
Sahur 2 3 f
tu sd
IP IPrdaayythe
,
General Guidelines: 2, 22002}3byareh
17 {R , NNov
2.1 EMov em m beer 4D, DRar Z
r
b _A OmE}
em b e
{U6.25 OT ovveem
b r
SE 4.9 E_er 2, ay , No O .9E bNyAM
N T
R_ by AD4, 2202 d y, M 54 _
NA O D 02 3
m R 3 f h utrusrda{R.1E6.2SER
ME ar } b fr rom TSa IP72 {U
} Za y om 1
reh IP
Therefore, because this result is a little too broad, and does not answer the user request in a
straightforward manner, MM to HM would be the best range for this result.
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
m
3 frroom
2 f
2 , 22002}3byareh
m beerr 4D, DRar Z T
Sahur
e b _A OmE}
ovvem tu sd
a y , N O .9E bNyAM
N o T IP IPrdaayy, N
,
u rsrdda{yRE6M . 25E4 R_ 17 {R , N ov
2.1 EMov em
TShatu IP72.1{US em b
{U6.25 OT
1 SE 4.9 Eb_erer 2,
IP R_ by AD4, 2202
NA O D 02 3
MEmarR} b3 frfrom
} Za y om
reh
R3 displays a list of videos from Aerosmith’s Vevo channel on YouTube. This is only one
option of where to PLAY AEROSMITH, however this result would still be helpful to many users.
Therefore, MM to HM would apply to this scenario.
The rating grades are listed below, which may be referenced to clarify the standards for each:
Rating Description
T rom
f rom applies to certain
FullySahMeets(FullyM)
t urs A special rating category, which 0 2 3only
3 f
u d
IP IPrdaayy, N 2 , 2 02 by reh
queries and results. All or ealmost 2 all} a users would be
17 {R , N ov
2.1 EMov em m b err 4D, DRar Z
em b b
veemE_A
immediately and fully osatisfied mby} the result and would not
{U6.25 OT
SE 4.9 Eb_erer 2, , N o vOT byAOME
need to view other 9
y , NM 4. _to
sdaayresults
E 5 R satisfy their need.
R_ by AD4, 2202 N
NA O D 02 3 ur rd {R 6.2 E
MEmarR} b3 frfrom TShatu IP72.1{US
Highly Meets (HM)} Zarye om P1
Very helpful for Imany or most users. Some users may wish to
h
see additional results.
Moderately Meets (MM) Helpful for many users OR very helpful for some users.
Some or many users may wish to see additional results.
Slightly Meets (SM) Helpful for fewer users. There is a connection between the
query and the result, but not a strong or satisfying
connection. Many or most users would wish to see additional
results.
Fails to Meet (FailsM) Completely fails to meet the needs of the users. All or almost
all users would wish to see additional results.
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
m
3 frroom
2 f
2 , 22002}3byareh
m beerr 4D, DRar Z T
Sahur
e b _A OmE}
ovvem tu sd
Another a y , N O .9E bNyAM
N
Example:
o T IP IPrdaayy, N
,
u rsrdda{yRE6M . 25E4 R_ 17 {R , N ov
2.1 EMov em
TShatu IP72.1{US em b
{U6.25 OT
1 SE 4.9 Eb_erer 2,
IP R_ by AD4, 2202
NA O D 02 3
MEmarR} b3 frfrom
} Za y om
reh
Here, the user wants to “play song happy”. There are many songs with the title “happy”, and a
Common Interpretation might be to generally play a song which is happy in nature. However,
“Happy” from Pharrell Williams is, at the moment, very well-known, and therefore would be the
Dominant Interpretation.
Although it is very likely user’s want to play this specific song, we cannot m assume this is the only
Th r o
Sa uruser intent, as intimated above. Therefore, for results like L1, which0only 2 3 ffromoffers results regarding
tur sda , 2202}3byareh
IP IPdayy, N 2
17 the song
2.{1RE, N oove by Pharrell Williams, the best range would be MM to mHM. beer 4D, DRar Z
r
b _A OmE}
M vemmb
{U6.25 OT ovveem
b e r
SE 4.9 E_er 2, ay , No O .9E bNyAM
N T
, 4 _
While R_ by AD4, 2202
NL2 Ois rsrdda{yRE6M
likewise specific to the Pharrell song, thehuoperative .2S5ERword of the query (ie play) is
AM mDa R0}233f fro u
TSat IP72 {U . 1
somewhatE}neglected,r Z by romm 1
therefore MM would be more appropriate.
are IP
h
Here are some steps you can also take when rating:
● Regularly review the General Guidelines
● Visit the reference materials available on the Education page of the portal
● Ensure you are taking your time when rating or evaluating a task. Please ensure you use
the Average Estimated Time too:
➔ Read through the Task Instructions
➔ Open the landing pages/videos
➔ Search the query outside of the task to have a better understanding of the user intent if
necessary
➔ Review your ratings
➔ Write an insightful comment explaining your ratings where suitable
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
