3
2 3
m
frforom
s
Device Action
2 , ,22002} bayniel
r
e 2 5 R D
mbber_ADaDvid} STahur
a y,
,N
M
v e
NooveOmTEby D
. 9 _NA
M E Task Guidet s
IP uIrPdadyay,
17 , N
s d y E 4 R 2.1{RENovove
ur da {R .25 E 6 M m
ThaturIP2.16{US {U.254OeTmbeber
17
TaskS Instructions: SE .9 E_ r 2 2,
R_ by AD5, 202
IP NADa D 202 3
MEvidR} b3 ffrom
In this task, you will be provided with a Prompt from
Usefulness on the Needs Met rating a user (e.g., aroquestion,
} Danyscalem or even a command)
iels
and potential results.
The Relationship Between Needs Met
Your job is to assign Needs Met rating to the results offered, reflecting how useful it would be to
and
the Freshness
user based on the Prompt.
Let’s have a look at the instructions for this kind of task:
As we can see, there’s nothing very specific here, but do not forget about the “contextual
headings”; at first glance, they can look a little strange, but they are sometimes included to
provide more information and context for your judgements.
Now, let’s have a look into the results offered for this query: m
STahur 3 frforom
t s 2
0023by iels
IP uIrPdadyay, 25, ,22R } n
17 {R, NNo
2.1 E ov ve b er 2 D d Da
r
m e D vi
6 M m
{U.254OeTmbeber N ovveemTbE_yADaME}
SE .9 E_ r 2 2, , o O b A
R_ by AD5, 202 r s daayy, NEM54.9R_N
NADa D 202 3 u d {R .2 E
MEvidR} b3 ffrom ThaturIP2.16{US
} Dany rom S 17
iels IP
R1 shows a number of clickable icons, or ‘buttons’.
It is very important to access links and icons provided by the task, as our goal is to assess the
user experience - therefore we must have a clear understanding of the options thereof available.
The first icon above, for example, leads users to the official Aerosmith YouTube Channel:
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
m
3 frforom
2 3 s
2 , ,22002} bayniel
r
e 2 5 R D
v e mbber_ADaDvid} STahur
E t s
NooveOmTEby D
,N M IP uIrPdadyay,
d a y, M 4 . 9 _NA 17 {R, NNo
s y E
ur da {R .25 E R 2.1 E ov ve
ThaturIP2.16{US 6 MOemmb
As the user {U.254wants
S 17 SE .9TEb_ere2r their
2 music played, but did not
IP R_ by AD5,, 202
specify the song NADor D album,
2 3 this would be a broad
MaEvidR}02b3 ffrom
query. } Dany rom
iels
HM to HM+ applies to this scenario as it is
authoritative and has the desired music, however it
does not play anything and therefore cannot be FullyM.
R2 presents options to see results related to an Overview of the band, Songs, Members, etc.
When icons on the result block don’t work, please take into consideration m
frforom the following from
STahur 2 3
t s 3 s
IP uIrPdadyaythe
,
General Guidelines: 2, ,22002} bayniel
17 {R, NNo r
e 25 R D
2.1 E ov ve
6.2 MOemmb v embber_ADaDvid}
E
{U 54 T be er
SE .9 E_ r 2 2, y ,NNooveOmTEby D AM
R_ by AD5, 202 a
d , M4 _ . 9 N
NADa D 202 3
R f h uurrsday{R16E.25SER
MEvid } b3 f rom T at IP2. {U
} Dany rom S 17
iels IP
Therefore, because this result is a little too broad, and does not answer the user request in a
straightforward manner, MM to HM would be the best range for this result.
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
m
3 frforom
2 3 s
2 , ,22002} bayniel
r
e 2 5 R D
v e mbber_ADaDvid} STahur
E t s
NooveOmTEby D
,N M IP uIrPdadyay,
d a y, M 4 . 9 _NA 17 {R, NNo
s y E
ur da {R .25 E R 2.1 E ov ve
ThaturIP2.16{US 6 M m
S 17 {U.254OeTmbeber
SE .9 E_ r 2 2,
IP R_ by AD5, 202
NADa D 202 3
MEvidR} b3 ffrom
} Dany rom
iels
R3 displays a list of videos from Aerosmith’s Vevo channel on YouTube. This is only one
option of where to PLAY AEROSMITH, however this result would still be helpful to many users.
Therefore, MM to HM would apply to this scenario.
The rating grades are listed below, which may be referenced to clarify the standards for each:
Rating Description
T
SahMeets(FullyM) f ro romm
Fully tuurs A special rating category, which 0 223only
3 yf ielsapplies to certain
IP IrPdadyay, , 2 0 ban users would be
17 {R, NNo queries and results. All or ealmost
b r 225,D2R}Dall
2.1 E ov ve r
mbe_ADavby i d
6 M m
{U.254OeTmbeber N
veem
immediately and fully osatisfied
v T E y D ME} the result and would not
SE .9 E_ r 2 2, need to view other y,, NoMO.9 b_toNA
R_ by AD5, 202
NADa D 202 3 sdayresults
E 54 R satisfy their need.
ur da {R .2 E
MEvidR} b3 ffrom ThaturIP2.16{US
Highly Meets (HM)} Danyi rom S many
Very helpful for 17 or most users. Some users may wish to
els IP
see additional results.
Moderately Meets (MM) Helpful for many users OR very helpful for some users.
Some or many users may wish to see additional results.
Slightly Meets (SM) Helpful for fewer users. There is a connection between the
query and the result, but not a strong or satisfying
connection. Many or most users would wish to see additional
results.
Fails to Meet (FailsM) Completely fails to meet the needs of the users. All or almost
all users would wish to see additional results.
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
m
3 frforom
2 3 s
2 , ,22002} bayniel
r
e 2 5 R D
v e mbber_ADaDvid} STahur
E t s
Another NooveOmTEby D
Example:
,N M IP uIrPdadyay,
d a y, M 4 . 9 _NA 17 {R, NNo
s y E
ur da {R .25 E R 2.1 E ov ve
ThaturIP2.16{US 6 M m
S 17 {U.254OeTmbeber
SE .9 E_ r 2 2,
IP R_ by AD5, 202
NADa D 202 3
MEvidR} b3 ffrom
} Dany rom
iels
Here, the user wants to “play song happy”. There are many songs with the title “happy”, and a
Common Interpretation might be to generally play a song which is happy in nature. However,
“Happy” from Pharrell Williams is, at the moment, very well-known, and therefore would be the
Dominant Interpretation.
Although it is very likely user’s want to play this specific song, we cannot mm assume this is the only
T r
f oo
Sahuruser intent, as intimated above. Therefore, for results like L1, which only 2233 yfr offers results regarding
tur sd 0 s
IP IPdayay, 2 , , 2 } baniel
2 0
17 the
2.1{R,ENsong
No
o v vem
by Pharrell Williams, the best range would be MM to HM.
m beerr 25DDviRd D
6 M
{U.254OeTmbeber N ovveemTbE_yADaME}
SE .9 E_ r 2 2, , o O b A
R_ by AD5, 202 s daayy, NEM54.9R_N
While NL2 AD isDlikewise
2 3 specific to the Pharrell song, thehuoperative
r d {R .2 E word of the query (ie play) is
MaEvidR}02b3 ffrom T aturIP2.16{US
somewhat }neglected, Da y omr S
therefore MM would be more appropriate.17
nie IP
ls
Here are some steps you can also take when rating:
● Regularly review the General Guidelines
● Visit the reference materials available on the Education page of the portal
● Ensure you are taking your time when rating or evaluating a task. Please ensure you use
the Average Estimated Time too:
➔ Read through the Task Instructions
➔ Open the landing pages/videos
➔ Search the query outside of the task to have a better understanding of the user intent if
necessary
➔ Review your ratings
➔ Write an insightful comment explaining your ratings where suitable
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
