Prompt Helpfulness Tasks Guide 3

2 3
m
frforom
, ,22002} bayniel
s
r 25 R
e 2 D
mbber_ADaDvid} STahur
Assessing the helpfulness and the presentation
y
v e
NooveOmTEby D
,N 9 _NA
M E t s
IP uIrPdadyay,
17 {R, NNo
s d ay , E M 4 . R 2.1 E ov ve
ur da {R .25 E 6 M m
ThaturIP2.16{US {U.254OeTmbeber
S 17 SE .9 E_ r 2 2,
IP R_ by AD5, 202
Task Instructions: NADa D 202 3
MEvidR} b3 ffrom
} Dany rom
iels
In this task, you will be provided with a Prompt from a user (e.g., a question, instruction,
statement) to an AI chatbot along with two potential machine-generated Responses to the
Prompt.

Your job is to understand the user intent of the Prompt and then assess which of the two
Responses is better for the Prompt, considering the following for each Response:

< Helpfulness >

To what extent does the Response provide useful information or satisfying content for the
Prompt?

Responses should:
● Address the intent of the user's Prompt such that a user would not feel the Prompt
was ignored or misinterpreted by the Response.
● Provide specific, comprehensive, and up-to-date information for the user needs
expressed in the Prompt, including relevant details needed to contextualize and
understand the Response.
● Be sensible and coherent. The response should not contain any nonsensical
information or contradict itself across sentences (e.g., refer to two different people
with the same name as if they are the same person).
● Adhere to any requirements indicated in the Prompt such as an explicitly specified
word length, tone, format, or information that the Response should include.
● Not contain inaccurate, deceptive, or misleading information (based on your current
knowledge or quick web search - you do not need to perform a rigorous fact check)
m
STahur● Not contain harmful, offensive, or overly sexual content 3 frforom
t s 2 s
IP uIrPd●adyay, Be in the correct language for the user. Generally the Response 2 02}3byniel be in the
, ,220should
17 {R, NNo r
e 25 D D R a
2.1 E same
6.2 MoOvevmem language as the Prompt unless the Prompt is clearly v embbrequesting
er_ADavid} a response in a
{U 54 T beber N ovemTE y D ME
SEdifferent
. E r 2 2, 2language. , o O b A
R9_ by _AD 5, 02 r s daayy, NEM54.9R_N
D2 3
NADaResponse
If the u d {R6.2 SE "Not at all Helpful" (this
MEvidR}02b3 ffrom is not in the correct language, please
ThaturIP2.1select {U
includes } Dan r
any English
om response to a non-English S
prompt).
P 17
i els I

A Response may sometimes intentionally avoid or decline to address the

question/request of the Prompt and may provide a reason for why it is unable
to respond. For example, "Sorry, there may not be a helpful answer to this
question." These responses can be considered helpful in cases where an
appropriate helpful response to the Prompt does not seem possible.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 m
3 frforom
2 3 s
2 , ,22002} bayniel
HelpfulnessmbRating r R
e 2 D D scale
5 Explanation
v e b er_ADavid} SThu
at rs
, NooveOmTEby D A ME IP uIrPdadyay,
y
ay, NEM 4.9R_N 17 {R, Nis
Not uat
rrsddaAll Response2.1 E N ouseless/irrelevant, contains even a single
{R .2Helpful
5E vevmem
6.2 MoO
Thatu IP2.16{US US54. TEbebrer 2
piece of{nonsensical/inaccurate/deceptive/misleading
S 17 E 9 b _A 25 , 2
IP information,Ris _Niny Dan 0
D , incorrect language for the
AMaviDR2}02233 fr
Prompt, and/or contains d o
E } Dany harmful/offensive/overly
b f rom m
sexual content. i e ls

Slightly Helpful Response is somewhat related to the Prompt, does

not address important aspects of the Prompt, and/or
contains outdated information.

Somewhat Helpful Response partially addresses the intent of the Prompt

(most users would want more information), contains
extra unhelpful information, and/or is lacking helpful
details/specifics (e.g., a list of relevant items without
explanations).

Very Helpful Response addresses the intent of the Prompt with a

satisfying response. Some users might want a more
comprehensive response with additional details or
context. It is comparable to a response an average
human with basic subject-matter knowledge might
provide.

Extremely Helpful Response completely addresses the intent of the

Prompt and provides helpful details/context. It is
comparable to a response a talented/well-informed
human with subject-matter expertise might provide.

m
STahur 3 frforom
t s 2 3 s
IP uIrPdadyay, 2, ,22002} bayniel
17 {R, NNo r
e 25 R D
2.1 E ov ve
6.2 MOemmb v embber_ADaDvid}
E
{U 54 T be er
SE .9Remember:
E_ r 2 2, y NooveOmTEby D
,N AM General
R_ by AD5, 202 Helpfulness rating scale works differently a from
d , EM54 R_ . 9 Nthe
NADa D 202 3 uurrsday{R1the.2 E
Needs R Met frating
MEvid } b3 f rom scale. It is important to T at IP2. 6{US characteristics of
h
understand
r
} Dany om S 1 7
each rating ielsand to what extent user intent is fulfilled
IP with the response.

Also, in-between ratings are available. Please do use them when

applicable.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 < Presentation> m
frforom
3
To what extent is the , ,22002content
2 3 y ls
e of the Response conveyed well?
r 25 R } bani
e 2 D
v e mbber_ADaDvid} STahur
Responses , Nshould:
ovemTE y D M
o O b A
E t s
IP uIrPdadyay,
y
ay, NEM 4.9R_N 17 {R, NNo
● Be u rrsddaorganized
{ R .2 5E
S
in a structure that is easy to consume
2.1 E ov ve and understand. Providing
6.2 MOemmb
h
T atheu P 1 6 U
S 1I 72most helpful information first and flowing{Uin S5E4a
.9Tlogical
Eb_er 2r 2, order with good use of
t . { e
P
formatting
I such paragraphs, lists, or tables.
R _Ny D D5D, 220023
b A
AMavi R} 23 fr
om
E}d D by frdirect,
● Be clearly written in a friendly, neutral tone that’s engaging, an om and
iels
inclusive. Avoiding off-putting tones, such as sassy, unprofessional or
judgmental voices.
● Have a consistent style with natural phrasing and transitions as if a helpful,
knowledgeable person was speaking to the user.
● Not be rambling, repetitive, or contain clearly off-topic information. Similar
information is grouped together, presented in a logical way, and not repeated
multiple times.
● Not include notable language issues or grammatical errors
● Be in the correct language for the user.
If it is in an incorrect language for the Prompt, please select "Poor" (this includes
an English response to a non-English prompt).

Remember: Presentation rating scale also has

in-between ratings.

Presentation Rating scale Explanation

Poor Response is poorly written or has notable structural,

formatting, language, or grammar issues. Or
Response has an awkward or inappropriate tone. Or
the Response repeats similar information.
mm
Or only a
o
STahur small portion of the Response r
f ro
2233 yfcontains helpful
t s 0 l s
IP uIrPdadyay, information. 2 0
2, 2 } b n i e
17 {R, NNo er 5, R Da2
2.1 E ov ve
6 M m embber_ADaDvid}
{U.254OeTmbeber v
o m E y D ME
oveO.T9 been
Nhave
Adequate
SE .9 E_ r 2 2,
R_ by AD5, 202
Response could y ,N
day, EM54 R_N
a b A written/organized better or
NADa D 202 3 r s
MEvidR} b3 ffrom may haveThminor 6.2 SE
u rd {Rlanguage/grammar issues. A minimal
atu 1IP72.1 {U
} Dany rom S
amount of less
iels IP helpful information may be present.
Users would still feel the content of the Response was
easy to consume.

Excellent Response is very well written and organized.

Sentences flow in a logical order with smooth
transitions and consistent style. The content of the
Response is conveyed in a way that is comparable to
a response a helpful, knowledgeable person might
produce.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 m
3 frforom
2 3 s
2 , ,22002} bayniel
r
e 2 5 R D
v e mbber_ADaDvid} STahur
E t s
NooveOmTEby D
,N M IP uIrPdadyay,
d a y, M 4 . 9 _NA 17 {R, NNo
s y E
ur da {R .25 E R 2.1 E ov ve
ThaturIP2.16{US 6 M m
S 17 {U.254OeTmbeber
SE .9 E_ r 2 2,
IP R_ by AD5, 202
< Overall SxS rating > NADa D 202 3
MEvidR} b3 ffrom
Overall, you should consider both factors in your SxS rating }of Dawhich
nie m response is
y ro
better. ls
● A more concise response presenting the most helpful information directly and
clearly is usually better than a longer response that may be harder to consume
and/or contains clearly off-topic information.
● Responses with Poor Presentation (e.g., rambling, inappropriate tone) should play
a significant role in your assessment of which side is better.

Tip: SxS rating in this task may work differently from

the General SxS rating guidelines.

Tip: Imagine the user is chatting with a real

person and consider which Response most
users would prefer to receive from a real
m
STahur person. 3 frforom
t s 2
0023by iels
IP uIrPdadyay, 25, ,22R } n
17 {R, NNo
2.1 E ov ve b er 2 D d Da
r
m e D vi
6 M m
{U.254OeTmbeber N ovveemTbE_yADaME}
SE .9 E_ r 2 2, , o O b A
R_ by AD5, 202 r s daayy, NEM54.9R_N
NADa D 202 3 u d {R .2 E
MEvidR} b3 ffrom ThaturIP2.16{US
} Dany rom S 17
iels IP

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Examples
m
3 frforom
2 3 s Tip: Please do a quick research for the accuracy of the
2 , ,22002} bayniel
beerr 25DDviRd D information providedTh before rating the responses.
e m
v b_ aE A } S a u
t rs
y , NooveOmTEby D A M IP uIrPdadyay,
a N 4.9R_N 17 {R, NNo
u rrsdday{,RE.M 2 5E
S
2.1 E ov ve
6 M m
h u P
T at I 2Prompt
S 17 {
. 1 6 U Response A {U.254OeTmBbeber
Response Side-by-side Rating
SE .9 E_ r 2 2,
I P R b A 5 2
_Ny D DD, 20023
AMavi R} 23 fr
What are 4 things I Architecture, Industrial If you love art Eand
d want toom Response A is Better
can major in if I love Design, Computer Science,
} D bny from
make a good salary,asome iels
art, and decent at and Mechanical potential majors to
Response B does not
math and Engineering are all lucrative consider include: Graphic
directly address the math
engineering, and majors requiring math and Design, Fashion Design,
and engineering aspects
want to make a good engineering skills which Interior Design, and Art
of the Prompt and
salary? can also involve a great History. These are 4 good
contains a somewhat
deal of artistic expression. options if you love art and
repetitive sentence
want to make a good
Helpfulness: salary, but it's important to
Somewhat keep in mind that the
specific salary for any
Presentation:
given career will vary
Excellent
depending on a number of
factors, such as location,
experience, and industry.
The response directly
addresses the intent of Helpfulness: Slightly
the Prompt and is
Presentation:
written clearly, but is
Adequate
only Somewhat helpful
because it does not
provide enough detail
about the majors. The The response does not
Presentation is directly address the
Excellent for this short math and engineering
response, but would aspects of the Prompt,
likely be better in a list does not provide
format if more details enough details about
were provided for each the majors, and is
major. repetitive with the
phrase "if you love art
and want to make a
good salary".

m
STahur 3 frforom
t s 2
0023by iels
IP uIrPdadyay, 25, ,22R } n
17 {R, NNo
2.1 E ov ve b er 2 D d Da
r
mbe_ADavi }
6 M m
{U.254OeTmbeber Remember:y, NNot ovveemTaddressing
E D E the
SE .9 E_ r 2 2, a No O.9 byNAM
R_ by AD5, 202 d , M 4 _
y RE.25 ER of the Prompt is
NADa D 202 3 ursdaaspects
important {
MEvidR} b3 ffrom ThaturIP2.16{US
} Dany rom one of the
S 1 characteristics of
7
iels IP
Slightly Helpful.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 m
3 frforom
2 3 s
2 , ,22002} bayniel
r
e 2 5 R D
v e mbber_ADaDvid} STahur
E t s
Prompt NooveOmTEby D M Response A IP uIrPdadyaResponse B Side-by-Side Rating
s d ay
y,
,N
M 4 . 9 _NA 17 {R,yN, No
E
ur da {R16.25SE R 2.1 E ov ve
ThaturIPWhen 6 MOemmb
. U
S 172 { are England's bank England's bank holidays in {U.25There ere8r 2bank
SE4.9TEbare
_ 2 ,2023,
holidays Response B is Slightly
IP holidays in 2023? 2023 are: 7 April, 10 April, 1 inREngland
_Ny D DD, 20023 they are
b A in
5 2 Better
May, 8 May, 29 May, 28 Good A MaEvidR
Friday on} 2Friday,
f
b3y frroom
August, 25 December, and } DaEaster
April 7th, 2023;
nie m Although Response A has
26 December. Monday on Monday, lsApril much better Presentation,
10th, 2023; Early May bank
Response B contains more
holiday on Monday, May
Helpfulness: helpful information such as
1st, 2023; Bank holiday for
Somewhat the names of the holidays
the coronation of King
and the days of the week
Presentation: Charles III on Monday, May
on which they fall.
Excellent 8th, 2023; Spring bank
holiday on Monday, May
29th, 2023; Summer bank
holiday on Monday, August
The response directly 28th, 2023; Christmas Day
addresses the intent of on Monday, December
the Prompt and is 25th, 2023; and Boxing Day
written clearly, but is on Tuesday, December
only Somewhat helpful 26th, 2023.
because it is missing
relevant details about
which holiday and day Helpfulness:
of week each date Extremely
corresponds to.
Presentation: Poor -
Adequate
Since only the dates
are provided, it is
Excellent presentation The response directly
to have them listed out addresses the intent of
in a sentence rather the Prompt with a very
than having a table or satisfying amount of
bulleted list with only useful details and
1-2 words for each context. However, it
item. would be better if the
information were
presented in the form
of a table or bulleted
list since there are
multiple pieces of
information provided romm
STahur f o
for each holiday. 0233 fr ls
t s
IP uIrPdadyay, 02} bynie
25, ,22R a
17 {R, NNo e r
ber DDvid D
2
2.1 E ov ve m
6 M m
{U.254OeTmbeber N ovveemTbE_yADaME}
SE .9 E_ r 2 2, , o O b A
R_ by AD5, 202 r s daayy, NEM54.9R_N
NADa D 202 3 u d {R .2 E
MEvidR} b3 ffrom ThaturIP2.16{US
} Dany rom S 17
iels IP

Tip: Please do a quick research on the accuracy of the

information provided before rating the responses.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 m
3 frforom
2 3 s
2 , ,22002} bayniel
r 5
be 2 Did D R
Prompt vember_ADavE } Response A STahur Response B Side-by-Side Rating
t s
y , NooveOmTEby DA M IP uIrPdadyay,
a N 4.9R_N 17 {R, NNo
urrsdday{,RE.M 2 5E 2.1 E ov ve
m
How to convert from To convert from Fahrenheit
{U.To
h u P
T at I 2. { 1 6 U S 6 2 Mconvert
OeTmbebefrom Celsius to Response A is Much
S Celsius7 to Fahrenheit to Celsius, subtract 32 from
5 4
Fahrenheit,
S E_ r 2rsubtract
2, 32 Better
1 E . 9
IP A 5 2 2
R by Celsius
the Fahrenheit reading and from_the NADaDD, 200reading
3
then multiply by 5/9. and then M v R 23 ff5/9.
multiply rom
E}id D} bby While Response A does not
any rom
iels interpret the Prompt
correctly, a correct related
Helpfulness:
answer is much better than
Somewhat Helpfulness: Not at all a wrong answer that might
Presentation: mislead users.
Presentation:
Adequate - Excellent Adequate - Excellent

The response The response

misinterprets the interprets the Prompt
Prompt describing a correctly as the user
correct conversion wanting to understand
from Fahrenheit to how to convert from
Celsius rather than the Celsius to Fahrenheit,
requested Celsius to however the provided
Fahrenheit. Although relationship is
not appropriately inaccurate!
framed for the Prompt,
the formula provided
could still be somewhat
useful to the user as
they could derive the Remember : Inaccurate
desired relationship information is one of the
from it with extra
effort. The characteristics of Not at
Presentation could All Helpful.
benefit from showing
an equation.

m
STahur 3 frforom
t s 2 3 s
IP uIrPdadyay, Tip: Please do a quick research on the accuracy of the , ,22002} baynielprovided
information
2
17 {R, NNo r
e 25 R D
2.1 E ov vebefore rating the responses.
6.2 MOemmb v embber_ADaDvid}
E
{U 54 T be er
SE .9 Don’t
E_ r 2 2, always assume a response is factually correct. y ,NNooveOmTEby D AM
R_ by AD5, 202 a
d , M4 _ . 9 N
NADa D 202 3
R f h uurrsday{R16E.25SER
MEvid } b3 f rom T at IP2. {U
} Dany rom S 17
iels IP

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.