ESCI Search Query:
Introduction & Training
2021
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared 1
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Table of Contents
• General Information
• Judgements
• Defects
• Next Steps
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Important Terminology
• Query = input in Amazon Search (e.g., iphone 11 accessories)
• ASIN = Amazon Standard Identification Number (i.e., product identifier)
Product ID B08L5M9BTJ
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
3
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: General Information
• The Marketplace Category needs to be
considered when evaluating a query.
• If a marketplace category is included in the
search, it means the customer selected that
category before entering the query. This
information can help clarify a broad query.
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
4
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: General Information
As a Rater, you will be presented with a
customer’s query and a product to be judged for
relevance to that query. Before making The judgements as highlighted by the
judgments, you must understand the query in ESCI Guidelines are as follows:
order to anticipate what kind of product the
customer is looking for. In instances you are • Exact
unsure, you may use Google to clarify a query.
• Substitute
• Complement
• Irrelevant
Evaluate products according to the ESCI • Unjudgeable
Guidelines
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
5
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Judgements
EXACT
This product fully satisfies the query. A broad or ambiguous query may be satisfied by a diverse set of products, so long as it
is reasonable to expect that a customer who entered that query might buy that product. A very specific query will only be
satisfied if the product matches all characteristics included in the query.
Query: adidas t-shirt green short sleeve
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
6
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Judgements
SUBSTITUTE
This product differs from the one intended, but it is functionally equivalent. This will be the same type of product, but may
differ by brand, model, color, title, or other detail. If the requested product is included in a bundle with other associated
products, it is judged as Substitute, so long as it is reasonable that the customer may want to buy both together.
Query: google home
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
7
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Judgements
COMPLEMENT
This product is not the one intended, but it could be useful to the customer if they did buy the intended product. This could
be an accessory, a component, replacement parts, or even another item designed to be used with the product.
Query: adidas shoes
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
8
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Judgements
COMPLEMENT
This product is not the one intended, but it could be useful to the customer if they did buy the intended product. This could
be an accessory, a component, replacement parts, or even another item designed to be used with the product.
Query: 24 inc tv
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
9
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Judgements
IRRELEVANT
This product does not satisfy a reasonable understanding of the customer’s intent. It could be entirely unrelated, or it could
differ by one crucial feature that would make it unsuitable for the customer.
Query: samsung s8 case
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
10
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Judgements
IRRELEVANT
This product does not satisfy a reasonable understanding of the customer’s intent. It could be entirely unrelated, or it could
differ by one crucial feature that would make it unsuitable for the customer.
Query: cat litter scoop
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
11
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Judgements
UNJUDGEABLE
This judgment is only to be used if the product cannot be judged, either because the query cannot be understood, or
because the product is lacking enough information to determine if it matches the query. An Unjudgeable query is one that is
complete gibberish, is seeking help, is attempting to navigate to a website, is in a non-market language, or is a string of
numbers that does not correspond to an ISBN or model number in Google searches.
Query: samsung s8 case
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
12
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Defect Table
• A defect is an aspect or feature of a product that detracts from its helpfulness or renders it irrelevant in relation to
the query and/or marketplace category.
Defect Judgement
Bundled product Substitute
Accessory/related product Complementary
Wrong color Substitute
Foreign Language Unjudgeable
Wrong size (one size up/down) Substitute
Wrong size (more than one size up/down) Irrelevant
Correct product, wrong brand (brand specified query) Substitute
Wrong product, correct brand (brand specified query) Irrelevant
Wrong gender/age Irrelevant
• Please be aware that this information may not fit perfectly to every scenario, so please base your overall judgement
on the query, marketplace category and the product itself.
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
13
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Tool
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
14
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Tool
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
15
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Search Query Annotation: Next Steps
• Familiarize yourself with the Guidelines. Queries and products should be considered
carefully while always keeping the Guidelines and ESCI approach in mind. Also, remember
to consider context and user intent!
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
Thank You!
This material contains information that is proprietary and confidential to TELUS International. It cannot be shared
17
with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.