Image Search Guidelines

Version 2.2 – July 2021

Part 1: Introduction to Image Search

0.0 The Image Search User Experience

Part 2: Image Search Rating Tasks

1.0 Overall Steps in Image Search Rating
2.0 Image Satisfaction Rating
2.1 Understanding Image Search Queries and Image Search Results
2.2 The Image Satisfaction Rating Scale
2.3 Image Satisfaction Rating Examples
Example: [blueberries], English (US)
Example: [chickenpox rash], English (US)
Example: [black and white nike shoes], English (US)
Example: [mona lisa], English (US)
Example: [piglet], English (US)
3.0 Landing Page Rating Scales
3.1 Prominence of the Image on the Landing Page
3.2 Examples of Prominence Ratings
3.3 Helpfulness of the Landing Page
3.4 Examples of Landing Page Helpfulness Ratings
Example: [blueberries], English (US)
Example: [black and white nike shoes], English (US)
Example: [red velvet cake recipe], English (US)
Example: [backyard shed ideas], English (US)
3.5 Page Quality of the Landing Page
4.0 Assigning Flags to Images and Landing Pages
4.1 Image Flag: Porn (P)
4.2 Image Flag: Upsetting-Offensive (U-O)
4.3 Image Flag: Not-for-Everyone (N-E)
4.4 Landing Page Flags
5.0 Image Satisfaction Rating for Porn / U-O / N-E Tolerant Queries
Example: [nazi propaganda poster], English (US)
Example: [leg punctured by branch], English (US)
6.0 Special Types of Image Search Results
6.1 Still Frames from Videos
6.2 Animated GIF Results and GIF-seeking Queries
6.3 Examples of Image Satisfaction Ratings for Animated GIF Results
Example: [beyonce gifs], English (US)
Example: [nintendo switch], English (US)
Example: [funny gifs], English (US)
7.0 Image SxS Rating
7.1 Image Search Result Set Considerations
7.2 Image SxS Example

Proprietary and Confidential – Copyright 2021 1

Part 1: Introduction to Image Search

0.0 The Image Search User Experience

Image search allows users to visually browse, find inspiration, and
complete real world tasks. Users may search repeatedly over time, going
on a journey that may begin with browsing images for styles or designs,
then end months later with large purchases or completed DIY projects.

Here are some of the important tasks or user journeys that image search
supports:

● Crafting and other DIY ("do-it-yourself") projects

● Menu planning and recipe finding
● Fashion and styling ideas
● Home and garden projects and design
● Vacation planning
● Researching and purchasing products, from everyday items to
more expensive items such as cars or jewelry

In this guideline, we use the phrase image search result to mean both the
image shown on the search results page and the landing page the image
represents. The images on the search results page may be the start of the
user's task or journey. When users click on an image search landing page,
they should be able to make progress on their task or journey. Helpful
landing pages inform users about the object(s) in the image, allow users to
purchase products, provide users with how-to's, etc.

Image search should provide users with results that represent all reasonable user intents, while showing a diverse
set of helpful, beautiful, inspirational, high quality, and visually appealing images. Users can easily scroll,
browsing many images to explore and ultimately find what they are looking for. Compared to web search, image
search is much more open to a variety of query interpretation and intents. Diversity is essential to image
search.

Image search users should be inspired, informed, and delighted. Image search results should never surprise
users with unpleasant, upsetting, offensive, or disturbing content. Hateful, violent, or sexualized images are only
appropriate if the query clearly seeks this type of content and has no other reasonable interpretation. Be sensitive
even to images that are uncomfortable, unpleasant, or not family-friendly. Even mildly unpleasant surprises lead
to a poor image search user experience. Users can't "unsee" the images on the search results page.

u=hhpopnlm

Proprietary and Confidential – Copyright 2021 2

Part 2: Image Search Rating Tasks

1.0 Overall Steps in Image Search Rating

In each image search rating task, you will:

1. Understand the query and the user task or journey. This will inform all of your subsequent ratings.
2. For each image search result, assign up to four types of ratings:
○ Image Satisfaction: How satisfying is the image for the query?
○ Image Prominence: How prominent is the image on the landing page?
○ Landing Page Helpfulness: How helpful is the landing page for the underlying user task or journey,
assuming that the user is interested in the image?
○ Landing Page PQ
3. Assign image and landing page flags.
4. Assign an overall Side-by-Side Preference rating if requested.
5. In some tasks, you will also be asked to assign Added Value ratings, Freshness ratings, and/or a Diversity
Assessment rating. See guidance inside the rating task for these steps.

2.0 Image Satisfaction Rating

2.1 Understanding Image Search Queries and Image Search Results

Users come to image search with a variety of needs, ranging from accomplishing
a quick task to making progress on a long term project. When thinking about the
user intent, keep in mind that an individual query may be part of a brief task or a
longer-term journey. Please think about the bigger picture of user needs.

In order to give high quality ratings, you must first

understand the query. Sometimes it is obvious
what users are looking for. Most of us can easily
understand the user intent behind image search
queries such as [cute kittens], [bathroom mirror],
or [quilt patterns].

Other queries may need research. For example,

you may not immediately understand the user
intent behind the query [vera wang].

Likewise, understanding how well particular

images fit a query may be easy. Cute kittens are
identifiable at a glance. However, you are not
expected to recognize a Vera Wang wedding
dress without research. If the image contains items or objects you don't recognize
or understand, please research the image itself to help you develop a better
understanding. See here for tips on how to search the web for a particular image.

Finally, please consider all reasonable query interpretations and user intents. Unless an interpretation or intent is
very unlikely or unreasonable, please assume that users may be interested in that interpretation or intent.

Proprietary and Confidential – Copyright 2021 3

2.2 The Image Satisfaction Rating Scale
The Image Satisfaction rating scale is used to rate individual images based on how well they fit the underlying
user task(s) or journey(s) for a query, including all reasonable interpretations for that query.

Important: Image Satisfaction ratings should be based only on the image you see in the rating task, not its
landing page or any text you see below the image. Occasionally you may need to research the object(s) in an
image to understand what they are, but you should not consider the content, quality, page type, or website domain
of its landing page when assigning an Image Satisfaction rating.

Image Satisfaction Rating Description

Fully Satisfying (FullyS) A special rating category that only applies to queries that are looking for one
very specific image.

Highly Satisfying (HS) A very satisfying image and a good fit for the user task or journey.

Highly Satisfying images must have one or more of the following

characteristics: extremely helpful, beautiful, inspirational, or visually appealing.

Highly Satisfying images may address any reasonable interpretation or intent

of a query, including reasonable minor interpretations.

Moderately Satisfying (MS) A satisfying image and a reasonable fit for the user task or journey.

Moderately Satisfying images are generally average to good. There is nothing

"wrong" with a Moderately Satisfying image.

Moderately Satisfying images may address any reasonable interpretation or

intent of a query, including reasonable minor interpretations.

Slightly Satisfying (SS) A less satisfying image and/or a less reasonable fit for the user task or journey.

Slightly Satisfying images may be less helpful for a reasonable interpretation

of the query, lower in quality, less visually appealing, etc.

Fails to Satisfy (FailsS) A satisfying image for very few to no image search users who issue the query.

Fails to Satisfy images may be unrelated to the query, have extremely low
image quality, or may contain unsolicited unpleasant or upsetting content.

Important: Any unpleasant images, uncomfortable images, upsetting or

offensive images, or porn images should be rated Fails to Satisfy unless the
query clearly indicates the user is seeking that type of content.

Please have high standards and reserve Highly Satisfying for truly outstanding images. Use in-­between ratings
if you think the rating of an image search result falls between two labels. You can either drag the slider or click on
the point that you want the slider to land on.

Proprietary and Confidential – Copyright 2021 4

2.3 Image Satisfaction Rating Examples
Note: All examples in this guideline use screenshots of image search results as they will be shown to you in rating
tasks, including any text (such as titles, descriptions, URLs, etc.) that may appear below the image. For Image
Satisfaction rating, please remember that your rating should be based only on the image you see in the rating
task, not its landing page or any text you see below the image.

Example: [blueberries], English (US)

User task or journey: Users may be seeking visually appealing images of blueberries. Users may also have
fresh blueberries on hand and need inspiration for what to do with them, which could be provided by images of
delicious desserts or other recipes that are made with blueberries.

Satisfaction Rating Images Explanation

Fully Satisfying N/A Because the query [blueberries] does not

(FullyS) seek one very specific image, no image can
be rated Fully Satisfying.

Highly Satisfying
(HS)

This is a visually appealing image of

blueberries. The green background, the open
hands, and the large, deeply colored berries
are a beautiful visual presentation.

This is a visually appealing image of

blueberries. The dark bowl, the highlighted
fresh blueberries in the foreground, and the
muted deeply blue berries in the background
combine for a very satisfying image.

This is a visually appealing image of ripening

blueberries on a blueberry bush. The range
of colors and the visual presentation of
ripening berries against a natural background
make this a visually appealing and very
satisfying image.

Proprietary and Confidential – Copyright 2021 5

Satisfaction Rating Images Explanation

Highly Satisfying
(HS)

(CONTINUED)

This is an image of an appetizing blueberry

smoothie. This image would inspire many
blueberry recipe-seeking users.

This is an image of an appetizing blueberry

pie. This image would inspire many blueberry
recipe-seeking users.

This is an image of an appetizing blueberry

ice cream. This image would inspire many
blueberry recipe-seeking users.

Proprietary and Confidential – Copyright 2021 6

Satisfaction Rating Images Explanation

Highly Satisfying
(HS)

(CONTINUED)

This is an image of an appetizing blueberry

jam. This image would inspire many
blueberry recipe-seeking users.

This is an image of an appetizing blueberry

ice cream. This image would inspire many
blueberry recipe-seeking users.

Proprietary and Confidential – Copyright 2021 7

Satisfaction Rating Images Explanation

Moderately This is an image of blueberries. While the

Satisfying (MS) image is OK, no particular aspect of it is
outstanding. Compared to the blueberry
images in the Highly Satisfying section,
there seems to be less effort and artistic skill
evident in this photograph. The blueberries
do not appear to be chosen or arranged, and
little effort appears to have been made in
composing the photograph, bringing out the
richness of color, etc. On the other hand,
there is nothing wrong with it. It would be
satisfying for many image search users.

Slightly Satisfying
(SS)

This is an image of a school notebook with a

blueberry-themed cover. While it is
connected to the query, blueberry-themed
school supplies are a less reasonable user
intent.

Fails to Satisfy
(FailsS)

An image of cherries is too unrelated to the

query [blueberries] to be satisfying for users.

Proprietary and Confidential – Copyright 2021 8

Example: [chickenpox rash], English (US)
User task or journey: This is a YMYL query (see "Your Money or Your Life" General Guidelines Section 2.3):
users may be trying to figure out whether a child has chickenpox or more generally want to know what the
chickenpox rash looks like.

The most important aspect of images for this query is the quality of visual information. Images of chickenpox
might not be pleasant to look at, but they should be informative for the query and no more shocking or unpleasant
than needed to help users with their task.

Satisfaction Rating Images Explanation

Fully Satisfying N/A Because the query [chickenpox rash] does not
(FullyS) seek one very specific image, no image can
be rated Fully Satisfying.

Highly Satisfying
(HS)
This is a very helpful image for chickenpox
rash identification because it shows both the
general appearance of a child with chickenpox
as well as specifically what the rash looks like.
The photo is high quality, and the magnified
portion gives details of the chickenpox rash.

This is a very helpful image for chickenpox

rash identification because it shows both the
general appearance of a child with chickenpox
as well as specifically what the rash looks like
on the skin. This high quality medical
illustration would be very helpful for users
concerned about what the chickenpox rash
looks like.

Moderately This is an image of a child with chickenpox

Satisfying (MS)
and would be helpful for many users.
However, the image doesn't show the
chickenpox rash in detail. Notice how the
lighting from the side that obscures the detail
on the face, the cropping of the child's head
and arm, and the distance from the camera all
limit the visual information quality and make
this photo a Moderately Satisfying image,
rather than a Highly Satisfying one.

Proprietary and Confidential – Copyright 2021 9

Satisfaction Rating Images Explanation

Slightly Satisfying
This is a low quality photo with limited visual
(SS)
information. The very high contrast bleaches
the image towards the middle and fades to
black on the edges, making it difficult to see
the rash on the skin. The black border above
and below the photo also looks out of place
and adds nothing to the informational value.
The low quality and low level of visual
information make this image helpful for few
users.

Fails to Satisfy
(FailsS)

This is a picture of shingles, not chickenpox.

This result is unhelpful for users who want to
know what chickenpox looks like.

Proprietary and Confidential – Copyright 2021 10

Example: [black and white nike shoes], English (US)
User task or journey: Users may be shopping for shoes or looking for stylish photos.

For this query, inspiring images may showcase the shoes themselves or style them in an appealing way. Highly
satisfying images include photos that display the shoe details clearly, or show how the shoes can be combined
with clothes and accessories to complete a stylish outfit.

Satisfaction Rating Images Explanation

Fully Satisfying N/A Because the query [black and white nike
(FullyS) shoes] does not seek one very specific
image, no image can be rated Fully
Satisfying.

Highly Satisfying
(HS)

This is a styling photo of someone wearing

black and white Nike shoes. The photo
shows how black and white Nike shoes can
be part of a stylish look, including what to
wear with them and accessories that match
well.

This is a photo of black and white Nike shoes

on a model. The photo displays the shoe
details clearly and in a visually appealing way.

Proprietary and Confidential – Copyright 2021 11

Satisfaction Rating Images Explanation

Highly Satisfying
(HS)
This is a photo of black and white Nike shoes,
(CONTINUED) showing the front, sole, and sides. An effort
was made in this product photo to show users
the features of the shoes.

This is a photo of a new (2017) type of black

and white Nike shoe. An effort was made in
this product photo to show users the shoes in
an artistic and visually appealing way.

Moderately
Satisfying (MS)
This is a typical product photo of a black and
white Nike shoe. The image is average to
good, but not outstanding in any dimension.
This image would be satisfying for many
image search users.

Slightly Satisfying
(SS)

The query is very specific about choice of

color, so it seems unlikely that users would be
satisfied by this black, white, and pink shoe.

Fails to Satisfy
(FailsS)

These are not black and white Nike shoes.

Proprietary and Confidential – Copyright 2021 12

Example: [mona lisa], English (US)
User task or journey: Most users are looking for an image of the famous painting Mona Lisa by Leonardo da
Vinci. Reasonable minor intents include learning more about the painting, such as the artist's biography, the
location where it is displayed, how it was painted, etc.

For this query, highly satisfying images should inform users about the painting, and perhaps inspire a love for the
arts.

Satisfaction Rating Images Explanation

Fully Satisfying
(FullyS)

This is the specific painting requested in the

query.

Highly Satisfying
(HS)

This image combines a portrait of the artist

with his most famous painting. Many users
would very satisfied by this paired image.

Moderately This photo shows the user the experience of

Satisfying (MS) seeing the actual Mona Lisa painting in the
Louvre. This would be helpful for users who
are interested in planning a trip or are curious
about what the experience is like to see the
Mona Lisa in person.

Slightly Satisfying
(SS)
This is a business logo for a company with
"Mona Lisa" in its name. While it is connected
to the query, it is a less reasonable user intent.

Fails to Satisfy
(FailsS)
This is a painting by a different artist
(Picasso). It is too unrelated to the query to
be satisfying.

Proprietary and Confidential – Copyright 2021 13

Example: [piglet], English (US)
User task or journey: Users may be interested in images of Piglet (the character from Winnie the Pooh) or
photos of cute baby pigs. Reasonable minor intents include Piglet/baby pig costumes, Piglet/baby pig toys and
products, or Piglet/baby pigs as a design element in a crafting or home project such as a Piglet-themed nursery.

For this query, inspiring images should cheer you up, delight you, or make you say, "How cute!"

Satisfaction Rating Images Explanation

Fully Satisfying N/A Because the query [piglet] does not seek one
(FullyS) very specific image, no image can be rated
Fully Satisfying.

Highly Satisfying
(HS)

Users may be interested in images of Piglet,

a character from Winnie the Pooh. This is a
visually appealing image of Piglet frolicking
with a flower.

This is a very visually appealing image of

Piglet with his best friend Winnie the Pooh.
The character of Piglet is known as Pooh's
best friend, and this image captures that
special friendship in a lovely way.

Proprietary and Confidential – Copyright 2021 14

Satisfaction Rating Images Explanation

Highly Satisfying
(HS)

Users may be interested in pictures of baby

pigs. This image is very cute and very
satisfying.

Users may be interested in pictures of baby

pigs. This image is very cute and very
satisfying.

This is a photo of a baby wearing a piglet

costume. It is very cute and appealing. It
addresses a more minor but still reasonable
user intent: piglet costumes and piglet
crafting projects.

Proprietary and Confidential – Copyright 2021 15

Satisfaction Rating Images Explanation

Moderately
Satisfying (MS)
This photo of a piglet is average to good and
would be satisfying for many image search
users. There is nothing wrong with it, but it is
not particularly adorable or inspiring.

This is an image of Piglet having an "Oh-No!"

moment. While there is nothing wrong with
this image and it captures a less common
side of Piglet's character, it may not be as
inspiring or visually appealing as other
images of Piglet from Winnie the Pooh.

Slightly Satisfying
(SS) This image is lower quality: the image is a bit
grainy or pixelated; the steps are somewhat
confusingly arranged and difficult to follow;
there is both a pink decorative font as well as
handwritten annotations. The lower quality of
this how-to image and the slightly more niche
intent of balloon animal crafting combine to
make this image Slightly Satisfying.

This is an image of a type of nail polish

named Piggy Polish, which features a pig on
the bottle. This is a very unlikely
interpretation or intent. Few users would be
satisfied with this image for the query [piglet].

Fails to Satisfy
(FailsS)
An image of a kitten, even an extremely cute
one, is too unrelated to the query [piglet] to
be satisfying for users.

IMPORTANT: The next page of this guideline includes examples of images that may be slightly unpleasant to
view. If you do not feel comfortable viewing these images at this time, please skip to Section 3.0.

Proprietary and Confidential – Copyright 2021 16

Satisfaction Rating Images Explanation

Fails to Satisfy
(FailsS)
This image prominently features piglet
genitalia in the center of the image. Nothing
(CONTINUED)
in the query suggests users are interested in
this type of content.

This photo features a severely deformed

piglet. This image could unpleasantly
surprise, upset, or disturb some users.
Nothing in the query suggests users are
interested in this type of content.

Proprietary and Confidential – Copyright 2021 17

3.0 Landing Page Rating Scales
For most images you will see in image search rating tasks, there are many pages available on the Internet that
contain the image. Image search should provide the most helpful landing page(s) for any particular image. Your
job is to evaluate the landing page on the following three scales:

1. Image Prominence: How easily can the image be found on the landing page?
2. Landing Page Helpfulness: How helpful is the landing page for the underlying user task or journey,
assuming that the user is interested in the image?
3. Landing Page PQ: Evaluate the Page Quality of the landing page as described in the General
Guidelines.

Important: In an image search rating task, you may see multiple results showing the exact same image, but with
different landing pages. In this case, your image satisfaction ratings should be the same for each image, while
your landing page ratings should vary according to the content and quality of each landing page.

3.1 Prominence of the Image on the Landing Page

The Image Prominence rating scale is used to indicate how easily the image can be found on the landing page.

Important: Image Prominence ratings should be based only on the image and the landing page, not the query.
You do not need to think about the query when assigning Image Prominence ratings to results.

Prominence Rating Description

Main Feature The image is the main feature of the landing page. All users would easily find the
image.

Easy to Find The image is easy to find on the landing page, but it is not the main feature of the
page. Most users would find the image.

Hard to Find The image can be found on the landing page, but it requires effort. Many to some
users would find the image.

Missing The image cannot be found anywhere on the landing page, even with effort.

This rating should also be used if the image is so difficult to find that few or no
users would find it, even though it is present somewhere on the landing page.
For example, if the image is not in the first few scrolls of a very long page with
many images, you may consider it Missing without investigating further.

Proprietary and Confidential – Copyright 2021 18

3.2 Examples of Prominence Ratings
Here are examples of images and landing page links (screenshots) for each prominence rating. For your
convenience, the image is framed with a red rectangle in the screenshot to show where it appears on the page.

Image and LP Prominence Rating Explanation

Main Feature This is a page from an educational website featuring this

image with information below it. The image is the main feature
of the landing page.

Landing Page

Easy to Find This is an informational page about the painting in the image.
The image is under the first paragraph, and it is easy to find on
a long page that mostly contains text.

Landing Page

Hard to Find This page has so many images related to the Mona Lisa that
it's hard to find this particular image.

Landing Page

Missing This is an article about a British Television Series: 100 Great

Paintings. The page does not display this image, though there
is text and a link to the painting from this page.

Landing Page

Proprietary and Confidential – Copyright 2021 19

Image and LP Prominence Rating Explanation

N/A Main Feature Example could not be found for this image at this time

Easy to Find This is a recipe page that shows step-by-step instructions on

how to make the recipe. This image is one of several on the
landing page, but it is easy to find.

Landing Page

Hard to Find This is a search results page full of images. This image is
towards the top, but it is still hard to find since it is one of many
similar images on the page.

Landing Page

Missing This image is technically included in the slideshow on the

landing page, but it is many clicks deep through a very long
set of slides. Because it is not immediately visible and takes
many clicks to find, many users would think it is missing.

Landing Page

Proprietary and Confidential – Copyright 2021 20

3.3 Helpfulness of the Landing Page
The Landing Page Helpfulness rating scale is used to rate how helpful the landing page would be for the
underlying user task or journey, assuming that the user is interested in the image.

Important: The helpfulness of the landing page only matters when the corresponding image would satisfy at least
some users and can be found somewhere on the page. As a result, when an image is Fails to Satisfy (FailsS)
or Missing, its landing page should automatically receive an Unhelpful (UH) rating without further
consideration. For LP Helpfulness ratings, you only need to consider the content of the landing page when the
corresponding image is at least Slightly Satisfying (SS) and can be found somewhere on the page.

The remainder of this section only applies to landing pages that correspond to images that are at least Slightly
Satisfying (SS) and can be found somewhere on the page.

When rating the helpfulness of the landing page, please start by considering the possible user task(s) or
journey(s) behind the query. Image search users may be trying to accomplish a specific task, such as buying a
product, or they may be on a longer-term journey, such as completing a project. As image search users look
through the results, users click on images to learn more. Helpful landing pages allow users to learn more about
the image or object(s) in the image and make progress on their task or journey.

Then, please assume that the user is interested in the image, and ask yourself whether the landing page
would help them complete their task or make progress on their journey. Helpful landing pages don't just provide
the image. Helpful landing pages have in-depth content about the image or object(s) in the image.

Putting it all together, helpful landing pages must be helpful both for the user task or journey AND for users
interested in the image. If a landing page is not helpful for the user task or journey, or if it is not helpful for users
interested in the image, then the LP Helpfulness rating should be Unhelpful (UH).

Here are some examples of potentially helpful landing pages (depending on the user task or journey) for different
types of images:
● For an image of a product, helpful landing pages should be about the product in the image and may
include information such as specs, reviews, etc. Helpful landing pages may offer the product for
purchase directly, give information about how or where to purchase it, provide reviews about it, etc.
● For an image of a recipe or dish, helpful landing pages may be about the specific recipe in the image.
Helpful landing pages may include step-by-step instructions on how to prepare the dish, reviews of the
recipe, tips on presentation, alternate ingredients, nutritional information, etc.
● For an image depicting a medical condition, helpful landing pages should give authoritative health
information about the condition in the image. This may include symptoms, diagnosis criteria, when to
seek formal medical attention, etc.
● For an image of a celebrity or musical group, helpful landing pages may explain when the photo was
taken or where the image came from, mention who is included, and give biographical information or
recent news about the people pictured. Helpful landing pages may also provide a relevant video featuring
the person, such as a music video from a musical group.
● For an image of a DIY, craft, or other project, helpful landing pages may offer detailed how-to
information, such as materials and resources, so that a user could complete a similar project of their own.
This may include step-by-step instructions, video tutorials, blueprints or technical drawings, etc.

Proprietary and Confidential – Copyright 2021 21

There are a few types of landing pages that are Unhelpful (UH) for users interested in the image:
● Any landing page that does not help users along their task or journey. Helpful landing pages must be
helpful both for the user task or journey AND for users who are interested in the image.
● Any landing page that consists only of the image itself. Users have already seen the image on the image
search results page. Helpful landing pages must offer much more than just the image.
● Any landing page that has little information specific to the image or the object(s) in the image. Helpful
landing pages must be about the image or the object(s) in the image.
● Any landing page filled with many similar images. Remember that image search users have just browsed
through image search results and have chosen an image of interest. A landing page that requires
users to search again by visually scanning a large number of similar images is unhelpful.
● Any landing page on which the image cannot be found at all. A landing page that does not allow users
to find the corresponding image is unhelpful.

Landing Page Helpfulness Rating Description

Very Helpful (VH) Landing Page The landing page is very helpful both for the user task or journey and for
users interested in the image. Most users would be helped along their
task or journey, and the landing page has a substantial amount of very
helpful content or information about the image (or the object(s) in the
image).

Note: A landing page is only eligible for this rating if the image is at least
Slightly Satisfying (SS) and can be found somewhere on the page.

Moderately Helpful (MH) Landing The landing page is helpful both for the user task or journey and for users
Page interested in the image. Many users would be helped along their task or
journey, and the landing page has helpful content or information about
the image (or the object(s) in the image).

Note: A landing page is only eligible for this rating if the image is at least
Slightly Satisfying (SS) and can be found somewhere on the page.

Slightly Helpful (SH) Landing Page The landing page is slightly helpful for the user task or journey and for
users interested in the image. Some users would be helped along their
task or journey, and the landing page has some content or information
about the image (or the object(s) in the image).

Note: A landing page is only eligible for this rating if the image is at least
Slightly Satisfying (SS) and can be found somewhere on the page.

Unhelpful (UH) Landing Page For any image that is Fails to Satisfy (FailsS) or Missing, its landing
page should automatically receive a rating of Unhelpful (UH).

For images that are at least Slightly Satisfying (SS) and can be found
somewhere on the page, the landing page should be rated as Unhelpful
(UH) when either of the following are true:
● The landing page is not helpful for the user task or journey. Few
or no users would be helped along their task or journey.
● The landing page is not helpful for users interested in the image.
The landing has little or no content or information about the
image (or the object(s) in the image).

Proprietary and Confidential – Copyright 2021 22

3.4 Examples of Landing Page Helpfulness Ratings

Example: [blueberries], English (US)

Image and LP LP Helpfulness Rating Explanation

Very Helpful (VH) Users who are interested in this image may want to make
the ice cream pictured in the image. This is a detailed
recipe page that would be very helpful for anyone wanting
to make this ice cream.

Landing Page

Slightly Helpful (SH) This landing page is a list of blueberry recipes, each with a
link to other websites for the recipe. The list is short, and
this image corresponds to the first recipe shown. There is
some content on the page specific to this image and
recipe.

However, the landing page itself isn't that helpful for users
interested in making this ice cream.

Landing Page

Unhelpful (UH) This is a search results page with many images and very
little information about each. A user who is interested in
this particular image would have to visually scan the
landing page to locate the image, then click on it to find
more information or content about the actual recipe.

Landing Page

Unhelpful (UH) This landing page is the image itself, with very little
additional information or content about this image. Even
though the image is extremely prominent, the page is
unhelpful for users trying to accomplish their task.

Landing Page

Proprietary and Confidential – Copyright 2021 23

Example: [black and white nike shoes], English (US)

Image and LP LP Helpfulness Rating Explanation

Very Helpful (VH) This is the product information and purchase page for the
shoe in the image on the official Nike website. This page
has product information, reviews, and offers users a way
to purchase these shoes. This page would be very helpful
for users looking for information/reviews or for those who
want to purchase this product.

Landing Page

Moderately Helpful (MH) This page has styling advice on sneakers and specifically
describes the type of black and white Nike sneakers
pictured. This page would be helpful for users interested
in styling or fashion.

Landing Page

Slightly Helpful (SH) This is an article about the shoes and clothes worn by
tennis superstar Serena Williams. It does mention that
Serena Williams wears Nike shoes, but the landing page
does not describe which Nike shoes are shown in this
image. This landing page is Slightly Helpful to users who
are interested in the shoes shown in this image.

Landing Page

Unhelpful (UH) This is a page full of shoe options from a popular shoe
retailer. To get information about the specific shoe in the
image, users would have to scan the listing page to find
this particular shoe and then click into the page specific to
this shoe.

Landing Page This landing page is Unhelpful for this image because the
landing page has very little content specific to this shoe.

Proprietary and Confidential – Copyright 2021 24

Example: [red velvet cake recipe], English (US)
User task or journey: Users may be planning to bake a red velvet cake, and they are most likely looking for a
beautiful cake along with a detailed recipe and instructions for preparing it. For this query, the word "recipe"
corresponds to an important part of the underlying user journey. Most users want to bake the cake themselves,
rather than purchasing an already prepared cake, finding restaurants that serve red velvet cake, etc.

Image and LP Image Satisfaction Rating and LP Helpfulness Rating and

Explanation Explanation

Highly Satisfying (HS) Very Helpful (VH)

This is a high quality, thoughtfully This page is a recipe page for the red
arranged, inspirational photo of an velvet cake depicted in the image.
appetizing red velvet cake. This The page is comprehensive and
image would be helpful for many high-quality; it comes from a
users who issue this query. trustworthy site, contains helpful
reviews, and even features a video
tutorial demonstrating how to prepare
the cake. This page would be helpful
for users on a cake-making journey
Landing Page who are interested in this image.

Moderately Satisfying (MS) Unhelpful (UH)

This is a decent image of a red velvet This page is a high-quality product

cake. The photo quality is not quite page that allows users to purchase
as good as the example above due to the red velvet cake depicted in the
dim lighting and slightly too much of a image. However, remember that the
zoom, but the cake still looks user is on a journey to bake a cake
professionally made and appetizing. themselves, not a journey to purchase
a prepared cake. As a result, the
page would not be helpful for the user
journey, so it should be rated
Unhelpful (UH).
Landing Page Note: If the query were just [red velvet
cake] (without the word "recipe"), this
landing page would be helpful for that
broader user journey, and the page
would instead be rated Very Helpful
(VH).

Fails to Satisfy (FailsS) Unhelpful (UH)

This is a zoomed-in image of a glass Because the image is Fails to Satisfy,

jar. The image clearly does not depict this landing page should automatically
a red velvet cake, so it would not be receive an Unhelpful (UH) rating. No
satisfying to users. additional consideration is necessary
regarding the helpfulness of this
landing page.

Landing Page

Proprietary and Confidential – Copyright 2021 25

Example: [backyard shed ideas], English (US)
User task or journey: Most users issuing this query are interested in backyard sheds. Users may want to
purchase a fully assembled backyard shed, or they may be seeking shed designs that they could build
themselves. Among users who have a DIY project in mind, some may be new to building and would like to find
detailed tutorials, while others may be skilled builders who are ready to get started and are simply looking for a
variety of images to inspire them.

Image and LP Image Satisfaction Rating and LP Helpfulness Rating and

Explanation Explanation

Highly Satisfying (HS) Very Helpful (VH)

This is a beautiful, inspirational, and This page is a very helpful tutorial

helpful photo of a backyard shed. describing how to build the backyard
The image features the shed in the shed depicted in the image. The
context of a backyard and with a page contains clear, detailed
person standing in the corridor, instructions, accompanied by
which provides a sense of scale as high-quality photos of each step.
well as an idea of what the shed Overall, the content on this landing
might look like in an actual page would be very helpful for users
backyard. on a shed-building journey who are
interested in this image.
Landing Page

Moderately Satisfying (MS) Very Helpful (VH)

This is a basic photo of a backyard This page is a very helpful product

shed. It's a decent image for the page focused on the backyard shed
query, but is not particularly depicted in the image. The landing
beautiful or inspiring. page provides helpful product details,
reviews, and the ability to purchase
this shed, all of which would be
helpful for users on a
shed-purchasing journey who are
interested in this image.
Landing Page
Note: An image does not need to be
Highly Satisfying (HS) in order for its
landing page to be Very Helpful (VH).

Fails to Satisfy (FailsS) Unhelpful (UH)

This image depicts a children's Because the image is Fails to Satisfy,

outdoor playset, which is very the landing page should automatically
different from a backyard shed. receive an Unhelpful (UH) rating. No
This would benefit very few to no additional consideration is necessary
users who issue this query. regarding the helpfulness of this
landing page.

Landing Page

Proprietary and Confidential – Copyright 2021 26

3.5 Page Quality of the Landing Page
Please see the General Guidelines for more information on how to assign Page Quality ratings to landing pages.

Important: Landing Page PQ ratings should be based only on the landing page, not the query or the image. You
do not need to think about the query or the image when assigning Landing Page PQ ratings to results.

Proprietary and Confidential – Copyright 2021 27

4.0 Assigning Flags to Images and Landing Pages
Image search rating tasks have special flags for images, as well as for landing pages in some tasks. You will
assign Porn (P), Upsetting-Offensive (U-O), and Not-for-Everyone (N-E) flags to images when appropriate.

You may also be asked to assign the Porn, Foreign Language, Didn’t Load, and Upsetting-­Offensive (U-O)
flags to landing pages when appropriate, as defined in the General Guidelines (Section 15.0).

All flags should be assigned based on the image or the landing page and do not depend on the query.

4.1 Image Flag: Porn (P)

Please flag porn images based on the standards of your locale. If you are not sure whether an image qualifies as
porn (such as racy images or borderline porn), the Not-for-Everyone (N-E) flag discussed in Section 4.3 may be
more appropriate than the Porn flag. Please use your judgment.

Different locales have different cultural sensitivities and standards, but the following types of images should be
flagged as Porn:
● Photos, cartoons, or other depictions of people engaging in sexual activity or in poses that are highly
suggestive of sexual activity, even if bare body parts are not exposed.
● Sexually suggestive depictions of naked or minimally clothed people in photos, cartoons, or drawings.
● Pornographic text, such as an image displaying the words, "HOT SEX".

4.2 Image Flag: Upsetting-Offensive (U-O)

Please use your judgment and knowledge of your rating locale to flag images that users in your locale may find
Upsetting or Offensive. Because U-O images may be disturbing or even have the potential to cause
psychological harm, U-O images must be shown with extreme care and only when users have issued queries that
clearly request U-O content. Even if a query seeks U-O content, image search results should include the least
upsetting or offensive content to satisfy the query.

Please flag all U-O images, even if the query explicitly seeks U-O images. Flagging these images is important so
that image search engines understand what types of images are U-O for users in your locale.

Upsetting-Offensive content includes:

● Graphic and/or disturbing depictions of harm, violence and gore including dead bodies, torture, etc.
● Graphic and/or disturbing depictions of invasive medical procedures, bodily fluids, severe medical
conditions or abnormalities, etc.
● Images that depict profanity, racial slurs, or offensive gestures.
● Images that promote hate or violence against a group of people based on criteria including (but not limited
to) race or ethnicity, religion, gender, nationality or citizenship, disability, age, sexual orientation, or
veteran status. This includes upsetting or offensive memes, infographics, photoshopped images,
animated or faked images, caricatures, etc.
● Images that incite or encourage violence or harm to self or others, including how-to images for harmful
activities such as bomb-making, etc.
● Any other images that users in your locale would find upsetting and/or offensive.

IMPORTANT: The next page of this guideline includes examples of images that are Upsetting-Offensive
(U-O). If you do not feel comfortable viewing these images at this time, please skip to Section 4.3.

Proprietary and Confidential – Copyright 2021 28

Image Flag Explanation

U-O This image has both profanity and an offensive

gesture.

U-O This image is a very offensive and hateful statement

about a racial group.

U-O This is a gory image of a severe medical injury. Users

may be upset by this graphic depiction of harm.

Proprietary and Confidential – Copyright 2021 29

4.3 Image Flag: Not-for-Everyone (N-E)
People search for images from all over the world: people of all ages, nationalities, races, religions, etc. People
search for images at home, at school, in the workplace, and in public. It is very important to be sensitive to results
that may make users feel uncomfortable for any reason. No user should ever be shown images search results
that make them feel unpleasantly surprised or uncomfortable.

Not-for-Everyone (N-E) images may be racy or sexual, distasteful, unpleasant, or related to sensitive topics that
are not appropriate for all ages or all environments. N-E images may unpleasantly surprise sensitive users and
result in a poor user experience if shown when users are not looking for that type of content.

Use the N-E flag for any content that you feel may be unpleasant or uncomfortable for some users. Use this flag
even if the query explicitly requests such content. Flagging these images is important so that image search
engines understand what types of images may be unpleasant or uncomfortable for users in your locale.

Flag even mildly uncomfortable or unpleasant images related to any of the following:

● Racy or sex-related images including images that show human or animal genitalia, even if the image is
blurred, slightly obscured, clearly photoshopped or "faked", illustrated, shown for the purposes of
education, etc.
● Images of medical procedures, bodily fluids, medical conditions, or abnormalities, even if illustrated for
educational purposes, clearly photoshopped, created with special effects, etc.
● Mild depictions of harm including self-harm, animal or human suffering, animal or human injuries, etc.,
even if clearly photoshopped, illustrated, created with special effects, etc.
● Images that negatively or unflatteringly portray groups of people, including memes, infographics, or
photoshopped images.
● Images that promote conspiracy theories or other factually unsupported or misleading information,
including memes, infographics, or photoshopped images.
● Mild profanity or mildly rude gestures.
● Any image that would not be appropriate in a public space, a work environment, a library, a school, etc.
● Any other image that would make sensitive users in your locale uncomfortable.

Important: Please note that the Porn and U-O flags are considered stronger than the N-E flag.

● If you flag an image as Porn or U-O, the N-E flag for that image will be automatically disabled in the rating
task.
● If you flag an image as N-E, you should also consider whether the Porn and/or U-O flags are appropriate
for that image as well. Please represent the sensitivities of users in your rating locale.

IMPORTANT: The next two pages of this guideline include examples of images that are Not-for-Everyone
(N-E). If you do not feel comfortable viewing these images at this time, please skip to Section 4.4.

Proprietary and Confidential – Copyright 2021 30

Note: All of the images below should receive the Not-for-Everyone (N-E) flag. Depending on the sensitivities of
your rating locale, the Porn or U-O flags may be appropriate as well. Please use your judgment.

Image Flag Explanation

N-E This is a racy photograph of a woman with exposed cleavage.

This image may be inappropriate for children or users in a
professional workplace.

N-E This sex position chart may be inappropriate for children or

users in a professional workplace.

N-E This medical illustration of human stool may be appropriate for

medical queries. However, images of stool, urine, blood,
vomit, etc. should be flagged as N-E.

N-E This picture of piglet genitalia may be distasteful for some

users.

N-E This photo of animal sex may be inappropriate for children or

in a professional workplace.

N-E Users may find this image of a severely deformed piglet

unpleasant or disturbing.

Proprietary and Confidential – Copyright 2021 31

Image Flag Explanation

N-E This image shows methods of suicide. This image may be

inappropriate for children and other sensitive users.

N-E This sign has mild profanity and expresses sentiments that
may be inappropriate for young children and other sensitive
users.

N-E This is a Nazi propaganda poster. While the image may have
historical importance, the ideology of racial supremacy
depicted may not be appropriate for all audiences.

Proprietary and Confidential – Copyright 2021 32

4.4 Landing Page Flags
Please refer to the General Guidelines (Section 15.0) for use of Porn, Foreign Language, Didn’t Load, and
Upsetting-­Offensive flags for landing pages.

5.0 Image Satisfaction Rating for Porn / U-O / N-E Tolerant Queries
Some queries may indicate that the user is explicitly seeking Porn, U-O, or N-E images. For these explicit
queries, Porn, U-O, or N-E images may be the most helpful results, and thus may receive high Image Satisfaction
ratings. For example, the most helpful results for the query [photos of dead bodies] are likely to be U-O or N-E.

Please select the Porn, U-O, and/or N-E flags for all such images, even if the query explicitly seeks Porn, U-O, or
N-E images.

Have high standards and think about the user intent and overall user experience. Assume that users have a
genuine educational or informational intent whenever possible. As always, image search results (including Porn,
U-O, and N-E results) must be helpful and satisfying for the query to receive high Image Satisfaction ratings.

Do not assume that Porn / U-O / N-E tolerant queries "deserve" the most graphic, offensive, or shocking images.
Do not assume that Porn / U-O / N-E tolerant queries are issued by people who hold upsetting or offensive views.
Do not assume that users are merely seeking to validate an offensive or upsetting perspective.

A very offensive Porn / U-O / N-E image should only be given a high satisfaction rating if it is very clear that the
user is seeking offensive images and such images are necessary to satisfy the user intent. Image search results
should be no more shocking, disturbing, or upsetting than necessary in order to satisfy the user intent and
underlying user task or journey.

IMPORTANT: The next page of this guideline includes examples of images that are Upsetting-Offensive
(U-O) and/or Not-for-Everyone (N-E). If you do not feel comfortable viewing these images at this time,
please skip to Section 6.0.

Proprietary and Confidential – Copyright 2021 33

Example: [nazi propaganda poster], English (US)
User task or journey: Most users are looking for historical examples of propaganda posters from World War
II-era Germany, when the Nazi regime was in power (approx. 1930s–40s). Reasonable minor intents include
learning more about Nazi propaganda in general, including how it was disseminated, different types and
techniques, museums where it is displayed, etc.

For this query, assume that users have an educational or informational intent. Highly satisfying images should
provide factually accurate examples of this topic. Image Satisfaction ratings should be approached with care,
however, due to the sensitivity of the topic.

Image Satisfaction Rating Flags Explanation

Highly Satisfying N-E This is a historically accurate and

(HS) representative image of a Nazi
propaganda poster. The photo is high
quality, and the image would be very
helpful for users seeking visual examples
of this topic.

Note that this image depicts the ideology

of racial supremacy from a very upsetting
time in history, so it would not be
appropriate for all queries. The N-E flag is
appropriate for this image. However, in
this case the user explicitly requests this
type of sensitive content, so Highly
Satisfying is appropriate.

Example: [leg punctured by branch], English (US)

User task or journey: The query clearly indicates that users are looking for gory images. The underlying user
need may be educational. For example, users may be seeking photos for medical training purposes, materials for
designing special effects, etc.

Image Satisfaction Rating Flags Explanation

Highly Satisfying U-O This image is very helpful for the query, so
(HS) Highly Satisfying is appropriate.

However, this image is a graphical

depiction of severe and potentially
disturbing harm, so it should be flagged
as U-O.

Proprietary and Confidential – Copyright 2021 34

6.0 Special Types of Image Search Results

6.1 Still Frames from Videos

In some tasks, you may be asked to rate an image result that is a still frame (a single static image) taken from a
video on the landing page. Here are some examples of what these results will look like in ratings tasks:

Please apply the following guidance to your ratings for still frame images from videos:

● Image Satisfaction should be rated as usual. For example, if a still frame image is blurry, contains
distracting/unhelpful backgrounds or borders, or has other qualities that make it less visually appealing,
then these attributes should contribute to a lower Image Satisfaction rating, just as they would for any
other image result.
● Image Prominence should capture how easily the image can be found on the landing page. This will
require you to consider both of the following: (1) How prominent is the video on the landing page? and (2)
How easily can you find the image within the video?
○ If the image can be easily found within the video—or corresponds to the video's thumbnail
image—then your Image Prominence rating should just correspond to how prominent the video is
within the landing page, and may be as high as Main Feature.
○ The harder it is to find the image within the video, the lower your Image Prominence rating should
be, even if the video itself is highly prominent on the landing page. For example, if the image is
very difficult to find in the video even after skipping around a bit, you may consider it Missing
without investigating further.
● Landing Page Helpfulness should be rated as usual. For still frame images that are at least Slightly
Satisfying (SS) and can be found somewhere within the video, your LP Helpfulness rating should
consider the helpfulness of the landing page as a whole, including the video content.
○ Examples of helpful video content include:
■ a helpful video review of a product featured in the image.
■ a relevant music video from a musician depicted in the image.
■ a helpful tutorial video on how to create a home garden design portrayed in the image.
○ Examples of unhelpful video content include:
■ a video that has nothing to do with a celebrity featured in the image.
■ a low-resolution, blurry video that is too low quality to be helpful.

Proprietary and Confidential – Copyright 2021 35

6.2 Animated GIF Results and GIF-seeking Queries
In some tasks, you may be asked to rate image result(s) that are animated GIFs. In general, when you are
presented with animated GIFs, you should consider reasonable user intents as well as the animated nature of the
result when providing an Image Satisfaction rating. You should ask yourself: Would some or most users benefit
from the animation? Or is the animation distracting or unhelpful?

Some queries clearly seek animated GIF results, such as [beyonce gif] or [funny cat animations]. However, many
queries do not clearly express a need for animated GIF results, such as [dramatic chipmunk], [dancing baby],
[strawberry shortcake], and [nintendo switch]. For these queries, animated GIFs may or may not be satisfying to
users depending on the query and the particular result.

● When the user is clearly seeking animated GIF results:

○ You should reserve Highly Satisfying ratings for satisfying animated GIFs that fit the query.
○ Non-GIF results should automatically receive a rating of Fails to Satisfy, since the animation
aspect of the user intent is not met.
● When the user is NOT clearly seeking animated GIF results:
○ Non-GIF results should receive Image Satisfaction ratings as usual; see Section 2.0.
○ Animated GIF results conveying a message, thought, or emotion that satisfies a reasonable user
intent may receive ratings up to Highly Satisfying. Please use your judgment.
○ Animated GIF results conveying a message, thought, or emotion that does not satisfy a
reasonable user intent should be considered distracting and unhelpful. In these cases, your
Image Satisfaction rating should be lowered due to the distracting animation aspect, and typically
should not exceed Slightly Satisfying.

6.3 Examples of Image Satisfaction Ratings for Animated GIF Results

Note: All examples in this section use static screenshots of animated GIF results. Please click the link below
each image to view the animation as it would appear to image search users.

Proprietary and Confidential – Copyright 2021 36

Example: [beyonce gifs], English (US)
User task or journey: Users are looking for animated GIF results that feature the celebrity Beyoncé. For this
query, highly satisfying results should be animated GIFs of Beyoncé that are visually appealing, well-composed,
and express an emotion, message, or scene from her life as a pop culture singer-songwriter and actress.

Image (Animated GIF) Satisfaction Rating Explanation

Highly Satisfying This is a high-quality, visually appealing GIF focused

(HS) on Beyoncé.

View animated GIF

Highly Satisfying This is another high-quality GIF featuring Beyoncé,

(HS) and it captures a segment from one of her most iconic
music videos.

View animated GIF

Moderately This animated GIF result depicts Beyoncé smiling at

Satisfying (MS) an awards ceremony. This GIF shows what Beyoncé
looks like, but it doesn't convey any specific message,
and it isn't particularly inspiring or amusing.

View animated GIF

Moderately This is an animated GIF result featuring Beyoncé.

Satisfying (MS) This result conveys a specific message that satisfies a
reasonable minor intent. However, this GIF does not
have the best photo resolution.

View animated GIF

Proprietary and Confidential – Copyright 2021 37

Image (Animated GIF) Satisfaction Rating Explanation

This animated GIF result shows Beyoncé sitting

Slightly Satisfying beside her husband, Jay-Z. This result captures a
(SS) brief interaction between the two of them, but the
resolution is grainy, it's difficult to see Beyoncé's face,
and it doesn't convey any real message or emotion.

View animated GIF

Slightly Satisfying This is an animated GIF result featuring Beyoncé.

(SS) However, Beyoncé's face is poorly lit and somewhat
less recognizable, and the content does not convey
anything particularly meaningful or interesting.

View animated GIF

Fails to Satisfy This result is a static image of Beyoncé. It's a visually

(FailsS) appealing, high-quality image, but it's not animated,
and thus would not satisfy the user's clear intent to
see animated GIF results.

Remember: For GIF-seeking queries like this one,

non-GIF results should automatically receive a rating
This result is NOT animated of Fails to Satisfy, since the animation aspect of the
user intent is not met.

Fails to Satisfy This result is animated, but it focuses on Beyoncé's

(FailsS) husband, Jay-Z, rather than on Beyoncé herself as
requested by the query.

View animated GIF

Proprietary and Confidential – Copyright 2021 38

Example: [nintendo switch], English (US)
User task or journey: Users may be shopping for a Nintendo Switch game console, or they may be interested in
learning more about it such as what it looks like, how it works, popular games, and various options available (e.g.,
sizes, colors, accessories, etc.). For this query, while the user did not directly express a preference for animated
GIFs, GIF results may still be quite helpful if they serve a reasonable user intent in a satisfying way.

Image (Animated GIF) Satisfaction Rating Explanation

Highly Satisfying This result features a Nintendo Switch, and it is a

(HS) beautiful, high-quality, and satisfying animated GIF.
Even though the user is not clearly seeking an
animated GIF result, the animation aspect is not
distracting, and in fact is actually helpful in
demonstrating how the different parts of the Switch
handheld device are put together.

View animated GIF

Moderately This result features a Nintendo Switch, and the

Satisfying (MS) animation demonstrates a key feature of the game
console. The composition and quality are somewhat
lacking due to the low resolution and the use of
grayscale. However, the animation is amusing without
being distracting, and the message it conveys could
be helpful for many users.

View animated GIF

Slightly Satisfying This result contains a Nintendo Switch, but the

(SS) dancing animation is distracting, and the caption
conveys a specific message that would not benefit
most users.

View animated GIF

Fails to Satisfy This animation is high quality and features several

(FailsS) handheld devices made by Nintendo, but none of
them are the Nintendo Switch.

View animated GIF

Proprietary and Confidential – Copyright 2021 39

Example: [funny gifs], English (US)
User task or journey: Most users are looking for animated GIFs that express something funny and could be
shared with others to make them laugh. Humor is subjective, so for this query, you should use your best
judgment to determine whether some or many users in your locale might find each result funny.

Image (Animated GIF) Satisfaction Rating Explanation

Highly Satisfying This result is a high-quality animated GIF that many

(HS) users are likely to find funny. It also has a clever
concept, good execution, and is visually appealing, all
of which contribute to a rating of Highly Satisfying.

View animated GIF

Highly Satisfying This is another animated GIF that is clearly aimed at

(HS) making people laugh. Although the humor in this
result may be considered more nuanced than the
example above, many users would likely find it funny.
Notably, this result also has a clever concept, an
impressive attention to detail, and good execution, all
of which contribute to a rating of Highly Satisfying.

View animated GIF

Moderately This is another animated GIF that many users may

Satisfying (MS) find funny. However, due to the grainy/low resolution
and the imperfect graphics (such as the shaky hat that
doesn't stay aligned with the baby's head), the quality
is lacking, which brings the rating down to Moderately
Satisfying.

View animated GIF

Moderately This is an animated GIF that many users may find

Satisfying (MS) funny, even though the humor may be considered
strange or unconventional. The result has some
shortcomings related to its composition, such as the
distracting ceiling fan in the background that doesn't
add anything to its message, and the shakiness and
poor lighting on the subject's face in the second half of
the animation. This is still a good result, but its less
appealing attributes make Moderately Satisfying the
most appropriate rating.
View animated GIF

Proprietary and Confidential – Copyright 2021 40

Image (Animated GIF) Satisfaction Rating Explanation

Slightly Satisfying This animated GIF depicts Whitney Houston doing a

(SS) fake laugh. It's not exactly humorous itself, but it's
somewhat related to the concept of humor, so some
users may find it satisfying.

View animated GIF

Fails to Satisfy This animated GIF is not really funny, but rather
(FailsS) depicts someone who seems bored and unimpressed.

View animated GIF

Fails to Satisfy Many users would likely find this result funny.
(FailsS) However, it's not animated, so it would not satisfy the
user's clear intent to see animated GIF results.

Remember: For GIF-seeking queries like this one,

non-GIF results should automatically receive a rating
of Fails to Satisfy, since the animation aspect of the
user intent is not met.

This result is NOT animated

Proprietary and Confidential – Copyright 2021 41

7.0 Image SxS Rating
In some rating tasks, you may be asked to give a SxS rating to compare two collections of image search results.
This section describes some special considerations for image search SxS ratings, followed by an example to
illustrate these concepts.

7.1 Image Search Result Set Considerations

Image search rating tasks typically require Image Satisfaction ratings and Landing Page assessments
(Prominence, Helpfulness, and Page Quality). In addition, SxS tasks ask for an overall preference rating to
compare two sets of image search results. Recall that an image search result refers to both the image as well as
the landing page it represents.

How to assign an Overall SxS Preference rating: Please indicate which set of images is most satisfying to
users for the query, considering the images, the landing pages, and the diversity of the result set. Image search
should allow users to visually browse all reasonable intents for a query. As long as the major intents are satisfied,
please value visual diversity and the inclusion of reasonable minor intents in your Overall SxS Preference rating.

Here are some important result set considerations for image search:

Result Set Consideration Description

Helpfulness of results Images and landing pages must be helpful for the query and user intent. Please
think about what users are trying to accomplish. The query may be part of a
quick task or longer-term journey.

Order of the results The most likely interpretations or intents should be addressed by results that
appear in the top positions of the image search result set.

Diversity of results Diversity in the result set is one of the most important aspects of image SxS
rating. A satisfying result set addresses all reasonable user intents.

Beauty, inspiration, and Beauty, inspiration, and visual appeal are very important for topics such as
visual appeal of the images home and garden, crafting, styling, design, products, travel, recipes, food, etc.
A satisfying set of images for these types of topics should be beautiful,
inspirational, and visually appealing.

Helpfulness and Page Landing pages must be helpful for the underlying user task or journey and for
Quality of the landing pages users interested in the images. High quality landing pages enable users to have
more trust in the content of those pages, which contributes to a higher quality
result set.

Porn, U-O, and N-E images Image search results should never unpleasantly surprise users with upsetting,
offensive, or disturbing content. Think carefully about user intent and consider
whether Porn, U-O, and/or N-E content is appropriate. When in doubt, assume
it is not appropriate.

Proprietary and Confidential – Copyright 2021 42

7.2 Image SxS Example

Query: [blueberries], English (US)

User task or journey: Users may be seeking visually appealing images of blueberries, blueberry recipes, or
uses for blueberries.

Note for this example: Please assume all images are prominent on helpful and high quality landing pages.

Proprietary and Confidential – Copyright 2021 43

Overall SxS Preference: Please indicate which side you think most users issuing this query would prefer
overall. Make sure to consider the images, the landing pages, and the diversity of the result set on each side.

Much Better Better Slightly Better About the Same Slightly Better Better Much Better
⬤ ◯ ◯ ◯ ◯ ◯ ◯

Discussion: The images on the left are Moderately to Highly Satisfying. The images on the right are Highly
Satisfying. However, the left side is Much Better because it has both blueberry images as well as blueberry
recipes. The right side is missing a very likely user intent: images of blueberries. Please pay close attention to
reasonable user intents and the diversity of images.

Proprietary and Confidential – Copyright 2021 44