Please ensure that you are using the latest version of the guidelines from BaseLine, which are

found in the upper right corner of every task.

6 l
Guidelines for Search – Video

24 tia
Table of Contents
Please ensure that you are using the latest version of the guidelines from BaseLine, which are
found in the upper right corner of every task. ......................................................................... 1
Guidelines for Search – Video ......................................................................................... 1

78 n
Search Video Evaluation Guidelines .............................................................................. 3
(1) Introduction .................................................................................................................. 3
e
(1.1) The importance of your work as a Rater ................................................................................................... 3

(1.2) Primary and Secondary Intent..................................................................................................................... 4

(1.3) Query Classification ...................................................................................................................................... 4

63 id
(1.4) Query Research ............................................................................................................................................ 6

(2) Rating Process ............................................................................................................. 9

(2.1) Aspects to Consider ........................................................................................................................................... 9
(2.1.1) Kids Content ............................................................................................................................................... 9
nf

(2.1.2) Franchise Ratings ...................................................................................................................................... 9

(2.1.3) Result filters .............................................................................................................................................. 10

(2.1.4) Sports Ratings .......................................................................................................................................... 11

(2.1.5) Apple TV+ Content................................................................................................................................... 11

Co

(2.1.6) Title Translation ........................................................................................................................................ 11

(2.1.7) Pre-Release Content ............................................................................................................................... 12

(2.1.8) Movie Bundles .......................................................................................................................................... 12

(2.1.9) Classic Content ........................................................................................................................................ 12

(2.1.10) Apple Event / WWDC ............................................................................................................................ 12

(2.1.11) Spell Correction ...................................................................................................................................... 13

(2.1.12) Explicit Content ...................................................................................................................................... 13

(2.1.13) Non-Content Queries ............................................................................................................................ 13

(2.1.14) Reporting Metadata Issues .................................................................................................................. 13

 (2.1.15) Seasonal Results ................................................................................................................................... 14
(2.2) Similarity Rating ............................................................................................................................................... 15
(2.2.1) Movie Navigational .................................................................................................................................. 15

6 l
(2.2.2) TV Navigational ........................................................................................................................................ 16

(2.2.3) Character ................................................................................................................................................... 16

24 tia
(2.3.1) Movie Navigational .................................................................................................................................. 17

(2.3.2) TV Navigational ........................................................................................................................................ 18

(2.3.3) Character ................................................................................................................................................... 19

(2.3.4) Person........................................................................................................................................................ 21

(2.3.5) Ambiguous ................................................................................................................................................ 23

78 n
(2.3.6) Channel/Studio ......................................................................................................................................... 26

(2.3.7) Live Event.................................................................................................................................................. 28

e
(2.3.9) Genre ......................................................................................................................................................... 32

(2.3.11) Language ................................................................................................................................................ 35

(2.3.13) Awards ..................................................................................................................................................... 38

63 id
nf
Co
 Guidelines update 6/28/23 to sections 2.1.1 to not demote kids content from acceptable to off-topic

Guidelines update 2/7/23 to sections 2.1.4 and 2.3.7 to provide additional clarity to
live event guidelines

6 l
Search Video Evaluation Guidelines

24 tia
(1) Introduction
In this document, we explain relevance rating guidelines for video search on Apple TV.
If you are not familiar with the Apple TV app, please refer to https://www.apple.com/apple-tv-app/ for
an overview and basic information about this app.

78 n
(1.1) The importance of your work as a Rater
e
Each of the judgements you complete will be used to build and improve artificial intelligence systems
such as search algorithms and machine learned rankers that power the user experience for Apple
TV users. Your attention to detail, research and language skills as well as your cultural knowledge of
63 id
the market are all critical to the success of our projects.

Your judgements should represent those of an Apple TV user who is using the Search feature. Ask
yourself if you would be content with the results returned for a particular search query. Is there a
significant relationship between the query and content returned? Would you be content if you see
this content appear as a search result? Stay curious and complete thorough research.
nf

Our ultimate goal is to surprise and delight our customers by improving search quality and
enhancing customer satisfaction, and you play an important role in this.

Please keep in mind that your tasks will be spot-checked for quality, and measured against those of
your peers.
Co
 Mandatory Comment
Each rating must be explained in the comment box [even if “optional” is indicated
- rating comments are always mandatory.
The comment should be concise and must only explain why the rating chosen is the correct one in
application of the guidelines. See example below:

6 l
24 tia
78 n
Comment Example

(1.2) Primary and Secondary Intent

The primary intent of a query is the most likely intent, i.e., the intent of most users who enter that
e
query in Apple TV search. Some queries may have multiple primary intents.

A secondary intent is less likely, or would be a less popular intent compared to a primary one. A
secondary intent could be:
63 id
• Content relevant to a smaller group of users than for the primary intent. For queries like
[shows] and [movies] the primary intent is usually media content for grown-ups. Content
for children would be considered secondary intent, except if the intent is obviously kids-
related (such as cartoons, animated films, etc.).
• Complimentary content such as trailers, reviews, cast members, or interviews with the cast
on how the movie was made.
nf

• Lower quality/lesser known content that is relevant to the query but is not the primary
intent, a content that is dated or less popular.

(1.3) Query Classification

Each query is assigned a ‘Query type’.
Co

On the left hand side in BaseLine, inside the input metadata section you will find the classification of
the query.
The classification is already set (you do not have to classify the query yourself).

You should rate the content relevancy based on (1)the classification of the query and (2)how
relevant is the content in relation to the query and the query type.

The ‘Query type’ aims to help you determine the primary intent of the query. Please also consider
potential secondary intents when doing your ratings.
Should you disagree with the ‘Query type’ classification, please explain in the comment section why
it should be different. You can then apply the appropriate rating scale to complete your judgement.
Example Comment: “I believe that James Bond shall be classified as character rather than Movie
navigational. I used this query type for the rating above.”

Examples:
 • Query: "Bi" → Classification: Ambiguous - Unclear intent (the intent is not clear).
• Query: “big little” → Classification: TV Navigational.
The intent is clear, The user is searching for a specific TV show 'big little lies'.
• Query: “Come” → Classification: Genre
The most likely primary intent is for comedy content. However, a movie or tv show that

6 l
starts with this token can also achieve good ratings. (e.g. Come True (2020))

24 tia
78 n
Metadata Example
Note that we have updated our classification guidelines, however, you may still see tasks that are
classified using older classifications.
e
Reference the table below to see what “new” classification you should use when rating these tasks.
63 id
nf
Co

Old Classifications to New

 (1.4) Query Research
To complete query research, tap into your local market knowledge in addition to online
sources such as IMDB, YouTube, Wikipedia, video streaming services, local content evaluators and
social media. Consider how a user of Apple TV engages the search feature to navigate to a
specific set of content, or as a means to browse a larger catalogue of titles.

6 l
Online Research

24 tia
• IMDB
o Popularity Ranking
o Rating Counts
o Storyline, Taglines
o Genre
o Release Date
o For topic, decade, etc: (IMDB Sort By feature)
• Box Office Mojo

78 ne o Local Box Office Rankings

• Common Sense Media

• Wikipedia
o Age Rating → for determination of kids content

o Information of the movie/show: plot, cast, awards, nominations, crew

• Social Media
o Trending films
o Followers on Twitter, Instagram, TikTok, etc. can help to determine a person’s
63 id
popularity
• YouTube trailer views
o Make sure to factor in time, consider the monthly average views since its upload
• Local content evaluators
o To identify popularity in the market

Market Knowledge
nf

Use the knowledge of your market to identify shows which are trending or historically popular. Be
aware of the shows offered on Apple TV and other streaming platforms in your country. Leverage
your understanding of the language to identify misspellings, translations, or aliases related to the
query. When completing online research, domestic sources should typically be prioritized over global
sources. For example, take into account the significant bias of IMDB for Hollywood productions and
English-language content.
Co

Set of Relevant Content

Query research includes making a determination about the entire set of content which could be
applicable to the query. For example, the query “the” can be related to a large set of popular content
such as “The Morning Show”, “The Avengers”, and “The Batman”. On the other hand, the query
“simps” has much small set of content which can be relevant. Your ratings should take this into
account: for a query that is associated with a large amount of popular/relevant content, you can be
more “harsh” with your ratings, compared to a query where the set of relevant content is smaller.
 Types of Rating
For each task, determine which type of rating should be used. To make the choice, look at the query
and the output, and determine how this output is related.
1. Overall Relevance: There is a relationship between the query and the returned content,
typically the text of the title, character, names of Cast & Crew, alias/nickname, genre or

6 l
topic of the content. Whenever there is a query relevance, please use overall relevance
rating.

24 tia
1. Example:
1. [modern] → “modern family”
2. [danie] → “No time to Die” (Daniel Craig stars as James Bond)
3. [spencer] → “Talentos Ocultos” (Character in the returned movie)
2. Similarity: There is no relevance between the query and the output, the intent of the
query is clear and the query classified as TV Navigational, Movie Navigational or
Character. In this case, the relationship between the intended content and returned
content shall be determined in these aspects: target audience, factual aspects and/or
theme. The highest rating to be used is good.

78 n
Query Relevance
e 1. Example: ‘modern family’ → ‘the goldbergs’

Relevance describes the relationship between the query and returned content. This does not only
include the text match with the title.

Examples:
• Relevant:
63 id
o [John] → “The Problem with Jon Steward” (*misspelled)
o [Back] → “Back to the Future”
o [th] → “The Big Conn” / “Thor: Ragnarok”
o [part] → “Anthony Bourdain: Parts Unknown”
o [acti] → “Top Gun: Maverick”
o [disn] → “Frozen II”
o [Show] → “The Morning Show”
nf

o [george c] → “Ocean’s Eleven”

o [Magic] → “They Call Me Magic”
• Somewhat relevant
o [Back] → “The Boss Baby: Back in Business”
o [Back] → “Fraggle Rock: Back to the Rock”
o [th] → “Winning Time: The Rise of the Lakers Dynasty”
Co

o [part] → “Toy Story Toons: Partysaurus Rex”

o [th] → “Avengers: Infinity War” (Commonly known as “The Avengers”)
• Not relevant:
o [back] → “Strike Back: Origins“ (It is highly unlikely that a user would use this
query to search for the returned content)
o [part] → “The Twilight Saga: Breaking Dawn - Part 2“ (‘insignificant’ part of the
title)
o [and] → “Hobbs and Shaw”
o [modern family] → “The Italian Job”
 Misspelling/Alternate spelling
Some words can be spelled in multiple ways or frequently misspelled. Since search is able to handle
misspellings, please evaluate the intent rather than the exact text.
• [gray] - Should also be evaluated as [grey]
• [John] → Should also be evaluated as [Jon]. E.g. The Problem with Jon Stewart

6 l
• [rhe] → consider that the user might have meant to write “the”
• [4], [IV], [Four] → consider synonyms

24 tia
• In case you are rating on an alternative spelling, please do not use similarity
o Why? Too many steps from query → result to trace in auditing

78 ne
63 id
nf
Co
 (2) Rating Process

6 l
(2.1) Aspects to Consider

24 tia
(2.1.1) Kids Content
As kids content satisfies the requests for a subset of users, it shall not get the same ratings for
generic queries as content for all age groups. Therefore, kids content shall be demoted by one
rating level unless the query is regarding kids content either directly (navigational, character) or
indirectly (“kids” token in query, animation). Please make use of the “Kids&Family” genre, Common
Sense Media and your own knowledge to determine whether content is targeting kids specifically. Do
not demote relevant content from Acceptable -> Off-Topic, as relevant content should always be

78 n
rated as at least acceptable.
This only applies to queries that are classified as “topic”, “genre”, “functional” and “year/decade”.

Examples:
e
• [Adventure] → “Yakari”: The returned show is in the intended genre and very popular,
however dated and hence could be rated as good. However, since it is kids content, it will
be rated acceptable.
63 id
(2.1.2) Franchise Ratings

General Guideline:
• Perfect → The first and last item of core franchise.
Please make an honest determination of the order in which users would typically watch
nf

the collection or franchise of content and use this to complete the rating. Use comment to
justify decision.
• Excellent → Other content in core franchise
• content in the franchise, but not core
o excellent → popular and recent
o good → popular or recent
Co

o acceptable → neither popular nor recent

 6 l
24 tia
78 ne
Franchise Rating Overview.jpg 116 KB
63 id
nf
Co

Franchise Rating.jpg 398 KB

(2.1.3) Result filters

You may come across queries that include the tokens [“new”, “best”, “movies”, “shows”]. Additionally
to the core intent, these function like filters. Please find examples in the sections below.
 When this is a part of the query, please consider the implications for your rating:
• [new] → emphasize on recency aspect, the popularity aspect is less important
• [best] → emphasize on the popularity aspect, the recency aspect is less important
• [movies] → consider only content in that content type (here: movie) as relevant. Shows in

6 l
this case can be rated as off-topic

24 tia
(2.1.4) Sports Ratings
You may come across sporting events in your rating tasks that are not classified as '“Live Event”.
Please determine if the query is directly relevant to the sporting event (clear Primary Intent) or if the
intent of the query can reasonably be multiple things, including non-sports events (Partial Intent or
Secondary Intent).

The following guidance can help determine the proper rating, however, please rely primarily on your

78 n
own market knowledge and query relevance to make your judgement.

Clear Primary Intent

see rating guidelines for live events
e
Partial Intent or Secondary Intent
Note that all queries that have live events as a potential secondary intent, shall be rated based on
the Live Event guidelines. The perfect rating category is not available in this case.
63 id
Consider what rating you would have given the result if the query was completed (i.e. query is “ho”
and the result is an NHL hockey game, what rating would the query have received for “hockey”?)
and give that rating to the query, with the exception of the “perfect” rating which should be changed
to “excellent”.
• [foot] → “football”
• [murr] → “Andy Murray”
nf

• [n] → “nba” / “nfl” / “ncaa” / “notre dame” / “brooklyn nets”

(2.1.5) Apple TV+ Content

Users are using search on Apple TV which indicates a preference for ATV+ content such as Ted
Lasso, Coda, Severance or Tehran. Therefore
Co

• Popularity and relevancy is slightly less strict for ATV+ content

• If you are in doubt about “Good” or “Excellent” for an ATV+ result, select “Excellent”

(2.1.6) Title Translation

Content may be rated based on relevancy to the English title or the language title of the locale.
However, consider how likely a user is to use a different language for the given query.
A higher rating may apply if the content is widely promoted or advertised using the the English title
(or vice versa for the local title).
 (2.1.7) Pre-Release Content
Sometimes the result is the preview page for content which is not yet available for viewing. The
timing of the release can be ignored. Use the guidelines for the query type, however ratings are
limited to "Excellent" at best. Please leave the date of your rating as part of your comment.
• Complete research to determine expected popularity (ex. YouTube trailers, production

6 l
budget, famous cast and crew)
• Original content from ATV+, Netflix, Hulu, or other popular production company - generally

24 tia
assumed to be popular.
Example:
• [christmas] → “Spirited”: Rated Excellet, this movie is an Apple TV+ original ant it stars Will
Ferrell and Ryan Reynolds and can therefore be assumed to become popular. Rating
completed on 11/9/22.

(2.1.8) Movie Bundles

78 n
Movie bundles shall be rated for the content included that is related to the query. Once the rating for
the content is determined, demote rating by one level
e• If the movie bundle contains the primary intended content it can receive an Excellent rating
o Example: No time to Die → James Bond 10 Film collection → Excellent
• If the movie bundle contains a piece of content that could potential be a primary intent of the
query, determine rating for individual movie first and demote by one level
o Example: Mav → Top Gun 2 Movie Collection → Good
63 id
• If the movie bundle contains a piece of content that individually would be rated as
“Acceptable”, the bundle receives the “Unacceptable: Off-Topic” rating

(2.1.9) Classic Content

Ignore recency for classic movies and tv shows that were among the most popular in the decade
nf

they were released or have high popularity even today.

• Christmas → “Home Alone” is relevant and very popular, because it is a classic, it can be
rated “Excellent”
• Drama → “The Godfather” is in the intended genre and very popular, because it is a classic,
it can be rated “Excellent”
Co

(2.1.10) Apple Event / WWDC

If this is the primary intent (query = “Apple Event”, “wwdc”, “developer conference”), rate based
on recency. Rating score decreases per year until “Acceptable”. Year is considered the last 12
months, so the past year on October 31, 2022 would be any events from November 2021 through
the current date.
• Query = “Apple Event”, “wwdc”, “apple developer conference”
• Perfect = Events within a year from the current date of rating
(ex. Apple Event 09.07.22, WWDC 2022)
• Excellent = Events older than a year, less than 2 years gap from the current date of rating
(ex. Apple Event 06.06.21, WWDC 2021)
• Good = Events older than 2 years, less than 3 years gap from the current date of rating
(ex. Apple Event 03.08.20)
• Acceptable = Apple Event 09.10.19, WWDC 2019
If this is the secondary intent, decrease the ratings above by 1.
 • Query = “Apple”, “iPhone”, “new technology”, “web conference”, “ww”
If the Apple Event / WWDC content is relevant to the query but unlikely to be the intent, rate as
“Acceptable”.
• Query = “2022”, “w”, “September 2021”
*date of publishing these examples: 10/28/22

6 l
24 tia
(2.1.11) Spell Correction
Some queries are misspelled. Determine intent and complete the rating based on the correct
spelling of the query.
• [grays anatomie] → [grey’s anatomy]

78 n
(2.1.12) Explicit Content
Rate all pornographic content as Unacceptable: Off-Topic. Exception: If the query intent is
specifically for erotic content, or porn → Please use above guidelines to find appropriate rating.
e • [drama] → any pornographic content is Unacceptable: Off-Topic
• [erotica] / [porn] / [navigational] → erotic content can receive any rating
• [18+] / [movies for adults] → Intent here is not specifically for pornographic or erotic content
63 id
(2.1.13) Non-Content Queries
You may come across queries such as [cancel subscription], [settings], or [log out] which are not
relevant to any video content. For such queries, please use Problem: Other. Please use with
caution and explain your judgement in a comment.
nf

(2.1.14) Reporting Metadata Issues

You may come across a preview page which has incorrect or missing data such as wrong Cast and
Crew, wrong Release Date, or the wrong artwork. For such issues, please use the rating Problem:
Other and describe the issue in the comment section. For release date issues, please only flag them
Co

when the indicated release date is 2 or more years apart from the researched correct release date of
the content.
 6 l
24 tia
78 ne
Metadata Reporting
63 id
(2.1.15) Seasonal Results
In some locales there are seasonal results which have additional relevance and popularity at specific
times of year, for example holiday movies are very popular in the US at the end of the year. Please
consider the seasonal popularity when rating results and consider increasing the rating due to
seasonal popularity.
nf

Examples (updated on 12/19/22):

• [s] → “Spirited”: Always “excellent”, new and very popular result, excellent is the best result
for an “Ambiguous - Intent Unclear” query
• [c] → “A Charlie Brown Christmas”: If rating the result in July this would be a “good” result as
it is relevant and popular, but not quite at the level of a classic that would receive an
Co

excellent rating. However, if this result surfaces in the November / December timeframe it
should be rated as excellent as it is a holiday classic and an excellent result at that time
• [bo] → “A Boyfriend for Christmas”: If rating in the summer this would be an “acceptable”
result, it is relevant but not recent or popular. During the Holiday season it would be rated
as "good", being a Christmas Romantic Comedy, it has additional popularity during that
period
 (2.2) Similarity Rating
Evaluate Similarity between the intended content and the returned content based on the following
aspects:
• Target Audience

6 l
o Genre
o Age Rating (Kids (PG) vs Everyone (PG-13, TV-14) vs Adults (R))

24 tia
• Factual Aspects
o Cast & Crew: Actors, Producers, Studio, etc.
o Setting: Location AND Time Period in which the content plays
• Theme
o What is the content about?

(2.2.1) Movie Navigational

78 n
General Guideline:
• Good → Similarity in Target Audience, Factual Aspect and Theme
• Acceptable → Similarity in 2 of the three categories (Target Audience, Factual Aspect and
e Theme)
• Off-Topic → Similarity in less than two aspects
63 id
nf
Co

Movie Similarity Rating Overview.jpg 52.5 KB

 6 l
24 tia
Movie Similarity Rating.jpg 434 KB

78 n
(2.2.2) TV Navigational
e
63 id
nf

TV Similarity Rating.jpg 378 KB

Co

(2.2.3) Character

character similarity rating.jpg 243 KB

 (2.3) Overall Relevance Rating
(2.3.1) Movie Navigational

6 l
General Guideline:

24 tia
• Perfect → If navigational content and result is certainly primary intent of user, rate it perfect,
regardless of popularity and recency. For Franchise related queries, please follow the
Franchise specific guidelines as outlined above.
• Excellent → If the returned content is a sequel or prequel of the intended content, or the
content is part of a movie bundle with the intended content, rate the result as excellent.
• Good → If the returned content is relevant to the query, and either recent or popular, or can
be considered a secondary intent, rate it as good. One example would be a title matching
show for a movie query.

78 n• Acceptable → If the returned content is relevant to the query, but neither recent nor popular

e or can only poorly satisfy a secondary intent, it can be rated as acceptable.

• Off-Topic → If the content is not relevant to the query or it is very unlikely that a user would
search for the returned content with the given query, it can be rated off-topic.
• In cases where the main character name and the title of the content match, character shall
be the dominating classification.
63 id
nf
Co

Movie Navigational
Rating Overview.jpg 82.7 KB
 6 l
24 tia
78 n
Movie Navigational Rating.jpg 460 KB
e
(2.3.2) TV Navigational
63 id
nf
Co

TV Navigational Rating.jpg 321 KB

 (2.3.3) Character
Intent for “Character” queries can be both Movies and TV Shows. Often, the “Franchise Ratings”
(Appendix) will apply to ratings for characters. In cases where the main character name and the
title of the content match, character shall be the dominating classification.

6 l
General Guideline:
• Perfect:

24 tia
o The most popular and recent content featuring this character in a major role.
(Apply “Franchise Query” Rule).
o If only one content with the character is produced, this content can be rated
perfect
• Excellent:
o Sequels/prequels for the show in which the character is best known
o Other high-quality content featuring the character
o Person page for well known actor/actress who plays the character

78 n• Good:

e o Content which is in the same franchise but about a different character

o If the character has their own spinoff, the ‘parent’ show where the character first

• Acceptable:
appeared

o Show/movie features the character in insignificant role

63 id
nf
Co

Character
Rating Overview.jpg 123 KB
 6 l
24 tia
78 n
Character Rating.jpg 457 KB
e
63 id
nf
Co
 (2.3.4) Person
Not currently a classifier type. Queries with this intent are currently classified as “Actor/Actress”,
“Producer”, “Director”, “Writer” or “Topic”. Please rate tasks that are classified as one of the latter
according to the person guidelines.

6 l
General Guideline:
• Excellent:

24 tia
o Content with the intended person as lead in cast & crew
o Popular and recent content with intended person in cast & crew
o Most popular documentary about the person (more than 1 possible if equal in
popularity/quality)
o Recent and popular live event with person
o Content where person is a significant guest star. Hosted by reputable content
creator.
o Set of most popular content inspired by the person

78 n• Good:

e o Person page

o Documentary about the person that is popular, but not most popular or recent
o Popular live event with person
o Popular content inspired by the person
o Content with the intended person as cast & crew (not as lead) that is popular or

• Acceptable:
recent
63 id
o Unpopular content about/with the person
• Content with the intended person as cast & crew (not as lead) that is neither popular nor
recent

Note, Directors and Writers can all be considered leads.

nf
Co

Person Rating Overview.jpg 57.2 KB

 6 l
24 tia
78 ne
63 id
Person Rating.jpg 729 KB
nf
Co
 (2.3.5) Ambiguous
Ambiguous - Multiple Classifications

These queries have intents that fit in multiple classifications. Please use the rating scale which
matches to the output content.

6 l
24 tia
78 ne
63 id
Ambiguous Multiple Classifications Rating.jpg 432 KB
nf
Co
 Ambiguous - Intent Unclear

If the intent is unclear or the query is relevant to a very large set of content, the maximum rating for
the query is Excellent and only relevance ratings can be applied (no similarity ratings).

6 l
General Guideline:
• Excellent → The returned content is relevant to the query (not only by title), popular and

24 tia
recent
• Good → The returned content is relevant to the query, and either popular or recent.
• Acceptable → The returned content is relevant or somewhat relevant to the query and
neither popular nor recent.
• Off-Topic → Regardless of popularity and recency, if it is unlikely that the user would use the
query to search for the returned content or there is no relationship between the query and
the content.
• Special case: If the intent is unclear and the query incomplete, ratings cannot be higher than
good when person pages are returned, if the person has recent content and is popular.

78 ne
63 id
nf

Ambiguous Intent Unclear Rating Overview.jpg 41 KB

Co
 6 l
24 tia
78 ne
63 id
Ambiguous_intent unclear.jpg 837 KB
nf
Co
 (2.3.6) Channel/Studio

General Guideline:
• Excellent → popular and recent content from either the intended channel or studio
• Good → popular or recent content from the intended channel or studio

6 l
• Acceptable → neither popular nor recent content that is available on the intended channel or
produced by the intended studio

24 tia
• Off-Topic → content that is not related to the intended channel or studio

78 ne
63 id
nf
Co

Channel-Studio
 Rating Overview.jpg 120 KB

6 l
24 tia
78 ne
63 id
Channel-Studio Rating.jpg 561 KB
nf
Co
 (2.3.7) Live Event
Note that the examples below may have already taken place in the past. Please assume that live
assets are either ongoing or upcoming.

When considering a sporting events popularity please consider multiple aspects that may affect

6 l
popularity, here are some examples of factors:
• Event: Some sports are very popular in the Olympics and not so much otherwise

24 tia
• League: An British user is likely more interested in the Premier League or Champions
League than Ligue 1 in France
• Level: The highest level professional sports are often more popular than lower leagues or
college sports
• Competition: Preseason is generally less popular than regular season events and playoffs /
championship events are often the most popular
• Sport: Some queries may point to multiple sports ("NCAA", "Sports", "Olympics"), what is the
relative popularity of events / sports that fit the query

78 n
Note that this popularity is scaled based on the possible intents left, if the query points directly to a
niche sport or event then we should not demote results because the specific sport/event is not
popular overall, however, in broader live event queries and queries where live events are a potential
e
secondary intent lower popularity live events should receive correspondingly lower ratings.
63 id
nf
Co

live event.jpg 171 KB

General Guideline:
• Perfect → live event when query is for a team or single day event
• Excellent → popular live events when query is for a sport, league or multi-day event. Also
video on demand assets for recent championship events.
• Good → less popular live events related to the query that satisfy a secondary intent. Also
movies and tv shows related to the query that is popular and recent
• Acceptable → video on demand content for relevant non-championship live events. Also
movies and tv shows related to the query that are not popular and recent
 6 l
24 tia
78 ne
63 id
nf

live event rating

overview.jpg 208 KB
Co

Live Event Rating.jpg 514 KB

 (2.3.8) Topic
Intent for Topic Queries can be very broad (“World War”) or specific (“Pearl Jam”). Adjust your rating
scale based on the amount of content available. For ‘broad’ queries, ratings “Excellent” can apply to
many results but they must be very high quality. For ‘specific’ queries, only a small set can receive

6 l
an “Excellent” rating.
“Perfect” rating is not available for this query type.

24 tia
Please consider the kids content guidelines as explained in the “Aspects to Consider” section above.

General Guideline:
• Excellent → The topic is prominent and important for the plot of the returned content. It is
also popular and recently released
• Good → In one scenario the topic is important for the plot of the returned content and the
content is either popular or recent. Another scenario is if the content is somewhat related
to the topic, but popular and recent

78 n• Acceptable → The content is related to the topic, but neither popular nor recent. Also content

e that is somewhat relevant to the content and recent can be considered acceptable.
63 id
nf
Co

Topic Rating Overview.jpg 77.2 KB

 6 l
24 tia
78 ne
Topic Rating.jpg 533 KB
63 id
nf
Co
 (2.3.9) Genre
The lines between queries with an intent for a genre, topic or country of origin can be blurred at
times. Please also consider the other possibilities when evaluating the returned content. Please
keep in mind the special rules that apply for “Classic Content” as explained above.
Please consider the kids content guidelines as explained in the “Aspects to Consider” section above.

6 l
General Guideline:

24 tia
• Excellent → the returned content is in the intended genre, popular and recent
• Good → the returned content is in the intended genre and either popular or recent
• Acceptable → the returned content is in the intended genre, but neither popular nor recent
• Off-Topic → the returned content is not in the intended genre and not matching the query in
any other way
• if the content type is defined, this must be matched for the returned content to be eligible for
a rating of acceptable or better. Content types are movies and shows.

78 ne
63 id
nf

Genre Rating Overview.jpg 81.8 KB

Co

Genre Rating.jpg 307 KB

 (2.3.10) Functional
The rating of results to queries in this classification depends on the query.
Please consider the kids content guidelines as explained in the “Aspects to Consider” section above.

6 l
General Guideline:

24 tia
• Excellent → content satisfies all of the functional requirements of the intent and is popular
and recent. Content type must match if specified (movie/tv show)
• Good → content satisfies all of the functional requirements of the intent and is popular or
recent. Content type must match if specified (movie/tv show)
• Acceptable → content satisfies all of the functional requirements of the intent but is neither
popular nor recent
• Off-Topic → content does not match content type, or neither of the other requirements are
satisfied
• Queries that include the string “free”

78 ne o Excellent → all AppleTV+ originals

o Good → all Apple Live Sports, non AppleTV+ content that is popular and recent
o Acceptable → non AppleTV+ content that is popular or recent
o Off-Topic → non Apple TV+ content that is neither popular, nor recent
• Queries that include the token “new”
o Content that was recently released can be considered new. Content older than
1 year cannot be considered new. Use your judgement to see how new the
content is or if newer popular content would be available. Popularity still plays a
63 id
role here.
• Content which is keyword-matching to words like “new”, “random”, “free” etc. are not eligible
for excellent ratings unless you believe it is actually the user’s intent. Ratings can reach
from Good to Acceptable.
nf
Co

Functional Rating Overview.jpg 78.8

KB
 6 l
24 tia
78 ne
Functional Rating.jpg 522 KB
63 id
nf
Co
 (2.3.11) Language
General Guideline:
• Excellent → Movies and tv shows that is originally produced in the intended language and is
recent and popular
• Good → Movies and tv shows that is originally produced in the intended language and is

6 l
recent or popular
• Acceptable → Movies and tv shows that is originally produced in the intended language and

24 tia
is neither recent nor popular. Also content that is not originally produced in the intended
language, but is available with audio in the intended language and is recent and popular

78 ne
63 id
nf
Co

Language Rating
Overview.jpg 105 KB

Language Rating.jpg 267 KB

 (2.3.12) Year/Decade
Intent for queries which are ‘time’ is typically (A) Content with a setting/theme from the year/decade
or (B) Created or Popular in the year/decade.
Please consider the kids content guidelines as explained in the “Aspects to Consider” section above.

6 l
Ratings for (A) Content with Setting or Theme related to the Year/Decade

24 tia
Please defer to guidelines for “Topic” queries.

Ratings for (B) Created or Popular in Year/Decade

Release date or TV season must be relevant to year/decade.
• Excellent:
o Content which won major award(s) in the given year/decade
o Ultra Popular (Approximately):
▪ Top 50 most viewed shows/films associated with decade
▪ Top 10 most viewed films associated with year

78 ne• Good:
o Ultra popular show has >=50% of seasons/episodes in decade

o Popular shows/movie associated with year/decade

o Show has 3+ seasons (or >50%) in decade
o Popular (Approximately):
▪ Top 100 most viewed in decade
o Top 30 most viewed films from year
• Acceptable:
63 id
o Content released in year/decade
o Show has 1+ season in decade
• Off-Topic:
o Content not from year/decade
nf
Co

Year-Decade Rating
Overview.jpg 66.5 KB
 6 l
24 tia
78 ne
Year-Decade Rating.jpg 444 KB
63 id
nf
Co
 (2.3.13) Awards
Users searching for an award show are generally interested in (1) watching nominated
movies/shows before the award event, or (2) watching movies/shows which won the most recent
award event. Recent winners should receive higher ratings, unless the query specifies a specific
edition of the award event.

6 l
Definitions:

24 tia
• If award event upcoming: Nominations have already been announced for the next upcoming
award event
• If no award event upcoming: Nominations for next upcoming award event are not announced
yet.

General Guideline:
• Perfect:

78 ne o Most recent award event show

o Currently ongoing or upcoming live award event show
• Excellent:
o If award event upcoming: nominees (movies, tv shows or actors) for the
upcoming award event. The returned content itself needs to be nominated
▪ if movie returned, movie needs to be nominated
▪ if artist page is returned, artist needs to be nominated
o If no award event upcoming: winners of the most recent award event
63 id
• Good:
o If award event upcoming: winners of the most recent award event
o If no award event upcoming: winners of previous award events, nominees of
most recent award event
o Similar award event shows
• Acceptable:
o Movies/Shows featuring people associated with award show
nf

o Nominees of previous award events

• Exceptions:
o If a year is specified in the query, consider only results relevant to the year
▪ Perfect → content that won an award that year
▪ Excellent → content that was nominated for that year
▪ Off-Topic → content has no association with the award in the given
Co

year
 6 l
24 tia
78 ne
63 id
nf
Co

Awards Rating Overview.jpg 129 KB

 6 l
24 tia
Awards Rating.jpg 281 KB

78 ne
63 id
nf
Co