[CAQ] Answer Relevance EN [3 Label] Tutorial - UHRS
[CAQ] Answer Relevance EN [3 Label] Tutorial – UHRS
Instructions
In [CAQ] Answer Relevance EN you will be given a user Query and an Answer taken from a Source Webpage. Your task in this HitApp will be to use the Search Results provided in the tabs above to determine the likely intents of the user's query. Then you will use the Answer and Source Page to rate how relevant the Answer is to the user's intent. Below is the process flow and some details to help you decide which label is most applicable:
Process to Follow
- [Identify Likely User Intent
Using Search Results]
Use the Bing Search Results found in the tabs of the HitApp to determine likely intents of the User’s query. - [Does the Answer have Content
Detrimental to the User?]
Check to see if the Answer or Query has content that would be detrimental to the user. If it does, select and submit a final rating of "Bad". - [Are the Query and Answer in
the Same Language?]
If the Query and Answer are both in different languages, choose a final rating of "In Wrong Language". - [Are the Query and Answer in
either English or the Market Language]
First identify the HitApp Langauge (for example, the FR HitApp's language is "French"). If the Query or Answer are in a language other than the Market Language or English, select a final rating of "In Wrong Language". - [Is the Answer Relevant to the
Query?]
Now that you understand the intent of the Query and the Topic of the Answer from investigating the Souce Page, determine if the Answer is relevant to the Query. If the topic of the Answer is not relevant to a likely intent of the User's Query, rate the final answer as "Bad". - [Is the User Looking for a Very
Specific Query?]
A specific query is a query where the user is looking for a single piece of information (such as "Average US Male Height", "Capitol of California", "Biggest Animal", etc.) If the user asks a specific query, the answer should provide them with the information they need in order to be relevant. If the Answer provides them with the information, rate the final rating as "Good" - otherwise rate the Hit as "Bad". - [Is the Answer the Same Scope
as the Query?]
The scope of a query is determined by how "Broad" or "Narrow" a query topic is. For example "Dogs" would be an extremely broad query as there are many possible intents. "How tall are pit bulls" would be an extremely narrow query as there is only a single possible intent of the query. If the topic of the Answer is slightly different in scope than the intent of the query, the final rating would be "Fair". If there is an extreme difference in scope between the two the final rating would be "Bad".
HitApp Ratings
In Wrong Language
If both the Answer and the Query are in
English, they should always be worked as normal regardless of the Market
Language. For example, if the Market is fr-FR (French Market), Hits in English
should still be worked.
f the query and language are in different languages, the final
rating should be "In Wrong Language". This is true even if the
languages are English and the Market Langauge as seen below.
Query: New to skiing
backcountry
Market: fr-FR (French)
Answer: Planifiez votre première sortie dans l'arrière-pays:
rassemblez un groupe, consultez les prévisions d'avalanche et utilisez des
guides, des sites Web et des cartes pour planifier votre première aventure.
Étape 1: Informez-vous sur les avalanches. Étape 2: Évaluez votre condition
physique et vos compétences. Étape 3: Préparez-vous pour le ski hors-piste et
le snowboard.
The Query or Answer are in a language other that the market
language or English, the final rating should be "In Wrong Language".
In the below example, the Answer is in Chinese so the final rating would be
"In Wrong Language".
Query: New to skiing
backcountry
Market: en-US (English)
Answer: 计划您的首次野外郊游：聚在一起，检查雪崩预报并使用指南，网站和地图来计划您的第一次冒险。第1步：接受雪崩教育。步骤2：评估您的身体素质和技巧。步骤3：为越野滑雪和单板滑雪做好准备。
Good
Auditor Comments:
In order for an answer to be rated as "Good", it needs to address the
most likely intent of the query. In the below example, the answer is very
obviously relevant to the query's intent.
Query: How to become a barber
without going to school
Answer: This is a series of first steps you need
to take in order to become a barber without going to school. Of course after
you gain some skills and a sense for cutting hair as well as as client
interaction, you will need to pursue your education into a barber school that
will provide you with a Certificate of Completion.
In order for an Answer to be seen as "Good" its topic
should generally match the topic size and scope of the topic of the query.
Broad, general queries shoudl ideally have broad, general answers while narrow,
specific queries should have Answers that match that scope.
Fair
When the user's query is broad, it's important
that the answer is also broad. In the below example, the user asks how to write
a cover letter and they recieve information on how to write a cover letter for
a high-security government job. The scope of the answer is narrow compared the
broad query, so the final answer would be "Fair". If the Answer and
Query were reversed, this would also be true.
Query: how to write a cover
letter
Answer: When writing a cover letter for a high-security
government job it's important to read the job description, research
the agency, include specific information...
If a user asks for multiple items in a query and the answer only
references one of the options the final rating would be fair. In the below
example a user asks for multiple all-purpose seasonings but they only recieve
information about one in the answer (checking the source page also shows that
there are multiple different types of seasonings.) This results in a final
rating of "Fair".
Query: mccormick all purpose seasonings
Answer: Similar
to their Himalayan Salt all purpose seasoning, the McCormic Garlic, Onion,
Black Pepper & Sea Salt All Purpose Seasoning is great to add to anything. Designed
to be used as a flavor boost...
Auditor Comments:
In the below example, a user types in a query "Beatles". Even if the
Search Results were completely about the band, there is obviously an animal by
the same name. We would see this sort of intent as a "common sense"
intent. No user would be surprised to recieve information on the animal in this
case. If there were no links about the animal available in the Search Results,
this "common sense" rating would be ranked as "Fair", as
even though it isn't in the search results it would be an acceptable intent for
an Answer.
Query: Beatles
Answer: Beetles
are insects that form the order Coleoptera in the superorder Endopterygota.
Their front pair of wings are hardened into wing-cases, elytra...
Bad
If the Answer is irrelevant to the Query, the
final rating should be "Bad". In the below example, the user is
asking for Movie Maker 10 information but recieves information on Unity - a
game-development program.
Query: Movie Maker 10
Answer: Unity is so much more than the world's best
real-time development platform – it's also a robust ecosystem designed to
enable your success. Join our dynamic...
If the Answer, Query,
or Source page contains detrimental content to the user, the final rating would
be "Bad". Detrimental content consists of any content with excessive
foul language, racism, fake news, overly-adult content, scams, or otherwise
embarrassing content to show to a user.
f the user asks for a specific piece of information, the query must
provide the information that was asked in order to be relevant. A good
indication of a specific query is if the answer can be provided to the user
with a single answer point. In the below example, the user asks for the Height
of Michael Jackson. The answer would be rated as "Bad" as it doesn't
give his height (even though it's information about MJ).
Query: Michael Jackson
Height
Answer: Michael Joseph Jackson (August 29, 1958 – June
25, 2009) was an American singer, songwriter, and dancer. Dubbed the "King
of Pop", he is regarded as one of the most significant cultural figures of
the 20th century.
The Answer can be irrelevant to the user's
query due to a difference in languages. In the case below the User is located
in Seattle Washington in the United States and the answer gives information on
the price of the dogs in India. This would not be useful to the user due to
location differences and should be rated as "Bad".
Query: Average Price of
Poodle Puppy
Location:Seattle,
Washington, USA
Answer: The average price of a poodle puppy in India
is ₹35,000...
Just having keywords from the User's Query isn't enough to make an answer Relevant. The Answer must also be relevant to the intent of the User's query. In the below example the intent of the User's Query is to find out how many controllers come with a PS4.
Query: How many
controllers does a PS4 come with?
Answer: The ps4 otherwise supports 4 controllers, but if you play a
multiplayer game as wwe or fifa, etc it can go up to 16 controllers
Some example You can Prefer.
|
S. No. |
Query |
Judgement |
|
1 |
commbank |
Good |
|
2 |
tiktok dances | Bad |
|
3 |
t4 | Good |
|
4 |
thread and butter |
Good |
|
5 |
uterine fibroid' |
Good |
|
6 |
game pickle |
Good |
|
7 |
lose meaning |
Bad |
|
8 |
netflix best movies |
Fair |
|
9 |
दिशा परमार विकिपीडिया |
In Wrong Language |
|
10 |
how to get a cold |
Good |
Please do like & subscribe.
