Annotation guideline

The quality of the annotation is strictly controlled. New assignments will not be issued to dishonest or inaccurate users.

The task is to determine the relationship between two news titles.

Two titles are called the same if they are about the same things or differ in minor details.

The first title causes the second title if the second title is impossible without the first one. If the first event didn't happen, then the second event should not be happening too. For example, this relationship includes opinions, details, comments, true causes. This relationship should be evident from the titles only.

The second title refutes the first one if the second title makes the first one irrelevant. For example, this relationship includes refutations, updates in the death toll, updates in the match score. This relationship should be evident from the titles only.

Other possible relationships include the same characters, titles with the common cause, titles with unknown order. You should choose this relationship in case if you are in doubt.

You should use the "No relationships" option if two titles are not connected or connected with a vast theme. For example, if coronavirus is the only common thing in two titles, you should use the "No relationships" option.

Overall, there are seven possible answers:
  1. Same titles
  2. B refutes A
  3. A refutes B
  4. A causes B
  5. B causes A
  6. Other relationship
  7. No relationship

Examples

#1
A: Judge issues order to allow indoor worship in NC churches
B: Some local churches resume indoor services after judge lifted NC governor’s restriction
Result: A causes B

#2
A: UGC Guidelines for Colleges, Universities for Academic Session and exams to be issued next week
B: UGC guidelines for universities and colleges to be issued today or tomorrow, NDTV
Result: B refutes A

#3
A: Police launch criminal negligence probe into death of Jodian Fearon
B: Post-mortem for Jodian Fearon pushed back
Result: Other relationship
Comment: The order of events is unknown

#4
A: Google Play Store bug prompting repeat Android app updates
B: A new Play Store bug is causing repeat app updates for the same versions
Result: Same titles

#5
A: US records 1,330 coronavirus deaths in 24 hours: Johns Hopkins
B: Gazan flower growers watch trade wither amid COVID-19 crisis
Result: No relationship
Comment: Coronavirus theme is too big to be considered as a relationship

#6
A: Gov. Kevin Stitt defends $2 million purchase of malaria drug touted by Trump
B: Oklahoma spent $2 million on malaria drug touted by Trump
Result: B causes A

#7
A: Man charged with posting fake 'intel' on COVID-19 measures on Facebook
B: Man to be charged for posting false message about COVID-19 circuit breaker measures
Result: A refutes B
Comment: "charged" refutes "to be charged"