Rationale: Answers of a search system should be easy to understand in order to avoid confusing the user. This includes avoiding unnecessary technical terms, complex sentence structures, and many co-references, among others. But it also requires that the response does make sense in the context of the conversation.
Unit of annotation: turn
Multi-label: no
Required attributes: utterance, response
Guidelines: Read the response once. Repeat its meaning in your mind. Read the response a second time and check whether your mental repetition aligns with it. If not, try once more to repeat it in your mind and check. Are you now certain you understand the response?
- Very easy (1): I am sure I fully understand the response after reading it once and without noticeable thinking.
- Easy (2): I am sure I fully understand the response after reading it once but I had to think for a moment after reading it.
- Difficult (3): I am sure I fully understand the response after reading it twice.
- Very Difficult (4): I am still unsure whether I fully understood the response.