Some Background

Searching for domain-specific information on the web is,

• Kind of Tough
• Requires a dedicated platform

Some Background

We have different search methods over community documents but they are,

• Generally evaluated over efficiency, not user-experience.
• Very different in implementation to be compared.

Context of the Research

The research is in context of the WikiDisability Project to make disability specific documents more accessible to NGOs, stakeholders and people.

The documents involved are either web based blogs or electronic documents, represented in free text (as PDF) and in structured data (in RDF, on a wikibase instance)

We wished to compare different search methods for the best user-experience of the stakeholders involved.

Evaluation Environment

The wikibase dataset and the documents were uploaded for every user.

There were 24 documents and 17 candidates for the experiment.

Two different questionnaires were provided to the user,

• Search Instruction Questionnaire
• User Experience Questionnaire

Search Instruction Questionnaire

• Instructions to search were search-method agnostic.
• A 7-point Likert Scale was provided to record the relevancy of the information retreived.
• 6 search instructions were provided per search method (5 had answers, 1 did not)
• 2 minute was the decided threshold for each search.

User Experience Questionnaire

• Chosen as it provides a benchmark to compare.
• UEQ lets users express feelings, impressions and attitudes towards a method.
• The questionnaire contained 26 items, dividing into 6 scales.
• UEQ finds how the candidates feel about the attractiveness (Pragmatic value) and the ease of use (Hedonic Value).

Results

ESDoc provided the most and relevant answers followed by QAnswer KG and QADoc

ESDoc also provided a false sense of information for instruction with no answers

Results

Results

Results

The scores obtained from UEQ on different scales

The scores obtained in the experiment belong to categry bad for all scales in the UEQ benchmark.

Results

One way ANOVA test

• No statistically significant difference between the methods on scales of Attractive, Dependability, Stimulation.
• Statistically significant differences on the scales of Perspicuity, Efficiency and Novelty.
• We further do Tukey-Kramer Test to find differences between each method to the other.

Results

q_tukey values from Tukey-Kramer Test

critical value is 3.425

We see that there is a significant difference between,

• ESDoc vs QAnswerKG and ESDoc vs QADoc for Perspicuity scale
• ESDoc vs QADoc for both Novelty and Efficiency scale

Results

• QAnswer KG was found to be the most efficient and useful.
• QADoc was found to be the most pleasent to interact with.

Conclusion

We presented a user-experience focused evaluation of search methods on domain-specific documents.

Elastic Search over Documents provided relevant answers but also provided a false sense of relevancy

For non-exploratory question answering with an exact answer, we need more than ESDoc.

QADoc was perceived to be innovative, but didn't perform well for information retreival.

Conclusion

We believe that there is a need to combine various search methods for different types of questions.

We therefore developed a demo to combine the different search methods, and introduced fallback for one to the other.

We employed wikibase to store data around the document, QADoc for the data inside the document.

If there is no confident answer from both, we do an elastic search where the keywords are highlighted.

We further plan to introduce a new set of documents to repeat the experiment with the concluded combined search demo to evaluate the differences.