Qualitative evaluation of thesaurus-based retrieval

This paper reports on a formative evaluation of a prototype thesaurus-based retrieval system, which involved qualitative investigation of user search behaviour. The work is part of the ongoing ‘FACET’ project in collaboration with the National Museum of Science and Industry and its collections database. The main thesaurus employed in the project is the Getty Art and Architecture Thesaurus. The aim of the evaluation is to analyse at a micro level the user’s interaction with interface elements in order to illuminate problems and inform interface design decisions. Data gathered included transcripts of think-aloud sessions, screen capture movie files, user action logs and observator notes. Key incidents from the sessions are analysed and the qualitative methodology is discussed. The evaluation analysis informs design issues concerning the allocation of search functionality to sub-windows, the appropriate role of thesaurus browsing in the search process, the formation of faceted queries and query reformulation. The analysis suggests that, although the prototype interface supports basic level operations, it does not provide non-expert searchers with sufficient guidance on query structure and when to use the thesaurus. Conclusions are drawn that future work should further support and suggest models of the search process to the user.