Metrics for Evaluating Cyber Security Data Visualizations in Virtual Reality

Cyber security analysts use data visualizations to speed up ingestion of security data. These visualizations typically take the form of 2D graphics displayed on computer monitors. Virtual reality has the potential to improve these visualizations with immersive 3D environments and unique interaction mechanics. However, research into this newly synergised area lacks evaluation, leading to unfounded claims of effectiveness. A potential cause for these missing evaluations was identified as a lack of guidance detailing how evaluations should be conducted in this area. Additionally, the small amount of research that does include evaluation incorrectly relies on subjective participant opinions to objectively measure system effectiveness. An example of this misuse is asking participants which system they thought was quicker, rather than timing them. The objective of this paper was to propose a solution to these issues in the form of a surveyed, categorised, and analysed set of evaluation metrics. A total of 49 metrics were identified from 41 papers. The presented metrics detail which dependent variables should be considered when evaluating works in the combined fields of cyber security, data visualization, and virtual reality. These metrics can be used to produce more accurate evaluations in future works in this area.