|
V13 N2 Paper 1
|
Annals of the MS in Computer Science and Information Systems at
UNC Wilmington
|
Fall 2019
|
An Empirical Study of Factors Impacting Cyber Security Analyst Performance in the Use of Intrusion Detection Systems
William Roden
Committee
Abstract
Cyber security attacks are needles in a haystack. A modest computer network generates over 1,000,000 network events per day, with less than 0.1% of those events involving some sort of malicious action against the network. Human analysts cannot process the sheer volume of information travelling across a network, so organizations use Intrusion Detection Systems (IDS) to alert on abnormal or potentially malicious behavior. However, prior research has shown that it is not uncommon for 99% of IDS alerts to be false alarms. This study seeks to understand to what extent the false alarm rate of IDSes affects human analyst performance. I created Cry Wolf, a simulated IDS web application, to display and capture user responses in triaging IDS alerts. I used Cry Wolf to conduct a controlled experiment wherein 51 participants were divided into two groups, one with a 50% false alarm rate and one with a 96% false alarm rate, and asked to classify whether the alerts were benign, or malicious, in nature. I analyze participants’ performance with regard to sensitivity, specificity, precision, and time-on-task. Results indicate the group with the 50% false alarm rate had ~ 60% higher precision and were 39% faster in time-on-task than the 96% false alarm rate group. The sensitivity of the 96% group approached 100% and the 50% group was also high, around 90%. Specificity appeared to be unaffected by the false alarm rate. Expertise appears to play a role in these performance measures, but more data is required to quantify the differences. These results indicate a tradeoff: IDSes that are overtuned and generate excess alarms may actually improve analyst sensitivity in identifying anomalous activity at the price of miscalssifying some false alarms as true alarms. This reflects the industry standard of placing a high priority on high sensitivity at the expense of low precision and specificity for intrusion detection, regardless of the circumstances for individual networks. I believe there is evidence to suggest from these results, and from personal experience, that analysts become comfortable with high false alarm rates as it reinforces what normal activity looks like and highlights abnormal activity.
download
(pdf)
Recommended Citation:
Roden, W., Layman, L, Cummings, J., Ebrahimi, E. (2019) An Empirical Study of Factors Impacting Cyber Security Analyst Performance in the Use of Intrusion Detection Systems. Annals of the Master of Science in Computer Science and Information Systems at UNC Wilmington, 13(2) paper 1. http://csbapp.uncw.edu/data/mscsis/full.aspx.
|
V13 N2 Paper 1
|
Annals of the MS in Computer Science and Information Systems at
UNC Wilmington
|
Fall 2019
|
