Zachary Vance and Bonnie Anderson, Information Systems

The comprehension of security messages can be a difficult task to decipher, as comprehension works through a series of stages. The four major steps in comprehension are attention, semantics, syntax, and pragmatics, as outlined by figure 1 below. Through the use of eye-tracking, we were able to focus on parts of security messages that inhibited understanding and make changes to ensure users were following through each step of comprehension and therefore making wise security decisions. Eye-tracking proved to be an excellent tool for comprehension as the data gives immediate measures of attention, with difficult words or phrases being stared at or reread multiple times compared to simple sentences.

For our experiment, stimuli were presented on a BenQ monitor with a refresh rate of 60Hz and a resolution of 1600 × 900 pixels. An Eyelink 1000 Plus eye tracker was used to collect eye movement data sampling from the right eye at a rate of 1000 Hz. The host computer connected to the eye tracker ran Eyelink 5.00 tracking software. The experiment was programmed using Experiment Builder software. Participant responses were collected on a computer keyboard.

An experiment was created where multiple security messages were created with different wordings of the same message. For example, changing the syntax of a sentence can place emphasis on the possible security breach, or on the possible consequences of the security breach. We hypothesized that changing the focus of a warning would change comprehension, which would be measured by the eye tracker. Secondly, comprehension would also be measured with a follow-up quiz afterwards.

Our study had a total of 38 participants, who were all college-age individuals with 14 males and 24 females. Security warnings were created by drawing from four message types found within Google Chrome and Apple Safari web browsers alerting users of malware, phishing, SSL, and unwanted software. The text for the warnings was manipulated by changing the subject, verb, and object of the statements. For example, warnings that focus on the attacker as the subject may say, in the case of an SSL warning, that, “Attackers might be trying to steal your information from expired.badssl.com (for example, passwords, messages, or credit cards).” Other warnings, like those from Safari, use the website rather than the attacker as the subject, in a security message that may say, “This website may be impersonating expired.badssl.com to steal your personal or financial information. You should go back to the previous page.” In addition, warnings could include focus on potential consequences of ignoring the warnings, with the example above of stolen information.

The four warning types (i.e., malware, phishing, SSL, and unwanted software) for two browsers (i.e., Chrome and Safari) across three different conditions (i.e., attacker focus, consequence focus, and site focus) provided 24 different warnings. All references to a specific website were changed to “this website” for ease of presentation. The text for each warning was overlaid on an image for each trial. An example is shown as figure 2. Each participant viewed each of the 24 warnings in random order and then answered a question each one after it was presented. The question asked was, “If this were a real threat and I ignored this warning, an attacker could…” and presented four options. Each answer represented the consequences of the four warning types and were presented in a randomized order each time.

The results of the experiment will be analyzed using both behavioral methods and the eye tracking data to measure comprehension. For the behavioral analysis, we will calculate the proportions of correct answers for each warning type and warning focus. Repeated measures ANOVA tests will be run to test these factors to ensure that there is a large enough number of responses for each trial. With the eye tracking we will determine if the focus of each warning determines how many times it will be reread, by inputting the number of times a warning is reread (a regression) as a dependent variable into a linear regression model with focus type as the independent variable. We will also whether the number of regressions correlates to accuracy of the comprehension questions using a similar linear regression model.

After this study, it seems that users remain a weak link in cybersecurity, as users respond inappropriately to warnings often because of misunderstanding. This research has the potential to inform the design and evaluation of warnings that more effectively help users respond properly to security messages, improving the security of individuals and companies. While our project was unable to utilize fMRI as previously expected, we believe our eye tracking data will provide insight and point us towards our next step in determining comprehension of security warnings.

Figure 1: The four stages of comprehension and how to evaluate user understanding of warnings.

Figure 2: An example of the warning messages