Phishing Awareness: The Most Important Thing

Dec 05 2012 Mike Uncategorized 2 Comments

In my previous couple of blogs, I talked about our research that shows that embedded training doesn’t do much more than act as a trigger.  You might get the idea that I’m not a big fan of products like those offered by PhishMe, Wombat and ThreatSim.

But you couldn’t be more wrong.

As I said in my first blog on the subject, there are two reasons to use these products: the training impact on the users is completely insignificant compared to the measurement impact of performing regular phishing training.    In fact, if you do nothing else to improve your awareness program, measuring the response to phishing gives you access to incredibly important information.

In fact, there are two measurements that you should be using well-designed, randomized phishing testing for on a regular (at least quarterly) basis:

  1. Response Rate:  How likely are your users to fall victim to phishing attacks?
  2. Report Rate: How likely are your users to tell you about phishing attacks?

While everyone touts the response rate metric (i.e.  what % of users clicked on the link), it’s actually the less important of the two measurements because it’s so variable across instances.  The response rate to a phishing email varies widely based on factors such as time of day, quality of the phishing email, etc.  So, a user who’s susceptible to one email at one point is likely less susceptible to a different email at a different point, so showing quarter-on-quarter improvement, while interesting, is often an artifact of the design of the test more than it is a measurement of user behavior.  (Of course, good test design can minimize this type of impact – hence why we have people like Kati around who are trained in experiment design methodologies)

What is more interesting and useful is the report rate – that is, of the users who received the email, what percentage of them reported the attack to the appropriate authority (either the security team or help desk in most organizations).  This can be further broken down by the users who reported without clicking on the link and those who reported when they did click on the link to give an idea of what portion of users reported issues before they fell prey and what portion reported after they were attacked.

In my experience, this is the most important measurement we can take for stopping phishing, because it tells us something incredibly important about the actual security of our organization on a day-to-day basis.  That metric acts as an estimate of the proportion of phishing attacks (both successful and unsuccessful) that are being reported by your users.  From this, we can use some simple math to predict what the likely impact of the actual phishing attacks against your organization will look like.

As Lance pointed out recently, the human firewall is an important part of the security ecosystem, and this detection rate is just as important as the detection rate in your IDS/IPS.

Note that these measurements only work if you turn off the embedded training altogether.  Because the act of embedding training (or even just notification) skews the measurement that you’re trying to gather.  

Just one more reason to skip the embedded training (most of the time) when you’re phishing your users.


Leave a comment
David Ward

October 22, 2013 at 5:20 am

Hi, great post. I’m currently running a mock phishing awareness programme at my organisation using a product from one of the companies you mention. The point you make about real metrics is very valid. One thing that surprised me was the speed with which users click on e-mails when they hit their in-box. We were getting responses within seconds of the e-mails going out with people clearly not even having time to read the title, never mind thinking about the content, a phishers nirvana. Unless you use a tool it is very hard to see that sort of information. We’ve also noticed an increase in the number of genuine internal e-mails being reported as phishing due to the way they have been written, so clearly at least some of the message is striking home.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>