Phishing Awareness: Embedded Training isn’t Really Training

The recent trend in helping organizations protect their users against threats is to perform some sort of automated spear phishing attack.  Services like PhishMe, WombatSecurity and countless others are popping up, claiming to offer two major benefits:

  1. Provide usable metrics for the organization to track security awareness 
  2. Train users close to the event to protect themselves more effectively against attacks.   This is known “embedded” training

While the first is incredibly useful, it’s the second that has always seemed fishy (pun intended) to me.  Because it just doesn’t track with what I know (and have learned more about since hiring behavioral scientists) about human behavior.

But just because I don’t think so doesn’t mean that it’s true.  Even though I had already seen studies that suggested that embedded training wasn’t giving the results these vendors were claiming, I hadn’t seen proof yet.  So, we set out to measure it.

We had a real client who was willing to let us do some experimenting with their userbase.  We designed a two-step phishing assessment that tested 60 users’ willingness to click on phishing emails and tailored our emails and embedded training based on the leading offerings from the vendors in the field.   Then we divided the users in to two groups:

  1. Notification Only:  when the user clicked on the link in the phishing email, they were directed to a page that informed them that they had been phished.
  2. Embedded Training: when the user clicked on the link in the phishing email, they were directed to embedded training similar to the kind provided by the typical vendors in the space.
The results were as shown. 

Users who received embedded training were more likely to report phishing, but were similarly more likely to click on the second  phishing email and overall were more likely to be compromised than those who were simply notified that they had been phished.  

Due to the small sample size, the exact numbers have some margin for error, but one thing is  evident from these numbers:

Embedded training is no more effective than just telling your users that they’ve been phished.

This actually makes perfect sense, despite what all of the phishing vendors are running around saying.  I’ll explain why an upcoming post.  For now, it’s important to realize that the usefulness of phishing your users isn’t to change their behavior – it’s to gain a metric.  (And, note that you can gain this metric far more effectively if you don’t actually tell them that you’re doing it because it’s a more effective real-world metric – not to mention that you can do it at a much better value without going to training services).


bad sciencebehavior designmeasurementmetricsphishingsecurity awarenessspear phishing


Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>