How does the Awareness level work?

Nimblr's "Awareness Level" reflects a user's current understanding of Cyber Security and related topics. Like other attention-demanding processes, this focus can change over time. The Awareness Level isn't a direct measure of knowledge, but there is a correlation between knowledge and awareness. Factors influencing the Awareness Level are:

Positive impact:

• Not triggering simulations
• Active course participation
  
  

Negative impact:

• Triggering simulations (click rate)
• Knowledge decay (based on a generic forgetting curve)

The formula to determine the Awareness Level considers all user activities since their training commenced. However, more recent security behaviors carry greater weight than older ones. For example, consider a user who, a year ago, frequently triggered simulations and received many course reminders but has since improved their participation and stopped clicking simulations. This user might have a higher Awareness Level than another user who continues to trigger simulations, but has done so less frequently overall.


Awareness status

Awareness status can be a valuable metric for categorizing users' security awareness.  It serves as a tool to identify individuals who may require additional support and assistance. By evaluating users based on their awareness status, administrators can easily pinpoint individuals who may benefit from further guidance.

Please note: During the initial months, it is advisable not to place significant emphasis on the Awareness Status metric, as it relies on cumulative data gathered over time. Accurately and fairly assessing a user's awareness level requires several months of data collection.

Critical (<= 25%): Users in this category require extra attention, as they often overlook course invitations and reminders while also triggering simulations on a regular basis. This level indicates a potential lack of awareness or understanding of security protocols.

Low (26% - 50%): Users in this category exhibit a lower level of awareness compared to the average. They may engage with course materials occasionally, but their overall involvement is typically low. It would be beneficial for administrators to closely monitor these users' progress and further development.

Normal (51% - 85%): Users with a Normal Awareness Status demonstrate a sufficient level of engagement with security training. They regularly participate in courses and show an understanding of security concepts, although there is room for improvement in certain areas.

High (86% - 100%): Users classified in this category display a commendable level of awareness and engagement with security training. They consistently participate in courses, avoid triggering simulations, demonstrate a strong understanding of security principles, and actively apply them in their actions.

For the Nerd: How Is the Awareness Level Calculated?

To fully understand how the Awareness Level is calculated, you'll need to familiarize yourself with the metrics used in the formula. Below, we describe these metrics and explain the overall calculation of the “current Awareness Level”.

Course Score

The Course Score measures how well a user has completed the courses assigned to them. It is calculated as the ratio of completed courses to total courses sent or reminders sent. The score is normalized between 0 and 1, where 1 indicates all courses have been completed (better performance) and 0 indicates no courses have been completed (lower performance). If no course was sent, the user will automatically receive a course score of 1.

Score Ratio = 0.7

Course Score = Courses Completed / Courses Sent

Final Course  Score = Course Score* Score Ratio

Simulation Score

The Simulation Score measures how well a user performs in simulated phishing tests. It is calculated as the ratio of simulated clicks to simulations sent. The score is normalized between 0 and 1, where 1 indicates no clicks (better performance) and 0 indicates all clicks (lower performance).

Score Ratio = 0.3

Simulation Score  = 1 - (Simulations clicked / Simulations Sent)

Final Simulation Score = Simulation Score* Score Ratio

Awareness Level

The Awareness Level accounts for knowledge decay, and calculates the present level of awareness. It combines the weighted Awareness Levels from previous months with the last month's Awareness Level, as reflected in the Evolving Awareness Metric, to provide a balanced and updated measure of awareness. The Awareness Level is recalculated monthly, and historical data are stored for auditing and review purposes.

Awareness Level = (Final Course Score  + Final Simulation Score ) * 100

Additional Rules and Calculations

Initial Awareness Level for New Users

• Rule: All new users start with an Awareness Level of 30%.
• Purpose: This initial value serves as a baseline for measuring the impact of engagement and educational interventions on user awareness over time, providing a starting point from which changes can be assessed.

Multiple Clicks on the Same Simulation

• Rule: If the same simulation is clicked multiple times by a user, it does not produce additional impact on the Awareness Level.
• Rationale: This prevents inflation of engagement metrics from repeated interactions with the same content.

Awareness Level for Inactive Users

• Rule: Inactive users—those who neither click on any simulation nor complete any courses—are assigned an Awareness Level of 30%, which is slightly above the critical threshold.
• Rationale: This designation acknowledges the absence of engagement data without assuming active negative behaviors (triggered simulations). It also recognizes the potential for knowledge decay over time.

Default Course Score

• Rule: If no courses or reminders are sent within a month, the course score will be 1.
• Rationale: This default value is set to prevent the overall Awareness Level from being penalized due to the absence of educational content being sent out.

Default Simulation Score

• Rule: If no simulations are sent within a month, the  simulation score will be 1.
• Rationale: This ensures that the overall Awareness Level is not adversely affected by the absence of simulations.

Course Timing and Awareness Level: Understanding the End-of-Month

Course invitations are sometimes distributed toward the end of the month, and if an end-user does not complete the course immediately, their Awareness Level may be temporarily impacted. This situation may  occur when a course is assigned just before a weekend, with the new month beginning shortly thereafter.

While this may cause temporary inconvenience, it is important to note that the Awareness Level will return to normal as long as the course is completed during the following month. Although timing can occasionally present challenges, the system is designed to fairly balance awareness over time, ensuring that users who complete their courses are not penalized in the long term.

The objective is to maintain a structured approach while allowing users sufficient opportunity to catch up and preserve their awareness standing.