How Awareness Level is Calculated

Factors influencing the Awareness Level are:

Positive impact:

Not triggering simulations
Active course participation

Negative impact:

Triggering simulations (click rate)
Knowledge decay (based on a generic forgetting curve)

The formula to determine the Awareness Level considers all user activities since their training commenced. However, more recent security behaviors carry greater weight than older ones. For example, consider a user who, a year ago, frequently triggered simulations and received many course reminders but has since improved their participation and stopped clicking simulations. This user might have a higher Awareness Level than another user who continues to trigger simulations, but has done so less frequently overall.

Awareness status

Awareness status can be a valuable metric for categorizing users' security awareness. It serves as a tool to identify individuals who may require additional support and assistance. By evaluating users based on their awareness status, administrators can easily pinpoint individuals who may benefit from further guidance.

Please note: During the initial months, it is advisable not to place significant emphasis on the Awareness Status metric, as it relies on cumulative data gathered over time. Accurately and fairly assessing a user's awareness level requires several months of data collection.

Critical (<= 25%): Users in this category require extra attention, as they often overlook course invitations and reminders while also triggering simulations on a regular basis. This level indicates a potential lack of awareness or understanding of security protocols.

Low (26% - 50%): Users in this category exhibit a lower level of awareness compared to the average. They may engage with course materials occasionally, but their overall involvement is typically low. It would be beneficial for administrators to closely monitor these users' progress and further development.

Normal (51% - 85%): Users with a Normal Awareness Status demonstrate a sufficient level of engagement with security training. They regularly participate in courses and show an understanding of security concepts, although there is room for improvement in certain areas.

High (86% - 100%): Users classified in this category display a commendable level of awareness and engagement with security training. They consistently participate in courses, avoid triggering simulations, demonstrate a strong understanding of security principles, and actively apply them in their actions.

For the Nerd: How Is the Awareness Level Calculated?

To fully understand how the Awareness Level is calculated, you'll need to familiarize yourself with the metrics used in the formula. Below, we describe these metrics and explain the overall calculation of the “current Awareness Level”.

Course Score

The Course Score measures how well a user has completed the courses assigned to them. It is calculated as the ratio of completed courses to total courses sent or reminders sent. The score is normalized between 0 and 1, where 1 indicates all courses have been completed (better performance) and 0 indicates no courses have been completed (lower performance). If no course was sent, the user will automatically receive a course score of 1.

Score Ratio = 0.7

Course Score = Courses Completed / Courses Sent

Final Course  Score = Course Score* Score Ratio

Simulation Score

The Simulation Score measures how well a user performs in simulated phishing tests. It is calculated as the ratio of simulated clicks to simulations sent. The score is normalized between 0 and 1, where 1 indicates no clicks (better performance) and 0 indicates all clicks (lower performance).

Score Ratio = 0.3

Simulation Score  = 1 - (Simulations clicked / Simulations Sent)

Final Simulation Score = Simulation Score* Score Ratio

Awareness Level

The Awareness Level accounts for knowledge decay, and calculates the present level of awareness. It combines the weighted Awareness Levels from previous months with the last month's Awareness Level, as reflected in the Evolving Awareness Metric, to provide a balanced and updated measure of awareness. The Awareness Level is recalculated monthly, and historical data are stored for auditing and review purposes.

Awareness Level = (Final Course Score  + Final Simulation Score ) * 100

Additional Rules and Calculations

Initial Awareness Level for New Users

Rule: All new users start with an Awareness Level of 30%.
Purpose: This initial value serves as a baseline for measuring the impact of engagement and educational interventions on user awareness over time, providing a starting point from which changes can be assessed.

Multiple Clicks on the Same Simulation

Rule: If the same simulation is clicked multiple times by a user, it does not produce additional impact on the Awareness Level.
Rationale: This prevents inflation of engagement metrics from repeated interactions with the same content.

Awareness Level for Inactive Users

Rule: Inactive users—those who neither click on any simulation nor complete any courses—are assigned an Awareness Level of 30%, which is slightly above the critical threshold.
Rationale: This designation acknowledges the absence of engagement data without assuming active negative behaviors (triggered simulations). It also recognizes the potential for knowledge decay over time.

Default Course Score

Rule: If no courses or reminders are sent within a month, the course score will be 1.
Rationale: This default value is set to prevent the overall Awareness Level from being penalized due to the absence of educational content being sent out.

Default Simulation Score

Rule: If no simulations are sent within a month, the simulation score will be 1.
Rationale: This ensures that the overall Awareness Level is not adversely affected by the absence of simulations.

Course Timing and Awareness Level: Understanding the End-of-Month

Course invitations are sometimes sent near the end of the month, and if a user doesn’t complete the assigned course before the month closes, their Awareness Level may be negatively affected. While this impact is not entirely eliminated over time, the system is designed so that the effect gradually diminishes, thereby mitigating its long-term effects as users continue to complete courses.

While this may cause temporary inconvenience, it is important to note that the Awareness Level will return to normal as long as the course is completed during the following month. Although timing can occasionally present challenges, the system is designed to fairly balance awareness over time, ensuring that users who complete their courses are not penalized in the long term.

The objective is to maintain a structured approach while allowing users sufficient opportunity to catch up and preserve their awareness standing.

Note: Smoothing Algorithm: The system does not allow dramatic drops. It calculates the average of the current month's score (X%) with the previous month's score (X%).

How Awareness Level is Calculated

Factors influencing the Awareness Level are:

Positive impact:

Not triggering simulations

Active course participation

Negative impact:

Triggering simulations (click rate)

Knowledge decay (based on a generic forgetting curve)

Awareness status

Critical (<= 25%): Users in this category require extra attention, as they often overlook course invitations and reminders while also triggering simulations on a regular basis. This level indicates a potential lack of awareness or understanding of security protocols.

Normal (51% - 85%): Users with a Normal Awareness Status demonstrate a sufficient level of engagement with security training. They regularly participate in courses and show an understanding of security concepts, although there is room for improvement in certain areas.

For the Nerd: How Is the Awareness Level Calculated?

To fully understand how the Awareness Level is calculated, you'll need to familiarize yourself with the metrics used in the formula. Below, we describe these metrics and explain the overall calculation of the “current Awareness Level”.

Course Score

Simulation Score

The Simulation Score measures how well a user performs in simulated phishing tests. It is calculated as the ratio of simulated clicks to simulations sent. The score is normalized between 0 and 1, where 1 indicates no clicks (better performance) and 0 indicates all clicks (lower performance).

Awareness Level

Additional Rules and Calculations

Initial Awareness Level for New Users

Rule: All new users start with an Awareness Level of 30%.

Purpose: This initial value serves as a baseline for measuring the impact of engagement and educational interventions on user awareness over time, providing a starting point from which changes can be assessed.

Multiple Clicks on the Same Simulation

Rule: If the same simulation is clicked multiple times by a user, it does not produce additional impact on the Awareness Level.

Rationale: This prevents inflation of engagement metrics from repeated interactions with the same content.

Awareness Level for Inactive Users

Rule: Inactive users—those who neither click on any simulation nor complete any courses—are assigned an Awareness Level of 30%, which is slightly above the critical threshold.

Rationale: This designation acknowledges the absence of engagement data without assuming active negative behaviors (triggered simulations). It also recognizes the potential for knowledge decay over time.

Default Course Score

Rule: If no courses or reminders are sent within a month, the course score will be 1.

Rationale: This default value is set to prevent the overall Awareness Level from being penalized due to the absence of educational content being sent out.

Default Simulation Score

Rule: If no simulations are sent within a month, the simulation score will be 1.

Rationale: This ensures that the overall Awareness Level is not adversely affected by the absence of simulations.

Course Timing and Awareness Level: Understanding the End-of-Month

Final Result: (X% + X%) ÷ 2 = X%

If you have any further questions, feedback or requests just reach out to us at:    support@nimblrsecurity.com
   Or
   Submit a Ticket

How Awareness Level is Calculated

Factors influencing the Awareness Level are:

Positive impact:

Not triggering simulations

Active course participation

Negative impact:

Triggering simulations (click rate)

Knowledge decay (based on a generic forgetting curve)

Awareness status

Critical (<= 25%): Users in this category require extra attention, as they often overlook course invitations and reminders while also triggering simulations on a regular basis. This level indicates a potential lack of awareness or understanding of security protocols.

Normal (51% - 85%): Users with a Normal Awareness Status demonstrate a sufficient level of engagement with security training. They regularly participate in courses and show an understanding of security concepts, although there is room for improvement in certain areas.

For the Nerd: How Is the Awareness Level Calculated?

To fully understand how the Awareness Level is calculated, you'll need to familiarize yourself with the metrics used in the formula. Below, we describe these metrics and explain the overall calculation of the “current Awareness Level”.

Course Score

Simulation Score

The Simulation Score measures how well a user performs in simulated phishing tests. It is calculated as the ratio of simulated clicks to simulations sent. The score is normalized between 0 and 1, where 1 indicates no clicks (better performance) and 0 indicates all clicks (lower performance).

Awareness Level

Additional Rules and Calculations

Initial Awareness Level for New Users

Rule: All new users start with an Awareness Level of 30%.

Purpose: This initial value serves as a baseline for measuring the impact of engagement and educational interventions on user awareness over time, providing a starting point from which changes can be assessed.

Multiple Clicks on the Same Simulation

Rule: If the same simulation is clicked multiple times by a user, it does not produce additional impact on the Awareness Level.

Rationale: This prevents inflation of engagement metrics from repeated interactions with the same content.

Awareness Level for Inactive Users

Rule: Inactive users—those who neither click on any simulation nor complete any courses—are assigned an Awareness Level of 30%, which is slightly above the critical threshold.

Rationale: This designation acknowledges the absence of engagement data without assuming active negative behaviors (triggered simulations). It also recognizes the potential for knowledge decay over time.

Default Course Score

Rule: If no courses or reminders are sent within a month, the course score will be 1.

Rationale: This default value is set to prevent the overall Awareness Level from being penalized due to the absence of educational content being sent out.

Default Simulation Score

Rule: If no simulations are sent within a month, the simulation score will be 1.

Rationale: This ensures that the overall Awareness Level is not adversely affected by the absence of simulations.

Course Timing and Awareness Level: Understanding the End-of-Month

Final Result: (X% + X%) ÷ 2 = X%

If you have any further questions, feedback or requests just reach out to us at: support@nimblrsecurity.com Or Submit a Ticket

If you have any further questions, feedback or requests just reach out to us at: support@nimblrsecurity.com
Or
Submit a Ticket