How does the Awareness level work?

Nimblr's "Awareness Level" reflects a user's current understanding of Cyber Security and related topics. Like other attention-demanding processes, this focus can change over time. The Awareness Level isn't a direct measure of knowledge, but there is a correlation between knowledge and awareness. Factors influencing the Awareness Level are:

Positive impact:

• Not triggering simulations
• Active course participation
• Completed knowledge refreshers (Boosters)

Negative impact:

• Inadequate course participation (as indicated by the number of course reminders sent)
• Triggering simulations
• Knowledge decay (based on a generic forgetting curve)

The formula to determine the Awareness Level considers all user activities since their training commenced. However, more recent security behaviors carry greater weight than older ones. For example, consider a user who, a year ago, frequently triggered simulations and received many course reminders but has since improved their participation and stopped clicking simulations. This user might have a higher Awareness Level than another user who continues to trigger simulations, but has done so less frequently overall.

Awareness status

Awareness status can be a valuable metric for categorizing users' security awareness.  It serves as a tool to identify individuals who may require additional support and assistance. By evaluating users based on their awareness status, administrators can easily pinpoint individuals who may benefit from further guidance.

Please note: During the initial three months, it is advisable not to place significant emphasis on the Awareness Status metric, as it relies on cumulative data gathered over time. Accurately and fairly assessing a user's awareness level requires several months of data collection.

Critical (<= 25%): Users in this category require extra attention, as they often overlook course invitations and reminders while also triggering simulations on a regular basis. This level indicates a potential lack of awareness or understanding of security protocols.

Low (26% - 50%): Users in this category exhibit a lower level of awareness compared to the average. They may engage with course materials occasionally, but their overall involvement is typically low. It would be beneficial for administrators to closely monitor these users' progress and further development.

Normal (51% - 85%): Users with a Normal Awareness Status demonstrate a sufficient level of engagement with security training. They regularly participate in courses and show an understanding of security concepts, although there is room for improvement in certain areas.

High (86% - 100%): Users classified in this category display a commendable level of awareness and engagement with security training. They consistently participate in courses, avoid triggering simulations, demonstrate a strong understanding of security principles, and actively apply them in their actions.

For the Nerd: How Is the Awareness Level Calculated?

To fully understand how the Awareness Level is calculated, you'll need to familiarize yourself with the metrics used in the formula. Below, we describe these metrics and explain the overall calculation of the “current Awareness Level”.

Click Rate

Definition: The Click Rate represents the proportion of triggered simulations, including other unsafe activities such as entering credentials, relative to the total number of simulations presented to the user. A partially binary approach also assigns greater significance to a single triggered simulation, and slightly less significance if multiple simulations are triggered during the monthly interval.

Formula:

Click Rate = (0,7 x First Single Click) + (0,3 * (additional clicks / additional simulations presented to the user)

Course Completion Rate

Definition: The Course Completion Rate measures the efficiency of course completion by measuring completed courses and the time from course invitation to completed course.

Formula:

Completion time = Average number of days from sent courses to completed courses
Score ratio = 0,02 (will be subtracted for every day of delay) 

Course Completion Rate = (1 - (Completion time * Score ratio))/1

​

Evolving Awareness Metric

Definition: The Evolving Awareness Metric combines the Click Rate and Course Completion Rate from the last completed month to assess the most recent engagement and completion effectiveness of the courses or simulations.

Formula:

Evolving Awareness Metric = 2 / (Click Rate+Course Completion Rate of the  latest completed month)

​

Awareness Level

Definition: The Awareness Level accounts for knowledge decay,and calculates the present level of awareness. It combines the weighted Awareness Levels from previous months with the last month's Awareness Level, as reflected in the Evolving Awareness Metric, to provide a balanced and updated measure of awareness. The Awareness Level is recalculated monthly, and historical data are stored for auditing and review purposes.

Formula:

Current Awareness Level = (Previous months Awareness Level×0.7) + (Last completed Month’s Evolving Awareness Metric×0.3)

Additional Rules and Calculations

Initial Awareness Level for New Users

• Rule: All new users start with an Awareness Level of 30%.
• Purpose: This initial value serves as a baseline for measuring the impact of engagement and educational interventions on user awareness over time, providing a starting point from which changes can be assessed.

Impact of Clicked Simulations from Previous Months

• Rule: Clicks on simulations sent in previous months are attributed to the Awareness Level for the month in which the simulation was originally sent.
• Rationale: This ensures accurate tracking and impact assessment of simulations over time.

Multiple Clicks on the Same Simulation

• Rule: If the same simulation is clicked multiple times by a user, it does not produce additional impact on the Awareness Level.
• Rationale: This prevents inflation of engagement metrics from repeated interactions with the same content.

Calculation of Maximum Reminders

• Rule: The maximum number of reminders is determined by the number of course invitations sent per month, typically set to 1 course invitation a month. Trial accounts may experience a higher rate.
• Rationale: This standardizes the expectation for course engagement and allows for tailored intervention in trial accounts to enhance user experience and engagement.

Awareness Level for Inactive Users

• Rule: Inactive users—those who neither click on any simulation nor complete any courses—are assigned an Awareness Level of 26%, which is slightly above the critical threshold.
• Rationale: This designation acknowledges the absence of engagement data without assuming active negative behaviors (triggered simulations). It also recognizes the potential for knowledge decay over time.

Default Course Completion Rate

• Rule: If no courses or reminders are sent within a month, the Course Completion Rate will default to 0.5.
• Rationale: This default value is set to prevent the overall Awareness Level from being penalized due to the absence of educational content being sent out.

Default Click Rate

• Rule: If no simulations are sent within a month, the Click Rate will default to 0.5.
• Rationale: This ensures that the overall Awareness Level is not adversely affected by the absence of simulations.