How to Evaluate the Effectiveness of Safety Standards Implementation?

Colleagues, in this article, I would like to share our experience using the "organizational maturity assessment" method, which is applied in many large international and Russian companies to "assess the level of safety culture," with the aim of evaluating a production site's compliance with corporate Safety Standards (hereinafter referred to as Standards).

Many large (and not so large) industrial enterprises have long been implementing the good practice of developing and introducing their own corporate Safety Standards (hereinafter – Standards), which are developed based on existing experience, risk assessment, analysis of various types of incidents (accidents, emergencies, incidents, etc.), industry best practices, and international experience. Usually, when such a Standard is developed, it is stated at the very beginning of the document that the Standard's requirements are not exhaustive but contain minimum mandatory requirements that supplement existing state and local regulatory requirements. Essentially, the Standard is designed to unify and consolidate the main mandatory HSE and fire safety requirements when performing hazardous types of work (e.g., work at height, hot work, work in confined and enclosed spaces, etc.).

At the stage of implementing the Standard's requirements, the role of the "leader" (the responsible employee appointed as the "leader" of the implementation process) differs significantly from the stage when the Standard is already implemented and monitoring compliance is required – see Table 1.

Table 1 – The Role of the Standard "Leader"

Given this fact, as well as the actual results of internal audits conducted at ROUST Group, we found that while the Standard's requirements may be implemented, the effectiveness of this implementation can vary significantly from site to site. For example, ROUST Group has implemented a "Standard for Safe Equipment Lockout" (LOTO Standard). The responsible "Leaders" of the Standard from all the Group's sites submitted reports confirming that the Standard's requirements were implemented: employees were trained, locks and lockout devices specified by the Standard were purchased, equipment lockout charts (LOTO charts) were developed, etc. However, an internal audit revealed that not all operators trained in the LOTO Standard know and follow all 6 steps of equipment lockout, not all operators underwent the refresher training required by the Standard, some locks and/or lockout devices are missing at specific sections, making it effectively impossible to perform work safely if equipment is taken out for repair, etc. This prompted us to introduce detailed checklists developed on the "organizational maturity assessment" principle to evaluate the current level of site compliance with corporate Safety Standards. These checklists make it very easy to identify the specific weak points at a site regarding a particular Standard and what needs to be worked on.

Why did ROUST Group conclude that such a system for assessing compliance with Standard requirements was necessary?

We determined that all Standard requirements can be divided into 3 groups – Table 2.

Table 2 – Categorization of Standard Requirements into Groups

Safety Standards, as I mentioned above, clearly define which measures and requirements must be implemented and realized to minimize the risks of accidents, emergencies, and other incidents. However, Standards do not describe or contain information on how specific Standard requirements should be implemented. It is this aspect that allows us to effectively evaluate using a checklist developed based on the "organizational maturity assessment" method.

The Standard compliance assessment checklist consists of a list of requirements provided by the Standard, which are divided into three risk categories – Table 3.

Table 3 – Risk Categories

The checklist also provides for a 5-point assessment system ("maturity assessment") – Table 4, where:

• 0 – total non-compliance
• 2 – basic compliance
• 3 – partial compliance
• 4 – substantial compliance
• 5 – full compliance

Table 4 – Assessment System

Using specific examples, I would like to show how this assessment methodology allows for evaluating the effectiveness of implementing various Standard requirements, using the "Standard for the Safe Operation of Loading Equipment" as an example.

Table 5 – Requirement (Level 3)

Table 6 – Requirement (Level 2)

Table 7 – Requirement (Level 1)

As a result of the audit using the checklist developed on the "maturity assessment" principle, we understand how effectively a particular Standard requirement has been implemented.

Colleagues, how do you evaluate the effectiveness of Safety Standard implementation at your sites? Do you think it is appropriate to use such an assessment when auditing Safety Standard requirements?