who global benchmarking tool (gbt) for evaluation … · 34 indicator: ri01 legal provisions,...

1

WHO GLOBAL BENCHMARKING TOOL (GBT) FOR 2

EVALUATION OF NATIONAL REGULATORY SYSTEM OF 3

MEDICAL PRODUCTS 4

FACT SHEETS FOR REGULATORY INSPECTION 5

(DRAFT REVISION VI VERSION 1) 6

(DECEMBER 2017) 7

DRAFT FOR COMMENTS 8

Should you have any comments on the attached text, please send these to: Country Regulatory Support 9 (CRS) Group , Regulatory Systems Strengthening (RSS), World Health Organization, 1211 Geneva 10 27,Switzerland; Alireza Khadem ([email protected]); fax: (+41 22) 791 4856; and to Laura 11 Brown ([email protected]) by 28 February 2018. Working documents are sent out electronically and they 12 will also be placed on the Medicines website for comment. 13

14 © World Health Organization 2017 15

All rights reserved. 16

This draft is intended for a restricted audience only, i.e. the individuals and organizations having received this draft. 17 The draft may not be reviewed, abstracted, quoted, reproduced, transmitted, distributed, translated or adapted, in part 18 or in whole, in any form or by any means outside these individuals and organizations (including the organizations' 19 concerned staff and member organizations) without the permission of the World Health Organization. The draft 20 should not be displayed on any website. 21

Please send any request for permission to: 22

Dr Alireza Khadem Broojerdi, Scientist, Country Regulatory Support, Regulatory Systems Strengthening, 23 Department of Essential Medicines and Health Products, World Health Organization, CH-1211 Geneva 27, 24 Switzerland. Fax: (+41 22) 791 4856; email: [email protected]. The designations employed and the 25 presentation of the material in this draft do not imply the expression of any opinion whatsoever on the part of the 26 World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or 27 concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border lines 28 for which there may not yet be full agreement. 29

The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or 30 recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. 31 Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. 32

All reasonable precautions have been taken by the World Health Organization to verify the information contained in 33 this draft. However, the printed material is being distributed without warranty of any kind, either expressed or 34 implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall the 35 World Health Organization be liable for damages arising from its use. 36

This draft does not necessarily represent the decisions or the stated policy of the World Health Organization. 37

WHO global benchmarking tool rev. VI ver. 1, RI

06-REGULATORY INSPECTION (RI) 1

2

Inspection of establishments across the supply chain including medical products manufacturers, 3 distributors, importers, wholesalers and retailers is an essential regulatory function. Its purpose is to 4 ensure that operations at the mentioned establishments are carried out in accordance with the approved 5 standards, norms, and guidelines, as well as in compliance with the national medical products legislations 6 and regulations, which in turn should be in line with WHO recommendations and/or other internationally 7 recognized guidelines. So, it is worth mentioning that the scope of the function applies to different Good 8 Practices (GxP) and not limited to Good Manufacturing Practices (GMP). Rather, Good Distribution 9 Practices (GDP) and Good Clinical Practices (GCP) comes under the scope of this function. Nevertheless, 10 Good Vigilance Practices (GVP) is not addressed in this function but addressed under the vigilance 11 function. 12

National Regulatory Authorities (NRAs) should have the legal mandate to inspect and enforce Good 13 Practices (GXP) through the supply chain, this include making decisions concerning the issue, 14 suspension or withdrawal of establishment licenses, authorizations and/or certifications for their activities, 15 as well as providing advice and handling of suspected quality defects. Usually inspectors perform several 16 types of inspections: pre-licensing vs. post-licensing, pre-approval vs. post-approval, announced vs. non-17 announced, and domestic vs. overseas inspections. 18

Inspection activities should be established based on a risk management approach. Inspections disclose 19 weaknesses and deficiencies as well as actual or potential errors in the production, quality control, storage 20 or distribution of medical products, therefore inspection activities are fundamental to guarantee the 21 quality, safety and efficacy of medical products used by the population. There should be an appeal system 22 that is independent of the body that made the initial decision. 23

The credibility of the process depends to large extent on the technical competence and integrity of the 24 inspectors. The inspectorate should also ensure confidentiality of the information obtained in the course 25 of its inspection activities. 26

Networking with other international bodies and NRAs is an important method for acquiring, sharing, and 27 exchanging the relevant information on medical products quality and safety which in turn would 28 contribute to informed science based decisions. The inspectorate of the NRA should follow uniform 29 procedures incorporating quality system principles. 30

A general limitation for this function exists in countries where no domestic manufacturers are established. 31 In this case, regulatory inspection of GMP is not applicable however GDP would always apply. Similarly, 32 GCP inspections might not apply in countries where no clinical trials are conducted. 33

Indicator: RI01 Legal provisions, regulations and guidelines required to 34

define regulatory framework of inspection and enforcement. 35

Rationale: 36

The objective of this indicator is to assure that the NRA is supported by the necessary mandate to conduct 37 inspections. Furthermore, national GXP rules are in line with internationally recognized and accepted 38 Good Practices such as those of the WHO. 39


The NRA must show evidence that laws, regulations, decrees, agreements, or other mandatory legal 1 provisions regarding inspections for best practices and other oversight activities also legal provisions to 2 inspect the establishments of marketing authorization holders, manufacturers, importers and exporters, 3 and distributor’s national standards and guidelines on good practices are in place. 4

Category: 01. Legal provisions, regulations and guidelines 5

Sub-Indicator: RI01.01: Legal provisions authorize the inspectorate to inspect and 6

enforce GXP throughout the supply chain. 7

Scope: All 8

Maturity: 1 9

Requirement: 10

Legal provisions, regulations and guidelines 11

Rationale: 12

The objective of this sub-indicator is to assure that laws, regulations, decrees, agreements, or other 13 mandatory legal provisions regarding inspections for best practices and other oversight activities to 14 inspect the establishments (of marketing authorization holders, manufacturers, importers and exporters, 15 and distributors) are established to ensure the necessary mandate for the relevant regulatory activities 16 through national standards and guidelines on GXP throughout the supply chain. 17

Description: 18

The assessor should verify that legislation establishes fundamental functions in the regulatory authority 19 appointed to verify compliance with laws, regulations, standards and good practices namely, Good 20 Manufacturing Practices (GMP) for manufacturers, Good Distribution Practices (GDP) for distributors, 21 wholesalers and retailers and Good Clinical Practices (GCP) for sites where clinical studies are 22 conducted. 23

The assessor should also verify that legislation provides the obligation for manufacturers and marketing 24 authorization holders (MAHs) to inform the NRA of any quality defects or safety signal of a product as 25 well as any marketing/regulatory decisions taken in the country of origin or other countries where the 26 product is marketed. In line with these legal provisions, the assessor should check relevant guidelines 27 endorsed explaining the obligation and how, when and what quality defects or safety issues have to be 28 reported (e.g. recall notifications, reporting of study results from actual or previous studies, information 29 on supervision of clinical trials). Similar obligation should also be in force for the MAH for all products 30 marketed in the country. 31

The assessor should ensure the existence and the implementation of a set of enforcement actions in case 32 of non-compliance. The enforcement actions should include but not limited to recall, suspension, 33 withdrawal sanctions and prosecution. The assessor should identify if the law attributes to the inspectorate 34 the adequate mandate and power and authority to suspend or stop production. This may be the case e.g. 35 with respect to violations of legal requirements or to guarantee the product quality to be brought to the 36 market in case quality defects are suspected or in case of confirmed quality defects leading to a rapid 37 alert. 38

39


Evidence to review: 1

The assessor should check the following evidences: 2

1. Published legal provisions (e.g. law, act, executive order, etc.) establishing the mandate for the 3 inspectorate to inspect and enforce; 4

2. Legal provisions establishing to inspect according GXP throughout the supply chain. 5

Framework: 6

Structure/Foundation/Input 7

Rating Scale: 8

NI: There are no legal provisions for mandating the NRA to inspect and enforce GXP. 9

OI: The NRA is taking steps toward the establishment of legal provisions to allow for regulatory 10 inspections however no results yet exist for mandating the regulatory inspection function. 11

PI: The NRA has the legal provisions to conduct the regulatory inspection activities however such 12 activities are recently implemented so results associated with this sub-indicator are not yet consolidated. 13

I: Legal provisions exist for mandating the NRA to inspect and enforce GXP. 14

Limitations and Remarks: 15

Related indicators: 16

References: 17

1. 5- WHO TRS 790, Annex 6, 1990. Chapter 1.4 and 1.8 (b) 18

2. 8- WHO TRS 902, Annex 8, 2002. Chapter 5.2 (d) 19

3. 1- WHO TRS 885, Annex 8, 1999 20

4. 7- Effective Drug Regulation; 2002. Chapter 4.4 21

5. 4- ISO standard 9001:2015 22

Sub-Indicator: RI01.02: Legal provisions allow inspectors to enter at any reasonable 23

time, in any place within facilities, throughout the supply chain. 24

Scope: All 25

Maturity: 1 26

Requirement: 27


Rationale: 29

The objective of this sub-indicator is to assure that laws, regulations, decrees, agreements, or other 30 mandatory legal provisions regarding inspections for best practices and other oversight activities to 31 inspect the establishments are actually providing the necessary mandate to allow inspectors to enter at any 32


reasonable time, in any place within facilities, throughout the supply chain would significantly contribute 1 to inspection function efficiency and effectiveness. 2

Description: 3

The assessor should verify that the legal provisions provide the mandate, the power and the authority for 4 the inspectors to enter at any reasonable time, in any place within facilities, throughout the supply chain. 5

It is worth mentioning that the expression of “any reasonable time” means any time within normal 6 working hours of the establishment subjected to the inspection. For example, if the establishment is 7 working during night, then the legal provisions should ideally authorize the inspectors’ access at that 8 time. On the other side, access to unlicensed/unauthorized establishments and/or access to establishments 9 beyond normal working hours may not be mandated by the respective legal provisions and might need 10 special approvals from judicial departments. 11



1. Published legal provisions (e.g. law, act, executive order, etc.) 14

2. Review Inspection activities and identify examples. Select records of non-announce inspections, 15 following any issue related with quality, complaints or part of an investigation. 16

Framework: 17


Rating Scale: 19

NI: There are no legal provisions allowing the inspectors to enter at any reasonable time, in any place 20 within facilities, throughout the supply chain. 21

OI: The NRA is taking steps toward the establishment of legal provisions to allow inspectors to enter at 22 any reasonable time to any place within facilities; however, no results yet exist. 23

PI: Legal provisions allowing the inspectors to enter at any reasonable time, in any place within facilities, 24 throughout the supply chain are recently established; however, the results of this practice is not yet 25 consolidated. 26

I: There are legal provisions allowing the inspectors to enter at any reasonable time, in any place within 27 facilities, throughout the supply chain and the implementation of the same is proven. 28



LI01 31

References: 32




3. 1- WHO TRS 885, Annex 8, 1999 1


Sub-Indicator: RI01.03: Legal provisions allow inspectors to collect relevant evidences, 3

including samples, during GXP inspections. 4

Scope: All 5

Maturity: 2 6

Requirement: 7


Rationale: 9

The objective of this sub-indicator is to assure that the respective laws, regulations, decrees, or other 10 mandatory legal provisions regarding inspections for best practices and other oversight activities allow 11 inspectors to collect relevant evidence, including samples, during GXP. 12

Description: 13

The assessor should verify the existence of the critical attribute of designated inspectors to have the power 14 and authority to collect any important evidences including samples during their inspection and 15 investigation activities. The designated inspectors should also be able to collect product samples for 16 testing by the national control laboratory (NCL) at any phase of production or supply chain as part of 17 compliance verification. 18



1. Published legal provisions (e.g. law, act, executive order, etc.) 21

2. Review inspection reports and the records of evidences (documental or material) or samples 22 collect during inspection activities. 23

3. Review samples laboratory results and decisions based on that. 24

4. Important to verify if it is a systematic process well established. 25

Framework: 26


Rating Scale: 28

NI: There are no legal provisions allowing the inspectors to collect relevant evidences, including samples, 29 during GXP inspections. 30

OI: The NRA is taking steps toward the establishment of legal provisions to allow inspectors to collect 31 relevant evidences, including samples, during GXP inspections however no results yet exist. 32

PI: There is evidence that the NRA has the legal basis, elements, and capacity to perform the processes 33 mentioned in the indicator, but it has only limited, recent experience with it. 34


I: There are legal provisions allowing the inspectors to collect relevant evidences, including samples, 1 during GXP inspections and the implementation of the same is proven. 2



LI01 5

References: 6

1. 35- WHO TRS 957, Annex 1, 2010 7

2. 51-WHO TRS 986, Annex 2, 2014 8

3. 60- WHO TRS 961, Annex 5, 2011 9

4. 58- WHO TRS 970, Annex 2, 2012 10

5. 61- WHO TRS 961, Annex 6, 2011 11

6. 64- WHO TRS 961, Annex 14, 2011 12

7. 67-WHO TRS 957, Annex 2, 2010 13

8. 80-WHO TRS 902, Annex 9, 2002 14

9. 27- WHO TRS 957, Annex 5, 2010 Section 17 Recalls 15

10. 40- WHO TRS 850, Annex 3, 1995 (Chapter 1.5, 2,4.7, 4.8,11) 16

11. 1- WHO TRS 885, Annex 8, 1999 17

12. 31- WHO TRS 790, Annex 2, 1990 18

13. 4- ISO standard 9001:2015 19

14. 28- Guidelines for the development of measures to combat counterfeit drugs. 20 (WHO/EDM/QSM/99.1) 21


16. 127-WHO TRS 929, Annex 4, 2005 23

Sub-Indicator: RI01.04: Updated national GXP regulations, norms or guidelines are 24

mandatory. 25

Scope: All 26

Maturity: 3 27

Requirement: 28


Rationale: 30


The objective of this sub-indicator is to assure that national GXP regulations, norms and/or guidelines are 1 available and their implementation is mandatory for the relevant establishments. 2

Description: 3

The assessor should review the GXP regulations and guidelines which should be updated, published and 4 available to all stakeholders. Also, the assessor should verify that the guidance is in coherence with WHO 5 or other internationally recognized/accepted guidance. If applicable, differences should be identified. The 6 assessor should verify that the national regulations are mandatory and actually implemented. 7



1. Published GXP guidelines (e.g. Good Manufacturing Practice -GMP, Good Distribution Practice-10 GDP, Good Cold Chain Management Practices -GCCMP, Good Clinical Practice - GCP, Good 11 Laboratory Practice - GLP, etc.) 12

2. Evidence that the guidelines implemented and enforceable by the NRA. 13

Framework: 14


Rating Scale: 16

NI: There are no legal provisions allowing the inspectors to collect relevant evidences, including samples, 17 during GXP inspections. 18

OI: The NRA is taking steps towards establishment of national GXP guidelines however none of these 19 guidelines (GMP, GDP, and GCP) is yet established. 20

PI: Some national GXP guidelines (GMP, GDP and GCP) are established while others are not. 21

I: National GXP guidelines exist and are mandatory. 22



References: 25

1. 1- WHO TRS 885, Annex 8, 1999 26

2. 4- ISO standard 9001:2015 27



5. 27- WHO TRS 957, Annex 5, 2010 30

6. 35- WHO TRS 957, Annex 1, 2010 31

7. 51-WHO TRS 986, Annex 2, 2014 32

8. 58- WHO TRS 970, Annex 2, 2012 33


9. 60- WHO TRS 961, Annex 5, 2011 1

10. 61- WHO TRS 961, Annex 6, 2011 2

11. 67-WHO TRS 957, Annex 2, 2010 3

12. 80-WHO TRS 902, Annex 9, 2002 4


Sub-Indicator: RI01.05: Legal provisions and/or regulations allow the recognition of 6

and/or reliance on foreign NRA inspections and/or enforcement actions based on well- 7

defined criteria. 8

Scope: All 9

Maturity: 1 10

Requirement: 11


Rationale: 13

The objective of this sub-indicator is to assure that laws, regulations, or other mandatory legal provisions 14 are actually establishing the mandate for the recognition and/or reliance of foreign NRA inspections and 15 enforcement actions based on well-defined criteria. 16

Description: 17

The assessor should review the legal provisions and/or regulations which should be available and 18 published and verify that the guidance allow for the recognition and/or reliance of foreign NRA 19 inspections and enforcement actions based on well- defined criteria. The criteria might vary significantly 20 however the assessor should check the criteria and ensure that it is clear and rationalized. 21



1. Published legal provisions and/or regulation for recognition and/or reliance. 24

2. Identify, select and review some examples of recognition of foreign NRA inspections and 25 enforcement actions. 26

Framework: 27


Rating Scale: 29

NI: There are no legal provisions or regulations allowing the recognition of and/or reliance on of foreign 30 NRA inspections and/or enforcement actions. 31

OI: The NRA is taking steps toward the establishment of legal provisions and/or regulations allowing the 32 recognition of and/or reliance on of foreign NRA inspections and/or enforcement actions however no 33 results yet exist. 34


PI: Legal provisions or regulations allowing the recognition of and/or reliance on of foreign NRA 1 inspections and/or enforcement actions are recently established, but there is only limited, recent 2 experience (less than two years) with the implementation of the same. 3

I: There are legal provisions or regulations allowing the recognition of and/or reliance on of foreign NRA 4 inspections and/or enforcement actions. 5

Limitations and Remarks: In some countries neither recognition nor reliance is considered. In this case, 6 the assessor can score the sub-indicator as NOT APPLICABLE if evidence is provided to show effective 7 and efficient regulatory inspection function without recognition or reliance. 8


References: 10

1. 1- WHO TRS 885, Annex 8, 1999 11




5. 4- ISO standard 9001:2015 15

Indicator: RI02 Arrangement for effective organization and good 16

governance. 17

Rationale: 18

The objective of this indicator is to ensure the implementation of effective organization and good 19 governance practices at the structure(s) in charge of establishments licensing activities which in turn is 20 contributing to effective and efficient functioning of the regulatory inspection activities. 21

Category: 2. Organization and good governance 22

Sub-Indicator: RI02.01: There is a defined structure with clear responsibilities to 23

conduct regulatory inspection activities. 24

Scope: All 25

Maturity: 2 26

Requirement: 27

Roles and responsibilities of the structure(s) in charge of establishments licensing activities. 28

Rationale: 29

The objective of this sub-indicator is to assure effective organization and good governance of regulatory 30 inspection activities and that these are taken over by defined structure(s) with clear roles and 31 responsibilities. 32

Description: 33


The assessor should identify the organization set in place to establish, implement or maintain the 1 regulatory inspection function including the defined structure(s) undertaking different relevant activities. 2 Responsibilities, duties and roles of this/these structure(s) should be clearly defined and documented. In 3 case of involvement of more than a structure, the assessor should check the ways and approaches by 4 which coordination between these structures is taking place. 5


The assessor should ask for and review the following evidences: 7

1. Organization chart of the organization(s) responsible for the implementation of inspection activities 8 along with identification of the particular structure(s) implementing the function. 9

2. Documentation clarifying roles and responsibilities of the structure(s) implementing inspection 10 activities. This may include administrative decrees, terms of reference, others. 11

3. Documentation identifying established mechanisms of coordination (committees, internal work-12 sharing, and workflow) among structures, if any, which take part in regulatory inspection activities. 13

Framework: 14


Rating Scale: 16

NI: There is no defined structure(s) in charge of regulatory inspection activities. 17

OI: A mandate to establish a structure(s) in charge of regulatory inspection activities is available however 18 the structure itself is not yet established. 19

PI: Structure(s) in charge of regulatory inspection activities is newly established and mandated however 20 the regular work and practice of this structure(s) is not yet consolidated. 21

I: There is defined structure(s) in charge of regulatory inspection activities with clear well documented 22 roles and responsibilities. 23



References: 26

1. 1- WHO TRS 885, Annex 8, 1999 27




5. 4- ISO standard 9001:2015 31


Sub-Indicator: RI02.02: Documented procedures and/or mechanisms are implemented 1

to ensure the involvement and communication between all stakeholders relevant to 2

regulatory inspection activities. 3

Scope: All 4

Maturity: 3 5

Requirement: 6

Agreements, Memoranda of Understanding (MOU) and/or procedures for ensuring involvement of and 7 communication between stakeholders relevant to regulatory inspection function. 8

Rationale: 9

The objective of this sub-indicator is to ensure that documented procedures and/or other mechanisms are 10 implemented to ensure the involvement and communication between different organizations, institutions, 11 and/or departments for appropriate organization and good governance of the function. 12

Description: 13

The assessor should verify that documented procedures and/or other appropriate mechanisms are 14 implemented to ensure the involvement and communication between different organizations, institutions, 15 and/or departments relevant to regulatory inspection activities, within and/or outside the NRA (e.g. NCL, 16 vigilance, MA, police, custom authorities and judicial branch, professional associations, as well as 17 manufacturers/industry and others). 18

In case of decentralized licensing establishments function, an information exchange system, mechanism 19 and/or platform must be established and used so that appropriate communication between the central and 20 peripheral structure(s) is ensured (e.g. the decentralized entity can receive requests and/or guidance from 21 the central authority and report back to it) along with consistency among different peripheral structures. 22



1. Guidelines or SOPs – related with external and internal communication. 25

2. Example records of communication and collaboration proofing implementation of the above 26 mentioned procedures. 27

3. In addition, the assessor should check if regular formal and/or official communication and 28 meetings among above-mentioned key players. Important to verify if it is a systematic process well 29 established. 30

4. Documentations of paths of communication and reporting. 31

5. Platforms for information sharing and exchange. 32

Framework: 33


Rating Scale: 35


NI: There are no information exchange procedures and/or mechanisms among different stakeholders of 1 the regulatory inspection function and/or between the central authority and the decentralized entities. 2

OI: The NRA is developing an information exchange mechanism, but it is not yet ready or exchanges are 3 being conducted without an established methodology. 4

PI: The NRA recently established an information exchange procedure and/or mechanism and it is at the 5 implementation stage, so this practice is not yet consolidated. 6

I: There are established, implemented and maintained information exchange procedures and/or 7 mechanisms among different stakeholders of the regulatory inspection function and/or between the central 8 authority and the decentralized entities. 9



References: 12

1. 1- WHO TRS 885, Annex 8, 1999 13




5. 4- ISO standard 9001:2015 17

Indicator: RI03 Human resources to perform regulatory inspection 18

activities. 19

Rationale: 20

The objective of this indicator is to assure that entity (ies) within an NRA is/are adequately resourced 21 with trained, experienced and skilled workforce, as well as empowered to fully perform regulatory 22 inspection activities. This will ensure that regulatory inspection activities will be performed in accordance 23 with international best practices. 24

Category: 06. Resources (HR,FR, infrastructure and equipment) 25

Sub-Indicator: RI03.01: Enough competent staff (education, training, skills and 26

experience) is assigned to perform regulatory inspection activities. 27

Scope: All 28

Maturity: 3 29

Requirement: 30

Enough number and proper competence of the staff in charge of regulatory inspection activities 31

Rationale: 32


The objective of this sub-indicator is to assure the existing human resources for regulatory inspection 1 activities is adequate in terms of number, their experience and their specific competences to perform all 2 regulatory inspection activities. 3

Description: 4

The assessor should verify that appropriate human resources (staff), from both quantitative and qualitative 5 aspects, is designated for performing different activities related to the regulatory inspection. The assessor 6 should verify that the NRA estimated the adequate number of staff required to effectively and efficiently 7 perform regulatory inspection function and such number is actually recruited. In addition, the assessor 8 should check the appropriateness of the competence of the staff as represented by their education, 9 training, skills, and experiences. In addition, the assessor should verify that these competency 10 requirements are well set and maintained by the NRA 11

Metrics and statistics on the different activities performed as well as performance indicators can be used 12 for estimating the adequacy of the number of the assigned staff. The assessor should also verify that the 13 competency of the assigned staff is built, maintained and improved through induction as well as 14 continuous on-the-job training. 15


The assessor should check the following evidences more not limit to: 17

1. The number of human resources involved in each of the documented regulatory activities for 18 adequacy. 19

2. The systems/structures in place to ensure appropriate staff’s placement with respect to 20 competence and skills, and whether the system or structures have been implemented including the staff 21 records. 22

3. The professional profiles of the human resources engaged in regulatory inspection activities for 23 appropriateness of education, and skills/expertise to perform a particular function including the list of 24 requisite skills and training. 25

4. Recruitment plan. 26

Framework: 27

Process 28

Rating Scale: 29

NI: There is no enough staff to perform regulatory inspection activities AND/OR staff is not competent. 30

OI: The NRA has recently developed the plan to recruit adequate competent staff to perform regulatory 31 inspection, however the plan has not been implemented. 32

PI: The NRA recently recruited the adequate number of the staff to perform regulatory inspection 33 activities and their training programme or competency building activities is ongoing. 34

I: There is enough and competent staff (education, training, skills and experience) to perform regulatory 35 inspection activities. 36

Limitations and 37


Remarks The adequacy and appropriateness of the number of the staff is quite subjective and 1 should be linked to some process and/or output indicators. The assessor should consider the workload, 2 backlog, and delay in delivery based on established timeframes for estimation of staff adequacy. 3


References: 5

1. 1- WHO TRS 885, Annex 8, 1999 6




5. 4- ISO standard 9001:2015 10

Sub-Indicator: RI03.02: Duties, functions, and responsibilities of the staff, in charge of 11

regulatory inspection, are established and updated in the respective job descriptions.. 12

Scope: All 13

Maturity: 3 14

Requirement: 15

Duties, roles and responsibilities of the staff relevant to licensing activities. 16

Rationale: 17

The objective of this sub-indicator is to assure clear and well defined duties and responsibilities of the 18 staff in charge of regulatory inspection activities and their job descriptions are up to date with their 19 current duties, functions, and responsibilities, and adequately documented. 20

Description: 21

The assessor should verify that duties and responsibilities of different staff involved in regulatory 22 inspection activities are well established and defined in the respective job descriptions. Such format and 23 content should be established and implemented for all staff. The procedure should be supported with a 24 guidance document that provide direction on when and how to update the information, and where the 25 information is kept for easy access. The assessor should also check different job descriptions for 26 different positions within the hierarchical structure of the function or organizational chart. 27



1. The procedure and /or guidelines that guide placement of staff members within the NRA. 30

2. The professional profiles of staff (job descriptions), and relate it to their current roles/duties. 31

3. Procedures to guide the documentation of up to date duties and work schedules, and enforcing the 32 implementing of the documented guidelines /procedures. 33

4. Job description of designated staff. 34


Framework: 1

Process 2

Rating Scale: 3

NI: Staff duties, roles and responsibilities are not defined. 4

OI: The NRA started the process of defining the duties, roles and responsibilities of the staff in charge of 5 regulatory inspection function. 6

PI: The NRA has initiated implementation of this requirement but has not defined or followed for all staff 7 or the role and responsibility document including staff job descriptions are not up to date. 8

I: Staff duties, roles and responsibilities are well defined and documented in the respective job 9 descriptions. 10



References: 13

1. 1- WHO TRS 885, Annex 8, 1999 14




5. 4- ISO standard 9001:2015 18

Sub-Indicator: RI03.03: Training plan developed, implemented and updated at least 19

once a year. 20

Scope: All 21

Maturity: 3 22

Requirement: 23

Training plan and evidence of its implementation 24

Rationale: 25

The objective of this sub-indicator is to assure that the competence of staff in charge of regulatory 26 inspection activities is maintained and enhanced through an established training programme according to 27 a well-defined plan. Effective and efficient implementation of the regulatory inspection function is the 28 ultimate goal of this sub-indicator. 29

Description: 30

The assessor should verify that training plans are developed and implemented reflecting the current status 31 of education and experience of the staff. Assessor should ensure planning and implementation of 32 Induction training for new staff as well as continued on-the-job for staff on board for different regulatory 33 inspection activities. Continuous training program through participation in courses, seminars, workshops, 34


expert circle meetings, etc... is of vast importance and should be checked by the assessor. The assessor 1 should verify that the training plan is regularly updated, expectedly on annual basis. There should be 2 procedures to approve the training plan and the budget allocated as well as updating the plan. For 3 effective training programmes, training needs should be collected and thoroughly investigated prior to 4 development of training plan. The plan should present clearly defined training goals. This should involve 5 learning certain topics and /or skills to address deficiencies. Learning objectives, training 6 methods/activities, documentation / or evidence of learning and evaluation / assessment of training should 7 be documented. Measuring training effectiveness is another aspect which the assessor should check. The 8 assessor should verify that there is a system in place for monitoring the implementation and effectiveness 9 of the training plan along with the skills acquired in training activities for internal and external experts. 10

The assessor should verify that the minimum number of training days of inspectors in charge of GMP 11 inspections is 10 days per year as recommended by the relevant WHO, and also PICS, guidelines. 12 Furthermore, the NRA might need to establish the minimum number of days spent on inspections by 13 different inspectors in order to retain their competence. 14


The assessor should check the following evidences but not limit to: 16

1. The guideline for development, implementation and update of training plan at least once every 17 year as well as mechanism to measure training effectiveness. 18

2. The system/structures used to approve the training plan as well as the budget allocated to the 19 training activities for the purpose of adequacy. 20

3. The current/existing training plan/matrix of staff, and relate it to their respective job descriptions. 21

4. SOP for developing and maintaining the training plan. 22

5. Collection and/or investigation of training needs. 23

6. List of trainings performed. 24

7. Example records for training activities. 25

8. Number of training days per inspector as well as average training days. 26

9. Number of day spent on site for inspections per inspector as well as average number of days spent 27 on inspection. 28

Framework: 29

Process 30

Rating Scale: 31

NI: There are no requirements or training programs for staff involved in regulatory inspection. 32

OI: The NRA is preparing the requirements or position profiles and a training programme for regulatory 33 inspection activities, but there is no evidence of results associated with these activities. 34

PI: PI: The NRA has developed and initiated the training plan implementation. However, the NRA has 35 not fulfilled all required planned training and has been applying it for less than two years. 36


I: The NRA has a training plan for the staff in charge of regulatory inspection activities including (1) 1 induction training of new staff, (2) routine on-the-job training for recruited staff and (3), the effectiveness 2 of the training received is verified, and (4) the NRA consistently maintains documentation of the results 3 of related activities over time. 4

Limitations and Remarks: Training plans must be updated regularly, expectedly on annual basis, but not 5 beyond biennial (once every two years). 6


References: 8

1. 1- WHO TRS 885, Annex 8, 1999 9

2. 8- WHO TRS 902, Annex 8, 2002. Chapter 5.2 (d 10



5. 4- ISO standard 9001:2015 13

Sub-Indicator: RI03.04: The NRA performs and maintains records of staff training 14

activities. 15

Scope: All 16

Maturity: 3 17

Requirement: 18

Training records 19

Rationale: 20

The objective of this sub-indicator is to assure that training activities are recorded and the records are kept 21 up-to-date. Staff training records as an integral part of staff file is a tool for measuring and tracking staff 22 competency, development and adequacy. 23

Description: 24

The assessor should verify the implementation of the training plan and training activities through 25 checking of the relevant training records. Training records may include list of trainees, proof of 26 participation, training materials, training evaluation by the trainees, and training effectiveness evaluation. 27

The assessor should check there is an evaluation/ assessment mechanism to verify the quality of learning, 28 and to confirm whether learning objectives are achieved. An inventory (soft and/or hard) system to 29 document all impactful and non- impactful trainings, and members of staff who participated. A system to 30 measure /estimate impact of trainings should be established. 31



1. Guidelines or similar documents to guide the NRA to perform and maintain records of staff 34 training activities; 35


2. Training effectiveness evaluation; 1

3. The training inventory, and how the inventory is completed; 2

4. Records keeping or staff training archives and archiving system. 3

Framework: 4

Output 5

Rating Scale: 6

NI: There are no records for training activities of staff involved in regulatory inspection. 7

OI: Training activities are implemented as evident by the competency of the staff in charge of regulatory 8 inspection activities however such implementation is not well documented. 9

PI: Documentation of training activities of the staff in charge of regulatory inspection activities is recently 10 implemented however consolidated documentation over a time (at least two years) is not yet available. 11

I: Records for training activities of staff involved in regulatory inspections is available and maintained. 12

Limitations and Remarks: The assessor should note that some NRAs out-source training including staff 13 capacity development activities. In this case the assessor should request for the identity, as well as the 14 provider evaluation status. The assessor may request for the professional profiles of tutors or resource 15 persons used to offer training, as well as the records and archiving systems in place. 16


References: 18

1. 1- WHO TRS 885, Annex 8, 1999 19




5. 4- ISO standard 9001:2015 23

Indicator: RI04 Procedures established and implemented to perform 24

inspection and enforcement. 25

Rationale: 26

The objective of this indicator is to assure the NRA or Inspectorate is supported with appropriate 27 mandate, effective organization and governance, and essential resources. Different activities should be 28 implemented through standard procedures and work instruction in order to ensure consistency, 29 effectiveness, efficiency, impartiality and proportionality of the same. 30

The NRA should have established a system which allows the regular reviewing of quality aspects of the 31 authorized products on the market, including a process to review and share relevant data between key 32 players and taking appropriate action. 33


Category: 07. Regulatory process 1

2

Sub-Indicator: RI04.01: Different inspection activities for checking compliance with 3

GXP, as for inspection preparation, conduct and/or reporting is documented. 4

Scope: All 5

Maturity: 3 6

Requirement: 7

Regulatory process 8

Rationale: 9

The objective of this sub-indicator is to assure that all pre-inspection, inspection and post-inspection 10 activities including inspection planning, preparation (e.g. inspection announcement and inspectors' 11 designation), conduct (e.g. opening meeting, evidence collection), and reporting (e.g. classifying findings, 12 closing meeting, conclusion and recommendations) are well documented in order to enable regulatory 13 inspection of manufacturing and distribution operations to be carried out in accordance with the official 14 guidelines and national legislation and in accordance with a formal inspection plan, in order to ensure 15 consistency, effectiveness, impartiality and proportionality of practices. 16

Description: 17

The assessor should verify the existence of procedures for an efficiently working on planning, preparing, 18 conducting, reporting and monitoring of different GXP inspection activities throughout the supply chain. 19 The existence of documented procedures and resources to enable regulatory GXP inspection to be carried 20 out in accordance with the official guidelines and national legislation and in accordance with a formal 21 inspection plan. All instructions, standards or written procedures, worksheets, check lists and reference 22 data relevant to the work should be maintained up-to-date and be readily available to staff. As part of 23 inspection process and procedures the assessor should verify if inspection reports in accordance with 24 national or other requirements as appropriate. 25

In addition, the assessor should verify that reports are prepared in approved format, and signed and dated 26 by the relevant inspector. Ideally, the report should follow a standard format which in turn is in 27 compliance with the relevant WHO guidelines or other internationally recognized and accepted 28 guidelines. 29



1. GXP planning, conducting, monitoring, and performance SOPs and records; 32

2. GXP inspection reports and inspection review process (are analysed on a regular basis) and 33 follow-up; 34

3. instructions, worksheets, check lists and reference data relevant to the different inspection 35 activities. 36

Framework: 37


Process 1

Rating Scale: 2

NI: There is no evidence of activities, documentation for this indicator. 3

4

OI: The NRA is preparing to establish the procedures, so there is no assurance that there is a systematic 5 process. 6

7

PI: There is evidence that the NRA has the capacity to conduct the inspections mentioned in the indicator, 8 with procedures in place; however, its experience is limited or recent and therefore the relevant records 9 are limited. 10

11

I: The NRA demonstrates all the aforementioned elements and also consistently maintains documentation 12 of the results of related activities over time. 13



References: 16

1. 1- WHO TRS 885, Annex 8, 1999 17




5. 4- ISO standard 9001:2015 21

Sub-Indicator: RI04.02: Regulatory inspection follow up, decision- making including 22

certification and enforcement activities are documented. 23

Scope: All 24

Maturity: 3 25

Requirement: 26


Rationale: 28

The objective of this sub-indicator is to assure the availability and implementation of procedures related 29 to following up the regulatory inspection activities as well as regulatory decisions including certification 30 and enforcement actions. Evidence that actions were initiated, if needed, e.g. with regard to the marketing 31 authorization of products and/or the establishment license holders as well as relevant actions taken, if 32 applicable on deficiencies/violations to enforce compliance (including timeframes) and to issue notices on 33


the violations for the purpose of correct regulatory decisions make. Such documentation would in turn 1 contribute to consistency and effectiveness of the regulatory inspection function. 2

Description: 3

The assessor should check the availability and implementation of procedures related to following up the 4 inspection activities as well as regulatory decisions including certification and enforcement actions. There 5 should be written guidance e.g. on how to follow up deficiencies. The assessor should verify documented 6 evidence that actions were initiated, if needed, e.g. with regard to the marketing authorization of products 7 or the establishment license holders. The assessor should review examples of inspection reports, with 8 special attention to inspectors’ findings and observations relevant to the inspection, list of deficiencies, 9 recommendations, summary and conclusions. The assessor should find out results of relevant actions 10 taken, if applicable on deficiencies/violations to enforce compliance (including timeframes) and to issue 11 notices on the violations for the purpose of correct regulatory decisions make. 12

Any corrective or preventive actions taken as a result of audits or other reports of non-conformity are 13 implemented and documented and their effectiveness is verified. 14

The assessor should also verify the existence of internal procedures or mechanisms to ensure proper 15 related to GXP certification (where applicable). Evidence that a system in place for GXP certification. 16



1. SOP for GXP follow up 19

2. SOP for GXP certification 20

3. SOP for GXP enforcement 21

4. GXP certificates 22

5. GXP enforcement records. 23

Framework: 24

Process 25

Rating Scale: 26

NI: There is no evidence of activities and/or documentation for this indicator. 27

OI: The NRA is preparing to establish a procedure to document the processes mentioned in this indicator 28 but not yet implemented. 29

PI: There is evidence that the NRA has established the documentation relevant to the processes mentioned 30 in this indicator; however its experience with this is only recent or limited. 31





References: 1


2. 4- ISO standard 9001:2015 3



5. 1- WHO TRS 885, Annex 8, 1999 6

Sub-Indicator: RI04.03: Plan for inspections based on quality risk management is in 7

place. 8

Scope: All 9

Maturity: 3 10

Requirement: 11


Rationale: 13

The objective of this sub-indicator is to assure that planning of the frequency and scope of inspections is 14 based on quality risk management for enhancement of the inspection programme effectiveness and 15 efficiency. It is a methodology based upon the concept of rating sites on the basis of an estimated risk that 16 they may pose to patients, consumers, animals and users of medicines. The Inspectorates should prioritize 17 sites for inspection when planning the frequency and scope of GXP Inspections. 18

Description: 19

The assessor should verify the existence and implementation of Quality Risk Management (QRM) 20 throughout the process of inspection planning. 21

QRM scope is: 22

• Planning of routine GXP Inspections by the Inspectorates: 23

• Inspections of Investigational Medicinal Product (IMP) manufacturers ; 24

• Follow-up activities, such as assigning a new risk rating to the site following the receipt of new 25 information about the site or its products, this normally occurs between inspections and the types of new 26 information might include information on quality defects, product recalls, market surveillance test results, 27 among others. 28

This requires a complete knowledge of the GXP compliance status, it is considered that new sites should 29 not be rated for their initial inspection in accordance with this Quality Risk Management principles, 30 because the Inspectorate in question will not likely have the necessary and complete knowledge about the 31 site or products. Quality Risk Management should not normally be applied until a full inspection has 32 occurred. 33

The required inspection focus, depth, site and the duration together with the required number of 34 inspectors to be assigned and whether any specific competence or expertise will be required on the 35


inspection team when performing the next routine inspection are important questions to be answer during 1 inspection preparation process. Applying QRM principles, require collecting relevant data during 2 inspection preparation and is fundamental for risk based planning e.g. vigilance, lab, and earlier 3 inspection data. 4

The preparation requires the inspector to consider the following items before making his/her 5 recommendation: 6

The areas in which deficiencies were identified during the most recent inspection at the site, that were not 7 inspected and all new information such may relate to new quality defect reports, Marketing Authorization 8 variation applications affecting the site, product recall actions, non-conforming market surveillance test 9 results or other general indicators of non-compliance, such as a failure to implement a Marketing 10 Authorization variation on time. 11

Ideally, a meaningful and robust inspection plan based on QRM should be developed in conjunction with 12 other data of other regulatory departments (e.g. vigilance data, quality control data, market surveillance 13 data and others). 14



1. Annual (regular) inspection plan along with methodology for its development; 17

2. GXP inspection plan review process along with its update. 18

Framework: 19

Process 20

Rating Scale: 21

NI: There is no evidence of activities and/or documentation for this indicator. 22

OI: The NRA is preparing to establish procedures to plan inspections based on risk management but no 23 results are yet exist. 24

PI: There is evidence that the NRA has procedures to plan inspections based on risk management, as well 25 as the capacity to perform the procedures mentioned in the indicator; however its experience with this is 26 only recent or limited. If inspection planning has only recently been established, the rating should be PI. 27




References: 32

1. 4- ISO standard 9001:2015 33

2. 1- WHO TRS 885, Annex 8, 1999 34




5. 55- WHO TRS 981, Annex 2, 2013 2


7. 4- ISO standard 9001:2015 sub-clause 5.1.1 4

8. 123- ICH Q9 Quality Risk Management; 2005 5

9. 124-A recommended model for risk-based inspection planning in the GMP environment. PIC/S; 6 2012 7

Sub-Indicator: RI04.04: Multi-disciplinary team is used to ensure proper expertise for 8

inspection of specific medical products. 9

Scope: All 10

Maturity: 3 11

Requirement: 12


Rationale: 14

The objective of this sub-indicator is to assure the availability of the optimized human resources involved 15 in regulatory inspections which would contribute to effective and efficient implementation of the 16 function. 17

Description: 18

The assessor should verify the existence and implementation of a system for defining the composition of 19 the inspection team. The inspection team should be determined based on skills, experience in the type of 20 inspection required, After designate the lead inspector who coordinates, with the active collaboration of 21 the other members of the inspection team, the entire inspection process (i.e. from the planning to the 22 evaluation of the corrective measures. A follow-up inspection would trigger a new inspection process). 23 Sub-contracted personnel or experts may be employed as part of an inspection team to assist or advise in a 24 technical capacity, but that team should normally be led by a GMP lead inspector. Sub-contracted 25 personnel should be bound by the requirements of the quality system and there should be a written 26 contractual agreement between the parties. Persons or organizations to whom inspection activities are 27 contracted out and experts should be free from any commercial or financial pressures which might affect 28 their freedom to act. They should follow defined rules to avoid conflict of interests and regarding ethics. 29 Senior management of the Pharmaceutical Inspectorate should ensure that these persons are appropriately 30 qualified and experienced and that they are independent of any organizations which they might be asked 31 to inspect. 32

Important to designate the necessary experts based on QRM , to join the inspection team in case of 33 specialized highly sophisticated products e.g. vaccines, advanced therapy medicinal products (ATMPs), 34 others, in order to ensure consistency, effectiveness, impartiality and proportionality of inspection 35 process. The number of members of the inspection team is another aspect which should be considered for 36 optimal work of the team throughout the inspection process. 37




1. Procedure team inspections 2

2. Inspection plan versus team Inspection nomination 3

3. Inspection process review 4

4. Inspection records. 5

Framework: 6

Process 7

Rating Scale: 8

NI: The NRA does not use multi-disciplinary team for inspection of specific medical products. 9

OI: The NRA is taking steps to use multi-disciplinary team for inspection of specific medical products but 10 no results are yet available. 11

PI: The NRA recently established the practice to use multi-disciplinary team for inspection of specific 12 medical products however results of such practice is not yet consolidated. 13

I: The NRA regularly and consistently uses multi-disciplinary team for inspection of specific medical 14 products. 15

Limitations and Remarks: Inspection team may comprise experts from the quality control laboratory or 16 the reviewers or other personnel external to the NRA in order to ensure the access to certain specific 17 expertise; however, there is still a need for all members of the inspection team to be qualified as 18 inspectors with minimal inspection training (e.g. questioning techniques, writing observation, and others). 19


References: 21

1. 1- WHO TRS 885, Annex 8, 1999 22




5. 4- ISO standard 9001:2015 26

Sub-Indicator: RI04.05: Inspection findings/observations are categorized according to 27

quality risk management. 28

Scope: All 29

Maturity: 3 30

Requirement: 31

07. Regulatory process 32

Rationale: 33


The objective of this sub-indicator is to assure that quality risk management (QRM) consideration is not 1 limited to inspection planning, preparation and conduct but also to reporting which would significantly 2 contribute to inspection process efficiency and effectiveness including the categorization of the 3 findings/observations according risk approach. 4

Description: 5

The assessor should verify the existence and implementation of a quality system related to different 6 inspection processes and that inspection findings/observations are categorized according to quality risk 7 management following a systematic process as per WHO or other internationally recognized guidelines 8 (e.g. classification of observations as critical, major and other according to their actual and/or potential 9 impact on the patients/receivers. 10



1. Findings categorization procedure and criteria 13

2. Inspection reports review 14

3. Findings review versus categorization process and criteria. 15

Framework: 16

Process 17

Rating Scale: 18

NI: Inspection findings/observations are not categorized according to quality risk management. 19

OI: The NRA is taking steps to categorize inspection findings/observations according to quality risk 20 management but no results are yet available. 21

PI: The NRA recently established the practice to categorize findings/observations according to quality 22 risk management however results of such practice is not yet consolidated. 23

I: Inspection findings/observations are categorized according to quality risk management. 24

Limitations and Remarks: As per WHO guidelines (84), a critical deficiency may be defined as an 25 observation that has produced, or may result in a significant risk of producing, a product that is harmful to 26 the user. A major deficiency may be defined as a non-critical observation that: a) has produced or may 27 produce a product that does not comply with its marketing authorization and/or prequalification 28 application (including variations); b) indicates a major deviation from the GMP guide; c) indicates a 29 failure to carry out satisfactory procedures for release of batches; d) indicates a failure of the person 30 responsible for quality assurance/ quality control to fulfil his or her duties; e) consists of several other 31 deficiencies, none of which on its own may be major, but which together may represent a major 32 deficiency and should be explained and reported as such. A deficiency may be classified as other if it 33 cannot be classified as either critical or major, but indicates a departure from GMP. A deficiency may be 34 other either because it is judged as minor or because there is insufficient information to classify it as 35 major or critical. 36



References: 1


2. 4- ISO standard 9001:2015 3

3. 55- WHO TRS 981, Annex 2, 2013 4

4. 7- Effective Drug Regulation, 2002. Chapter 4.4 5

5. 1- WHO TRS 885, Annex 8, 1999 6


7. 123- ICH Q9 Quality Risk Management; 2005 8

8. 84-WHO TRS 996, Annex 4, 2016 9

Sub-Indicator: RI04.06: The same criteria are used for the inspection of domestic, 10

foreign, public and private facilities. 11

Scope: All 12

Maturity: 3 13

Requirement: 14


Rationale: 16

The objective of this sub-indicator is to assure that regulatory requirements are applied equally for all 17 inspections: domestic, foreign, public, or private. 18

Description: 19

The assessor should check that the same criteria in terms of regulatory requirements, actual inspection and 20 enforcement actions are applied equally for all enterprises: domestic, foreign, public, and/or private. In 21 order to assure that, the assessor is asked to perform a systematic review of inspection process with 22 several examples and compare. 23



1. Review inspection process and reports: domestic, foreign, public, or private and confirm that no 26 differences in application of regulatory requirements. 27

Framework: 28

Process 29

Rating Scale: 30

NI: There is no evidence of activities and documentation for this indicator. 31

OI: The NRA is preparing to establish the process as stated in the sub-indicator however no yet results are 32 available. 33


PI: There is evidence that the NRA has the capacity to perform the processes mentioned in the indicator; 1 however, it has only limited or recent experience with it. 2




References: 7

1. 1- WHO TRS 885, Annex 8, 1999 8


3. 4- ISO standard 9001:2015 10



Indicator: RI05Mechanism in place to monitor regulatory performance 13

and output. 14

Rationale: 15

The objective of this indicator is to assure the existence and implementation of a system or mechanism for 16 monitoring of regulatory performance and output as estimate of the effectiveness and efficiency of the 17 regulatory inspection function. 18

Category: 09. Monitoring progress and assessing outcomes & impact 19

Sub-Indicator: RI05.01: A database is established and regularly updated of all 20

establishments which may be subject to inspection, along with their relevant regulatory 21

decisions (certification and/or enforcement activities). 22

Scope: All 23

Maturity: 4 24

Requirement: 25

Monitoring progress and assessing outcomes & impact 26

Rationale: 27

The objective of this sub-indicator is to ensure consistency and traceability of regulatory inspection 28 activities through the establishment and maintenance of a database of all the inspected establishments 29 along with their regulatory decisions (certification and/or enforcement actions). 30

Description: 31

The assessor should check the existence of an updated database of all facilities and premises subject to 32 regulatory inspections throughout the supply chain as well as clinical trial sites. The database should be 33


comprehensive containing essential info of the inspected facility along with the inspection general 1 information, conclusions, and relevant regulatory decisions (certification or enforcement actions). 2 Furthermore, the assessor should verify that the databased is regularly updated as per a well established 3 procedures and/or mechanisms. 4



1. Inspected facilities database showing inventory of facilities along with regulatory decisions. 7

Framework: 8

Output 9

Rating Scale: 10

NI: There is no list or database of all establishments which may be subject to inspection along with their 11 regulatory actions (certification and/or enforcement). 12

OI: The NRA is preparing to establish a database, but there is no evidence of results from such activities. 13

PI: There is evidence that the NRA has the elements (procedures, documentation, management systems, 14 and information, etc.) and the capacity to perform the processes mentioned in the indicator, however, it 15 has only limited experience and/or a limited number of documented events. 16

I: There is list or database of all establishments which may be subject to inspection along with their 17 regulatory actions (certification and/or enforcement). Also, this list and/or database is regularly updated. 18

Limitations and Remarks: Some facilities may be subject to inspection without being licensed by the 19 NRA (e.g. clinical trial sites). However, these sites may not be licensed by the NRA but they are ideally 20 licensed by another governmental institution. Integration and communication within the government is 21 critical in the latter situation. 22


RI02.02 related to communication within the NRA and all stakeholders. 24

References: 25


2. 4- ISO standard 9001:2015 27



5. 1- WHO TRS 885, Annex 8, 1999 30

Sub-Indicator: RI05.02: Inspection reports are well archived and easily retrieved. 31

Scope: All 32

Maturity: 3 33

Requirement: 34



Rationale: 2

The objective of this sub-indicator is to assure that a mechanism or procedure is in place for defining a 3 standard format for inspection reports and set up an archival system so that reports are easily, effectively 4 and efficiently retrieved. 5

Description: 6

The assessor should verify that reports of inspections are well archived and easily retrieved. The assessor 7 should review how all the information collected during the inspection process is managed, how inspection 8 reports are registered and archived, in paper and/or electronic formats. 9



1. SOPs for archiving and retrieval of inspection reports 12

2. Space and suitability of the archive system 13

3. Time spent to retrieve inspection reports. 14

Framework: 15

Output 16

Rating Scale: 17

NI: Inspection reports are not properly archived and not easily retrieved. 18

OI: The NRA is in the process of establishing an inspection archival system. 19

PI: Inspection report archival system is recently established; however, consolidated documentation over a 20 time (at least two years) is not yet available. 21

I: Inspection reports are properly archived and easily retrieved. 22



References: 25


2. 4- ISO standard 9001:2015 27



5. 1- WHO TRS 885, Annex 8, 1999 30


Sub-Indicator: RI05.03: Inspection reports are subjected to a regular and robust review 1

by experts other than the designated inspection team. 2

Scope: All 3

Maturity: 4 4

Requirement: 5


Rationale: 7

The objective of this sub-indicator is to assure the existence of a systematic review of each individual 8 inspection report by a group of experts other than the designated inspectors who performed the inspection 9 activity and wrote the report. This targets objectiveness, fairness and un-bias. 10

Description: 11

The assessor should check the existence of a systematic review of each inspection report by a group of 12 experts other than the designated inspectors who performed the inspection activity and wrote the report. 13 The reviewing experts and/or committee may be internal or external provided that the necessary 14 confidentially and declaration of interest measures are considered as indicated in the transparency and 15 communication indicator. The review process should be technically efficient, thorough and deep. 16



1. SOP for GXP inspection reports review 19

2. Records of GXP inspection reports review. 20

Framework: 21

Process 22

Rating Scale: 23

NI: Inspection reports are not subjected to a regular and robust review by experts other than the 24 designated inspection team. 25

OI: The NRA is in the process to establish a regular and robust review by experts other than the 26 designated inspection team; however, such mechanism/process is not yet established. 27

PI: The NRA recently established a regular and robust review by experts other than the designated 28 inspection team; however, it is at the implementation stage, so this practice is not yet consolidated. 29

I: Inspection reports are subjected to a regular and robust review by experts other than the designated 30 inspection team. 31



References: 34




3. 84-WHO TRS 996, Annex 4, 2016 3

4. 1- WHO TRS 885, Annex 8, 1999 4

5. 4- ISO standard 9001:2015 5


Sub-Indicator: RI05.04: Inspection data/outcomes are systematically evaluated or 7

interpreted. 8

Scope: All 9

Maturity: 4 10

Requirement: 11


Rationale: 13

The objective of this sub-indicator is to assure a system is in place for checking and confirming the 14 effectiveness of the regulatory inspection programme through the systematic review, analysis, evaluation 15 and interpretation of the inspection reports. 16

Description: 17

The assessor should confirm that inspection reports and data including inspection findings and outcomes 18 are collectively reviewed, analysed, evaluated and interpreted in a systematic way to figure out gaps and 19 strengths as well as trends. It is worth mentioning that the output of this systematic review and analysis 20 represents the body of the inspection metrics. The benefit of the inspection metrics enabling the NRA to 21 identify the need for development and/or update of relevant guidelines, setting up and amending 22 inspection plans and resources, re-structure the format of inspection process among others. 23



1. Inspection data/outcomes analysis (inspection metrics). 26

Framework: 27

Output 28

Rating Scale: 29

NI: Inspection data are not systematically evaluated and interpreted. 30

OI: The NRA is in the process of establishing a system for inspection data evaluation and interpretation. 31

PI: The NRA recently established a system for evaluation and interpretation of Inspection data; however, 32 consolidated documentation over a time (at least two years) is not yet available. 33


I: Inspection data are systematically evaluated and interpreted. 1



References: 4


2. 4- ISO standard 9001:2015 6



5. 1- WHO TRS 885, Annex 8, 1999 9

Sub-Indicator: RI05.05: Performance indicators for regulatory inspection activities are 10

established. 11

Scope: All 12

Maturity: 4 13

Requirement: 14


Rationale: 16

The objective of this sub-indicator is to assure that a system or mechanism is in place for measuring the 17 performance of the regulatory inspection activities and regularly reviewed as part of quality indicators and 18 quality management review (QMR). 19

Description: 20

The assessor should confirm that a system or mechanism is in place to ensure that performance indicators 21 for regulatory inspection activities are established and regularly reviewed and are part of quality 22 indicators and quality management review - QMR. It is fundamental that the NRA promotes quantitative 23 and/or semi-quantitative KPIs, as example, no. of inspections per year, no. of inspection days, and 24 average no. of days to produce inspection reports, among others. 25

In general, quantitative indicators are preferred so as to avoid bias and/or misinterpretation. However, 26 qualitative indicators are also accepted (with or without scoring and/or scaling to render them semi-27 quantitative so more informative). 28

The assessor should ensure that indicators are measured on regular bases to monitor progress and 29 advancement. In addition, the assessor should verify analysis of the measured indicators to find out trends 30 and/or abnormalities including justifications and/or optimizations, if any. 31



1. Performance indicators procedure and process 34


2. Inspection data/outcomes analysis (inspection metrics). 1

Framework: 2

Output 3

Rating Scale: 4

NI: KPIs for regulatory inspection activities are not established. 5

OI: KPIs for regulatory inspection activities are established; however, are not measured or KPIs for 6 regulatory inspection are measured but not formally established. 7

PI: KPIs for regulatory inspection activities are well established and measured; however, are not 8 interpreted or no actions are taken when necessary. 9

I: KPIs for regulatory inspection activities are well established, measured, interpreted and actions taken 10 when necessary. 11

Limitations and Remarks: When they refer to outcomes, indicators may be ambiguous and difficult to 12 interpret, as they are the result of many factors that are difficult to disentangle. When they refer to 13 processes, indicators are often too specific, focusing on a particular intervention or condition or may 14 quickly become outdated as business models of care and technology develop. 15


References: 17


2. 4- ISO standard 9001:2015 19



5. 1- WHO TRS 885, Annex 8, 1999 22

6.23 125http://www.who.int/management/district/performance/PerformanceMeasurementHealthSyste24 mImprovement2.pdf. accessed 25

7. 126- http://www.euro.who.int/__data/assets/pdf_file/0009/244836/Health-System-Performance-26 Comparison.pdf. accessed 27

8. 115-ISO standard 9004: 2009 28

Indicator: RI06 Mechanism exists to promote transparency, 29

accountability and communication. 30

Rationale: 31

The objective of this indicator is to assure communication within and outside the NRA, as well as 32 transparency and outreach to the public as milestones reflecting the accountability of the NRA to its 33


mandate. Additionally these contribute to mutual understanding and involvement of all stakeholders 1 relevant to regulatory inspection activities. Consequently, confidence in the regulatory system is built. 2

Category: 08. Transparency, accountability and communication 3

Sub-Indicator: RI06.01: The list of inspectors is publicly available and/or the identity of 4

the designated team for each inspection is communicated to the relevant institutions 5

subject to inspections. 6

Scope: All 7

Maturity: 4 8

Requirement: 9

Transparency, accountability and communication 10

Rationale: 11

The objective of this sub-indicator is to assure confidence building and accountability of the licensing 12 structure(s) via the enhanced transparency: a mechanism to ensure the list and identity of inspectors 13 should be publicly available. 14

Description: 15

The assessor should verify that a proper mechanism is available, enacted and used together with internal 16 procedures to ensure that information related with the list and identity of the inspectors is publicly 17 available and/or the identity of the designated team for each inspection is afforded to the relevant 18 institutions subject to inspections. Such communication or information of designated inspectors may be 19 done via public communication aids (e.g. website, national gazette) and/or with each inspection visit via 20 letter of designation, inspection notification or other similar approaches. 21



1. Review the process for publishing the list of inspectors. 24

2. Comparison of inspectors’ list versus the publicly available list of inspectors. 25

3. Review of the process of communication of the identity of the designated team for each 26 inspection to the relevant institution subject to inspection. 27

4. Example of inspection notification, letter of designation or any other similar mechanism. 28

Framework: 29

Output 30

Rating Scale: 31

NI: The NRA is not notifying the inspectee of the designated inspection team and the list of the inspectors 32 is not published 33

OI: The NRA notifies the inspectee of the designated inspection; however, the list of inspectors is not 34 established, neither internally nor publicly available. 35


PI: The NRA notifies the inspectee of the with designated inspection team; however, the list of inspectors 1 is not published. 2

I: The NRA publishes an updated list of inspectors and a procedure for notifying/informing the inspectee 3 the designated inspection team. 4



References: 7


2. 4- ISO standard 9001:2015 9



5. 1- WHO TRS 885, Annex 8, 1999 12

Sub-Indicator: RI06.02: The updated list/database of all inspected facilities along their 13

regulatory decisions, actions and enforcement is regularly published and publicly 14

available. 15

Scope: All 16

Maturity: 4 17

Requirement: 18


Rationale: 20

The objective of this sub-indicator is to assure confidence building and accountability of the licensing 21 structure(s) via the enhanced transparency, through keeping the information related to regulatory 22 inspections available for the general public. 23

Description: 24

The assessor should check that a system is in place and well established to ensure that information related 25 to inspections performed is available for the general public. This particularly entails list of inspections 26 performed along with their conclusions, regulatory decisions, regulatory actions and enforcements 27 including recalls, production and/or license suspension, revocation, sanctions, prosecutions, and others. In 28 addition, the assessor should make sure that urgent regulatory decisions (e.g. related with quality, safety 29 or efficacy) and enforcement actions affecting the public health is immediately communicated to the 30 public to ensure high level of public health protection. 31




1. Latest publicly available list and/or database of inspected facilities along with their conclusions, 1 regulatory decisions, regulatory actions and enforcements. 2

2. Earlier publicly available list of inspected facilities. 3

Framework: 4

Output 5

Rating Scale: 6

NI: There is no publicly available list or database of all inspections performed along with their 7 conclusions, regulatory decisions, regulatory actions and enforcements. 8

OI: The NRA is preparing to establish a publicly available list or database of all inspections performed 9 along with their conclusions, regulatory decisions, regulatory actions and enforcements, but there is no 10 evidence of results from such activities. 11


I: There is a publicly available list or database of all inspections performed along with their conclusions, 15 regulatory decisions, regulatory actions and enforcements. 16



LI05.01 19

References: 20


2. 4- ISO standard 9001:2015 22



5. 1- WHO TRS 885, Annex 8, 1999 25

Sub-Indicator: RI06.03: Inspection metrics are regularly published and publicly 26

available. 27

Scope: All 28

Maturity: 4 29

Requirement: 30


Rationale: 32


The objective of this sub-indicator is to assure confidence building and accountability of the licensing 1 structure(s) via the enhanced transparency through a system or mechanism in place to inform the general 2 public on inspection metrics. 3

Description: 4

The assessor should check that a system or mechanism is well established to communicate information 5 related to inspections metrics or performance for the general public, and is regularly updated. 6

Considering the utilization of quality metrics as an input to inspection models as well as to predict 7 possible drug shortages, determine inspection schedules for a manufacturer, assess post-market change 8 reporting, and re-structure the format of inspection process. 9

Some examples of possible inspection metrics are suggested next: total number inspection, proportion of 10 the different inspections (routine, cause triggered, new product, new license, innovative pharma, generics 11 among others), number of critical – major – other findings, number of inspection findings based on the 12 inspection type, average number of findings reported by inspection type, average number of inspection 13 findings over time, among others, frequently quoted observations and others. Inspection metrics always 14 dependent on inspectorate mandate and activities planned. 15



1. Inspections metrics procedure and process 18

2. Review the process for published the inspections metrics or performance 19

3. Published inspection metrics. 20

Framework: 21

Output 22

Rating Scale: 23

NI: Inspection metrics are not regularly published and publicly available. 24

OI: The NRA is preparing to establish a publicly available inspection metrics, but there is no evidence of 25 results from such activities. 26


I: Inspection metrics are not regularly published and publicly available. 30



References: 33



2. 4- ISO standard 9001:2015 1



5. 1- WHO TRS 885, Annex 8, 1999 4

Sub-Indicator: RI06.04: Information on inspections conducted is regularly published 5

and publicly available in accordance with national confidentiality requirements. 6

Scope: All 7

Maturity: 4 8

Requirement: 9


Rationale: 11

The objective of this sub-indicator is to build confidence and accountability of the licensing structure(s) 12 via the enhanced transparency through keeping information relevant inspections performed and reports 13 publicly available. 14

Description: 15

The assessor should check that inspections performed and reports are available for the general public 16 relevant to regulatory inspection activities. The information which might be shared with the public varies 17 according to the national legislations. On one side, full inspection reports, redacted or non-redacted, might 18 be published. On the other side, inspection summaries (excerpts) might be publicly available. Both 19 approaches are accepted; however, the earlier one is encouraged. It is worth mentioning that publication 20 of inspection reports or summaries would significantly contribute to regulatory reliance and recognition in 21 the area of regulatory inspection. 22



1. Review the process for published the list of inspections performed 25

2. Review the list of inspections performed 26

3. Review the actual published Inspection reports or excerpts (summaries). 27

Framework: 28

Output 29

Rating Scale: 30

NI: Inspection report and/or summaries (excerpts) are not publicly available. 31

OI: The NRA established a procedure and/or mechanism by which relevant inspection reports and/or 32 summaries (excerpts) could be publicly available; however, this is not yet implemented. 33


PI: Inspection reports and/or summaries are publicly available only upon request (i.e. pull rather than 1 push publication). 2

I: Inspection reports and/or summaries (excerpts) are regularly and consistently published and publicly 3 available. 4



References: 7


2. 4- ISO standard 9001:2015 9



5. 1- WHO TRS 885, Annex 8, 1999 12

who global benchmarking tool (gbt) for evaluation … · 34 indicator: ri01 legal provisions,...

Documents