utilize efficient audit trail monitoring as a part of … · utilize efficient audit trail...

©2017 Waters Corporation 1COMPANY CONFIDENTIAL

Utilize Efficient Audit Trail Monitoring

as a Part of Data Quality Review Processes

Heather Longden

Snr Marketing Manager,

Informatics and Regulatory Compliance


Gathering & Sharing Regulatory Information

RegulatoryBodies

Industry GroupsISPE/GAMP

Customer QA teams

Sales and Specialists

Professional Services

Product Functionality and Design


Disclaimer

This presentation is for informational purposes only and should not be taken as

advice regarding any particular course of action to be followed.

Waters does not make any representations or warranties, express or implied, to

any party, regarding use of the information contained in this presentation to make

decisions regarding the implementation and maintenance of effective quality

control systems and quality assurance testing programs, including but not limited

to the applicable Good Manufacturing Regulations that apply to the manufacture

of regulated products.


Data Integrity

Orphan Data and IOOS concerns

• Including Empower Audit trails

Audit Trail features

Technical Error Logs

Designing Data Review including audit trails

Periodic Review

Documenting Data Review


Data Integrity






Periodic Review



Data Integrity/ˈdeɪtə ɪnˈtɛɡrɪti/


What is Data Integrity?


Why the New Focus on Data Integrity?

Provide the controls to prevent but also

capability to detect undesirable users actions

Tools for QA and regulators

– Access levels

– System polices

– Audit Trails

Electronic Systems Improve Traceability

Agencies have lost the trust

that analysts behave with

honesty and integrity


CSV

Technical Controls

Sharing Accounts

Delete Privileges

Unsecured Data

No Audit Trail

Inspection themes

Procedural Controls

All Data: Good and Bad

Poor Review of Electronic Data

including audit trails

Poor OOS or Lab Error

Investigations

OOS found in orphan data

DATA MANIPULATION


Website Q and A 2015, DRAFT Guidance April 2016 GMP Data Integrity, March 2015

GxP Data Integrity, DRAFT July 2016

Medicines & Healthcare Products Regulatory Agency (UK)

Pharmaceutical Inspection Co-Operation Scheme

PI-041-1 (DRAFT 2),August 2016

Released June 2016,as WHO_TRS_996 Annex 5 Q and A: August 2016

For GLP, April 2016

Data Integrity Guidances

Points to Consider Series: Conduct: March 2016

Fundamentals: Sept 2016Data Integrity: In Progress

EPA QA/G-8,November 2002

GAMP: RDI GuidePublished

April 4th 2017

Coming Soon


What’s involved in Data Integrity?

People

– Culture for data integrity

– Governance and data review

– Unique user accounts

– Scientific skill

– Training

– Safeguards against fraud

Quality Separations

– Quality standards & reagents

– Instrument calibration & maintenance

– Qualification

– Method validation

– System suitability

– Quality columns

IT Components

– Secure centralized storage

– Network qualification

– Disaster recovery plan

– Backup and restore process

Laboratory Computerized Systems

– Built-in data integrity controls

– Computerized system validation

– Traceability

– Periodic review


Data Integrity






Periodic Review



FDA Draft Data Integrity Guidance:Rejection of Data & Repeat of Analyses

Question 2: When is it possible to exclude cGMP data from decision making?

Any data created as part of a CGMP record must be evaluated by the quality unit as part of release criteria

– Electronic data…should include relevant metadata

To exclude data....there must be a valid, documented, scientific justification for its exclusion

– Guidance: Investigating Out-of-Specification (OOS) Test Results for Pharmaceutical Production

The requirements for record retention and review do not differ depending on the data format;

– Paper-based and electronic ..are subject to the same requirements.

Data Integrity and Compliance with CGMP Guidance for Industry

DRAFT April 2016


FDA Draft Data Integrity Guidance:System Suitability & Test Runs

Question 13: Why has the FDA cited use of actual samples during “system suitability”

or test, prep, or equilibration runs in warning letters?

FDA prohibits sampling and testing with the goal of achieving a specific result or to

overcome an unacceptable result

– e.g., testing different samples until the desired passing result is obtained

– This practice, also referred to as testing into compliance is not consistent with CGMP

We would consider it a violative practice to use an actual sample in test, prep, or

equilibration runs as a means of disguising testing into compliance.

All data should be included in the record that is retained and subject to review unless there

is documented scientific justification for its exclusion.


DRAFT April 2016


Orphan Data

Data (paper or electronic) found in the laboratory (or trash bins) which is not included in final study reports/ quality certificates/ LIMS or ERP reports

Without documented scientific reasons for its invalidation, all orphan data is suspected as – ‘deliberately excluded to make results look better”

– apple polishing or cherry picking

Minimizing any failed tests or results that require repeat analysis reduces the amount of orphan data to be reviewed and addressed

Root causes of failed tests may include:– Poorly developed or validated analytical methods

– Inconsistent column separation performance

– Sample, standard, reagent or mobile phase preparation errors

– Instrument failures

– Analyst error


All the data….. Is it complete?

Orphan Data

Initial Sample Set

Result Set Result(1)

Project A

Project B

Technical controls (project access and project creation) are important, other

technical controls may not exist

Sample Set

Result Sets

Result(3)

Result(2)

Result Set Result(1)

Only this data is

Reported

Initial Sample Set

Result Set Result(1) Data is not saved,

reviewed, invalidated or

reported


1. Your firm failed to ensure that laboratory records include complete data derived from all tests necessary to assure compliance with established specifications and standards (21 CFR 211.194(a)(4)).

Reliance on incomplete data

– Our investigator reviewed the audit trails generated by your high performance liquid chromatography (HPLC) system for impurities testing that you conducted ..The audit trail showed that you performed this testing in duplicate.

– The audit trail indicated that you conducted a chromatography sequence analyzing impurities on samples of these lots on April14,2014. The audit trail showed that a new sequence was started approximately 24 hours later, on April 15, 2014, for impurities testing that again included samples for (the same) lots

– None of the 19 chromatograms generated in the first sequence were maintained and available for review. Only the second set of chromatograms was maintained and relied upon in releasing lots ….for use in the manufacture of products for the U.S. market.

– You could not provide any rationale for not maintaining the original data, and you failed to document a scientific justification for repeating the analysis.

Failure to appropriately maintain data

– You do not maintain electronic data on your UV spectrophotometer..and it does not have an audit trail.

Retesting into Specification, not maintaining the original

records; evidence in Audit Trails


“Our investigator reviewed the audit trails generated by your high

performance liquid chromatography (HPLC) system for impurities testing that

you conducted ..The audit trail showed that you performed this testing in

duplicate.

The audit trail indicated…..”

Even if data is missing, unavailable, or has been deleted, traces of orphan data

may be found in the audit trails or other logs

Regular review of audit trails may reveal incorrect processing of data and help

prevent incorrect results from being reported and identify the need for

additional training of personnel; (From WHO

What does an audit trail add to data review?


Data Integrity






Periodic Review



NIST Handbook : Review of Audit Trails

From a NIST publication*

– Audit trails are a technical mechanism that help managers maintain individual

accountability. .

– Users are less likely to attempt to circumvent security policy if they know that their

actions will be recorded in an audit log.

– “Determine how much review of audit trail records is necessary”

* Introduction to Computer Security: The NIST Handbook 1989


Annex 11 Audit Trails

Audit Trails are PART of the meta-data about the data

– Audit trails tell us WHO did WHAT, WHEN automatically

– Audit trails tell us WHY as defined by the user

They have two primary purposes:

– Give a history to the data, to help decide if it can be trusted

– They should deter wrongdoing (think of CCTV/ surveillance cameras)

o Without any review, they are not a deterrent


Reasons for Change in Empower

Empower automatically records what was changed, by whom and when

Reasons explain why users performed an action,

– such as: optimized peak width and threshold for better peak detection

– using older version of method to analyze retain sample.

Reasons created in advance

save users time

– users can select an appropriate

reason from a drop down list

– do not need to create their own

reason.


Reasons for Change in Empower

Restricted Comments

– User selected from a drop don list

– ensures that reasons recorded within the audit trails are consistent.

‘Unrestricted’ Comments

– Allows for more freeform, descriptive or accurate capture of the exact reason for an

action

– If set to “Unrestricted’ either freeform or drop down list reasons could be used

Confirm Identity requires password

as well as reason

– Additional level of security of user

identity

– NOT a electronic signature


Common mistakes

Giving free rein and no guidance for reasons to be used

– Users entering their user names

– Users entering WHAT they did rather than WHY

No review of proper use of reasons

QA role,

– To advise where and how peers should review the reasons

– To suggest predefined reasons where appropriate

o Ensures common reasons are used

o Permits searching for reasons

– To periodically review the reasons being used


Each Application has unique Audit Trail /History design

System Level

System Audit Trail

Archived System Audit

Trail

Project level

Summary level audit trail

Methods

Sample Set

• History

• Compare

• Sample History

Instrument

• Audit trail

• Compare

• Acquisition Log

Processing

• Audit trail

• Compare

Result Sets

Chromatograms

Calibration Curves

Individual results

Manual results

All calculated peak values

Signed off Reports

Summary of Results

Created by Empower

Verify againste-data

What was approved


Recording changes during acquisition

• Record human initiated changes to instrument

parameters during acquisition

• Permanently tied to the channel/chromatogram


CDS Audit Trails

Not always in a table labelled “Audit Trail”

– Can it be exported / printed into secure formats?

– Does an export preserve the secure fidelity?

– If imported to Excel how can its completeness be proved?

Details of change (incl before and after values / reasons etc) may be in lower level

change histories

Sometimes the only sure way to see changes is to compare two content items.

Are all audit trails archived?

– With data?

– In a separate procedure?

– To a completely secure format /location?


Data Integrity






Periodic Review



ISPE RDI Guide: Audit Trail vs System Logs

Data audit trails normally record the creation, modification, or deletion of records and

data.

Technical system logs normally record various system, configuration, and operational

events.

Technical system logs should not be regarded as equivalent to data audit trails. This

distinction is consistent with existing regulations and normal IT good practice and

terminology.

Where systems lack appropriate audit trails, alternative arrangements to verify the accuracy of data

should be implemented, e.g., administrative procedures, secondary checks, and controls.

Technical system logs may be helpful, e.g., in case of investigations or in the absence of

true audit trails.

Records and Data Integrity GUIDEPublished April 4th 2017


Audit Trails vs. Error Logs

Audit Trails are designed to record changes to records by humans:

– Record CREATION, MODIFICATION and DELETION of data

– Who changed what (including before and after values), when and why

– Critical to record authorized or unauthorized changes to data

Error Logs

– NOT AN AUDIT TRAIL but still could contain useful information beyond simply data

creation and modification

• Missing ‘before and after values’ ‘who’ and ‘why’


Error Logs

Eg Message Center in Empower

Instrument /System messages from either the instrument, Empower software or

linked toolkit applications

Very useful for troubleshooting: ‘What’ happened ‘when’

– Often ignored by laboratories in daily use

– Persistent messages should be investigated and addressed

Complete story or root cause requires

– All information in the messages including users, times and details

– All associated messages generated at time of error or action

– Detailed review of any data or method referenced by the message

– Corroboration with the analyst or instrument activities in other records


Data Integrity






Periodic Review



WHO Guidance: Reviewing Electronic Records Summary

A PDF or printout is fixed or static and the ability to expand baselines, view the full spectrum, reprocess and interact dynamically with the data set would be lost in the pdf or printout

Data integrity risks may occur when persons choose to rely solely upon paper printouts or PDF reports

– If the reviewer only reviews the subset of data provided as a printout or PDF, these risks may go undetected

Paper printouts of original electronic records from computerized systems may be useful as summary reports …verify that the printed summary is representative of all (electronic)results.

A risk-based approach to reviewing data requires process understanding and knowledge of the key quality risks.. requires understanding of the computerized system, the data and metadata and data flows.

Guidance on Good Data and Record Management PracticesReleased June 2016 As WHO_TRS_996 Annex 5


MHRA Draft GxP Guidance:Reviewing Electronic Records Summary

Data may be…

– Static (e.g. a ‘fixed’ record such as paper or pdf) or

– Dynamic (e.g. an electronic record which the user / reviewer can interact with).

Data must be retained in a dynamic form where this is critical to its integrity or

later verification.

(Once printed) chromatography records lose the capability of being

reprocessed and do not enable more detailed viewing of baselines or any

hidden fields.

GxP Data Integrity Definitions and Guidance for Industry

DRAFT July 2016


FDA Draft Data Integrity Guidance:Reviewing Electronic Records Summary

Static is used to indicate a fixed-data document (such as a paper record or an

electronic image), and

Dynamic means that the record format allows interaction between the user and

the record content.

– But defines as allowing the reviewer to change/edit things…???

(Printouts allowed if) includes associated metadata and the static or dynamic

nature of the original records

– Electronic records from certain types of laboratory instruments are dynamic records,

and a printout or a static record does not preserve the dynamic format


DRAFT April 2016


FDA Draft Data Integrity Guidance:

Reviewing Audit Trails

Question 7: How often should audit trails be reviewed?

– reviewed with each record and before final approval of the record.

– BUT: does not apply to all audit trails??

o include, but are not limited to, the following:

• the change history of finished product test results,

• changes to sample run sequences,

• changes to sample identification,

• changes to critical process parameters. (not “processing” parameters)

– routine scheduled audit trail review based on the complexity of the system and its

intended use.

Question 8: By WHOM?

– Personnel responsible for Record Review


DRAFT April 2016


MHRA draft GxP Guidance:

Audit Trail Review Summary

are not expected to implement a forensic approach to data checking on a routine basis

design and operate a system which provides an acceptable state of control

a paper based audit trail to demonstrate changes to data will be permitted.. where they

achieve equivalence to integrated audit trail.. or replace by end 2017

not necessary for audit trail review to include every system activity

Routine data review should include a documented audit trail review

– Could be through an exception report.. abnormal data which requires further attention or

investigation

QA should have sufficient knowledge…to review relevant audit trails, raw data and

metadata as part of audits

GxP Data Integrity Definitions and Guidance for Industry

DRAFT July 2016


WHO Guidance:


…may include discrete event logs, history files, database queries or reports

Regular review of audit trails may reveal incorrect processing of data and help prevent

incorrect results from being reported and identify the need for additional training of

personnel;

All GxP records held by the GxP organization are subject to inspection by health authorities. This includes original electronic data and metadata, such as audit trails.

Risk based…frequency, roles, responsibility and approach

.. periodic review of audit trails that track system maintenance activities,

…audit trails that track changes to critical GxP data..would be expected to be reviewed each and every time the

associated data set is being reviewed and approved – and prior to decision-making.



PIC/s Guide

PI 041-1 (Draft 2) 10 August 2016

Audit trails records should be in an intelligible form and have at least the following

information:

– Name of the person who made

– the change to the data;

– Description of the change;

– Time and date of the change;

– Justification for the change;

– Name of any person authorising the change. (this is new and not in other laws or guidances)

Good practices for data management and integrity in regulatedGMP/GDP environments PI-041-1 (DRAFT 2), August 2016


Reviewing Audit Trails

Purpose

– To uncover possible cases of fraudulent behaviour- Specifically OOS results in Orphan Data

o Multiple processing data

o Altering metadata to make results pass

o Hiding or altering meta data on reports sent to QA

o Uncovering persistent suspicious behaviour around security of data

o Ensuring only authorised users have access to certain functionality

o Deletion of data

o Altering system policies /configuration / settings without change control procedures

How?

– Two possible procedures:

o Review of relevant audit trails as part of data review

o Periodic review of system level audit trails by admin


Review of Audit Trails

Review audit trails as part of data review process

– Find anomalies before batch release

– Focus of user behaviour that affect results

– Peer Review / Manager review / QA review?

Periodic Review of overall/system level audit trails

– system level activity without correct documentation, change control, testing or approval

o e.g.. changing system policies, user access or deletion of data

Inspectors WILL look at the audit trails in electronic data systems

– and are likely to ask for an export or printout…….

Biggest Issue: Audit trails are often more a log of all activity

(to comply) and not designed for easy review


Review Audit Trails ElectronicallyPrint Audit Trails

Use the tools ( if any) built into the CDS

Review as PART of the data/integration

/method review

Write a clear SOP defining which audit

trails to review and when

– Only flagged or suspicious results?

Signing results includes declaration of

electronic review

Review of Audit Trails

Include data relevant audit trails in regular

reports

Periodically print out System level audit

trails to “review”

Sign reports as “evidence” of review


Adding Audit Trails to Reports


Risk based Approach to Data and Audit Trail Review

Identifying the higher risk results using IDs, codes and unique metadata

Monitoring consistency with IDs and DB relationships


Traceability offered with Relational DB designed CDS


The Review Tool:

Reviewing Data Electronically leveraging DB relations

Access to integrated chromatograms /results

– All integration positions

– Ability to zoom in to examine without reprocessing

Peak and Result level values

Used Instrument Method

Used Processing Method

Calibration Curves

Direct access Sample Set

and Sample Set History

Access to result audit trails with Result Audit Viewer

Direct connection to Preview for Sign Off capabilities


Data Review SOP suggestions

Should be performed on ELECTRONIC data in the application at least at Peer Review level

– Not relying on paper /pdf or Empower reports entirely

Define a Process

Look at final results (summaries, averages, CofA)

– Work back through the data from final quantitation, to areas and integration to SampleSet meta data to

audit trails

Specifically focus on suspect data

– Define a list of warning signs..

o Manual integration / multiple results / metadata changes

o Results that only just meet specification


Data Integrity






Periodic Review



MHRA Draft GxP Guidance:Periodic Review

Even if a system is “configured to prevent data manipulation (prevention)”

– “does not restrict a person from repeating the process by manipulating data to achieve a

desired result”

– Periodic reviews. May reduce risk from repeated events

Periodic Audits are capable of detecting opportunities for data integrity failures

Periodic system review might

– Assess the effectiveness of …organizational and technical measures

– verify the effectiveness of existing control measures

– consider the possibility of unauthorized activity

GxP Data Integrity Definitions and Guidance for Industry, DRAFT July 2016


Periodic Review

It’s like an internal audit on the compliance of the system

– Find concerns BEFORE the audit

– Find ways to improve the efficiency of systems and processes

– Documented evidence of actively searching for data integrity issues

– Eg Review System Audit Trail for correct use of Admin functionalities

Review major and minor changes to determine if any retesting or additional testing of

new functionality is required

– Has it significantly expanded or changed use

– Is the system still in control and in a validated state?

How often?

– Frequency may depend of maturity and criticality (3-18monthly)

A formal report must be written about the review

– Its a regulatory requirement


Each Application has unique Audit Trail /History design

System Level

System Audit Trail

Archived System Audit

Trail

Project level

Summary level audit trail

Methods

Sample Set

• History

• Compare

• Sample History

Instrument

• Audit trail

• Compare

• Acquisition Log

Processing

• Audit trail

• Compare

Result Sets

Chromatograms

Calibration Curves

Individual results

Manual results

All calculated peak values

Signed off Reports

Summary of Results

Created by Empower

Verify againste-data

What was approved

Periodic

Review

candidates


What might be of interest in a System Audit Trail?

Deleting project data only by designated administrators

Creating projects only by designated users

Regular archiving of projects / altering access or status of projects

Altering System Policies

User creation patterns

Password resetting activity

Unauthorised access to system (incorrect username or password)

Alteration of systems

Changes to user types

Unusual patterns /activity of Access to Empower ( ie unusual times)

Check on performance of IQ (Warning, Error)

Archive and Removal of Audit Trail

In the Empower Help: Actions recorded in system audit trails


Laboratory Analytics to aid Periodic Review

• Integrates professional services with data driven laboratory analytics tools

• Provides unprecedented visibility into the utilization, efficiency, compliance and financial needs of your

laboratory’s instruments and people.

Basic set of analytics in an easy-to-use dashboard in Empower 3 FR2.

Enhanced Dashboards (prebuilt and tailored) for continuous monitoring

Point in time extraction and targeted reporting service

Empower Analytics

Tailored and

Targeted Reports

Waters

Laboratory

Analytics

EDS 365 Laboratory Analytics


Built into Empower; Prebuilt Analytical Dashboards

– System summary

– System usage

– Project usage analysis

– User analysis (optional)

– Methods analysis

CDS Software Laboratory Analytics

Empower 3 Laboratory Analytics Allows You to:

Access critical system usage information

Identify training needs

Identify error messages that affect your workflows

Identify non-robust processing methods

Plan for capital expenditures

Identify opportunities to shorten run times with UPLC® technology

New in Feature Release 2


Data Integrity






Periodic Review



WHO Guidance:


…may include discrete event logs, history files, database queries or reports

Data review should be documented.

– For electronic records, this is typically signified by electronically signing the electronic data set

that has been reviewed and approved.

Paper printouts of original electronic records from computerized systems may be useful as

summary reports …verify that the printed summary is representative of all

(electronic) results.



PIC/s Guide

PI 041-1 (Draft 2) 10 August 2016

“Where data summaries are used for internal or external reporting, evidence

should be available to demonstrate that such summaries have been verified in

accordance with raw data.”

So the action of signing a report, is NOT to document that you reviewed the

content of the report ONLY

– But that you agree that the content or conclusions of the report match the content of the

electronic records you reviewed

Good practices for data management and integrity in regulatedGMP/GDP environments PI-041-1 (DRAFT 2), August 2016


How to document Data Review including Audit Trails

Review chromatograms, methods and relevant Audit Trails in Empower application

Document that process by SIGNATURE: WHO Guidance

– Sign a report to document that you have followed the review SOP

SOP should document what to review (including manual entries) and how it should be done by

your role

Similar to other laboratory tasks where there is no proof of the activity (such as

making mobile phases or sample preparation) other than a user attesting to their

completion of the task

I sign this data to attest that I performed/ reviewed / approved this data according to SOP 12345


Industrial Augmented Reality:

Ultimate Audit Trail/Recording..

Apprentice Field Suite

As a CMO, our customer can be challenged with various client demands at any given time. To provide

customer oversight prior to AFS, clients routinely flew to this facility to monitor their process.

Now this organization offers remote auditing through our hands-free application.

With AFS the CMO’s clients can see their process in real-time,

significantly reducing travel costs and wasted time

https://apprenticefs.com/industrial-augmented-reality-telepresence/


Summary


Ensuring Adherence to Rules

What makes me drive slower/safer?

– Technical Controls?

– Intense “Highway Review” procedure?

– Consequences?


Summary

Computer audit trails should be automatically created and not be tampered with

Most laboratory systems now are equipped with audit trails

– But do you EVER review them

How much review will ensure correct user behaviour

Not practical to attempt on paper: need electronic review

Don’t let the regulators be the first people to look at the electronic audit trail

0% 100%

% A

udit T

rail R

evie

w


Questions/Discussion

[email protected]

Waters Corporation

34 Maple Street

Milford

MA 01757

(508) 482 2000

utilize efficient audit trail monitoring as a part of … · utilize efficient audit trail...

Documents