yokogawa electric corporation exaquantum/batch validation planning guide copyright © yokogawa...
TRANSCRIPT
Yokogawa Electric Corporation
Exaquantum/Batch Validation Planning GuideCopyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Industrial Automation Systems Division
Exaquantum/BatchValidation Planning Guide
Page 2Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Purpose
Provide Yokogawa sales, marketing and project teams with recommendations on how to validate Exaquantum/Batch systems that conform with 21 CFR Part 11
Target audience:– Internal sales, marketing, project teams
• Serves as starting point for project specific documentation
Page 3Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Introduction
Purpose:– Provide Yokogawa sales, marketing and project teams with
recommendations on how to validate Exaquantum/Batch systems that conform with 21 CFR Part 11
There is no one solution that will work for all applications
Recommendations are a starting point and reflect Yokogawa’s opinion for best practices
Recommendations can be used in creating user requirement specifications and project specific implementations
Assumptions used for preparing this presentation– Reader is familiar with:
• Exaquantum/Batch features & functions• Stelex’s Exaquantum/Batch Part 11 Whitepaper
– This document is not meant to supersede our customer’s corporate policies and procedures regarding Part 11
Page 4Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Background Material
U.S. FDA Regulations - http://www.fda.gov/– Predicate Rules that apply for each process– 21 CFR Part 11
Corporate policies & procedures for using Part 11
GAMP 4– Good Automated Manufacturing Practices– Published by ISPE, www.ispe.org
“Achieving 21 CFR Part 11 Compliance using Exaquantum/Batch” whitepaper by Stelex validation consultants
“Use of Exaquantum/Batch in Applications Regulated by FDA 21 CFR Part 11” Exaquantum/Batch help topic
Part 11 Preamble– Provides background information not in the actual rule
FDA guidance document “Part 11, Electronic Records; Electronic Signatures – Scope and Application” published September 3, 2003
Page 5Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Validation
Process used to provide evidence of a high degree of assurance that a facility will consistently operate correctly– Process Automation Systems and Plant Information Systems are parts of the
facility• The facility is validated
• Validating individual systems is part of validating the facility
Traditionally validation consists of:DQ = Design Qualification
• “Documented verification that a facility, system and instrument are designed according to written and pre-approved specification (URS)” [Design Review]
IQ = Installation Qualification• “Documented verification that a system is installed according to written and pre-
approved specifications” [GAMP 4]
OQ = Operational Qualification• “Documented verification that a system operates according to written and pre-
approved specifications throughout all specified operating ranges” [GAMP 4]
PQ = Performance Qualification• “Documented verification that a system is capable of performing or controlling
the activities of the process it is required to perform or control, according to written and pre-approved specifications throughout all specified operating ranges” [GAMP 4]
Page 6Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
21 CFR Part 11
U.S. government regulation– Applied worldwide when products are to be sold in the U.S.
– Other countries and regions have similar regulations
Established criteria under which electronic records and electronic signatures are considered the equivalent of paper records and handwritten signatures– Potential for cost savings by reducing quantity of paper records
• Cost reduction by space saving and record management e.g. Saving of paper consumption (ISO14001) and management cost
• Saving of seaching and looking for time of necessary records• Prevention of deterioration and damage of records compared paper
based one
Part 11 requirements are intended to:– Make it difficult to falsify records
– Ensure strict data integrity so electronic records are trustworthy
– Ensure that electronic signatures cannot be readily repudiated by the signer
Page 7Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
When to apply 21 CFR Part 11
21 CFR Part 11 does not require records to be saved or signed– Predicate rules require records and signatures
When predicate rules require records and signatures
and a company decides to store them electronically,
then Part 11 is applied to determine the requirements for handling the electronic records and electronic signatures
Even when electronic records and signatures are not used, manufacturing plants must still be validated.
Page 8Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Predicate Rules
Rules that require records to be kept – May require signatures
When a predicate rule requires a record be saved or signed and it will be stored electronically then 21 CFR Part 11 is applied to determine the requirements for the storage of the electronic record and rules for the electronic signature
U.S. FDA Regulations - http://www.fda.gov/– “Predicate Rules” must be understood for each process
• Used to identify electronic records requirements• Used to identify electronic signature requirements• Used in risk analysis to determine if Part 11 applies to the
Exaquantum/Batch system• For Exaquantum/Batch typically include Good Manufacturing
Practices 21 CFR 210 and 21 CFR 211
– Sec. 211.182 Equipment cleaning and use log:» “…The persons performing and double-checking the cleaning and maintenance
shall date and sign or initial the log indicating the work was performed.”
Page 9Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Validating Exaquantum/Batch Systems
Exaquantum/Batch was designed for use in applications that require use of the U.S. Food & Drug Administration’s 21 CFR Part 11 regulation
Considerable work is required to validate any computer system used in a FDA regulated environment. – No pre-prepared list of recommendations or actions will be
acceptable for all companies and installations.
– This set of recommendations may be used as a starting point for each Exaquantum/Batch installation.
Each installation must be validated individually– No vendor can provide a pre-packaged validated solution
– Yokogawa will work with customers to validate each Exaquantum/Batch installation
The first step in validating an Exaquantum/Batch system is:
PLAN
Page 10Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 11Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Risk assessment
FDA recommends need for Part 11 controls be based upon a risk assessment. From Final Part 11 Guidance August 2003:
“We suggest that your decision to validate computerized systems, and the extent of the validation, take into account the impact the systems have on your ability to meet predicate rule requirements. You should also consider the impact those systems might have on the accuracy, reliability, integrity, availability, and authenticity of required records and signatures. Even if there is no predicate rule requirement to validate a system, in some instances it may still be important to validate the system.
We recommend that you base your approach on a justified and documented risk assessment and a determination of the potential of the system to affect product quality and safety, and record integrity. For instance, validation would not be important for a word processor used only to generate SOPs.”
Section III, C, 1 Validation, Lines 222-232
Page 12Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Risk assessment
Most processes controlled by PASs will be high risk– This means a deviation can have large negative impact on
product quality
– Will require validation and enforcement of Part 11 based on predicate rules
– This is not a reflection on PAS system quality
End user responsibility– Usually corporate policy dictates who, when, how risk
assessments are performed.
Yokogawa may assist with risk assessments– However risk assessment will go beyond PAS boundary and be
process specific
Page 13Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 14Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Project Plan
Project plan is required:– Identifies tasks
– Used to develop schedule
– Required for validation
GAMP 4– Good Automated Manufacturing Practices
– Published by ISPE• ISPE = International Society of Pharmaceutical Engineers
– Widely recognized by regulators and pharma companies as basis for manufacturing practices
– All project teams must have access to and training in GAMP 4
Page 15Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Project Plan – V Model
All Exaquantum/Batch projects should follow the GAMP 4 “V Model”– Required for cGMP– Recommended as a good practice for non-regulated projects
PlanningURS
FS
DS IQ
OQ
PQ
Verifies
Verifies
Verifies PerformanceQualification
OperationalQualification
User RequirementSpecification
InstallationQualification
FunctionalSpecification
DesignSpecification
System Build
Verifies
This Planning Guide is targeted to support User Requirements Specification
Page 16Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Project Plan – GAMP 4 Software Categories
Exaquantum/Batch – Category 4 Configurable Software Package
Major Exaquantum/Batch sub-systems:– Automatic Batch Data Collection (ABDC)
• Category 3 Standard functions• No configuration
– Custom Batch Data Collection (CBDC)• Category 4 Configurable Software Package• Configuration required
– BatchWeb• Category 3 Standard functions• No configuration
– Reporting• Category 4 Configurable Software Package• Complex reports may be considered Category 5 Custom (Bespoke)
Software
– Custom MES Applications• Category 5 Custom (Bespoke) Software• Not a standard function therefore does not impact product
categorization
Category
Software Type
1 Operating System
2 Firmware
3 Standard Software Packages
4 Configurable Software Packages
5 Custom (Bespoke) Software
GAMP 4 Software Categories
Page 17Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Project Plan – Configurable Software Package
Validation approach– Record version (and configuration of environment) and verify
operation against user requirements• Windows 2000 and Windows 2003 Server are Category 1 Operating System
– Typically qualified for use by customer– “Challenged indirectly by the functional testing of the application” (GAMP 4)
– Vendor audit is welcome• Master records help by QA group in Tokyo
– Manage any custom (bespoke) programming as Category 5• Typically this would be complex reports performing functions outside of
Exaquantum/Batch– Examples:
» File system access» Access to other databases
SQL Server 2000 is considered an embedded component of Exaquantum/Batch– Does not have to be separately validated
DCS, SCADA and MES systems are typically considered Category 4
Page 18Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 19Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Master Validation Plan
Customers will prepare a master validation plan– Depending upon the scope of the project this may focus on an
entire facility or the Exaquantum/Batch system
Master Validation Plan provides overall description of the project’s goals and the method to validate they can be achieved– State overall objectives, philosophies and approaches
– State, or reference, validation and Exaquantum/Batch system terminology and concepts required to understand required tasks
– State validation methodology to be used
– State major assigned tasks and responsibilities
– Establish means for creating and maintaining documentation developed during the validation process
Typically created by customer or their contractor– Yokogawa may assist with the Master Validation Plan
• As a minimum provide description of the Exaquantum/Batch system
Page 20Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 21Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
System Architecture
Scalability
Hardware configuration
System software– Part 11 license
Network architecture– OPC
– Redundancy/Fault Tolerance requirements
– Failure scenarios
– Time considerations
Page 22Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Scalability
Requirements drive system configuration– How valuable is the data?
– Reliability vs. Cost trade-off
Scaling options– Centralized Configuration
• All on 1 computer• Suitable for very small systems & for
development/training systems
– Distributed Configuration• Different options • Driven by requirements
Deployment Components:• Exaquantum/Batch
Server • Database Management
System • Web Server• Administration Tools
BatchWeb may be accessed from any computer
Page 23Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Centralized Exaquantum/Batch System
V-net or Vnet/IP
Control System LAN
FCS
CS Batch 3000
ExaopcStation
ENGMaster Recipe Storage
HIS Batch ServersHIS
Plant Office LAN
. . .
Corporate LAN/WAN
. . .
. . .
Exaquantum/Batch Serverwith Databases, Web Server& Administration Tools
BatchWebUsers
BatchWeb Users
Switch
Internet
Internet
Switch
BatchWeb Users
Corporate Firewall
Page 24Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
FCSCS Batch 3000
Exaopc Stations
Distributed Exaquantum/Batch System
HIS Batch Servers
HIS
Plant Office LAN
. . .
Corporate LAN/WAN
Exaquantum/Batch Server
ENGMaster Recipe
Storage
Administration Tools& BatchWeb Users
. . .
BatchWebUsers
BatchWebUsers
BatchWeb Users
Exaquantum/BatchWeb Server
Switch
Switch
Corporate Firewall
Internet
Internet
. . .
BatchWeb Users
. . .
BatchWeb Users
Exaquantum/Batch Database Server
Control System LAN
V-net orVnet/IP
Page 25Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Standard Deployment Options
Installation program supports:– Exaquantum/Batch Server & Database Management System on 1
computer
– Administration Tools may be installed on Windows 2000/2003 Professional or Server
– Web Server may be installed on its own computer
Project customization is required to install the Database Management System on a separate computer– Not a major effort
Page 26Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Deployment Recommendations (1 of 2)
Exaquantum/Batch Server & Database Management System on 1 computer– Sufficient for most applications
– Rule of thumb when to separate:• When number of concurrent BatchWeb or 3rd party database connections
exceeds 100 and number of concurrent batches exceeds 20
Separate Web Server from data collection & storage– Separate when expecting more than 100 registered BatchWeb users
and 3rd party database connections
– Separation benefits• Separates BatchWeb data requests do not content with data collection
tasks
• Increased cyber-security – Intruders look for web servers first, separation provides additional level of
isolation for data collection functions
• Easier to have separate DBA (Database Administrator) and Web Server Administrator
• Easier to apply Microsoft updates to the web server computer
Note: Rules will vary with computer specifications
Page 27Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Deployment Recommendations (2 of 2)
Administration Tools– Always load on Exaquantum/Batch Server
• For initial configuration and when restricting network access during disruptions
– Primary use of Administration Tools should be on remote computers
• Lessens load on Exaquantum/Batch Server– Less competition with data collection for memory and CPU utilization
• Access Control – Different Administration Tools can be enabled on each computer
» System Configuration Tool
» Equipment Configuration Tool
» Custom Batch Data Collection Configuration Tool
» Report Template Manager
– Not all persons using a Tool should have access to Exaquantum/Batch server and Database Management System Server
» Lessens chance of accidental disruption due to human error
• Administration Tools can be loaded on multiple computers– Warning: Individual tools not designed for concurrent use
» No locks on data
» Most recent “Save” could overwrite other’s work
Page 28Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
System Software
Windows 2000 Server SP 4 or Window 2003– Web server uses Microsoft IIS
– Net Framework Version 1.0 SP 2
Microsoft Internet Explorer 6 SP 1
Microsoft SQL Server 2000 SP 3
Microsoft Office 2000, 2003 or XP (for reporting)– Office 2000 SR-1, SP2 and SP3
– Office XP SP1, SP2 and SP3
Standards BasedOPC– Data Access (DA 2.05a)– Alarm & Events (A&E 1.1)– Historical Data Access (HDA
1.1)– Batch (Batch 1.0)
S88– Part 1 Models & Terminology– Part 2 Data Model
SQL– ANSI standard supported by
Microsoft SQL Server 2000World Wide Web Consortium– HTTP– XML
Page 29Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Exaquantum/Batch Part 11 License
Exaquantum/Batch has an optional license that enables features useful when working with Part 11
Features enabled with the license:– Electronic signature for manual data entry
– Signature manifest data and icon is displayed to indicate manually changed values
– Report output must be approved with an electronic signature
– Report check out/in feature is disabled to increase report data integrity
• Excel copies of reports may not be accessed by normal means • The PDF format report is retained as the master copy
Other features such as audit trails and access controls are part of the base product offering
See the Configuration Guide Presentation for details on the Part 11 license
Page 30Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Network Architecture
Exaquantum/Batch must be implemented as a closed system– Should be treated as an Intra-net web site
• DO NOT treat as an Internet web site
– Utilize strict and strong access control (more later)
Grant access using Windows 2000/2003 Domains– Provides consistent method for access control
– Single login from any computer can grant access
Consider using a firewall to isolate Exaquantum/Batch and the control system from the plant and/or corporate network– Isolation of data collection and storage from general plant and
corporate networks increases data integrity and lessens chances of disruption
Page 31Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Firewall Usage
Firewall may be used to isolate the control system LAN from the Plant Office & Corporate LANs
Place Firewall between the web server and intranet web clients– Firewall Ports Opened:
• HTTP (Port 80)• Windows Authentication
(Port 445)
If a Firewall is placed between the Exaquantum/Batch Server and web server DCOM ports must be left open– This is not recommended
ExaopcStation
Plant Office LAN
Exaquantum/Batch Server
BatchWebUsers
Firewall
Switch
SwitchCorporateFirewall
Internet
Internet
BatchWeb User
Control
System
LAN
Exaquantum/BatchWeb Server
Page 32Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Redundancy / Fault Tolerance
Exaopc Servers– Redundant Data Access, Alarm & Event and Historical Data
Access Servers
– Dual Batch Servers
Multiple Ethernet paths
Fault tolerant computers– Stratus fault tolerance gives highest server availability
RAID 5 data storage– Provides highest data availability
Level of Redundancy & Fault Tolerance
Is a
Cost – Benefit Trade-off
Examine failure scenarios and their impacts
Page 33Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Exaopc Configuration
Exaopc Stations are part of the CS Batch 3000 system
Exaopc supports OPC Foundation Specifications
– Data Access 2.05a (DA)
– Alarm & Events 1.1 (A&E)
– HDA 1.1 (includes Alarms & Events)
– Batch 1.0
High data availability options– Data buffering for DA and A&E servers
– Historical catch-up for DA server
– Redundancy for DA and A&E servers
– Dual Batch servers
Control System LAN
V-net orVnet/IP
FCS
CS Batch 3000
ExaopcStation
ENG
Exaquantum/Batch
Switch
HIS
Page 34Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
V-net or Vnet/IP
Ethernet
FCS
Unit Supervision
Exaquantum/Batch
Exaopc Servers
A B
Exaquantum/Batch
Exaopc Server
A
Exaopc Client
Exaquantum/Batch OPC Client
Exaopc Server
Data AccessAlarm & Event
Redundant OPC DA and A&E Servers
Exaopc Server
B
CENTUM CS 1000 or CS 3000 Batch
HIS & BatchServers
Page 35Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Exaquantum/Batch
Exaquantum/Batch OPC Client
Dual OPC Batch Servers
Manual switchov
er required
Exaopc Server
A
Exaopc Server
B
CENTUM CS 1000 or CS 3000 Batch
Ethernet
FCS
Unit Supervision
Exaquantum/Batch
Exaopc Servers
A BHIS & BatchServers
V-net or Vnet/IP
Page 36Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Exaopc Recommendations
Driven by importance of data
Option 1 Standard Availability:– Single Exaopc station providing Data Access, Alarms & Events
and Batch interfaces
Option 2 – Highest Availability:– Two Exaopc stations
• Redundant DA and A&E servers• Dual Batch servers
To recover from disruptions in connections between Exaopc station and Exaquantum/Batch Server:– Use history catch-up and data buffering for trend data (Data
Access)
– When redundant servers used provides highest availability • Note: maximum of 1 minute failure
– Use persistence of CS Batch 3000 batch data
Page 37Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Failure Scenarios
Partial list of failure scenarios to consider– Controlled shutdown of each computer for maintenance
– Unexpected shutdown of each computer – causes include• Power failure• Physical damage (building collapse, fire, sprinkler system discharges,…)
– Network disruption• Cables cut or unplugged• Competing programs use network bandwidth denying sufficient
bandwidth for Exaquantum/Batch
– Disk failure
– Programs manually stopped
– Unauthorized programs run on server conflict with Exaquantum/Batch
– Computer virus/hacker attack• File deletion• Web Server attacked• Passwords stolen• Unauthorized intrusion
Page 38Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Failure Scenarios – Data Collection Recovery
ImpactFailure
Batch Data Recovery Trend Data Recovery Alarm & Event Data Recovery
Exaopc Station Failure(No Exaopc Redundancy)
Upon recovery persistent batch data in CS Batch 3000
automatically recovered
Recovery not possibleMinimize risk using
redundancy
Recovery not possibleMinimize risk using
redundancy
Exaopc Station Failure- Single server of redundant pair fails(Exaopc DA and A&E RedundancyDual OPC Batch servers)
Data collection resumes after manual configuration change in
Exaquantum/BatchUpon recovery persistent batch
data in CS Batch 3000 automatically recovered
No interruption provided 1 server of redundant pair is
operational
No interruption provided 1 server of redundant pair is
operational
Exaopc station controlled shutdown
Upon recovery persistent batch data in CS Batch 3000 can be
recovered
Recovery not possibleMinimize risk using
redundancy
Recovery not possibleMinimize risk using
redundancy
Exaopc station controlled shutdown - Single server of redundant pair shutdown(Exaopc DA and A&E Redundancy)
Data collection resumes after manual configuration change in
Exaquantum/BatchUpon recovery persistent batch
data in CS Batch 3000 automatically recovered
No interruption provided 1 server of redundant pair is
operational
No interruption provided 1 server of redundant pair is
operational
Exaquantum/Batch Server Failure (not a disk failure)
Upon recovery persistent batch data in CS Batch 3000 can be
recovered
Data can be recovered using History Catch-up
(manual procedure)
Data can be recovered using History Catch-up
(manual procedure)
Database Disk Failure Recovery not possibleMinimize risk using RAID 5 Disks
Recovery not possibleMinimize risk using RAID 5
Disks
Recovery not possibleMinimize risk using RAID 5
Disks
Web Server Failure No Impact No Impact No Impact
Loss of Communication between Exaquantum/Batch Server & Exaopc
Upon recovery persistent batch data in CS Batch 3000 can be
recovered
Recovery possible for DA data only
(AE data, Server Calcs and Aggregations not recovered)
Recovery not possibleNotes:• Slide refers to use of Automatic Batch Data Collection. Custom Batch Data Collection
scenarios differ and are dependant upon the OPC server and control/information system used.
Page 39Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
High Availability Recommendations
Exaquantum/Batch Server– Stratus fault tolerant computer
– RAID 5 disks
– Dual power feeds
Exaopc Stations– Redundant DA and A&E servers
• Use buffering and catch-up features
– Dual Batch server• Do not delete batch from CS Batch 3000 until data collected
Isolate Control System LAN from Plant Office & Corporate LANS with a firewall
Web Server on separate computer from the Exaquantum/Batch Server
Administration Tools on multiple remote computers within data collection firewall
Page 40Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Recommended Small System Configuration
V-net and Vnet/IP
Control System LAN
FCS
CS Batch 3000
ExaopcStation
ENGMaster Recipe Storage
HIS Batch ServersHIS
Plant Office LAN
. . .
Corporate LAN/WAN
. . .
. . .
Exaquantum/Batch Serverwith Databases, Web Server& Administration Tools
BatchWebUsers
BatchWeb Users
Switch
Internet
Internet
Switch
BatchWeb Users
Corporate Firewall
Page 41Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
V-netor Vnet/IP
FCSCS Batch 3000
Exaopc Stations
Recommended Large System Configuration
HIS Batch Servers
HIS
Plant Office LAN
. . .
Corporate LAN/WAN
ENGMaster Recipe
Storage
. . .
BatchWebUsers
BatchWebUsers
BatchWeb Users
SwitchCorporateFirewall
Internet
Internet
. . .
BatchWeb Users
. . .
BatchWeb Users
Control System LAN
Firewall
Switch
Exaquantum/Batch Server & Database Server
Administration Tools& BatchWeb Users
Exaquantum/BatchWeb Server
Page 42Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Time Considerations
Records must be clear and consistent with regard to timestamps– Time zone– Standard Time vs. Daylight Savings Time (Summer Time)
Exaquantum/Batch stores all timestamps in UTC in the database– UTC = Universal Coordinated Time (~GMT)– When collected times are converted from local time to UTC and stored– When viewed on BatchWeb times are converted from UTC to the Web
Server’s local time
No changes required when switching between Standard & Daylight (Summer) Time
All computer’s must have synchronized time to avoid discrepancies
More detail in Configuration Guidance Presentation
Page 43Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 44Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Electronic Records/Signatures
User Requirements Document is key for planning– Without clear requirements project is in jeopardy
Identify electronic records– All Exaquantum/Batch data are potentially electronic records
– What is required by predicate rules?
– What will be handled electronically vs. with paper?
Identify electronic records requiring signatures– Will electronic signatures be used?
– If so, what electronic records must be signed?• Only human actions require signatures
– Done by
– Reviewed by
– Checked by
Page 45Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Identification of Electronic Records
Identification of electronic records is specific to each application
Exaquantum/Batch is designed so all data in the databases can be considered electronic records
Possible Exaquantum/Batch electronic records– Batch data
– Master recipe data
– Equipment data
– Trend data
– Alarms & Event data
– Reports
Page 46Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Electronic Records – Data Integrity
Stored securely– Stored in SQL Server 2000
– Long-term on-line storage
Access control– Windows 2000/2003 security used to limit access
– Administrators set access rights
Data Collection– Changed formula values not overwritten
• Save each changed value with timestamp
– Property value changes recorded in audit trail
– Automatic Batch Data Collection• Product automatically collects data, no configuration – reduces
validation effort
– Custom Batch Data Collection• System specific configuration increases validation effort
Page 47Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Electronic Records – Configuration Data
Configuration/Engineering data not covered by an audit trail– System Configuration Tool
• Small amount of simple data
– Equipment Configuration Tool• Definition of tag aliases for use with batch trends and reports
• Entry of custom equipment hierarchy for use with Custom Batch Data Collection
– Report Template Manager• Report definitions
• May range from simple to complex, an number of report templates possible
– Custom Batch Collection Tool• Optional package
• Definition of collection lists and their triggers
• May range from none to large amounts of data
– Administration Tools• Defines OPC gateway, tag template and tag definitions
• Typically simple data, may be expanded to more complex applications
Page 48Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Electronic Records – Configuration Data
Requires SOPs to document changes– Normal change control procedures acceptable
Reports can include configuration data– Use to document configuration
Page 49Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Electronic Signature
Applied using account name & password– Uses Windows 2000/2003 domain accounts & passwords
Required for manual entry of operational data:– Formula value changes
– Adding new formula items
– Approving report templates
– Approving report output
– Entry of manual performance rating
All operational data manual entries recorded in an audit trail
Exaquantum/Batch only supports 1 signature per entry– SOPs required to document checked by or approved by
Signature manifest pop-up window– Provides information about electronic signatures
– Satisfies § 11.50 Signature manifestations
– Manifest data may be included in reports
Page 50Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 51Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Access Control
Different types of access control
Physical– Physical access to Exaquantum/Batch and Web servers should be
limited
Networked– Network access to servers should be limited
• Require Intra-net access– Corporate VPN can be used to allow remote access
• Restrict access to Exaquantum/Batch Server using a firewall
Privileges– Grant individuals access to specific functions using Windows
2000/2003 Groups– Create roles
• Each role is represented by a Windows 2000/2003 Group• Add individual accounts to the role’s group• Individuals can still be given specific privileges outside of the group
– Windows 2000/2003 domain makes managing access control simpler
Page 52Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Exaquantum/Batch Privileges
Each Windows 2000/2003 Group corresponds to an Exaquantum/Batch privilege
Group Privilege
QUserGroup Access BatchWeb
QDataWriteGroup Write data to the databases
QExplorerDesignGroup
Access to Exaquantum/Explorer, not required for Exaquantum/Batch
QAdministratorGroup Access to configuration tools
QBReportApprove Approve reports in BatchWeb
QBReportCheckout Check-out and Check-in report spreadsheet files (Note: This function is disabled when the Part 11 license is used)
QBReportRun Run reports from BatchWeb
QBReportTmplApprove
Approve report templates in the Report Template Manager
QBTrendTmplAdmin Create, modify and delete batch trend templates in BatchWeb
QBPerfRatingModify Modify performance rating values in BatchWeb.
QBFormulaAllWritable Override individual lockouts to change formula item values in BatchWeb
QBFormulaCreate Create new formula items in BatchWeb
QBFormulaParamMod Modify existing parameter formula item values in BatchWeb
QBFormulaResultMod Modify existing result formula item values in BatchWeb
Page 53Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Use Roles to Assign Privileges
Each Windows 2000/2003 Group corresponds to an Exaquantum/Batch privilege
GroupDB
AdminEnginee
rOperat
or
Operations
Supervisor
Office Staff
QUserGroup X X X X X
QDataWriteGroup
QExplorerDesignGroup
QAdministratorGroup
X
QBReportApprove X
QBReportCheckout
QBReportRun X X X X X
QBReportTmplApprove
X
QBTrendTmplAdmin X X
QBPerfRatingModify X
QBFormulaAllWritable
X X
QBFormulaCreate
QBFormulaParamMod
QBFormulaResultMod
Page 54Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Roles
DB Admin– Responsible for database
maintenance (backups, disk space mgt,…)
– No operational responsibilities or privileges
Engineer – Responsible for configuration,
but not for operations
– Approve report templates
GroupDB
AdminEnginee
r
QUserGroup X X
QDataWriteGroup
QExplorerDesignGroup
QAdministratorGroup
X
QBReportApprove
QBReportCheckout
QBReportRun X X
QBReportTmplApprove
X
QBTrendTmplAdmin X
QBPerfRatingModify X
QBFormulaAllWritable
QBFormulaCreate
QBFormulaParamMod
QBFormulaResultMod
Easy to change privileges and individuals assigned to roles
Page 55Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Use Roles to Assign Privileges
GroupOperat
or
Operations
Supervisor
Office Staff
QUserGroup X X X
QDataWriteGroup
QExplorerDesignGroup
QAdministratorGroup
QBReportApprove X
QBReportCheckout
QBReportRun X X X
QBReportTmplApprove
QBTrendTmplAdmin X
QBPerfRatingModify
QBFormulaAllWritable
X X
QBFormulaCreate
QBFormulaParamMod
QBFormulaResultMod
Operator– Use BatchWeb to view
historical data
– Run reports
– Enter specific manual entries
Operations Supervisor – Operator’s privileges,
plus:
– Approve reports and batch trend templates
Office Staff– Only access to BatchWeb
and running reports
Page 56Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Using Local Groups & Domains
Fred’s Windows Domain account is made a member of the Local Group “Engineer”
Fred now may log into the Exaquantum/Batch server or web server from any location
Local Group:QBEngineer Privileges:
• QUserGroup• QAdministratorGrou
p• QBReportRun• QBReportTmplAppro
ve• QBTrendTmplAdmin• QBPerfRatingModify
Individual Domain Account:Fred Smith
Page 57Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Using Local Groups & Domains
Fred’s Windows Domain account is made a member of the Domain Group “Engineer”– The Domain Group has no privileges by itself
The Domain Group is made a member of the Local Group– This gives all members of the Domain Group “Engineers” the
privileges set in the Local Group “Engineer”
Fred now may log into the Exaquantum/Batch server or web server from any location
Local Group:QBEngineer Privileges:
• QUserGroup• QAdministratorGrou
p• QBReportRun• QBReportTmplAppro
ve• QBTrendTmplAdmin• QBPerfRatingModify
Individual Domain Account:Fred Smith
Domain Group:QBEngineer
Page 58Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Access Control – System Administrators
Multiple System Administrators– Limits capability of 1 person to circumvent controls
– Enables different organizational reporting structures to reduce motivation to circumvent controls
System Administrators:– Domain Admin
– Operating System Admin
– Database Admin
– Exaquantum/Batch Admin
Administrators are trusted individuals– No system is fool-proof, ultimately we depend upon trust
Page 59Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Domain Administrator
Responsibilities– Maintain integrity of the Windows Domain
• Implement policies
– setting up and maintaining• User accounts • User groups • Administering Windows 2000/2003 group policy
– Assign individuals to domain groups
– Monitor Windows 2000/2003 log files• System log• Application log• Security log
– Ensuring there is no unauthorized access to Exaquantum/Batch from beyond the company's domain
– License management
Page 60Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Operating System Administrator
Responsibilities– Maintain integrity of the Local Computer
• Should be 1 person responsible for all servers• Implement policies
– Assign Domain Groups to Local Groups
– Assign privileges to Local Groups
– Monitor Windows 2000/2003 log files• System log• Application log• Security log
– License management
Page 61Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Database Administrator
Responsibilities– Ensure smooth operation of the Exaquantum/Batch relational
database • Typically requires a low level of on-going activity • Maintain integrity of the Database
– Database backup• Establish schedule• Follow schedule to backup database • Securely store backups on-site and off-site
– Preparation for disaster recovery• Include practicing
– Restoration of backup data as necessary
– Control access to database• Assign Read/Write data access rights
– Monitor database logs
– Monitor disk space• Ensure free space is always available
Page 62Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Exaquantum/Batch Administrator
Responsible for – Installing, configuring, maintaining Exaquantum/Batch.
Uses Exaquantum/Batch configuration tools – System Configuration Tool
– Report Template Manager
– Equipment Configuration Tool
– Custom Batch Data Collection Tool
– Administration Tools
– Other Exaquantum configuration tools may optionally be used:• Exaquantum Explorer, Database Creation Tool, Graphics Editor,…
Monitor system operation– Regularly review Windows application and system log files for
Exaquantum/Batch errors and warnings
– Create reports to record configuration data
– Disk space usage
Page 63Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 64Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Audit Trail
Different audit trails must be checked– Domain level
• Check for unauthorized access attempts
– Server Operating Systems – Windows 2000/2003• System Event Log• Security Event Log (must enable security auditing in Windows
2000/2003)• Application Event Log
– Exaquantum/Batch messages are found here
– Operational entries recorded in QBatch database• Property History table records changes to object properties• Electronic Signature Manifest records manual entries
– Report template manager preserves old report formats• Useful to reference what queries were used to produce a report
FDA information requests– Typically will be satisfied with data in the databases
– Run reports to produce reports to satisfy FDA requests
Page 65Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Reporting
Standard Operating Procedures (SOPs)
Page 66Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Reporting Package
Approvals part of reporting lifecycle
All versions of report templates saved in the database
All report output (created when reports are run) are saved in the database
Report files can be saved to the file system to be made available to FDA inspectors
3rd party report packages may be used with Exaquantum/Batch– To comply with corporate standards
Page 67Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Reporting Lifecycle
Report template created– Must be approved before reports are run
Reports run– Report property specifies if individual
reports must be approved
Report template modified– Must be approved before reports are run
Reports run– Must be approved before reports are run
Report template placed out of service– Template can not longer be used
– May be placed back in service to be used again Report
Template Out of Service
Create Report Template
Run Report
Modify Report
Template
Run Report
Approval
Approval
Approval
Approval
Access Control On all Lifecycle
Steps
Page 68Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Report Types
Many uses of reports– Regulatory
– Business
– Production efficiency
– Time based summaries
– Batch End Reports
Regulatory reports must be validated– Other reports may or may not be validated depending upon
company policy
– Report master copies stored in database• Validating that Exaquantum/Batch can consistently display and user
can save to file system an exact copy of the report should be performed
Page 69Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Providing Data to FDA Inspectors
Part 11 requires the “ability to generate accurate and complete copies of records in both human readable and electronic form suitable for inspection, review and copying by the agency
Reports can be designed to:– Document all aspects of batch execution, including master
recipe, trend and event data
– Produce PDF format output
– Macros may be used to write Excel, CSV or other format files to disk
Method used to provide data to FDA inspectors will be heavily influenced by company policy
Page 70Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Standard Operating Procedures (SOPs)
Page 71Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Standard Operating Procedures (SOPs)
SOPs define manual tasks are to be performed and documented
SOPs will be required to augment Exaquantum/Batch automated actions – Provide consistent manual actions
– Administratively enforced
List of required SOPs derived from– User requirements
– Functional specification
– Design specification
Page 72Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Typical Exaquantum/Batch SOPs
SOPs recommended for Exaquantum/Batch installations– User account management
• New accounts, maintaining existing accounts• Recommended account settings• Assigning Exaquantum/Batch privileges
– Keeping passwords and accounts in Windows & CS Batch 3000 in synchronized
– Data backup
– Data restoration
– Disaster recovery
– System monitoring• Windows log files• SQL Server logs• System state of health
– Report creation & maintenance
– Change Control of Software and Computer Systems
Page 73Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Planning Checklist
Risk assessment
Project Plan
Master validation plan
System architecture
Identification of Electronic Records
Identification of Electronic Signature requirements
Access control
Audit trail
Standard Operating Procedures (SOPs)
Page 74Copyright © Yokogawa Electric Corporation1st Edition April 18, 2005
Exaquantum/Batch Validation Planning Guidance(PM36J04B90-01E_001)
Summary
Planning for validation is critical
Exaquantum/Batch is designed to operate under Part 11 regulations
Yokogawa will work with customers to validate Exaquantum/Batch systems
More detailed information is available:
– Exaquantum/Batch Configuration Guide Presentation
– Exaquantum/Batch Help System