assessing the reliability of computer-processed data gao ... · preface • obtain an understanding...

United States General Accounting Office

GAO Report to Office of Policy

April 1991

Assessing theReliability ofComputer-ProcessedData

GAO/OP-8.1.3

Preface

Auditors and evaluators rely on data to accomplishtheir assignment objectives. In today’s computer age,more and more of the data available to them arecomputer-based and processed. Such data may comefrom a microcomputer, minicomputer, or mainframeand may range from a collection of questionnaireresponses to large national data bases.

In considering the use of computer-based data, thefollowing logical questions arise:

• How do the data relate to the assignment’sobjective(s)?

• What do we know about the data and the system thatprocessed them?

• Are the data reasonably complete and accurate?

The Government Auditing Standards—generallyreferred to as the “Yellow Book”—provide thestandards and requirements for financial andperformance audits. A key standard covers the stepsto be taken when relying on computer-basedevidence.

The purpose of this guide is to help GAO staff meetthe Yellow Book standard for ensuring thatcomputer-based data are reliable—complete andaccurate. The guide also provides a helpfulconceptual framework to expedite job performanceand help staff address standards for assessing internalcontrols and compliance with applicable laws andregulations.

The key steps in assessing reliability are:

• Determine how computer-based data will be used andhow they will affect the job objectives.

• Find out what is known about the data and the systemthat produced them.

GAO/OP-8.1.3Page 1

Preface

• Obtain an understanding of relevant system controls,which can reduce the risk to an acceptable level.

• Test the data for reliability.• Disclose the data source and how data reliability was

established or qualify the report if data reliabilitycould not be established.

The work described in this guide should normally bedone by auditors and evaluators as an essential partof assignment planning. However, in those caseswhere computer specialist skills are needed, everyeffort should be made to secure these skills.

The major contributors to this guide were DanJohnson and Charles M. Allberry. For furtherassistance, please call 275-6172.

Werner GrosshansAssistant Comptroller General for Policy

GAO/OP-8.1.3Page 2

GAO/OP-8.1.3Page 3

Contents

Preface 1

Chapter 1 Introduction

6Government Auditing Standards 6Using This Guide 7System Review Versus a Limited Approach 8When Data Reliability Should Be Determined 9Who Should Evaluate Computer-Based Data

Reliability9

Terms Defined 10

Chapter 2 AssessingReliability Risk,UnderstandingSystem Controls,and DeterminingData TestingRequirements

13Conceptual Framework 13Planned Use of Data 14Knowledge/Experience With System or Data 16System Controls 17

Chapter 3 Data Testing

21Objectives and Methods of Data Testing 21Various Levels of Data Testing 27Special Considerations in

Computer-Programmed Data Tests29

Chapter 4 Reporting onData Reliability

32Reporting on Computer-Based Evidence 32

GAO/OP-8.1.3Page 4

Contents

Chapter 5 Case Study:GuaranteedStudent Loans

36Case Example 36Determining Reliability Risk 38Understanding System Controls 40Data Testing 41

Appendixes Appendix I: Examples of Data Tests 46Appendix II: Special Considerations in

Understanding Computer System Controls49

Figure Table 2.2: Factors in DeterminingExtensiveness of Data Testing

14

GAO/OP-8.1.3Page 5

Chapter 1

Introduction

This chapter discusses

• standards and requirements for using computer-basedevidence contained in GAO’s “Yellow Book”,

• the purpose of this guide,• distinctions between a full system review and a more

limited effort,• when (during the assignment) data reliability should

be determined including options to consider whendata is unreliable,

• who should determine data reliability, and• definition of terms.1

GovernmentAuditingStandards

As part of the evidence standard for performanceaudits, GAO’s Government Auditing Standards (theYellow Book) and chapters 4 (“Standards”) of theGeneral Policy Manual and the Project Manualinclude requirements for determining the reliability ofcomputer-based information.

The Yellow Book gives the following guidance:

When computer-processed data are an important

or integral part of the audit and the data’s

reliability is crucial to accomplishing the audit

objectives, auditors need to satisfy themselves

that the data are relevant and reliable. This is

important regardless of whether the data are

provided to the auditor or the auditor

independently extracts them. To determine the

reliability of the data, the auditors may either

(a) conduct a review of the general and

application controls in the computer-based

systems including tests as are warranted; or (b)

if the general and application controls are not

reviewed or are determined to be unreliable,

conduct other tests and procedures.

1This guide supercedes the publication, Assessing the Reliability ofComputer Output (AFMD-81-91) dated June 1981.

GAO/OP-8.1.3Page 6

Chapter 1

Introduction

When the reliability of a computer-based system

is the primary objective of the audit, the

auditors should conduct a review of the system’s

general and application controls.

When computer-processed data are used by the

auditor, or included in the report, for

background or informational purposes and are

not significant to the audit results, citing the

source of the data in the report will usually

satisfy the reporting standards for accuracy and

completeness set forth in this statement.

Using This Guide This guide helps staff ensure that their use ofcomputer-based data meets Yellow Bookrequirements. It applies to both performance andfinancial assignments.

Staff should not assume that computer-based data arereliable. When using computer-processed data asevidence, staff must take steps to providereasonable—not absolute or complete—assurancethat the data are valid and reliable. Effectively carriedout, the steps discussed in this guide will provide thatassurance. They will not—nor do they needto—ensure that all data errors are detected.

The effectiveness of carrying out the steps in thisguide depends on judgment in determining how muchto rely on system controls, how to test data, and howmuch testing to do. Errors in judgment haveundesirable consequences—too much audit effortwastes valuable resources, while too little jeopardizesthe credibility of our work.

GAO/OP-8.1.3Page 7

Chapter 1

Introduction

System ReviewVersus a LimitedApproach

There are basically two approaches to assessing thereliability of computer-based data, the system

review generally performed by specialists and themore limited approach (which this guide addresses)designed for evaluators/auditors.

A systems review assesses and tests all controls in acomputer system for the full range of its applicationfunctions and products. These reviews (1) examine acomputer system’s general and application controls,(2) test whether those controls are being compliedwith, and (3) test data produced by the system. Whilethis approach provides the best understanding of asystem’s design and operation, it tends to be timeconsuming. When the assignment’s objective(s)dictate a complete system review, specialists shouldbe consulted.

The limited review is targeted to particular data. Asa result, it normally requires a less extensiveunderstanding of general and application controls.Pertinent controls are examined to the extentnecessary to judge the level of data testing needed todetermine data reliability. This can usually beperformed by generalist staff.

For most assignments using computer-basedevidence, the more limited approach described in thisguide is adequate. However, if GAO staff use aspecific set of computer-based data for many differentassignments during an extended period, the lengthand cost of a full system review may be warranted.Such a review (periodically updated) might be lessexpensive in the long run than individualdeterminations of data reliability using theprocedures in this guide.

GAO/OP-8.1.3Page 8

Chapter 1

Introduction

When DataReliability ShouldBe Determined

The reliability of computer-based data should bedetermined early in the planning phase of anassignment. If an assignment relies oncomputer-based evidence, staff must know if the dataare reliable. If the data are not sufficiently reliable tomeet the assignment’s objective(s), they cannot beused as the primary evidence and staff will need toplan alternative approaches. The following optionsshould be considered and discussed withmanagement and, as necessary, with customers:

• Seek evidence from other sources. Staff would needto determine the reliability of such data.

• Collect primary data to meet the assignment’sobjective(s), rather than use secondary source data.This would be possible only if the work could becompleted in time to meet requester’s needs.

• Redefine the assignment’s objective(s) to eliminatethe need to use unreliable data.

• Use the data, but explain their limitations and refrainfrom drawing unreasonable conclusions orrecommendations. It is preferable to draw noconclusions or recommendations.

• Terminate the assignment if no other alternative ispossible.

Assignment proposals should include adequate stafftime and identify the specific skills necessary tocomplete reliability determinations in a timelymanner.

Who ShouldEvaluateComputer-BasedData Reliability

This guide is designed for the use ofevaluators/auditors. If expert help is needed,however, in carrying out this more limited approach,it should be obtained promptly. As with all evidence,evaluators/auditors are responsible for its reliability;computer-based data should be no different. Thebasic tests of evidence apply. It should be best

GAO/OP-8.1.3Page 9

Chapter 1

Introduction

evidence, competent, relevant and sufficient. Inevaluating the competence of evidence theevaluator/auditor should carefully consider whetherany reason exists to doubt its validity, completeness,and accuracy.

Terms Defined Definitions of terms used in this guide are as follows:

Data reliability: A state that exists when data aresufficiently complete and error free to be convincingfor their purpose and context. It is a relative conceptthat recognizes that data may contain errors as longas they are not of a magnitude that would cause areasonable person, aware of the errors, to doubt a

Computer system controls: Policies and proceduresthat provide reasonable assurance thatcomputer-based data are complete, valid, and reliable.They include general and application controls.

General controls: The structure, methods, andprocedures that apply to the overall computeroperations in an agency. They include organizationand management controls, security controls, andsystem software and hardware controls.

Application controls: Methods and proceduresdesigned for each application to ensure the authorityof data origination, the accuracy of data input,integrity of processing, and verification anddistribution of output.

Systems review: An assessment of general andapplication controls, test of the degree of compliancewith those controls, and appropriate data tests.

Compliance testing: Verifying whether controls arebeing complied with during the system’s operation.

GAO/OP-8.1.3Page 10

Chapter 1

Introduction

Compliance testing does not directly test whetherparticular computer data are valid and reliable.

Data testing: Testing to determine if particular dataproduced by a computer system are valid and reliable.Data testing does not establish the existence oradequacy of system controls or whether such controlsare being complied with, but may reveal indications ofcontrol weaknesses.

Source record: Information, in manual or electronicform, which is the basis for original entry of data to acomputer application.

Attribute test: An examination of a data element for alogical or defined characteristic; also referred to as anunconditional test. For example, the status of a loanapplication must be “approved”, “denied”, or“pending”.

Relationship test: A comparison of values to validate alogical or defined correlation; also referred to asconditional tests. For example, an invoice date mustbe the same as or earlier than the related paymentdate.

Data element: An individual piece of information thathas definable parameters (e.g., a social securitynumber).

Data record: A collection of data elements relating toa specific event, transaction, or occurrence (e.g.,name, age, social security number, school, dateenrolled, loan date, loan amount, amount repaid, andloan balance).

Data file: A collection of data records relating to aspecific population (e.g., student loan applications forMaryland schools).

GAO/OP-8.1.3Page 11

Chapter 1

Introduction

Attributes: Characteristics of a data element definedby the data dictionary (e.g., numeric or alpha,acceptable values, and length).

GAO/OP-8.1.3Page 12

Chapter 2

Assessing Reliability Risk,Understanding System Controls, andDetermining Data TestingRequirements

This chapter introduces the process and decisionpoints for conducting a reliability assessment ofcomputer-processed data. It examines the elementswhich influence the level of data testing requiredincluding

• a conceptual framework,• the planned use of the data relative to the

assignment’s objective(s),• the existing knowledge base relating to the data and

system, and• the adequacy of system controls.

ConceptualFramework

When computer-processed data are being consideredfor use in an assignment, staff must initially determinethe reliability risk—the risk that the data areunreliable for the planned use. As illustrated in table2.1, reliability risk is determined by considering bothplanned use and present knowledge of the data or thecomputer system. èáB÷(1 5802áB÷(2.1: Factors in DevelopingReliability Risk

The second step (illustrated in table 2.2) is tounderstand system controls and determine if theylower the reliability risk. If system controls arestrong, they can lower the reliability risk to anacceptable/prudent level and decrease the datatesting that would normally be required in a high riskenvironment.

GAO/OP-8.1.3Page 13

Chapter 2

Assessing Reliability Risk,

Understanding System Controls, and

Determining Data Testing

Requirements

Table 2.2: Factors in Determining Extensiveness of Data Testing

Details of assessing reliability risk and determiningthe extensiveness of data testing needed to reducethat risk to an acceptable level appear in the sectionsthat follow.

Planned Use ofData

When an assignment requires computer-processeddata, the first step is to decide how the data will beused—how will they contribute to meeting anassignment’s objective(s)?

Normally, data are used as

• the sole evidence supporting a finding,• corroborative or supporting evidence, or• background information.

GAO/OP-8.1.3Page 14

Chapter 2




Requirements

If computer-processed data are the sole support foran assignment’s objective(s), the need for confidencein their reliability is greatest. For example, assumeGAO is asked to determine whether mine safetyinspections are being made within a legislativelyprescribed time frame. The agency under review hasinformation in a computerized data base that directlyaddresses this question. If GAO plans to useinformation in that data base without corroboratingevidence, establishing the reliability of the data baseis critical to the assignment’s objective(s).

When computer-based data are supported by otherevidence, the need for complete confidence in thatdata varies depending on how effectively the otherevidence—standing alone—could support the finding.For example, staff discussions with unionrepresentatives might reveal that regular mineinspections were not occurring, but such discussionswere unable to clearly establish the interval betweeninspections. These discussions corroborateinformation in the agency’s computerized data basesand add to its persuasiveness. The finding still leansheavily on the computerized data base, anddetermining its reliability is important.

The lowest risk of using computer-based data occurswhen the data are used in the report for backgroundor informational purposes and are not vital to auditresults. In these cases, citing the data source in thereport and ensuring that the data are the bestavailable will satisfy reporting standards for accuracyand completeness unless there is reason to believethat inaccuracies in the data would jeopardize thereport’s credibility.

GAO/OP-8.1.3Page 15

Chapter 2




Requirements

Knowledge/Experience WithSystem or Data

After deciding how the computer-based data will beused, the next step is to find out what is alreadyknown about the data and the system that processedthem. This information combined with the planneduse of the data, determines the reliability risk.

The following examples illustrate ways in which staffcan learn more about the data or the system controls:

• GAO used the same data to support a prior findingafter adequately establishing their reliability. The riskof using the data in a current report would be low.But updating would be needed to ensure that the datahad not changed since they were last used.

• GAO recently established the adequacy of systemcontrols used to process data critical to theassignment’s objective(s). After determining that nosignificant changes had occurred in the system sincethe assessment, the reliability risk would be low.System controls could be considered good, andminimal data testing would be required.

• In a recent report, GAO cited information developedfrom a different application of the same computerizeddata base. Work previously done to determine theadequacy of general controls after updating couldreduce the present data’s reliability risk. Additionalwork would need to be done to establish theadequacy of relevant application controls and thelevel of data testing required.

• The inspector general or other audit/evaluation groupstudied the system controls or used the data. Theresults of the work could establish reliability risk, butthe Yellow Book’s due professional care requirementinvolving reliance on work performed by otherswould need to be met. (See Government AuditingStandards, pp. 3-14 through 3-16.)

GAO/OP-8.1.3Page 16

Chapter 2




Requirements

System Controls While some data testing is always necessary whencomputer-based evidence is used to meet anassignment’s objective(s), satisfactory systemcontrols can reduce the data testing required toestablish reliability. In such cases, less data testing isneeded than when those controls are weak orundetermined.

According to the Yellow Book,

The degree of testing needed to determine data

reliability generally increases to the extent that

the general or application controls were

determined to be unreliable or were not

reviewed.

UnderstandingSystem Controls

Staff should understand system controls and theirpurposes to determine whether they can be relied onto reduce data testing. This understanding includesboth general and application controls that relate toassignment evidence.

An understanding of general controls would includeknowledge of the following items which affect datareliability.

• Management commitment to system design andoperation: This includes management’s methods formonitoring and following up on performance,including corrective action on internal auditrecommendations and on user complaints.

• Organization of the system functions, includingassignment of responsibilities and separation ofduties: This provides that key duties andresponsibilities in authorizing, processing, recording,and reviewing transactions are assigned to differentindividuals.

GAO/OP-8.1.3Page 17

Chapter 2




Requirements

• Physical security of the computer facility and itscomponents, including restrictions on access:Restricting access helps ensure data reliability byreducing the risk of unauthorized data entry ormodification.

• Supervision: Effective supervision requires a clearcommunication of duties and responsibilities, regularoversight—particularly at critical points—andperiodic performance evaluations.

In understanding application controls, staff shouldconsider matters such as the following:

• procedures to ensure that application software andsubsequent modifications are authorized and testedbefore implementation;

• frequency of system modification and the reasons forit;

• whether program changes are controlled andpromptly documented;

• the review, approval, control, and editing of sourcetransactions to ensure completeness and preventerror;

• tables used in computer processing, their sources,and the frequency of updating;

• the existence of current narrative system descriptionsand flowcharts;

• reconciliation of output records with input entries;• error detection and correction procedures;• data user’s views of data reliability; and• internal audit reports and other evaluations or

studies.

Regardless of how well-conceived and designedsystem controls may be, they are ineffective if appliedincorrectly and inconsistently. For example, a systemmay have a control that requires a data quality controlgroup to verify that source data are accounted for andthat they are complete and accurate, have been

GAO/OP-8.1.3Page 18

Chapter 2




Requirements

appropriately authorized, and transmitted in a timelymanner. But if that group is bypassed, the controlcontributes nothing to ensuring the integrity of dataentry.

Many reasons exist for bypassing or overridingcontrols such as time pressures, fatigue, boredom,inattention—or even collusion for personal gain. As aresult, staff should select the most significant controlprocedures and confirm adherence to them. Althoughit is unnecessary to test all procedures, staff shouldconduct sufficient tests to afford a reasonable basisfor reducing testing by relying on the adequacy ofcontrols. Observing the work environment for anordered and businesslike atmosphere can be helpful.

Documentation of a well-controlled system should becomplete and current. Absence of suchdocumentation may indicate that controls do not existor, if they do, that they are not understood oradequately applied. Other red flags that suggestvulnerability to data errors include

• old systems with high program maintenance;• large volumes of data;• frequent processing and updating activity;• numerous transaction types and sources;• large number of coded data elements;• high employee turnover (e.g., data entry clerks,

operators, and analysts) and inadequate training;• complex or messy data structures; and• lack of EDP standards, especially related to security,

access, and program change control.

Discussions with knowledgeable agency personnelcan provide an effective beginning in gaining anoverall system understanding. Their testimonialevidence, however, should be corroborated throughindependent observations or tests whenever possible.

GAO/OP-8.1.3Page 19

Chapter 2




Requirements

Relying on SystemControls to ReduceData Testing

Some data testing is essential whenevercomputer-based data will be used as evidence. Evenwhen system controls are well designed and generallyadhered to, data accuracy is not ensured. While staffcan rely on good controls to reduce data testing,control reviews cannot substitute for data testing.

After reviewing controls, staff should evaluate theirstrength, that is, whether controls can be reasonablyexpected to prevent errors and to detect those that dooccur. This evaluation determines whether extensive,moderate, or minimal data testing is needed.

Staff should keep the purpose of reviewing thecontrols in mind as they progress. If it is determinedthat (1) system controls cannot be relied on to limitdata testing or (2) continuing the controls review ismore costly than expanding data testing, the reviewshould cease. In this case, staff must proceed withdata testing as if system controls were weak ornonexistent.

Documenting theBasis forExtensiveness ofTesting

The workpapers should be documented to disclose:

• What assignment objective(s) willcomputer-processed data likely support?

• How will the data support that objective?• Will that objective be supported by other evidence?

What is the other evidence?• What is known about the data?• What did staff do to understand the system and its

controls?• Did staff determine the reliability of system controls?

If so, are the controls strong, adequate, or weak?

GAO/OP-8.1.3Page 20

Chapter 3

Data Testing

This chapter discusses

• the objectives and methods of data testing,• the appropriateness of varying testing levels, and• factors to consider when using computer-assisted

testing techniques.

Objectives andMethods of DataTesting

Yellow Book standards require evidence (regardlessof its source or format) to be competent, relevant, andsufficient. Data reliability focuses on assessing thecompetency of data. Data testing is intended toestablish that evidence relied on is suitably accuratefor its specified purpose.

While it is unlikely that any computer system containserror-free data, the concept of reliability does notrequire perfect data. It should, however, include stepsto assess data completeness, data authenticity, andthe accuracy of computer processing.

Tests of data completeness confirm that the universecontains all data elements and records relevant to theassignment’s objective(s) and to the period coveredby the audit. Missing data is particularly harmful if itrepresents a specific segment of the total population(e.g., all grant recipients from California).

An analysis of data authenticity determines if thecomputer-based data accurately reflect the sourcerecords.1 This means that information in sourcerecords should match that entered in computer-basedrecords and that each computer-based record shouldbe supported by a source record.

1Appropriate steps should also be taken to insure that theinformation contained in source records is factual. If this is notdone, related limitations on the data should be fully disclosed in thereport.

GAO/OP-8.1.3Page 21

Chapter 3

Data Testing

Steps aimed at the accuracy of computer processingare designed to verify that all relevant records werecompletely processed and that computer processingmet the intended objectives.

There are two varying approaches to testingcomputer-based data. They are characterized asauditing around the computer or auditing with thecomputer. The appropriate approach or combinationof approaches is dependent on the nature of therelated system.

Auditing Around theComputer

Auditing around the computer assumes thattechniques and procedures the computer uses toprocess data need not be considered as long as thereis a visible audit trail and/or the result can bemanually verified. This approach bypasses thecomputer in either of two ways.

In the first way, computer output is compared to orconfirmed by an independent source. This approachconfirms computer-processed data with third partiesor compares data with physical counts, inspections,records, files, and reports from other sources.Physical counts and inspections can verify quantity,type, and condition of tangible assets. Reports ongovernment programs and activities issued by outsidecontractors, universities, audit and privately-fundedorganizations, and others can contain a useful basisfor comparison.

Examples of sources from which confirmation can beobtained include

• banks (cash balances on hand or amounts of loans);• warehouses (assets stored or volume of transfers);• training institutions (number of students or dollar

volume of contracts);

GAO/OP-8.1.3Page 22

Chapter 3

Data Testing

• common carriers (rates for freight shipments orvolume of passengers between selected locations);

• medical facilities (daily rates for patient care or typesof outpatient services);

• private business concerns (billings for utility servicesor wholesale prices of generic drugs); and

• other government agencies (checks cancelled by aU.S. Treasury Department disbursing center orstatistics on an agency’s use of General ServicesAdministration automobiles).

Staff can also conduct common-sense examinationsof printed data output to reveal potential reliabilityproblems. These inspections can establish datareliability when a low to very low level of data testingis required. When a moderate to high level of testing isrequired, these tests should be supplemented by moreextensive procedures. The following questions areexamples of common-sense data tests:

• Are amounts too small (cost per mile to operate a1-ton truck equals $.004)?

• Are amounts too large (a student loan for $150,000)?• Are data fields complete (a loan payment amount is

blank)?• Are calculations correct (inventory value is a negative

amount)?

Although confirmations and comparisons directly testthe accuracy of computer output and effectivelydisclose fictitious data, they may not detectincomplete data input. When data completeness is indoubt, confirmations or comparisons should besupplemented by tracing a sample of source recordsto computer output.

The second way to bypass the computer in confirmingdata reliability is to select source transactions,manually duplicate the computer processes, and

GAO/OP-8.1.3Page 23

Chapter 3

Data Testing

compare the results with computer output. Examplesinclude

• benefit payments for selected grant recipients,• loan balances and delinquent amounts,• resale prices of foreclosed and repossessed

properties, and• salary payments.

Although this approach can test the completeness ofcomputer output as well as the accuracy of computerprocessing, it does not disclose fictitious data (i.e.,data that have been entered into the computer but arenot supported by source records). If fictitious data arean issue, tracing data from the computer to sourcerecords should be considered.

The usefulness of auditing around the computerdiminishes as the number and complexity ofcomputer decisions increase. It may be impracticalwhen sophisticated data processing activities areinvolved.

Auditing With theComputer

Auditing with the computer means that computerprogrammed tests are used, in part, to measure datareliability.

After determining the completeness and accuracy ofcomputer input by manually tracing data to and/orfrom a sample of source records, this approach usesauditor-developed computer-programmed tests toexamine data reasonableness and identify defects thatwould make data unreliable.

An advantage of auditing with the computer is that itcan be used regardless of the computer system’scomplexity or the number of decisions the computermakes. Auditing with the computer is also fast and

GAO/OP-8.1.3Page 24

Chapter 3

Data Testing

accurate, permitting a much larger scope of testingthan would be practical with other methods.

The first step in developing computer-programmedtests is to identify what computer information is to beused as evidence and what data elements were usedto produce it. Staff should test all data elements thataffect the assignment’s objective(s).

When an audit-significant data element is derived (i.e.,calculated by the computer based on two or moredata elements), staff should also test the source dataelements. For example, the element “net pay” mightbe planned for use as evidence to meet anassignment’s objective(s). Review of the system’s datadictionary shows that a computer program uses threeother data elements to calculate net pay—“hourlyrate”, “hours worked”, and “deductions”. Errors in anyof these data elements would make “net pay”incorrect. Therefore, staff should determine theaccuracy of each.

After identifying the relevant data elements, the datadictionary can be examined to define the attributes ofeach and identify rules which each should meet. If adata element fails these requirements, the computermay exclude it or process it in a way that does notensure an accurate result. Computer programsfrequently have default logic that may cause a missingor defective data element to be erroneouslyprocessed.

For example, data to be entered into a computer mayidentify whether a project is ongoing or completed. Ifthe data element is not entered for a specific record, acomputer program prescribes treatment of themissing data. The record could be put in an error fileuntil the missing data is provided, or a programmedassumption could be made about its status (i.e., if the

GAO/OP-8.1.3Page 25

Chapter 3

Data Testing

status is blank, then the project is ongoing). If thatassumption is incorrect in enough records, that dataelement will be unreliable.

Understanding a data element also makes it possiblefor staff to develop reasonableness assumptions thatcan be programmed as common-sense tests—forexample, can a student loan recipient be a12-year-old? Common-sense tests do not establishthat a data element is erroneous. They raise red flagsfor follow-up. Although it is possible for a 12-year-oldto be a college student, it is unlikely. (Other examplesof common sense tests are included on page 22.)

Data attributes should also consider expectedrelationships among data elements. Althoughdeveloped independently, a data element may have areasonable relationship to another data element. Forexample, some kinds of medical procedures are age-or gender-related. Determining and testingrelationships can reveal errors by disclosing irrationalor unlikely relationships such as a hysterectomy on amale patient.

When staff have learned about each of the dataelements that affect the information relied on, testsare developed to detect errors. Tests are of two types:those that disclose failures of data elements to meetestablished requirements and those that discloseillogical relationships. (See appendix I for discussionand examples of these test types).

After data tests are developed, the computer isprogrammed to apply them. The programmed datatests must be validated and tested to ensure thaterrors revealed during the data testing are the resultof incorrect data and not the result of invalid testprograms.

GAO/OP-8.1.3Page 26

Chapter 3

Data Testing

Data tests can be developed without knowledge of thetechnical design of the data base, its structure, andlayout. This knowledge, however, is needed toprogram the tests. If assignment staff are unfamiliarwith the necessary programming techniques, supportis available from their division’s design, methodology,and technical assistance group (DMTAG) or region’stechnical assistance group (TAG).

Whether a microcomputer or a mainframe should beused to process data tests depends on factors such asthe size of the data base, the number and complexityof data tests, required processing speed, computeraccessibility, and team expertise. If a mainframe isrequired, staff will almost certainly need to getsupport from their DMTAG or TAG.

Commonly available retrieval or analysis applicationsmay be used for programming tests. These includeproducts such as Lotus 1-2-3, dBASE, SAS, SPSS, andDYL-280. While some programs have beensuccessfully used in testing data bases of over amillion records, staff should take care to ensure thattest requirements are properly matched to theapplication and to the operating environment (micro-versus mainframe computer).

Various Levels ofData Testing

As stated in chapter 2, the level of data testingdepends on the reliability risk (based on data use andexperience with the data) and staff judgment of theadequacy of system controls. The greater thereliability risk, the more assurance is required toreduce the risk to an acceptable level.

If a low level of data testing is adequate to establishthe reliability of computer-processed data, it may bemost appropriate to test only those items which in theauditor’s judgment are most likely to have errors. At

GAO/OP-8.1.3Page 27

Chapter 3

Data Testing

this level of testing, reliance for data acceptabilityrests primarily on staff judgment of system controls.Data tests provide some confirmation that relied-onsystem controls were operating effectively. Ajudgmental sample size, which is randomly selected,can give this confirmation but will not define theconfidence or precision levels achieved by the testing.

If data test results detect no errors or suggest an errorrate that is acceptable for the data’s planned use, thedata could be considered reliable. If, however, the testerror rate is high, staff evaluation of system controladequacy—on which reliance was placed—may havebeen in error. In this case, sample size and scope oftesting should be increased or a statistically validapproach used to provide a defensible basis for adecision on data reliability.

If moderate to high data testing is needed, reliance isprimarily on data testing rather than on systemcontrols. Sufficient tests should be performed toreasonably assure detection of significant errors. Ifsampling methods are used, a statistically validsample size would be necessary to permit appropriateprecision levels to be calculated and support the testresults.

A number of statistical approaches are discussed andillustrated in Transfer Paper 6, Using StatisticalSampling. The statistical approach depends onwhether GAO needs only to determine whether theerror rate is acceptable or whether it is necessary toquantify the error rate.

GAO/OP-8.1.3Page 28

Chapter 3

Data Testing

SpecialConsiderations inComputer-ProgrammedData Tests

Because of the computer’s speed,computer-programmed tests (used in auditing withthe computer to detect data defects and inconsistentrelationships) are usually not sampled but run againstall records for each data element tested. Only whenusing very large data bases would it be necessary tolimit testing to a sample of records. The testing level(high, moderate, or low) normally relates to thenumber of tests applied rather than to the number ofdata elements or records tested. If low-level datatesting is adequate, it might be limited to those teststhat disclose failures of data elements to meetestablished requirements. Moderate to high testinglevels would contain a wider variety of tests,including increased use of relationship tests. Seeappendix I for a discussion of various data tests.

In using results of computer-processed data tests,staff should consider whether the same record or dataelement failed more than one data test. If so, the errorrate may need to be adjusted. The following examplesillustrate this situation:

Assume that for the data element, “loan balance”,staff conducted two tests on a universe of 100records. A range test counted any loan balance below$0 or greater than $10,000 as an error. A derivationtest defined an error as any loan balance

GAO/OP-8.1.3Page 29

Chapter 3

Data Testing

which did not equal “original loan amount” plus“interest charges” minus “loan payments to date”.Results were as follows:

Since all errors relate to one test, the error rate is5 percent.

But assume the following results for the same tests.

In the second example, each test identified failures.Based on these results, from 5 to 10 percent of thedata are defective. The actual error rate depends onwhether a record failed one or both tests. If a10-percent rate would cause the data to be unreliable,staff would need to make additional tests todetermine if the same records were defective in thevarious tests.

Data tests that detect inconsistent relationshipsbetween data elements establish the likelihood oferror, but do not identify which data element isdefective. Staff must run additional tests or perform

GAO/OP-8.1.3Page 30

Chapter 3

Data Testing

other audit work to determine which data element torely on. Similarly, the failure of a data element tomeet an expected attribute signals a potential error.Additional follow-up (e.g., discussions withknowledgeable agency personnel) may identifyacceptable explanations. Only after defective data areconfirmed can the error rate be correctly calculated.

Whenever an error rate is unacceptable, staff mayconsider two actions to make the data usable:

• Repair the defective data elements. By doing this, theacceptability of the corrected data error rate could bedetermined.

• Exclude the defective data records from theassignment universe. By doing this, the data elementused to support audit findings, conclusions, orrecommendations includes only data not found to bedefective. This approach is inappropriate, however,when the data exclusion would introduce a systemicbias in assignment results. (See General PolicyManual and Project Manual, chapter 10,“Methodology,” for a discussion of systemic andrandom bias.)

GAO/OP-8.1.3Page 31

Chapter 4

Reporting on Data Reliability

This chapter discusses the reporting requirementswhen using computer-based data to meet theassignment’s objective(s). In addition, it suggestssample report language for cases in which the data is

• reliable,• unreliable but still usable,• unreliable and not usable, and• not assessed for reliability.

Reporting onComputer-BasedEvidence

Completeness and accuracy reporting requirements ofthe Yellow Book and GAO’s Communications Manual(12.8) require that data sources and the methods usedto determine data reliability should be stated in thereport. When material is included in a report forbackground or informational purposes and isinsignificant to audit results, staff can normally meetthis reporting standard by citing the data source in thereport.

For computer-processed data which is critical to theassignment’s objective(s), the report should assurereaders that the information relied on is credible andreliable. Specifically, it should

• identify the scope of work done when system controlsare relied on to reduce data testing, and

• describe the testing of the computer-processed data,including the tests performed, their purpose, and theerror rates disclosed.

• present any factors known to limit the data’sreliability and if significant, the sensitivity of theresults to the accuracy of the data.

If sampling was used to determine data reliability, thedescription should include the purpose of the sample;the universe and sample sizes; the basis of the samplesize (judgmental or statistical); the type of sample

GAO/OP-8.1.3Page 32

Chapter 4


(simple random, stratified, and so on); confidencelevels and precision; and errors detected.

Staff should include a summary of the above in theobjectives, scope, and methodology (OSM) section ofthe report. Technical details of complex samplingmethods and computer-programmed data tests mayappear in the body of the report or in a technicalappendix.

If data reliability was not determined or was notdetermined to the extent normally desired, theproduct should include a clear statement to thateffect as well as a qualified conformity statement. Inthese cases, statements of negative assurance1 may beuseful. Auditors/evaluators should consider theappropriateness of presenting any conclusions orrecommendations based on the data.

The following are examples of report language thatcan be used in the OSM to meet established reportingstandards.

Reliable Data IsUsed

“To achieve the assignment’s objective(s) weextensively relied on computer-processed datacontained in [cite data base used]. We assessed thereliability of this data including relevant general andapplication controls and found them to be adequate.We also conducted sufficient tests of the data. Basedon these tests and assessments we conclude the dataare sufficiently reliable to be used in meeting theassignment’s objective(s).”

1Negative assurance is a statement that nothing came to theauditor/ evaluator’s attention as a result of specified proceduresthat caused them to doubt the acceptability of the data. Theauditor/evaluator, by using other data and information, came to theconlusion that the data could be relied on to achieve theassignment’s objective(s).

GAO/OP-8.1.3Page 33

Chapter 4


Unreliable Data StillUsable

“To achieve the assignment’s objective(s) weextensively relied on computer-processed datacontained in [cite the data base used]. Our review ofsystem controls and the results of data tests showedan error rate that casts doubt on the data’s validity.

However, when these data are viewed in context withother available evidence, we believe the opinions,conclusions, and recommendations in this report arevalid.”

Unreliable Data NotUsable

“To achieve the assignment’s objective(s) weextensively relied on computer-processed datacontained in [cite the data base used]. Our review ofsystem controls and the results of data tests showedan error rate that casts doubt on the data’s validity.Since the assignment’s objective(s) require specificstatements based on this data and sufficientindependent evidence is not available, we wereunable to provide specific projections, conclusions, orrecommendations.

Reliability Is NotDetermined

“To achieve the assignment’s objective(s) weextensively relied on computer-processed datacontained in [cite the data base used]. We did notestablish the reliability of this data because [cite thereason(s)]. As a result, we are unable to provideprojections, conclusions, or recommendations basedon this data.2 Except as noted above, GAO’s work wasconducted in accordance with generally acceptedgovernment auditing standards.”

2There may be cases where sufficient other data could be relied onto draw conclusions and recommendations from such data,because the issues are broader (i.e., policy issues), wherepreciseness of data is not of paramount importance. In those rarecases, conclosure and recommendations may be appropriate, butfull disclosure is needed. Staff should also consider the limitationsdiscussed on page 20, footnote #1.

GAO/OP-8.1.3Page 34

Chapter 4


If the reliability of critical data is not determined, anexception to the generally accepted auditingstandards is necessary. Staff should discuss thecircumstances with the Assistant Comptroller Generalfor Planning and Reporting and obtain approvalbefore final processing.

GAO/OP-8.1.3Page 35

Chapter 5

Case Study: Guaranteed Student Loans

This chapter presents a case example of how todetermine the reliability of computer-based evidence.It discusses appropriate steps for

• assessing the reliability risk,• examining the adequacy of system controls, and• performing data testing at both an extensive and

minimal level.

Case Example The following case illustrates how to apply therequirements, concepts, and principles discussed inthis guide to an assignment. The circumstances of thiscase are hypothetical and are intended to illustratethe factors affecting the extent of data testing.

AssignmentObjectives

Assume that GAO has been requested to review theStafford Student Loan Program and determine if

• the Department of Education is paying the correctamount of interest and special allowance (interestsubsidy) to lenders,

• payments are made to lenders in a timely manner, and• interest payments are being made for defaulted loans.

Background Under the program, private lenders make loans atlower-than-market interest rates to qualified studentsattending approved educational institutions. TheDepartment of Education pays the interest while thestudent attends school and for a stipulated graceperiod thereafter. Education also funds specialallowance payments during the life of the loan toprovide lenders the difference between the loaninterest rate and the rate on 90-day Treasury bills,plus 3-1/4 percent. If borrowers default on their loans,Education repays the loan (usually through stateagencies) and stops paying interest and specialallowances.

GAO/OP-8.1.3Page 36

Chapter 5


The Department of Education makes interest andspecial allowance payments directly to lenders basedon detailed quarterly billings. Lenders’ billings areentered into Education’s computerized system, whichsummarizes and authorizes payments to lenders forinterest and special allowances. A separate data basemaintains information on defaulted loans.

Assignment Approach Since the computer-based data compiled byEducation contains information relating to interestpayments and defaults, staff have identified it as a keysource of evidence to support their objective(s).However, before beginning an analysis of thisinformation, auditors/evaluators must assurethemselves that the data are reliable. For example,

• Are individual loan amounts correct?• Are interest calculations accurate?• Do all records apply to the time period of our

audit/evaluation?• Are dates of loan defaults accurate?• Are lender identification codes correct?

Reliability assessment procedures should include:

• determining the importance of the computer-baseddata in meeting the assignment’s objective(s),

• determining what past experience and currentknowledge is available about the data and the systemwhich processes them,

• reviewing general and application controls to theextent they can be relied on to reduce the level ofdata testing, and

• developing and performing data tests.

These efforts are focused on providing reasonableassurance that the data does not contain significanterrors which would undermine the credibility of ouranalyses and conclusions.

GAO/OP-8.1.3Page 37

Chapter 5


DeterminingReliability Risk

The first step in meeting the case study objectives isto determine the reliability risk. This includes the riskthat Education’s computerized data do not accuratelystate amounts paid to lenders for interest paymentsand special allowances and the risk that default datadoes not accurately reflect the eligibility of loans forcontinuing interest payments. Reliability risk isdetermined by considering both the planned use ofthe data and the existing knowledge of the computersystem and its data.

Planned Use of Data In gauging how the planned use of computer-baseddata affects reliability risk, staff should considermatters such as the following:

• Will the data be important in determining theaccuracy and appropriateness of payments made tolenders? Will the data merely provide backgroundinformation or provide a context for the assignment’sconclusions? Background information normallysuggests a very low reliability risk.

• Is the computer-based data the only evidenceavailable regarding payments made to lenders? Is thecomputer-based data part of a broader body ofcorroborating evidence? Evidence used as solesupport suggests a high reliability risk, while thereliability risk of corroborative evidence is moderatedby the strength of the other evidence.

• Is the issue of student loan payments and eligibility sosensitive that the accuracy of any data presented(even when used as background) is likely to bechallenged? If there is reason to believe that the data’saccuracy will be questioned, regardless of its use inthe report, the reliability risk increases.

Knowledge andExperience WithData

The second component of reliability risk is recentexperience or knowledge of the data and relatedsystem. Favorable experience and/or knowledge can

GAO/OP-8.1.3Page 38

Chapter 5


reduce reliability risk, limit the review of systemcontrols, and reduce data testing. Unfavorableexperience and/or knowledge leads to increaseddoubts and requires greater assurance that data isaccurate.

In compiling information about the data and itssystem the auditor/evaluator should address thefollowing questions:

• Has GAO used this data base to provide supportingevidence in prior assignments? If so, what was ourassessment of its reliability at that time?

• Has the Department of Education’s Inspector Generalstaff reviewed the related system or assessed thereliability of the data? If so, what recommendations, ifany, did they make for improving system controls?Did Education officials take steps to implement theserecommendations? What opinion, if any, did the IGexpress regarding data reliability?

• What do Education officials and users say about thedata’s accuracy? How frequently do they encountererrors with the data? How serious are theseproblems? Do they rely on the data in performingtheir duties or do they maintain separate manualrecords?

• Have lenders, state agencies, or loan recipientsreported payment problems or concerns?

• Do corroborating sources of information tend tosupport or contradict the computer-based data?

When evaluated together, the planned use of the dataand the current knowledge about it help theauditor/evaluator identify a level of risk. Loweringthat risk to an acceptable level can be accomplishedby performing detailed tests of the data. While theneed for data testing can never be completelyeliminated from an assignment, the extent of testing

GAO/OP-8.1.3Page 39

Chapter 5


can potentially be reduced by assessing the system ofcontrols.

UnderstandingSystem Controls

Understanding and assessing controls is a normalauditing activity. Strong system controls can diminishthe reliability risk, thus reducing the amount of datatesting needed to determine data reliability. In turn,knowledge and experience with the data can helpdirect the review of system controls to areas wherethey are most likely to be weak.

System controls must be considered in terms of bothgeneral and application controls. Work should includegaining an understanding of those controls andobserving that significant controls are being followed.

A review of general controls should include thefollowing questions:

• Does Education’s management take an active role indecisions affecting ADP functions?

• Do external auditors and/or the IG routinely conductreviews of ADP functions? Have Education officialsimplemented all past audit recommendations relatedto ADP operations?

• Does Education’s organization provide adequateseparation of duties within the ADP operation?

• Does Education have standards for documenting ADPfunctions?

• Do formal procedures exist for requesting, approving,testing, and implementing system changes?

• Are appropriate measures in place to physicallysecure Education’s computer facility and control useraccess to the system and data files?

A review of application controls should consider:

GAO/OP-8.1.3Page 40

Chapter 5


• Does Education have formal documentation whichidentifies procedures for data collection,authorization, input, and error handling?

• Does Education’s system perform edit checks on dataprior to combining them with the existing data base?If so, what are those edits?

• Is data which fails to meet input requirementsidentified, corrected, and re-entered to the system in atimely manner?

• Are reconciliations performed to insure that allsource input is accounted for?

• Are system outputs reconciled against inputs toaccount for all data?

The amount of time and effort expended inunderstanding and assessing system controls isdirectly related to the potential reduction on detaildata testing. The “cost” of system control tasks shouldnot outweigh the “benefits” of reduced data testing.

The strength of system controls falls into a range withthe following end points.

• Strong controls: This judgment assumes missing orineffective controls (if any) are minor; the overallsystem could be expected to detect and correct anysignificant data errors.

• Weak controls: This judgment assumes that missingor ineffective controls provide an opportunity forsignificantly incorrect data to be introduced to thedata base. Control deficiencies could pervade theentire system or affect only parts of it.

Data Testing By considering the strength of system controls inrelation to the reliability risk, a level of data testing isestablished. The type of tests are dictated by thenature of the data and the ultimate data analysis to beconducted.

GAO/OP-8.1.3Page 41

Chapter 5


Case 1: ExtensiveTesting

Assume that auditors/evaluators have determined thatEducation’s computer-based data is the only existingsource of payment data. Since neither GAO nor the IGhave done any recent work with this data, thereliability risk is high. The auditors/evaluators havefurther determined that general and applicationcontrols are inadequate. In this instance, the results ofdata testing alone must provide the basis for reliance.Therefore, the number and scope of tests will beextensive.

After conducting procedures to determine thatinformation contained on the lender billingstatements is factual, staff should conduct tests todetermine the accuracy and completeness with whichthat data was entered into the computer. This testingshould generally be based on statistically valid samplesizes and methods. Specifically, tasks would include

• matching computer-based records againstcorresponding source records to measure the datainput error rate, and

• matching source records against correspondingcomputer-based records to determine that all relevantdata had been entered into the computer.

Computer-assisted procedures could then beperformed on all computer-based records to verifythat

• billing and payment dates fall within the assignment’stime frame,

• key data elements are present in all records (i.e.billing date, payment date, payment amount, loanbalance, and so on),

• there are no negative payment amounts or zero loanbalances, and

• payment amounts and loan balances fall within“reasonable” ranges.

GAO/OP-8.1.3Page 42

Chapter 5


Further automated tests could be designed to

• re-compute lenders’ interest calculations,• sort and summarize payments by lender to identify

duplicate records,• match lenders against Education’s list of eligible

institutions,• compare payment dates against billing dates to

determine that billing dates precede payment dates,• compare loan status against default date to insure

that all defaulted loans contain a default date, and• compare loan status against payment amount to

identify records showing payments on defaultedloans.

In addition, staff should review the automated errorfile to determine if it includes billings for the periodthat have not been processed.

The failure of a data element or record to pass areliability test does not prove the data is incorrect. Itmerely identifies a potential matter for furtherinvestigation.

The results of these tests and the follow-upinvestigations will provide numeric error rates. Basedon the error rate and the seriousness of errors, theauditor/evaluator will make a judgment about thedata’s reliability.

Case 2: MinimalTesting

Although the circumstances of this case do not lendthemselves to a discussion of minimal data testing,assume that GAO staff used data from the same database to support report findings within the last 6months. At that time, we concluded that systemcontrols were strong and the data was reliable. Underthis scenario, the reliability risk would be low. An

GAO/OP-8.1.3Page 43

Chapter 5


extensive system control assessment would not beperformed. Reliance would be placed primarily on ourprior knowledge and experience with the data.However, even at this low risk level, some testingshould be performed to update the results of previouswork and detect any conspicuous errors.

Our understanding of the system controls should beupdated to determine

• what, if any, modifications have been made to thesystem,

• that critical controls are still being adhered to, and• that any previous recommendations relating to

system controls have been implemented.

Tracing computer records to source records and viceversa to show completeness and accuracy of datainput could be accomplished through use of small(judgmental) randomly selected samples.

Computer-assisted procedures, aimed at locatinglarge errors, would verify that

• all billing and payment dates fall within theassignment’s time frame,

• key data elements are present in all records (i.e.billing date, payment date, payment amount, loanbalance, and so on),

• there are no negative payment amounts or zero loanbalances, and

• all payment amounts and loan balances fall within“reasonable” ranges.

If these basic tests produced significant error rates,the scope of testing would be expanded. Otherwise,based on the updating of prior reliability work,auditors/evaluators would conclude the data isreliable.

GAO/OP-8.1.3Page 44

GAO/OP-8.1.3Page 45

Appendix I

Examples of Data Tests

This appendix describes some data tests whichshould be considered in developing an overall testingplan. The number and combination of tests performedfor a given assignment will be influenced by therequired level of testing, the complexity and size ofthe data base, and established time frames.

UnconditionalData Tests

The following are examples of data tests that disclosefailures of data elements to meet establishedrequirements:

• Derivation tests identify data errors by using formulasor tables to recalculate computer-generated dataelements.

• Mode tests disclose data that are defective becausethey do not comply with the numeric or alpharequirement for the data element.

• Pattern tests disclose data errors evidenced byinconsistencies of a specific pattern of digits andcharacters. Calendar date checks are a pattern testthat has considerable significance for some dataelements.

• Presence/absence tests disclose data that aredefective because they lack required information orinclude information when they should not.

• Sign tests detect data defects that result from aninappropriate positive or negative value.

• Value/range/limit tests detect data that are defectivebecause they are not within a required set of specificvalues; a set of values that fall into a given range; or aset of values encoded in a list, table, or file.

It is generally useful to test audit-significant dataelements against each of the requirements defined forthem. (Consult the data dictionary.)

GAO/OP-8.1.3Page 46

Appendix I


Conditional DataTests

The following are examples of tests that compare twodata elements that have a logical relationship:

• If college graduation date is given, the type of degreemust be identified.

• If loan date is between July 1, 1989, andSeptember 30, 1989, the interest rate must be12.5 percent.

• Loan approval date must be the same as or later thanthe loan application date.

• If order quantity is greater than 5,000, the discountrate must be 40 percent.

• If payments to an individual under a given entitlementprogram exceed $10,000 in fiscal year 1988, theeligibility code must be “C.”

• The number of program graduates must equal thenumber enrolled minus program dropouts.

These tests are not limited to comparisons of twodata elements in the information system’s data base.They can include data rules that compare particulardata elements with program or legislative criteria orwith information from another data system.

In developing data rules staff should considerwhether reverse relationships exist among dataelements. When information systems are developed,data rules built into the system establish requirementsfor data elements and for relationships among them.At times, reverse relationships are not considered,and reversing data rules is not part of system logic. Inthose cases, the likelihood of data errors is increased.

Well-thought-out reverse rule tests can effectivelydisclose data inconsistencies (overlooked in systemsdesign) that have contributed to data basecontamination over time. A data rule could, forexample, test the requirement that if status isdeceased, the date of death must be present and valid.

GAO/OP-8.1.3Page 47

Appendix I


A reverse data rule could reasonably test that if a dateof death is present and valid, the status must bedeceased.

Staff must exercise care, however, because seeminglyreasonable reverse relationships do not always exist.For example, the rule, “if status is eligible, thenannual income must be less than $10,000,” could betested. But eligibility restrictions may involve factorsother than income, for example, age. If that is thecase, the reverse data rule—“if annual income is lessthan $10,000, then status must be eligible”—could notbe used.

GAO/OP-8.1.3Page 48

Appendix II

Special Considerations inUnderstanding Computer SystemControls

This appendix presents a sample of possible questionsrelating to general and application controls. They areintended to help relate the internal controlapproaches generally followed in performance auditsto the computer environment. 1

General Controls General controls apply to all computer processingcarried out at a facility and are independent ofspecific applications. They relate to organization;system design, development, and modification; andsecurity.

Organization Does top level management take an active role in ADPfunctions?

Does the ADP function received continuing auditcoverage?

Is there evidence of effective actions to follow-up onpast audit recommendations?

Is there adequate separation of duties within the ADPoperation? The following functions are usuallyperformed by a different individual or group:

• system analysis,• application programming,• acceptance testing,• program change control,• data control,• source transaction origination,• system software maintenance,• computer files maintenance, and• computer equipment operation.

1Further guidance is available in GAO’s Evaluating InternalControls in Computer-Based Systems, June 1981 (under revision).

GAO/OP-8.1.3Page 49

Appendix II

Special Considerations in

Understanding Computer System

Controls

System Design,Development, andModification

Controls in this category are intended to insure thatsystems meet user needs, are developedeconomically, are thoroughly documented and tested,and contain appropriate internal controls. Reviewtasks might include the following questions.

Does the agency have a formal approach for systemdevelopment?

Are users involved in the development of systemrequirements?

Do standards exist for documenting different ADPfunctions?

Is the system documentation current and does itinclude:

• functional requirements documents,• data collection requirements,• design characteristics of the systems and component

subsystems,• a user manual,• a system operating manual,• the strategy for testing the computer-based system

including test procedures and evaluation criteria, and• test analyses reports documenting test results and

findings?

Are requests for modifications to existing programsdocumented and approved by appropriatemanagement levels?

Security These controls should provide assurances thatcomputers and the data they contain are properlyprotected against theft, loss, unauthorized access, andnatural disaster? Reviews might consider:

GAO/OP-8.1.3Page 50

Appendix II



Controls

Is a periodic risk analysis performed anddocumented?

Have responsibilities for computer security beenformally assigned?

Is access to the computer room controlled throughuse of some physical device (i.e., locked door,security badges, etc.)

Are two persons present in the computer room at alltimes?

Is the responsibility for storing magnetic data clearlydocumented?

Does the agency have an emergency disaster recoveryplan?

Is the disaster recovery plan periodically tested?

Is computer software used to control access to thecomputer system by identifying and verifying peoplewho try to gain access?

ApplicationControls

Controls which are incorporated directly intoindividual applications are intended to insureaccurate and reliable processing. They address thethree major operations of data input, data processing,and data output.

Data Input Controls in this category are designed to insure thatdata is converted to an automated form and enteredinto the application in an accurate, complete, andtimely manner. Review tasks might address thefollowing questions.

GAO/OP-8.1.3Page 51

Appendix II



Controls

Do documented procedures exist for entering datainto the application?

Are controls in place which permit the number ofrecords input to the application to be reconciledagainst the number presented for entry?

Do all source records contain some indication ofauthorization (either physical or electronic)?

Are security measures in place to limit access to inputterminals and validate user sign-on?

Is data validation and editing performed on all datafields before entry into the system?

Are uses of methods to override or bypass datavalidation and editing procedures recorded andanalyzed for appropriateness and correctness bysupervisory personnel?

Do documented procedures exist that explain theprocess of identifying, correcting, and reprocessingdata rejected by the application?

Is all data that does not meet edit requirementsrejected from further processing and written to anautomated suspense file?

Is the automated suspense file used to controlfollow-up, correction, and reentry of rejected data?

Is the automated suspense file regularly analyzed todetermine the rate of data input error and the statusof uncorrected records?

Are corrective actions taken when error rates becometoo high?

GAO/OP-8.1.3Page 52

Appendix II



Controls

Are counts of rejected items produced and reconciledwith accepted records to account for all input?

Data Processing Processing controls are designed to insure that data ishandled by the computer in an accurate, complete,and timely manner. Review tasks might include thefollowing questions.

Do documented procedures exist to explain themethods for proper data processing of eachapplication program?

Does a history log record events performed by thecomputer and its operators during applicationprocessing?

Are application programs secured against direct inputfrom operator consoles?

Do on-line systems protect against concurrent fileupdates?

Are controls in place to prevent operators fromcircumventing file checking routines?

Are file completion checks performed to make surethat application files have been completelyprocessed?

Do processing controls make sure that output countsfrom the system equal input counts to the system?

Is relationship editing performed between inputtransactions and master files to check forappropriateness and correctness before updating?

GAO/OP-8.1.3Page 53

Appendix II



Controls

Data Output Output controls are used to insure the integrity ofsystem output and the correct and timely distributionof outputs. Review tasks could address the followingquestions.

Do documented procedures exist that explain theprocedures for balancing, reconciling, anddistributing output products?

Are users questioned periodically to determine theircontinued need for the product?

Is each output product labelled to identify the productname, recipient’s name, and time and date ofproduction?

Do documented procedures exist that explainmethods for reporting, correcting, and reprocessingoutput products with errors?

Are input record counts and controls totalsreconciled against output record counts and controltotals to insure that no data was lost or added duringprocessing?

Are system outputs reviewed for completeness andaccuracy before release to users? Does this reviewinclude reconciling record counts and control totals?

Are source documents retained and stored in a logicalsequence for easy retrieval?

GAO/OP-8.1.3Page 54

Ordering Information

The first copy of each GAO report and testimony

is free. Additional copies are $2 each. Orders

should be sent to the following address,

accompanied by a check or money order made

out to the Superintendent of Documents, when

necessary. Orders for 100 or more copies to be

mailed to a single address are discounted

25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax number

(301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available

reports and testimony. To receive facsimile

copies of the daily list or any list from the past

30 days, please call (301) 258-4097 using a

touchtone phone. A recorded menu will provide

information on how to obtain these lists.

United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Mail

Postage & Fees Paid

GAO

Permit No. G100