Citation: McLeod, Julie, Childs, Sue and Heaford, Susan (2007) Records management capacity and compliance toolkits : a critical assessment. Records Management Journal, 17 (3). pp. 216-232. ISSN 0956-5698

Published by: Emerald

URL: http://dx.doi.org/10.1108/09565690710833116 <http://dx.doi.org/10.1108/09565690710833116>

This version was downloaded from Northumbria Research Link: http://nrl.northumbria.ac.uk/1079/

Northumbria University has developed Northumbria Research Link (NRL) to enable users to access the University’s research output. Copyright ©  and moral rights for items on NRL are retained by the individual author(s) and/or other copyright owners. Single copies of full items can be reproduced, displayed or performed, and given to third parties in any format or medium for personal research or study, educational, or not-for-profit purposes without prior permission or charge, provided the authors, title and full bibliographic details are given, as well as a hyperlink and/or URL to the original metadata page. The content must not be changed in any way. Full items must not be sold commercially in any format or medium without formal permission of the copyright holder. The full policy is available online: http://nrl.northumbria.ac.uk/policies.html

This document may differ from the final, published version of the research and has been made available online in accordance with publisher policies. To read and/or cite from the published version of the research, please visit the publisher’s website (a subscription may be required.)

1

Records management capacity and compliance toolkits: a critical

assessment

Julie McLeod, Sue Childs and Susan Heaford

Northumbria University

Abstract

Purpose:

This article presents the results of a project which critically evaluated a series

of toolkits for assessing records management capacity and/or compliance.

These toolkits have been developed in different countries and sectors within

the context of the e-environment and providing evidence of good corporate

and information governance.

Methodology:

A desk-based investigation of the tools was followed by an electronic Delphi

with toolkit developers and performance measurement experts to develop a

set of evaluation criteria. Different stakeholders then evaluated the toolkits

against the criteria using cognitive walkthroughs and expert heuristic reviews.

The results and the research process were reviewed via electronic discussion.

Findings;

Developed by recognised and highly respected organisations, three of the

toolkits are software tools the fourth is a methodology. They are all

underpinned by relevant national / international records management

legislation, standards and good practice including, either implicitly or explicitly,

ISO 15489. They all have strengths, complementing rather than competing

with each other. They enable the involvement of other staff thereby providing

an opportunity for raising awareness of the importance of effective records

management.

Practical implications:

These toolkits are potentially very powerful, flexible and of real value to

organisations in managing their records. They can be used for a ‘quick and

dirty’ assessment of records management capacity or compliance as well as

in depth analysis. The most important criterion for selecting the appropriate

one is to match the toolkit with the scenario.

2

Originality/value of paper:

This paper aims to raise awareness of the range and nature of records

management toolkits and their potential for varied use in practice to support

more effective management of records.

Keywords: Records management; Toolkits; Compliance; Assessment;

Benchmarking; Research

Category: Research paper

3

Context

Records support more effective and efficient business, underpin e-

government and service delivery, help to demonstrate accountability,

transparency and corporate governance, and are the source of information for

citizens in the context of open government and freedom of information. But,

only since the end of the 20th century and the current millennium has the

importance of effective records management been widely recognised:

“Good records management should be seen as a benefit, not a burden”

(Information Commissioner's Office, 2006, p.1)

In the UK at least this has been due to a combination of factors. Public sector

organisations realised that effective records management was key to them

being able to comply with new freedom of information legislation and

environmental information regulations (Great Britain, 2000 and 2004). Many

used the Model Action Plans (MAPs) produced by The National Archives

(TNA) (e.g. The National Archives, 2002) to benchmark their current practice

against the recommendations for compliance with the Code of Practice on

records management under Section 46 of the Freedom of Information Act

2000, issued by the then Lord Chancellor’s Office (Lord Chancellor’s Office,

2002).

Both public and private sector organisations have been affected by scandals,

failures and embarrassing situations which have involved, and in some

instances been the result of, mis-management of records. In the USA the

Enron and Worldcom financial scandals resulted in the US Congress passing

new legislation, the Sarbanes-Oxley Act or SoX (United States of America,

2002), imposing more stringent recordkeeping requirements (Stephens,

2005). The Bichard Enquiry (http://www.bichardinquiry.org.uk/) explored

issues arising from the Soham murders, including intelligence-based record

keeping and the lack of records to support information sharing between public

sector agencies. And Moss (2005) critically reviews the inadequacies of

recordkeeping as revealed by the Hutton Enquiry. There are many other

4

examples of negative scenarios which have raised the profile of records

management in today’s e-information society.

On a positive note ISO 15489 (2001) provides an authoritative source for

promoting effective records management as underpinning effective business

management. Since the publication of this important best practice standard,

guides have been produced and training opportunities provided to help

organisations successfully implement it. Examples in the UK are the

BIP0025:1-4 guides (BSI 2002, 2003 and 2007) and workshops provided by

BSI, the Records Management Society (www.rms-gb.org.uk) and TPFL

(www.tfpl.com).

In addition, since the publication of ISO 15489 (2001) a range of toolkits has

been developed for different yet related purposes within the broad context of

measuring records management capacity, compliance and/or readiness in the

electronic environment. They have been developed by different organisations

in different countries and from different sectors. This article shares the results

of a short project, undertaken from September 2005 to February 2006, which

investigated the development and application of some of these toolkits.

Project aims and research questions

The aim of the project was to critically evaluate four toolkits for assessing

records management capacity and/or compliance from both the theoretical

and practical aspects. In doing so it sought to answer the following questions:

Why were the toolkits developed? Why did their developers see a need for

a toolkit and invest in their development? What was the rationale?

What is the purpose of the toolkits and who are the intended users? Are

they the similar or different?

What models, theoretical frameworks and/or principles underpinned the

toolkits? Why were these chosen and were any others considered and

rejected?

5

What is the underlying design and technology used and why was this

chosen?

Who is actually using the toolkits, how and why? How practical are they to

implement? How effective are they? What value and benefits have been

gained by deploying them?

What are the strengths of the toolkits?

How do the toolkits compare in terms of appropriateness for different

scenarios or contexts?

The work built on the results of a previous project which assessed the impact

of ISO 15489 in the UK (McLeod, 2004a, b; McLeod, 2005; McLeod and

Childs, 2006) and highlighted the need for practical tools to help organisations

assess their compliance with the ISO standard.

Records management toolkits

The four toolkits selected for the study were (in alphabetical order):

Information Governance Toolkit (IGT)

developed by the UK National Health Service (NHS) and only available

via NHSnet; designed to assess Information management (IM)

governance in NHS organisations

https://www.igt.connectingforhealth.nhs.uk/

Information Management Capacity Check (IMCC) Tool and

Methodology

developed by the Library and Archives Canada; designed to assess IM

capabilities of Canadian federal organisations

http://www.collectionscanada.ca/information-management/002/007002-

2003-e.html

Records Management Capacity Assessment System (RMCAS)

developed by the International Records Management Trust; designed

to assess Records and Information Management (RIM) capacity in the

6

public sector, particularly for developing countries

http://www.nationalarchives.gov.uk/rmcas

RiskProfiler

developed by ARMA International and NetDiligence; designed to

assess RIM for compliance (available to all organisations for a fee)

http://www.arma.org/standards/eassessment.cfm

Two of the toolkits explicitly incorporate the requirements of ISO 15489 within

their design and two encompass the spirit of what the standard seeks to

ensure viz. “that appropriate attention and protection is given to all records,

and that the evidence and information they contain can be retrieved more

efficiently and effectively, using standard practices and procedures” (ISO

15489-1, 2001, p vi). The organisations producing the toolkits were keen to

participate in the project, believing it would add value for the profession and

wider potential user community.

A literature review at the start of the project found that records management

toolkits had not been widely discussed and literature on the four particular

toolkits was very limited, other than press releases on the Web.

Three articles were found referring to records management toolkits in general.

Harries (2001, p36) refers to ‘workflow and object-orientated toolkits’, in the

context of software packages that address electronic document and records

management. Barata and Cain (2003) discuss a wide range of records

management ‘toolkits’ which include methodologies, standards and codes of

practice. And Bailey (2003, p.27) refers to the development of an Electronic

Records Management Training Package as a tool for raising awareness and

promoting best practice in records management where there is none. (This

tool was in fact developed by Northumbria University (Hare, 2003) and has

since been replaced with a training guide for administrators (JISC infoNet

http://www.jiscinfonet.ac.uk/records-management/guide-for-administrators).

Other articles describe one or other of the toolkits covered in this project:

7

Carlisle (2004) on the ARMA toolkit; Demb (2004) and Griffin (2004) on the

RMCS toolkit; Wells (2004) on the IGT.

A few articles were found on evaluating toolkits. Thebridge (2004) and

Greenwood and Davies (2004) discuss toolkits in library contexts, the latter

providing interesting and relevant background from the toolkit developer’s

perspective. They describe designing a toolkit for evaluating a project as a

“formidable task” where affordability was key for both the funders and project

co-ordinators and concluded that the toolkit development process

“demonstrated the importance of properly framed evaluation in achieving

excellence and in advocacy” (Greenwood and Davies, 2004, p 110 and p112).

Their experience was relevant to both the evaluation process undertaken in

this research project and the understanding of the outcomes from the

perspectives of different stakeholders, viz. users and developers. Haswell and

Banwell (2004) report on an investigation into existing toolkits for ICT

evaluation but do not explain how to evaluate a toolkit.

Two articles were useful in exploring definitions of ‘evaluation’ and ‘toolkits’.

Banwell (2000, p173) describes ‘evaluation’ as a complex field, associated

with a range of other concepts including ”performance measurement and

benchmarking, quality, validity, effectiveness, value for money, best value and

audit”. Oliver and Conole (2000, p32) define ‘toolkits’ as “decision making

systems based on expert models” which they expand on saying:

“all toolkits include an expert model of a process derived from recognised

theory and best practice ... [they] produce documentary evidence of

assumptions, process and outputs ... for quality assurance and

assessment purposes ... bringing best practice within the reach of all

practitioners in a usable format” Oliver and Conole (2000, p35).

These definitions are important since the word ‘toolkit’ can vary from discipline

to discipline, the types of tool can vary and the scope of an evaluation must be

clear.

Methodology

8

To achieve the aims and answer the research questions the project

investigated:

(a) the context and purpose(s) of each tool

(b) the underlying principles and models of their design, and

(c) their utilisation and the benefits realised together with their strengths

and areas for improvement from the stakeholders’ perspectives.

The qualitative methodology comprised four main phases:

an initial desk-based investigation of the toolkits, entailing a focused

review of the literature on toolkits generally, and obtaining factual

information about the structure, design and output of each of the

specific toolkits from manuals, background information and contact with

the developers

an e-Delphi study with toolkit developers and performance

measurement experts to develop a set of criteria for evaluating any

records management toolkit

the toolkit evaluation undertaken by different stakeholders, using the

project-developed criteria in cognitive walkthroughs and expert

heuristic reviews

review of the results and evaluation of the research (project) process,

via electronic discussion.

Development of evaluation criteria via a Delphi study

The e-Delphi study was used to gather expert opinion on toolkit design and

development and to determine a set of evaluation criteria. This technique was

developed in the 1950s at the Rand Corporation to gather a consensus of

‘expert’ opinion (Gupta & Clarke, 1996, p.185). We used a relatively ‘classic’

Delphi technique (e.g. Linstone and Turoff, 2002) although it was conducted

electronically via email. This approach enabled experts located in different

9

parts of the world to participate in the anonymous generation and agreement

of evaluation criteria. They were a combination of three stakeholders involved

in the development of the toolkits and two objective experts, one a records

practitioner, the other a researcher with expertise in evaluation. The first round

was used to elicit ideas for evaluation criteria. The experts’ suggestions were

analysed qualitatively to produce a first set of criteria, organised under

categories. These were then fed back to them for ranking in order of

importance and to identify any gaps. These results were then analysed

quantitatively. However, there was little difference in ranking of the criteria;

they were deemed to be of equal importance. (The evaluation criteria are

given in Table 1). Though developed in the context of records management

toolkits, these criteria are sufficiently generic that they could be used to

evaluate any type of toolkit measuring an organisation’s performance of their

business processes.

Toolkit evaluation via a cognitive walkthrough and heuristic reviews

The evaluation criteria were then used to develop the structure of a cognitive

walkthrough (Wharton, 1994; Bias, 1994) used by two researchers to conduct

an independent and objective assessment of each toolkit. The cognitive

walkthrough approach to evaluation has its origins in software engineering

and involves a “detailed review of a sequence of actions” (Abowd, 1995). It is

based on the information processing model of human cognition i.e. a goal is

set, a system is searched for available action to meet the goal, the action is

selected, the action performed, the user evaluates performance and

remembers success or failure. The purpose is to evaluate the system not the

user.

The researchers conducting the cognitive walkthrough were not records

management experts though they were information management experts, and

were obviously records creators and users with particular

knowledge/awareness of the need to manage research records appropriately.

The decision not to use records experts for this part of the evaluation was

deliberate. It enabled them to evaluate the usability of the toolkit from a

layperson’s perspective, to become familiar with each toolkit, and to test the

10

suitability and clarity of the evaluation criteria. The researchers used their own

organisational context (the subject team within the School of Computing,

Engineering and Information Sciences at Northumbria University) as the

context for using and assessing each toolkit, commenting on each step as

they systematically worked their way through the toolkits sequentially from

‘introduction to conclusion’.

Two further assessments were undertaken: one by a researcher with records

management expertise, but no experience of using the toolkits under study;

the other by a real toolkit user (one for each of the toolkits). With the help of

the toolkit producers one expert user was identified for each toolkit in all but

one case, where the expert user was a person known to the researchers.

Asking the producers to identify these experts had the potential to introduce

bias. However, it was important to identify users who were able to give a full

evaluation, given only one expert user was to be used per toolkit, and this was

a pragmatic approach to identifying them. The expert users had all used the

particular toolkit in a real situation to ensure their evaluation was well informed

and not superficial. Some were more experienced in using a toolkit than

others.

These assessments took the form of a modified expert heuristic review

(Nielsen, 1994).

“Heuristic review is a type of expert evaluation, where experts review a

product's usability. It is an easy to learn method that can be quickly applied

… to roughly determine the usability of … software products” (OCLC, no

date).

In a heuristic review the experts bring their own knowledge to their

assessment; they have previously learned and internalised appropriate

heuristics which they apply in a more informal way to the evaluation task.

They do not necessarily follow the set of sequential steps used by the

cognitive walkers but conduct their evaluation as a more random, non-linear

11

process. However, in our approach the project-developed criteria were also

used by the records expert and the expert toolkit users. Use of toolkit users

extended the normal scope of a heuristic review beyond an evaluation of the

user interface and ease-of-use of the toolkit by, for example, reviewing the

usability of the results and analysis, and the tangible and intangible value and

benefits of the process and outputs within the context of everyday activities.

All of the evaluation was conducted virtually, either electronically or via

telephone calls and email correspondence, with the exception of one toolkit,

the IGT. This required access over the NHS secure intranet and entailed two

on-site visits. In all but one case, the IMCC, it was possible to have face-to-

face discussions with the toolkit producers to either clarify factual queries or

learn about the toolkit. In both the cognitive walkthroughs and the heuristic

reviews the (electronic) evaluations were conducted independently and

individually for each toolkit by answering the questions which formed the

evaluation criteria.

The individual results were collated into one document for each toolkit. The

project staff discussed in depth the results for each toolkit, noting

commonalities and disagreements. A final consensus evaluation was then

agreed for each toolkit. Additionally, commonalities between the results for all

the toolkits enabled generic recommendations for good practice in developing

and revising records management toolkits to be drawn up (McLeod, Childs

and Heaford, 2006a). The fact that two researchers had completed a cognitive

walkthrough of each toolkit meant that in the analysis the project staff could be

alerted to any subjectivity in the data from the expert users that could have

been the result of familiarity with the toolkit. This methodological approach to

the evaluation involving three types of users (expert users, a records

management expert who was not a user of the toolkits in a real situation and

non-records management experts who were expert researchers) enabled

triangulation of the data collected and hence the robustness and validity of the

results; in other words there was a 360 degree evaluation of each toolkit.

12

Analysis of the results and research process via e-discussion

The final phase of the investigation was an electronic discussion with the

developers involved in the initial e-Delphi who were presented with an

analysis of the evaluation results. They were asked to comment on the results

and the approach taken to the research.

Findings

The results of the assessment are presented in two forms. First, is a brief

textual summary of each toolkit, which addresses some of the research

questions posed, in particular why the toolkits were developed, their purpose

and intended users. This is followed by a summary evaluation under each of

the remaining research questions. For further details of the toolkit features

and use see the separate guide to records management toolkits (McLeod,

Childs and Heaford, 2006b

http://northumbria.ac.uk/sd/academic/ceis/re/isrc/themes/rmarea/tlkit/)

Overview of each toolkit

Information Governance Toolkit (IGT)

Use of the IGT is internal to the NHS and NHS organisations. Its aim is to

assess NHS organisations’ information governance and their compliance with

legal and regulatory requirements; records management is included in the

assessment. The toolkit is applied organisation-wide. It was first issued in the

NHS financial year 2003-04 and is revised annually. Its use is a mandatory

annual requirement as part of the NHS assurance framework. The Healthcare

Commission (http://www.healthcarecommission.org.uk) also uses the toolkit

as part of their audit of NHS organisations’ performance. An NHS Records

Management Code of Practice has recently been developed (Department of

Health, 2006) and will inform the content of the IGT. The version assessed by

the project covers the following elements: information governance

management, records management (primarily health records), freedom of

information (including records management for administrative records), data

protection, confidentiality, data quality/accreditation, information security, and

the NHS National Plan for IT. The assessment, in the form of easy-to-answer

questions, is completed through teamwork involving a small number of

13

‘reviewers’ and ‘users’ and one administrator. Data is input into a Web-based

tool via radio buttons. Results are presented in an easy-to-understand way

using traffic lights. The toolkit provides a simple system for organisational

benchmarking/auditing, with comparison to last year’s results and guidance

for future improvement. Help and guidance on using the toolkit is available via

the Web, along with links to a comprehensive collection of internal and

external resources (e.g. standards, legislation, good practice, examples). The

results of other NHS organisations are available for comparison; NHS

organisations publish their results for the public to see and the Healthcare

Commission’s audits are also publicly available.

Information Management Capacity Check Tool and Methodology (IMCC)

The IMCC, although aimed at helping Canadian federal departments and

agencies to assess their current information management capabilities, is

freely available on the Web in a variety of textual forms (html, rtf, pdf, ppt); it is

a ‘methodology’ not a software tool. The tool comprises six key elements of

information management practices (i.e. organizational context; organizational

capabilities; management of information management; compliance and

quality; records and information life cycle; user perspective) with criteria under

each element, and assessment of each criterion at one of five capacity levels.

The methodology comprises setting up a project team, collecting data from

staff (through workshops and interviews) and by analysis of documentation,

establishing results and assessment through discussion, and producing a

report and action plan for future improvement. Guidance on conducting the

review is available but there are no explicit links to background resources

(other than the Library and Archives Canada Web site). The tool is very strong

on the ‘process’ of evaluating information management capacity and, by

involving so many staff members, develops teamwork and communication

channels, engages people with records management, and should result in the

setting up of a process of continuous improvement. However, such an

approach can be very human and time resource intensive.

Records Management Capacity Assessment System (RMCAS)

14

RMCAS is designed to help assess records and information systems capacity

in public sector organisations, particularly those in developing countries. It is a

downloadable software tool freely available from the Web. It focuses on

corporate governance issues: law, policies and procedures; ICT–records

management integration; resources and training; records management

programme management; awareness and ownership; business function–

records management integration. The toolkit is versatile and may be used at

different organisational levels (e.g. sector, organisation, department, project,

system). Questions can be answered via radio buttons or text replies. Each

capacity statement is traceable to internationally recognised best practice.

Assessment does not have to be completed in one go; the assessor can

return to the toolkit at their convenience. A wide range of outputs is produced,

e.g. 2-d / 3-d graphical summaries, detailed and customisable reports,

detailed reports for individual respondents’ further interpretation. RMCAS is a

powerful, detailed, sophisticated, comprehensive tool; however this means

that investment in learning how to use it is required for users to become

familiar with terminology and to reap the value of its complexity and depth /

breadth of coverage and analysis. A separate user guide in pdf is available

from the producer’s Web site and the software provides extensive online help.

RiskProfiler

RiskProfiler is a Web-based software tool available on payment of a fee, of a

level such that it should be within the means of all but the smallest of

organisations. The tool assesses an organisation’s records management

programme against internationally recognised standards with the aim of

identifying risks of non-compliance with legislation and regulations. It covers

the organisation’s demographic details; policies and procedures; overall

program structure; classification plan effectiveness; records security and

protection; active records program effectiveness; inactive records program

effectiveness; monitoring and training. The toolkit is very easy and intuitive to

use. Questions are easy-to-answer with radio buttons and some limited

textual input. Assessment does not have to be completed in one go; the

assessor can return at their convenience. Summary report cards are produced

showing scores for each section and any issues that should be addressed,

15

linked to a brief summary of best practice for future improvement. Traffic light

results are also provided. The tool also enables an organisation’s results to be

compared against the anonymised results of other organisations in their

sector, as well as with their own previous assessment.

Answers to research questions

What models, theoretical frameworks and/or principles underpin the

toolkits? Why were these chosen and were any others considered and

rejected?

Relevant national / international records management legislation, standards

and good practice underpin all of the toolkits. This provides authority and is a

quality indicator. ISO 15489 (2001) explicitly underpins RMCAS and

RiskProfiler and is referred to by the IGT, but is not explicitly referenced in the

IMCC toolkit. In some of the toolkits the evaluation criteria are clearly

traceable to specific statements in the legislation/standards/good practice,

with links to good practice guidance to enable change and improvement.

There was no indication that alternative models etc. had been considered and

rejected.

What is the underlying design and technology used and why was this

chosen?

Three of the toolkits (IGT, RMCAS, RiskProfiler) are software tools for data

input, analysis and report generation. However, in each case, the automated

analysis process is ‘hidden’, the tools functioning as ‘black boxes’. The IMCC

comprises documentation describing the process to be undertaken. In addition

to the transparent assessment of information management capabilities the

process engages staff with records management and encourages change.

Interestingly, none of the software toolkits explicitly catered for the needs of

those with impairments (e.g. visual or mobility) by, for example, following WC3

guidelines (http://www.w3.org/WAI/intro/accessibility.php).

16

Who is actually using the toolkits, how and why? How practical are they to

implement? How effective are they? What value and benefits have been

gained by deploying them?

The IGT has a clearly defined user group (UK NHS organisations) and its use

it mandatory. Producers of the other toolkits have their own (confidential)

information about their toolkit users. In some cases the users match their

target audience and in others (e.g. those freely available on the Web) they are

broader. In the four real examples provided by the expert users, each toolkit

was found to be very effective. They were used for different purposes:

an initial consultancy assessment for a small non-profit making

organisation where the outputs were valuable in communicating the

state of recordkeeping

to give “a rather complete overview of where we stood in the ‘battle of

compliance’ and thus in our goal to establish good overall record and

information management” in a large government department

to self-assess the status of information management maturity in two

national bodies prior to their merger and to determine a desired state of

maturity post merger. “It was a great tool to bring people together to

discuss issues of IM [information management] and to learn more

about what other people in the institution do”

for a mandatory annual assessment in an NHS Trust, which was an

“effective way of focusing the Trust across the range of IG [information

governance] requirements including RM”, the results forming an annual

work plan against which to measure progress.

What are the strengths of the toolkits? How do the toolkits compare in terms of appropriateness for different scenarios

All of the toolkits have strengths. Because they each have a different purpose,

audience and/or design, they ‘complement’ rather than ‘compete’ with each

other. One advantage of the toolkits is their flexibility; users can adapt them to

their own needs, using them either comprehensively or in a ‘quick and dirty’

fashion. The toolkits met their stated objectives, and were practical to

implement, albeit with the need for minor improvements. The real-life users

said the toolkits met their own particular objectives, were effective, and

17

assisted in improving records management within their organizations. They

would all use such toolkits again. However, the results from using any toolkit

depend on the thoroughness and accuracy of the data ‘input’ by the user. This

thoroughness and accuracy would clearly be improved by the involvement of

records management and archives staff in the process.

Conclusions and recommendations

Toolkits are so rich that ideally the evaluation of these toolkits would have

been much longer involving, for example, more expert users. A more in-depth

and lengthy application in our own (university) organisational context and/or

evaluating each toolkit using the scenario or approach they were

designed/targeted for in the university context, would have been interesting.

This would have meant involving senior management and other stakeholders

in using the IMCC and RMCAS toolkits; involving the university’s records

manager and consulting all of the documentation for ARMA’s RiskProfiler

toolkit; and involving multiple reviewers with the IGT. But the constraints of

the project design, existing knowledge of the toolkits at the time the project

was designed, and the desire to conduct a timely project precluded this.

Despite these limitations the project delivered some valuable outputs, viz.:

i. a set of evaluation criteria to use for evaluating and selecting any

records management toolkit and potentially, with some adaptation,

other information management toolkits (Table 1);

ii. a guide to four toolkits, including real case examples, and

recommendations for their practical use (McLeod, Childs & Heaford,

2006b);

iii. a series of generic good practice recommendations for toolkit

developers in the initial development or revision of a records

management toolkit (McLeod, Childs & Heaford, 2006a, Appendix B).

The results highlight the similarities of the toolkits (e.g. their design based on

best practice and internal/external standards) as well as their differences (e.g.

format and intended audience). They demonstrate the variety offered by just a

18

small number of toolkits and consequently their combined potential value for

many organisations.

None of the toolkits evaluated was suitable for all organisations and all

situations; indeed none made that claim. They are all relatively easy to use,

the more detailed ones requiring more subject expertise to gain maximum

benefit and ensure reliable and accurate results. They offer the potential to

assess compliance and/or capacity, benchmark against standards (in some

cases benchmark against other organisations), identify strengths, weaknesses

and areas for improving an organisation’s records management. The case

examples provided by the expert users illustrate how toolkits can be used for

different purposes and at different levels, e.g. in a ‘quick and dirty’ manner or

in detail. At the same time they can be used to work with others during the

data collection and/or analysis stages, to raise awareness, communicate and

build partnerships for managing records effectively.

So what should users look for in selecting an appropriate toolkit? The

conclusion was that the most important criterion is to match the toolkit with the

scenario, i.e. to select a toolkit whose purpose and intended audience

matches that of the user.

Developed by recognised and highly respected organisations, committed to

their use, development, support and maintenance, each one offers something

different and, together, they offer a valuable resource and powerful

opportunity for records managers, information managers, information security

managers, information and corporate governance officers, auditors and others

to assess, benchmark, monitor and develop better records management in

support of organisational goals.

Toolkits such as these are potentially very powerful and flexible and of real

value to organisations in managing their records. They deserve to be part of

the records professional’s total toolkit, indeed non-records professionals may

also use them. Their application is limited only by the imagination of those

who use them. But judging by the limited literature on them awareness of

19

these tools was not high at the time of the research. They were not well

known or well-established in terms of being automatically considered and

commonplace. However, since this project was completed the interest in

toolkits appears to have increased and new ones have been developed. For

example, Anderson (2007) has discussed the ARMA toolkit in this study and

the Local eGov Standards Body began work on an information governance

assessment toolkit for local government bodies

(http://www.legsb.gov.uk/blueprints/item.php?id=512) based on the NHS IGT

studied in our project. Also in the UK, Blake (2007) profiled a new self-

assessment tool developed by The National Archives in the UK. In the form of

an evaluation workbook, it is designed to help public authorities assess their

conformance to the Code of Practice on managing records in the context of

section 46 of the Freedom of Information Act 2000 (Lord Chancellor’s Office,

2002)). Despite these developments there remains little formal evaluation or

publications about their use.

A final interesting point was raised by one participant in the process of

evaluating the project:

“I would like to have seen more analysis of the problem of evaluating

records management specifically (i.e. RM is deeply integrated with

business functions and ICT, how do you best address that in a RM

evaluation? which viewpoint, direction do you take? How do you weigh the

value of variable results?) … I would have liked to have seen a rough

baseline of best practice for RM Evaluation.”

Whilst the evaluation of records management itself was outside the scope of

this project, it is an important issue. It encompasses other aspects, e.g. value,

performance measurement, quality, and would be an interesting area for

research.

20

Acknowledgement

The project was funded by the Arts and Humanities Research Board (now the

arts and Humanities Research Council - AHRC).

References

Abowd, G. (1995). Introduction to software engineering.http://www.cc.gatech.edu/computing/classes/cs3302/documents/cog.walk.html

Anderson J (2007). Auditing an organization's RIM program. InformationManagement Journal, 41(2), p30-6.

Bailey S (2003). JISC 09/02 Supporting Institutional Records Managementprogramme. Records Management Bulletin, 113, p27

Banwell L (2000). Evaluating information services. In: Booth A, Walton G (Eds).Managing knowledge in health services . London, Library Association Publishing, pp.173-181

Barata K, Cain P (2003). Records management toolkits from across the pond.Information Management Journal, Jul/August, 37(4), pp. 40-48

BIP 0025-1 (2002). Effective records management: a management guide to the valueof BS ISO 15489-1. BSI

BIP 0025-2 (2002). Effective records management: practical implementation of BSISO 15489-1. BSI

BIP 0025-3 (2003). Effective records management: performance management for BSISO 15489-1. BSI

BIP 0025-4 (2007). Effective records management: how to comply with BS ISO15489-1. BSI

Bias RG (1994). The pluralistic usability walkthrough: coordinated empathies. In:Nielsen, J., Mack, R. L. (Eds) Usability inspection methods, John Wiley & Sons, NewYork.

Blake, R. (2007). Assessing information governance. Electronic Document andRecords Management in the Public Sector Cimtech/TNA Conference 22-23 May,Hatfield, UK. (For the self assessment workbook seehttp://www.nationalarchives.gov.uk/recordsmanagement/code/assessing.htm)

Carlisle D (2004). The International Records Management Standard (ISO 15489):Early indications and developments. 15th International Congress on Archives,Vienna, 23-27 August 2004http://www.wien2004.ica.org/fo/speakers.php?ctNv1=48&IdSpk=163

Demb S (2004). An introduction to Records Management Capacity AssessmentSystem (RMCAS), ESARBICA Newsletter, Oct, (5), pp.8-13.http://www.geocities.com/esarbica/esarnews5.pdf

21

Department of Health (2006). Records management: NHS code of practice. Parts 1and 2. Department of Health.http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_4131747

Great Britain (2000). Freedom of Information Act 2000: Chapter 36. London, TheStationery Office. http://www.opsi.gov.uk/ACTS/acts2000/20000036.htm

Great Britain (2004). The Environmental Information Regulations 2004: StatutoryInstrument 2004 No. 3391. London, The Stationery Office.http://www.opsi.gov.uk/si/si2004/20043391.htm

Greenwood H and Davies JE. (2004). Designing tools to fill the void: a case study indeveloping evaluation for reading promotion projects. Performance Measurementand Metrics, 5(3, pp16-111.

Griffin A (2004). Records Management Capacity Assessment System (RMCAS).Archival Science, 4(1-2), pp71-97

Gupta, U. G. & Clarke, R. E. (1996) Theory and application of the Delphi Technique.Technological Forecasting and social change, 53, pp185-211.

Hare C (2004). JISC Circular 09/02, Supporting Institutional Records Management,Theme 3, Electronic Records Management Training Package. Final report. Newcasleupon Tyne, Information Management Research Institute, Northumbria University.http://www.jisc.ac.uk/uploaded_documents/JISC%20Final%20report%20Theme%203.doc

Harries S (2001). Strategic use of electronic records. Integrating electronic documentand electronic records management in the government sector. InformationManagement & Technology, 34(1), pp32-34,36,41

Haswell G and Banwell L (2004). Toolkits and the assessment of outcomes: the ICTEVALKIT Project. VINE: The Journal of Information and Knowledge ManagementSystems, 34(3), pp92-98.

Information Commissioner's Office (2006). Freedom of Information Act. AwarenessGuidance Number 8. Records Management FAQs. Information Commissioner'sOffice.http://www.ico.gov.uk/upload/documents/library/freedom_of_information/detailed_specialist_guides/awareness_guidance_8_-_records_management_faqs_v2001.pdf

ISO 15489 (2001). Information and documentation - records management. Part 1:General. and ISO 15489-2. Part 2: Guidelines. ISO.

JISC infoNet (s.d.) Managing information to make life easier: a practical guide foradministrators. http://www.jiscinfonet.ac.uk/records-management/guide-for-administrators

Linstone, H. A., Turoff, M. (Eds.) (2002) The Delphi method: techniques andapplications. A Web book - a reproduction of the original 1975 text. InformationSystems Department, College of Computing Sciences, New Jersey Institute ofTechnology, Newark. http://www.is.njit.edu/pubs/delphibook

22

Local eGov Standards Body. Information governance toolkit for local governmenthttp://www.legsb.gov.uk/ http://www.legsb.gov.uk/blueprints/item.php?id=512

Lord Chancellor’s Office, 2002. Lord Chancellor’s Code of Practice on theManagement of records under Section 46 of the Freedom of Information Act. London,Lord Chancellor’s Office. http://www.foi.gov.uk/reference/imprep/codemanrec.htm(Now at Department for Constitutional Affairshttp://www.dca.gov.uk/foi/reference/imprep/codemanrec.htm)

McLeod J (2004a). ISO 15489: helpful, hype or just not hot? Archives & Manuscripts,32(2), pp90-113

McLeod J (2004b). Assessing the Impact of ISO 15489 in Practice. Early indicationsfrom the UK. 15th International Congress on Archives, Vienna, 23-27 August 2004

McLeod J (2005). Using the ISO 15489 standard: strategic step or sterile selection?Records Management Conventie Seminar, Utrecht, June 2005

McLeod J, Childs S (2006). Assessing the impact of ISO 15489: the first internationalstandard for records management. Final project report, October 2005. Newcastleupon Tyne, Information Society Research & Consultancy Group, School ofComputing, Engineering and Information Sciences, Northumbria University.http://northumbria.ac.uk/sd/academic/ceis/re/isrc/themes/rmarea/iso/

McLeod J, Childs S, Heaford S (2006a). A critical assessment of recordsmanagement capacity and compliance toolkits. Final project report, June 2006.Newcastle upon Tyne, Information Society Research & Consultancy Group, Schoolof Computing, Engineering and Information Sciences, Northumbria University.http://northumbria.ac.uk/sd/academic/ceis/re/isrc/themes/rmarea/tlkit /

McLeod J, Childs S, Heaford S (2006b). A guide to records management capacityand compliance toolkits & recommendations for their practical use. Newcastle uponTyne, Information Society Research & Consultancy Group, School of Computing,Engineering and Information Sciences, Northumbria University.http://northumbria.ac.uk/sd/academic/ceis/re/isrc/themes/rmarea/tlkit/

Moss M, 2005. The Hutton Inquiry, the President of Nigeria and what the butlerhoped to see. English Historical Review, CXX (487), pp577-592.

Nielsen J (1994). Heuristic evaluation. In: Nielsen J and Mack RL. Usabilityinspection methods. Wiley.

OCLC. Heuristic evaluation.http://www.oclc.org/policies/usability/heuristic/default.htm

Oliver M, Conole G (2000). Assessing and enhancing quality using toolkits. QualityAssurance in Education, 8(1, pp32-37

Stephens DO, 2005. The Sarbanes-Oxley Act: Records management implications.Records Management Journal, 15(2), pp98-103

The National Archives (2002). Freedom of Information Act 2000. Model action planfor achieving compliance with The Lord Chancellor’s Code of Practice on theManagement of Records [3]. Model action plan for higher and further education

23

organisations. The National Archives.http://www.jisc.ac.uk/uploaded_documents/modelactionplan.pdf

Thebridge, S (2004). The eVALUEd toolkit: practical evaluation help for academiclibraries. Performance Measurement and Metrics, 5(2), pp72-80.

United States of America, 2002. Sarbanes-Oxley Act of 2002. Government PrintingOfficehttp://fl1.findlaw.com/news.findlaw.com/hdocs/docs/gwbush/sarbanesoxley072302.pdf

Wells, S. (2004). Information governance and records management in health andcare services. Paper presented at the National Archives Conference, October 2004,Brighton, UK.

Wharton, C., Rieman, J., Lewis, C., Polson, P. (1994).The cognitive walkthrough: Apractitioner's guide, In: Nielsen, J., Mack, R. L. (Eds) Usability inspection methods,Wiley.

24

Table 1: Toolkit Evaluation Criteria1. Provenance of Toolkit Producer’s name Producer’s category Producer’s track record in the RM field Sustainability of toolkit Process of development of toolkit Date of current version Toolkit kept up-to-date Vendor support Acceptable use statements Fees/extra charges for various support functions Any additional comments about provenance

4. Toolkit Content Based On Legislation / Standards / BestPractice Legislation used to develop the toolkit Standards used to develop the toolkit Sector policy, guidelines and compliance requirements used to

develop the toolkit Best practice used to develop the toolkit Clear traceability of the tools evaluation criteria to specific

statements in legislation / standards / policy, guidelines,compliance requirements / best practices, etc.

Any additional comments on toolkit content

2. Toolkit Audience Toolkit targeted at different sectors Toolkit targeted at different types of organisation within a particular

sector Toolkit targeted at different sizes of organisation Toolkit targeted at different staff categories within an organisation Any additional comments about toolkit audience

5. Toolkit Process / Format Information gathering process User’s own internal documentation to be consulted when using

the toolkit Toolkit is automated for data input and analysis Any additional comments about toolkit process / format

3. Toolkit Coverage Toolkit purpose Type of records Toolkit addresses the full life cycle or continuum of records

management processes Toolkit results accurately represent the state of the organisation's

RM situation Toolkit results completely represent the state of the organisation's

RM situation Any additional comments about toolkit coverage

6. Resource Requirements to Use the Toolkit Money People Time taken for evaluation process to be completed Time commitment of staff Any additional comments on resource requirements to use the

toolkit

25

7. Accessibility / Compatibility of the Toolkit Accessibility of the toolkit to disabled people or people with an

impairment Accessibility of the software tool from a technology viewpoint Compatibility of the software tool Any additional comments on accessibility / compatibility of the

toolkit

8. Usability of the Toolkit Clearly articulated methodology Background information provided Training available for operators and users Help files "How to guidance" to move through the toolkit from stage to stage Easy to learn Easy to remember Clear language Clear user instructions Options for both new and experienced users Visually pleasing Enjoyable Easy to recover from user errors Easy to understand and interpret the results of the evaluation The software tool is accessible from many different locations The contents of a software tool can be printed out The software tool can support large amounts of data With a software tool, the user can save evaluation drafts and return

to them at a later time With a software tool, the results of the evaluation can be exported

to other platforms Any additional comments on usability of toolkit

9. Evaluation Approach Who collects and inputs the data Toolkit can be customised Depth of the toolkit’s evaluation criteria Consensus is required on answers to the toolkit’s evaluation

criteria before data input Methods for providing answers to evaluation questions and

criteria Data can be input over a period of time, on a stop - restart

basis Process for analysing the data Ability of the software tool to accommodate conflicting

information during the evaluation phase Data can be analysed over a period of time, on a stop - restart

basis How the results are presented “How to guidance” to respond to the results of the evaluation

to improve practices and enable change The toolkit can be reused any number of times under the same

license Any additional comments on evaluation approach

26

(See Mcleod, Childs and Heaford (2006b) for the criteria with explanatory examples and notes)