review of the ista accreditation system: evaluation of the ... · pdf filereview of the ista...

Review of the ISTA Accreditation System: evaluation of the questionnaires completed by the laboratories and the Designated Authorities

Rita Zecchinelli

Report of the ISTA Accreditation Review Working Group

Background

During the Ordinary Meeting 2012 in Venlo (NL), the membership of ISTA approved a motion jointly proposed by the ISTA Executive Committee and the Designated Authorities from Australia and New Zealand. The motion proposed a review of the audit process, in order to improve its cost effectiveness, considering the overall cost to laboratories, without reducing standards. During the Ordinary Meeting 2013 in Antalya (TR), two questionnaires were presented, one for the Designated Authorities, and one for the laboratories. Both questionnaires have been distributed and responses received.

19.06.2014 – R. Zecchinelli

ISTA Accreditation Review Working Group

Composition (approved by the ISTA ECOM on 25.01.2013)

Rita Zecchinelli (Chair, ISTA ECOM member) Joël Léchappé (ISTA President) Craig McGill (ISTA Vice-President) Masatoshi Sato (ISTA ECOM member) Ronnie Don (ISTA Honorary Life member) Rasha El-Khadem (Head of the ISTA Accreditation and Technical Department) Graeme Smith (representative from the Australian DA) John Randall (representative from the New Zealand DA) Piero Sismondo (representative from ISF)


Questionnaires

Aims of the questionnaires: - Identification of the needs of ISTA labs, DAs, other

stakeholders related to accreditation. - Collection of information on other accreditation systems

relevant for the seed testing labs. - Evaluation of the ISTA accreditation system (e.g. audits,

scope, recognition, fulfilment of needs, services, costs), even in comparison with the other relevant accreditation systems.


1. Questionnaire to the laboratories A) General information Answers have been received from: > 50% of ISTA member labs (106/202) ≈ 70% of ISTA accredited labs (81/120) > 80% of ISTA member countries or

distinct economies (58/71)


0

10

20

30

40

50

60

70

80

90

100

Sampling Traditional Tests Special test Performance based tests

% o

f IST

A a

ccre

dite

d la

bora

tori

es

Tests/Methods laboratories accredited for by ISTA Status Participant labs

Accredited only by ISTA 38 35,8%

Accredited by ISTA + other body/ies 43 40,6%

Accredited by other body/ies (no ISTA) 10 9,4%

Not accredited at all 15 14,2%

Total 106 100,0%

1. Questionnaire to the laboratories


A) General information: accreditation other than ISTA Answers (n) Out of % Notes

35 106 Participating labs 33 ISO accredited

71 106 Participating labs 67 ISO non-accredited (21 out of 71 state that they have planned ISO accreditation)

29 81 ISTA accredited labs 36 Also ISO accredited

6 25 ISTA non-accredited labs 24 ISO accredited

6 35 ISO accredited labs 17 With “other tests” (seed testing and/or different tests) in the scope

Answers (n) Out of % Notes

25 98 Answering labs 26 Accredited on the basis of other standards (different from ISTA and ISO)*

*When requested, only 4 labs (4%) specified a relevant standard, 13 state ISO standards, 8 do not provide information

73 98 Answering labs 75 Not accredited on the basis of other standards

19 81 ISTA accredited labs 24 Also accredited on the basis of other standards

6 25 ISTA non-accredited labs 24 Accredited on the basis of other standards



A) General information: geographical distribution

8% 8%

4%

24% 47%

6%

3% Responses received

Africa

North America

South America

Asia

Europe

Oceania

No response

Africa 9accredited 4

not accredited 5

North America 8accredited -

not accredited 5

South America 4accredited 1

not accredited 3

Asia 26accredited 12

not accredited 14

Europe 50accredited 7

not accredited 43

Oceania 6accredited 1

not accredited 5

No response 3accredited -

not accredited 3

TOTAL 106

1. Questionnaire to the laboratories B) The needs: why ISTA Accreditation


0

10

20

30

40

50

60

70

80

90

100

Agree Disagree No response

Num

ber o

f lab

orat

orie

s

ISTA Accreditation is necessary because we issue ISTA Certificates

not ISTA accredited

ISTA accredited

not ISTA accredited

ISTA accredited

0

10

20

30

40

50

60


Num

ber o

f lab

orat

orie

s

ISTA Accreditation is necessary because it is requested by our National Authority or Administration

not ISTA accredited

ISTA accredited78%

43%

1. Questionnaire to the laboratories B) The needs: why ISTA Accreditation


43 out of 81 (53%) ISTA Accredited labs are also accredited by other bodies; 25 out of 43 (61%) would like to be accredited by only one accreditation body.

0

10

20

30

40

50

60

70

80

90

100


Num

ber o

f lab

orat

orie

s

ISTA Accreditation is very prestigious

not ISTA accredited

ISTA accredited

0

10

20

30

40

50

60

70

80

90

100


Num

ber o

f lab

orat

orie

s

ISTA accreditation is the most important recognition for a seed testing laboratory

not ISTA accredited

ISTA accredited

not ISTA accredited

ISTA accredited

87%

86%

1. Questionnaire to the laboratories B) The needs: why is ISTA Accreditation appreciated?


0

10

20

30

40

50

60

70

80

Num

ber o

f lab

orat

orie

s

In the framework of the ISTA accreditation system, I appreciate the Proficiency Test program.

not ISTA accredited

ISTA accredited

0

10

20

30

40

50

60

Num

ber o

f lab

orat

orie

s

In the framework of the ISTA accreditation system, I appreciate the competence of the

auditors.not ISTA accredited

ISTA accredited

Agree + Strongly Agree:

94%

Agree + Strongly Agree:

84%

not ISTA accredited

ISTA accredited

1. Questionnaire to the laboratories B) The needs: why is ISTA Accreditation appreciated?


Other reasons of appreciation: • focus on seed testing (95% agree/strongly agree) • being part of a community of seed testing labs (92%

agree/strongly agree) • receiving support from the Association (80% agree

/strongly agree). • the possibility to be accredited for performance based

approach (PBA) methods is also appreciated (74% agree/strongly agree).



0

5

10

15

20

25

30

35

40

Num

ber o

f lab

orat

orie

s

I don’t have any reasons to criticize the ISTA accreditation system.

not ISTA accreditedISTA accredited

Have no reasons to criticise ISTA Accreditation: • 14 Company labs labs (56%) • 35 Governmental labs (51%) • 6 Private Independent labs (46%) • 55 All labs (52%)

Have reasons to criticise ISTA Accreditation: • 10 Company labs (40%) • 16 Governmental labs (24%) • 3 Private Independent labs (23%) • 29 All labs (27%)

Area No reasons to

criticise Reasons to

criticise

N° % N° % Africa (9) 5 56 4 44 North America (8) 3 38 4 50 South America (4) 3 75 0 0 Asia (26) 16 62 3 12 Europe (50) 23 46 16 32 Oceania (6) 3 50 1 17 No response (3) 2 67 1 33

Total (106) 55 52 29 27

not ISTA accredited

ISTA accredited

B) The needs: why isn’t ISTA Accreditation appreciated?

1. Questionnaire to the laboratories B) The needs: why isn’t ISTA Accreditation appreciated?


0

5

10

15

20

25

30

35

40

Num

ber o

f labo

rato

ries

In the framework of the ISTA accreditation system, I don’t appreciate the costs.

not ISTA accredited

ISTA accredited

Costs are a concern for 52 labs (49%) of the ISTA laboratories. Other reasons of concern: • Way the audits are conducted :12 labs (11%) labs agree/strongly agree • Audit frequency: 28 labs (26%) agree/strongly agree

0

10

20

30

40

50

60

Num

ber o

f lab

orat

orie

s

In the framework of the ISTA accreditation system, I don’t appreciate the way the audits are conducted.

not ISTA accredited

ISTA accredited

05

1015202530354045

Num

ber o

f lab

orat

orie

s

In the framework of the ISTA accreditation system, I don’t appreciate the frequency of the audits.

not ISTA accredited

ISTA accredited

not ISTA accredited

ISTA accredited

1. Questionnaire to the laboratories EVALUATION OF THE COSTS

Only very few labs provided costs of other accreditation system. 19.06.2014 – R. Zecchinelli

IN THE FRAMEWORK OF ISTA ACCREDITATION, WE PARTICULARLY DON’T APPRECIATE THE COSTS

ALL LABS ONLY GOVERNAMENTAL LABS COMPANY + PRIVATE LABS

Area Agree+Strongly

agree Area Agree+Strong

ly agree Area Agree+Strongly

agree N° % N° % N° %

Africa (9) 6 67 Africa (8) 5 63 Africa (1) 1 100

North America (8) 7 88 North America (4) 4 100 North America (4) 3 75

South America (4) 2 50 South America (3) 1 33 South America (1) 1 100

Asia (26) 10 38 Asia (15) 6 40 Asia (11) 4 36

Europe (50) 23 46 Europe (33) 16 48 Europe (17) 7 41

Oceania (6) 3 50 Oceania (3) 2 67 Oceania (3) 1 33 Blank (3) 1 33 Blank (2) 0 0 Blank (1) 1 100 Total (106) 52 49 Total (68) 34 50 Total (38) 18 47

1. Questionnaire to the laboratories B) The needs: recognition of ISTA Accreditation


• In general the ISTA accreditation system is widely recognized (90%). • 19 (18%) labs think that the ISTA accreditation system is not recognized

as equivalent to other accreditation system/s. • 9 do not provide further information, 5 state that ISO is more

recognized (labs from Europe and North America), 5 state that a regional or national system is more recognized (labs from North America and Asia).

• 50 labs (47%) state that ISTA accreditation is recognized only for

traditional tests, 63 labs (59%) state that ISTA accreditation is recognized for all kinds of tests.

1. Questionnaire to the laboratories C) ISTA audits: labs’ experience. General feedback.


0

10

20

30

40

50

60

70

Num

ber o

f lab

orat

orie

s

The ISTA audits have always provided a true picture of our laboratory.

not ISTA accredited

ISTA accredited

0

10

20

30

40

50

60

Num

ber o

f lab

orat

orie

s

The ISTA audits have always been an opportunity to improve

not ISTA accredited

ISTA accredited

05

101520253035404550

Num

ber o

f lab

orat

orie

s

The laboratory management and staff always agreed with the audit findings.

not ISTA accredited

ISTA accredited

1. Questionnaire to the laboratories C) ISTA audits: labs’ experience. Use of PTs.


0

10

20

30

40

50

60

70

80

90


Num

ber o

f lab

orat

orie

s

The results achieved by the laboratory in the proficiency tests should be taken into greater

consideration for the purposes of the ISTA audits.

not ISTA accredited

ISTA accredited

0

10

20

30

40

50

60

70


Num

ber o

f lab

orat

orie

s

A laboratory with low ratings in proficiency tests should be audited by ISTA more frequently than a

laboratory with higher ratings.

not ISTA accredited

ISTA accredited

But during the audits labs with high PT ratings will show a few non-conformities → 27 labs agree – 68 labs disagree labs with low PT ratings will show a larger number of non-conformities → 37 labs agree – 59 labs disagree

75% 60% 33%

not ISTA accredited

ISTA accredited

1. Questionnaire to the laboratories C) ISTA audits: labs’ experience. Frequency.


0

10

20

30

40

50

60

70

80

No No response Yes

Num

ber o

f labo

rato

ries

In your opinion, is the frequency of ISTA audits (one in three years) appropriate?

not ISTA accredited

ISTA accredited

In comparison with other audit systems: • 20 labs state that ISTA audits are less frequent • 15 labs state that the frequency is the same • 10 labs state that ISTA audits are more frequent

YES 75

NO 26

1. Questionnaire to the laboratories SUGGESTIONS 1/3


Differentiate the fees: • The fees should be tailored to the species, the country, the scope of

accreditation, the size of the lab, the number of the certificates issued by the lab.

• Organizations having labs in different countries that have implemented the same QS in all labs should pay a lower fee.

• In case of labs audited by other bodies, ISTA should use the results of those audit.

• ISTA should recognise national ISO-17025 audits. • Labs preparing PT rounds should pay a lower audit fee. • Charge the audit basic costs and separately reimburse other

expenditures. • Increase the annual accreditation fee and reduced the audit fee. • The audit fee should be transparent and cover only the costs of the

audits.



Review of the audit system: • Group the audits in the same region. • For good labs (no “S” findings, good PT results): audit every 4 years. • For all labs: audit every 5 years (in between: PTs). • ISTA could use the PT system + desk audits. • The audit carried out by a local independent body. • Increase the PT and reduce the audit cost of the Lab. • Take into consideration the results/rating on PTs and the duration of

accreditation (risk based approach). • If cheaper, organize one audit day with two auditors or two audit days

with one auditor. • Possibility to organize desk system audits or video audits. • Increase the number of auditors to get better representation of all

sectors.



Add benefits, in order to make the audit fee more valuable : • Make a particular number of ISTA certificates free by including it

in the membership fee (e.g. for the annual laboratory membership fee ISTA includes in 2 free packs of ISTA certificates. The laboratory chooses whether they want both packs to be Orange, Blue or 1 pack of each).

• Provide a checklist of requirements before the audit. • Audits with a more consulting role. • Allocate time to receive questions from the staff.

2. Questionnaire to the Designated Authorities A) General information Answers have been received from 16 Designated Authorities (≈ 23%):

2 from North America (2/2) 3 from Africa (3/10) 2 from South America (2/7) 1 from Asia (1/15) 6 from Europe (6/35) 2 from Oceania (2/2)


• In all countries there is at least one ISTA accredited lab.

• In 9 countries out of 16 there is at least one ISO 17025 accredited seed testing lab.

• In 5 countries out of 16 there is at least one seed testing lab accredited basing on other standards. These standards are: -national standards of the different countries, NAL (1 European country)

• ISTA accreditation is recognized by the law in 9 countries (1 in S. America, 5 in Europe, 2 in Africa, 1 in Oceania).

• ISO 17025 is recognized by the law in 4 countries (1 in S. America, 2 in Europe, 1 in Africa).

• All accreditation systems (ISTA, ISO, other standards) are recognised by popular practice in several countries.

2. Questionnaire to the Designated Authorities B) The needs: why ISTA Accreditation


Out of 16 DAs: • 13 out state that ISTA accreditation is necessary in order to

issue ISTA certificates. • 6 state that ISTA accreditation is necessary because requested

by the law (4 in Europe, 2 in Africa). • 14 believe ISTA accreditation the most important recognition

for a seed testing lab. • 13 think that ISTA accreditation is very prestigious. • 2 state that for seed testing labs it is necessary to be

accredited by ISTA and by another body as well. • 4 think that ISTA accreditation is not necessary at all.

2. Questionnaire to the Designated Authorities B) The needs: why ISTA Accreditation is appreciated


• PT programme (14 DAs). • Focus on seed testing (15 DAs). • Support provided by the association to the labs (13 DAs). • Possibility to be accredited for performance based approach

(PBA) methods is also appreciated (11 Das). B) The needs: why ISTA Accreditation is not appreciated 6 DAs don’t have reasons to criticise the ISTA accreditation

system; 4 DAs have reasons to criticise it: • 3 DAs do not appreciate how the ISTA audits are conducted. • 5 DAs do not appreciate the frequency of the ISTA audits. • Costs are a concern for 8 DAs. • 1 DA states that ISTA accreditation system is inexpensive

compared to other systems. • 7 DAs state that ISTA accreditation system gives good value for

the costs.

2. Questionnaire to the Designated Authorities B) The needs: evaluation and use of PTs results


• 14 DAs state that PT results should be taken into greater consideration for the purposes of ISTA audits.

• 11 DAs think that labs with low PT rating should be audited more frequently that labs with higher ratings.

• 9 DAs expect that lab with low PT ratings will have a larger number of significant non- conformities.

• 6 DAs expect that lab with high PT ratings will have a smaller number of significant non- conformities.

2. Questionnaire to the Designated Authorities SUGGESTIONS 1/2


Review of the audit system: • One audit every three year for the beginning (e.g. for 3 audits), then

less frequent audits if -the PT ratings is good -no disputes/complaints have been received with regards to the certificates issued by

the lab. • Modulate the audit frequency on a risk based analysis. • Define when a full audit is necessary and when a remote system

audit could be enough. • ISTA should participate as technical assessors in the ISO 17025 audit. • Recognition of audits between ISTA and ISO 17025 accreditation

systems. • Use local/regional auditors. • Use the financial tool to increase the transparency of the fees.

2. Questionnaire to the Designated Authorities SUGGESTIONS 2/2


Add benefits, in order to make the audit fee more valuable: • The audit should be a learning occasion for the team at the

lab. • Audits conducted in Spanish. • Add new PTs for the tests today not covered in the

programme or covered only once every several year (e.g. SHT, electrophoresis).

• Enlarge the possibility to be accredited only for sampling to non-lab institutions.


ISSUES RAISED FROM THE EVALUATION OF THE QUESTIONNAIRES


Frequency of the ISTA audits: should we continue to discuss this issue? Use of PT results: do the opinions appear contradictory? Costs and benefits balance: how to address the needs? Comparison with other accreditation systems: are you interested in? Are the lab accredited to ISTA and other bodies ready to provide more information?


FUTURE WORKS AND GOALS 1. Proposals for the revision of the ISTA accreditation system: - fees (e.g. are they reasonable basing on the evidences provided by the

use of the financial tool? can we define which expenditure is paid by which of the two fees currently paid for being accredited? should we revise the structure of the fees?)

- audit frequency (should we change it?) - audit organization (e.g. can we increase the number of local auditors?

can we define other changes/initiatives in order to improve the efficiency of the system? can we address some requests, such as provide an audit checklist?)

- proficiency test programme (e.g. can we improve the programme?) - other issues

2. Improve the recognition of the ISTA accreditation system (mutual recognition ISTA/ISO national accreditation bodies, other issues).



From the lab’s suggestions possibilities for changes

Model based on 2014 fees Year 1 Year 2 Year 3

Laboratory Membership (CHF) 5‘214 5‘214 5‘214

ACC: Current Model (CHF) 1‘224 1‘224 14‘224

ACC: Model based on yearly payment (CHF) 5‘557 5‘557 5‘557

• Assuming a yearly payment of the actual audit visit fee

• Pros: • Replace the large audit fee every three years and change to a

yearly fee • Reduce the year to year variation caused by the number of

audits (ISTA finances and workload)

• Can this be an issue for laboratories?

• Scope of Accreditation • Shall laboratories pay more if their scope covers many tests

and the auditors need to audit for more than one day?

• Risks when changing the system to a scope based approach: • Will labs stay with their current scope of accreditation if they

could save money removing tests they do not need for ISTA Certificates?

• More administration in Secretariat needed.

• Can this help us to be more cost effective?

From the lab’s suggestions possibilities for changes

THANK YOU FOR YOUR ATTENTION

review of the ista accreditation system: evaluation of the ... · pdf filereview of the ista...

Documents