report of external evaluation and review - new … of external evaluation and review nz flooring...
TRANSCRIPT
Report of External Evaluation and Review NZ Flooring Industry Training Organisation trading as DecorateNZ and FloorNZ
Confident in ITO performance
Confident in capability in self-assessment
Date of report: 11 September 2013
Final Report
2
Contents
Purpose of this Report ................................................................... 3
Introduction ................................................................................... 3
1. ITO in context ........................................................................................... 3
2. Scope of external evaluation and review .................................................. 6
3. Conduct of external evaluation and review ............................................... 7
Summary of Results ...................................................................... 8
Findings ...................................................................................... 11
Recommendations ...................................................................... 25
Appendix ..................................................................................... 26
MoE Number: 8164
NZQA Reference: C11468
Date of EER visit: 10 and 11 June 2013
Final Report
3
Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the industry training organisation’s (ITO) performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, trainees, communities, employers, and other interested parties. It is also intended to be used by the ITO itself for quality improvement purposes.
Introduction
1. ITO in context Name of ITO: NZ Flooring Industry Training Organisation trading
as DecorateNZ and FloorNZ
Location: Palmerston North
Gazette recognition date: 27 November 2008-2013
Gazette coverage: ‘Pursuant to section 5 of the Industry Training Act 1992, full recognition for the NZ Flooring Industry Training Organisation Inc. has been granted for a period of five years, with effect from 27 November 2008-13, to set standards at Levels 1 to 8 on the National Qualifications Framework for the flooring industry in the areas of production, distribution, sales, and installation of resilient/soft and associated flooring surfaces excluding the production, distribution, sales, installation and maintenance of flooring surfaces such as slate, cement/clay tiles and pavers, ceramic tiles and natural stone, and liquid floor coatings, seamless flooring and screeds.
An extension of coverage was granted from 9 February 2011 to 27 November 2013 for the painting and decorating industry with respect to painting and decorating in all areas of production, distribution, sales and application of paint, wall coverings, specialised coatings and industrial coatings including resins.’
Number of qualifications registered on NZQF:
Four
Final Report
4
Number of standards registered on NZQF:
71 Flooring
37 Painting and Decorating
Number of trainees: 1,283 trainees (638 Standard Training Measures), 846 under DecorateNZ and 437 under FloorNZ. Approximately 20 per cent of trainees are Māori, and 5 per cent are Pasifika.
Number of staff: 14 full-time equivalents
Number of registered workplace assessors:
Seven field staff: regional coordinators or Modern Apprenticeship coordinators based in regions who are also registered assessors
Distinctive characteristics: NZ Flooring ITO monitors and reviews existing qualifications to ensure they are current and meet industry requirements. In March 2011, a new version of the National Certificate in Flooring (Level 4) with strands in Carpet Installation, Fine Sand and Finishes, Flooring Planning and Design, Resilient Flooring Installation, Substrate Preparation, Timber Overlay Installation, and Resin Flooring Application, was introduced.
In August 2011, the National Certificate in Painting and Decorating (Level 4) was created to replace one existing qualifications. The above qualifications with strands or an elective section enable companies to meet their specific skill requirements.
The National Certificate in Flooring (Trade Supply) (Level 3) was developed in 2010 and introduced on a trial basis in November 2011. It followed a comprehensive industry survey of the needs of employees required to advise customers and industry groups on the technical aspects of flooring products within the trade supply sector of the flooring industry, but who do not need the full suite of skills required to carry out flooring installations for floor planning and design.
Field staff are the Modern Apprenticeship coordinators for FloorNZ nationally, and for DecorateNZ in the Wellington region only. While quarterly visits to modern apprentices are a requirement of the Modern Apprenticeship coordinator contracts, the ITO is keen to ensure that the same level of service is available to all
Final Report
5
trainees. Conventional apprentices are guaranteed to receive a minimum of two visits per year, but in reality more are made in response to the needs of the employer and apprentice.
NZ Flooring ITO’s industries are part of the allied trades sector in the construction industry. Unlike other parts of this sector (e.g. building, plumbing) they are not regulated. However, employees may be contracted by, and work in association, with licensed builders. The ITO is therefore keen to ensure that the training is to its standards and qualifications, and not only meets the requirements of individual commercial and residential customers but also the standards of the construction sector overall. A recent initiative, if approved, would introduce licensing similar to the Licensed Building Practitioner programme. This would improve confidence and qualification profiles and contribute to ongoing professional development.
The National Flooring Association is the business association representing the flooring industry. In 2010, it established FloorNZ as an all-encompassing, comprehensive organisation to meet the needs of flooring professionals.
Master Painters New Zealand is the business association representing the painting and decorating industry. In 2012, DecorateNZ was established.
The combined industry is characterised by small to medium-sized enterprises estimated at 7,400 companies employing approximately 17,800 workers. These small to medium-sized enterprises are the main users of the ITO’s qualifications.
NZ Flooring ITO has a high number of Skills Recognition (DecorateNZ) and Recognition of Current Competence (FloorNZ) candidates (approximately 30 per cent) and is currently reviewing its processes in order to provide greater assistance to these candidates.
NZ Flooring ITO reinstated off-job training for DecorateNZ in 2011, and now arranges off-job training in both sectors with four providers.
Final Report
6
Recent significant changes: NZ Flooring ITO merged with the painting and decorating sector from the former Creative Trades ITO in February 2011.
NZ Flooring ITO is actively engaged in the NZQA Allied Trades Targeted Review of Qualifications.
Previous quality assurance history:
NZ Flooring ITO has not been quality assured previously by NZQA. NZQA’s Tertiary Assessment and Moderation unit has no concerns in regard to assessment. NZ Flooring ITO has consent to assess, but has not reported any results against unit standards administered by NZQA.
Other: The economic recession had a negative impact on the capacity of employers to train new staff or maintain existing apprenticeships. Consequently, this saw a marked decrease between 2009 and 2011 in NZ Flooring ITO’s numbers in training and completions. Currently the situation has improved significantly and opportunities for training and employment are enhanced by the Christchurch rebuild and improved business confidence. The ITO has actively researched and analysed data and trends in training and employment to inform all stakeholders and to assist its own planning.
2. Scope of external evaluation and review The scope of the external evaluation and review (EER) consisted of three focus areas. These were:
• Governance, management and strategy
• National Certificate in Flooring (Level 4) with strands in Carpet Installation, Fine Sand and Finishes, Flooring Planning and Design, Resilient Flooring Installation, Substrate Preparation, Timber Overlay Installation, and Resin Flooring Application
• National Certificate in Painting and Decorating (Level 4)
Governance and management was selected because of the recent merger between Decorate NZ and Flooring NZ. The other two national certificate programmes were chosen as these enrol the majority of trainees in the ITO and give coverage of the two main skills areas.
Final Report
7
3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA’s published policies and procedures. The methodology used is described fully in the document Policy and Guidelines for the Conduct of External Evaluation and Review available at: http://www.nzqa.govt.nz/for-providers/keydocs/index.html) [Refer to ITO supplement]. The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report.
A team of two evaluators spent two days at NZ Flooring ITO’s office in Palmerston North. The team met with management and support staff, members of the ITO board of directors, a Modern Apprenticeship coordinator, and an employer and his apprentice (painting and decorating). The team also visited a local flooring company to interview two apprentices and their supervisor. Telephone conversations were held with two regional coordinators, representatives of Master Painters New Zealand and the National Flooring Association, and two of the four providers contracted by the ITO.
A selection of documents, records, training and assessment resources and the industry training database was reviewed to complement and triangulate information gained from key stakeholder interviews. Discussions with the Tertiary Education Commission (TEC) and NZQA Tertiary Assessment and Moderation unit were held during the scoping phase.
Final Report
8
Summary of Results
Statement of confidence on ITO performance NZQA is Confident in the performance of NZ Flooring ITO.
NZ Flooring ITO is effective in carrying out its statutory functions. Its research, surveys and analyses of industry employment trends and skill needs provide a very useful context for both strategic and operational planning, and are used to provide advice to industry and government. The ITO also uses the information to highlight changes in demand for employment and training to its industry sectors, update the standards and qualifications, improve its marketing and communication strategies for the uptake of training, and monitor the demand for off-job training. Information about the ITO’s products and services, for example the Recognition of Current Competence and new literacy programmes, can be accessed by employers’ potential and current trainees on its websites and in leaflets. The results of the annual industry confidence/training surveys are published in the FloorNZ and DecorateNZ newsletters.
Overall, the information and analyses are useful to industry with regard to decisions about the investment in training by individual employers and for highlighting trends in employment and training needs to external stakeholders. This was verified in conversations with board members, employers and their professional associations, and providers.
NZ Flooring ITO has a very active board, and excellent communication and cooperation between the chief executive and the board means that the governance and management functions of the ITO are efficiently managed. Relationships with industry are maintained at both the strategic and operational levels through the board, the industry associations and regular visits by field staff to company sites.
The ITO used a comprehensive survey in 2010 of the flooring industry and its annual industry confidence/training surveys to inform its strategic and operational planning for this sector. The 2010 survey provided useful information about the impact of the recession, employer capacity for training and levels of confidence in the services of the ITO. While many employers regarded the uncertain economic environment as a barrier to training, they still valued highly the services of the ITO.
The merger with the painting and decorating sector of the former Creative Trades ITO, now DecorateNZ, was well managed and the sector has been fully integrated into the ITO at governance, management and operational levels. A new level 4 painting and decorating qualification was introduced in 2012, with new training and assessment resources. Industry representatives, employers and apprentices indicated high levels of satisfaction with the ITO management, support from field staff, and the training.
Final Report
9
Operationally, the work of field staff, the regional coordinators and the Modern Apprenticeship coordinators is considered by industry to be of high value, leading to improved training outcomes. The coordinators are well trained and supported.
The ITO has three current training programmes and is actively engaged in updating and maintaining the currency of its qualifications. This includes involvement in the current NZQA-led project reviewing national qualifications (Targeted Review of Qualifications) and has led to recent changes in the two apprenticeship programmes to meet changing needs. The ITO is also trialling the implementation of a level 3 qualification first introduced in 2012.
The number of trainees enrolled has increased between 2011and 2013, as has the number of completions, with a further rise in these areas predicted. The ITO has had relatively low numbers of qualification completions following the negative impact of the recession, which resulted in fluctuating numbers and low levels of enrolment. It is actively managing the situation through the marketing of the qualifications, additional mentoring of over-duration trainees, and improved management of the high number of Recognition of Current Competence candidates.
Increased demands from the Christchurch rebuild and improved business confidence has led to a big increase in the number of enrolments. The TEC has provided additional Standard Training Measure funding to meet the demand.
The engagement of the ITO with other groups in the tertiary education sector will lead to other training opportunities for core construction skills to meet the needs of trainees at the senior secondary school and trades-entry levels. Training is mainly conducted on the job. New resources for training and assessment have been introduced that provide a learning pathway for trainees in both in-trade and pre-trade programmes. Off-job training and assessment is provided through block courses with four providers.
Overall, the ITO demonstrated that it has adequately analysed skill needs within its sectors. Additionally, in association with its key stakeholders, the ITO has reproduced and revised new qualifications to meet those needs. A number of activities are occurring in the sector that will have an impact on the ITO. These include programme changes, the Targeted Review of Qualifications, mergers and amalgamations. These changes will take time for the ITO to embed as it moves towards developing and extending its influence. There is a level of uncertainty, but this is not seen as a serious impediment to the ITO and the evidence has shown that it can manage change effectively.
Final Report
10
Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of the NZ Flooring ITO.
NZ Flooring ITO has a good understanding of the needs of industry, trainees and government, and the value of its programmes to industry. It collects information through research and analyses of trends in industry engagement in training through formal and informal sources. The annual industry confidence/training surveys also provide information which is useful for planning purposes. Its current surveys do not include questions about the effectiveness of training that could be used to inform the ITO’s products and services.
Field staff are contracted to the ITO, supported with up-to-date resources and provided with professional development opportunities. They monitor trainees and provide regular updates on their progress. The information is used to identify training and assessment issues and any barriers to training. Feedback from field staff is analysed by quality assurance staff in regular meetings. Although informal, this is an effective process for analysing individual trainee progress.
Employers and apprentices discussed the importance to them of the training in gaining skills that were of value to the company and individual worker, and the support from the ITO’s field staff.
The ITO’s quality management system has been reviewed, and improvements to resources, assessment (including the workplace verification model), the Recognition of Current Competence programme and internal moderation are underway or planned. Overall, the ITO’s quality assurance processes, including moderation, provided evidence that trainee achievements are valid, reliable and consistent.
The ITO is aware of some gaps in self-assessment and is planning improvements. While the mix of formal and informal self-assessment works in general for an ITO of this size, there are questions about the capability of some systems relating to the recording, reporting and analysis of trainee achievement as to whether or not they are sufficient to meet the needs of the predicted growth in numbers. Greater use could be made of the annual surveys, information the ITO collects from field staff reports, and the information held on its database about trainees.
Final Report
11
Findings1
1.1 How well does the ITO understand and meet the needs of industry, trainees and government?
The rating for performance in relation to this key evaluation question is Excellent.
The rating for capability in self-assessment for this key evaluation question is Good.
NZ Flooring ITO was first established in 1997. It understands the needs of industry very well and actively manages its statutory functions to provide leadership in identifying the skills needed for its industries, developing standards and qualifications, and arranging training. These needs are informed by its close association with Master Painters New Zealand, the National Flooring Association, the Nuplex Resin Flooring Association, employers and providers. The members of the NZ Flooring ITO board are representative of all its sectors, and are experienced tradespeople who have been, or are, actively engaged in training. They are well informed through their professional associations and by the ITO management. They are supportive of the ITO and able to plan the strategic direction of the organisation. The chief executive advises the TEC investment manager and ensures that the ITO continues to meet government and regulatory requirements.
Flooring ITO understands the importance to industry of information about the demand for skilled workers, numbers in training, and current and predicted completions and future trends, especially in regard to individual employer decisions about whether or not to invest in the training of new staff. The ITO has surveyed and analysed the overall industry context to determine skill needs, and the impact of such events as the recession and the Christchurch rebuild on current and future demands for training. Data supplied by the ITO shows the number of new trainees predicted to grow over the next three years, a rise in completions of current trainees, with a corresponding reduction in withdrawals. This information is used to measure the fluctuations in demand for off-job training and provides the ITO and industry with up-to-date information about the likely number of skilled workers entering the industry. The information is provided to the ITO board, government agencies and employers, and is published in regular newsletters.
In 2010 the ITO conducted a comprehensive survey of the flooring industry. The survey of 496 flooring companies representing a workforce of approximately 2,830 employees was designed to gain insight into the industry’s demand for skilled labour, training participation and needs, and confidence in the activities and direction of the ITO. Ninety-five per cent of companies surveyed considered that the ITO is providing appropriate leadership. The survey identified a gap in training for trade supply (retail and wholesale) and resulted in the development of a new level 3 qualification. The survey also identified new areas for training in advanced
Final Report
12
installation, customer service, and sales and business management. Qualifications and training have yet to be developed in these areas, except for the level 3 trade supply qualification.
Annual industry confidence/training surveys also provide useful information. In 2012 for example, surveys for both sectors highlighted the fact that there is now an extreme shortage of skilled people in the industry. The consequent risks to training and industry skill levels, from the limited numbers of people in training, the likely potential for unskilled or semi-trained workers in the industry and the competition for skilled workers will have a negative impact on the quality of services.
The ITO regularly reviews its standards and qualifications, and since 2012 has been actively engaged in the Allied Trades Targeted Review of Qualifications. An outcome is the recommendation for the development of a level 3 trade entry qualification in allied trades-related skills. Major changes to the FloorNZ and DecorateNZ apprenticeship programmes are unlikely as a result of the Targeted Review of Qualifications, as these qualifications were reviewed and new versions introduced just prior to the review to meet industry and trainee needs.
The ITO is a foundation member of the Built Environment Training Alliance (BETA), an alliance of construction-related ITOs. The group aims to coordinate skills leadership across the industries, reduce the fragmentation in qualification design that results from industry ‘silo’-based training, to engage with the industry and providers in the Christchurch rebuild, and to provide clear pathways from the senior secondary school into the construction sector. New training initiatives will provide more options for training in both industry sectors.
A major focus for the ITO over the past two years has been the integration of the painting and decorating industry into its activities. In 2011 it gained this sector from the former Creative Trades ITO and since then has worked with Master Painters New Zealand, employers and trainees to ensure that their needs for relevant training are being met. Initially, considerable work and investment was required to restore training structures and confidence within the industry. This included visiting all employers and their apprentices to update their training progress, confirm employer capacity and consult over ongoing needs. The Master Painters New Zealand representative said the merger has lifted industry standards and provided a better understanding of what training can achieve. The annual DecorateNZ industry confidence/training survey showed that in 2012, 77 per cent of industry respondents valued the leadership of the ITO, which is a rise from 51 per cent in 2011 ‒ the year of the merger.
Training is mainly conducted on the job, and the ITO has developed new resources for trainees and their employers. These coincide with the introduction of new or revised qualifications. The integration of the on and off-job components is an essential feature of the training programmes, contributing to trainee success. The ITO has invested in a learning pathway supported by resources that can be used by industry and providers, both in-trade and pre-trade.
Final Report
13
Field staff work with employers, initiate the training, monitor trainee progress and provide regular updates to the ITO in the form of weekly and quarterly reports. Any problems with training and assessment are usually dealt with on site during visits to companies by field staff who are able to observe trends in the uptake of training, monitor the reasons for withdrawals and provide feedback to the ITO on the use of the training and assessment resources.
The ITO understands the clear link between literacy and productivity and now provides literacy and numeracy assessment on enrolment, and assistance for those requiring added support. Profiles of the literacy and numeracy demands of ‘typical’ workers in each sector are being developed and used to improve resources.
The ITO has excellent information which can be used for strategic and planning purposes, and its recent reviews of the qualifications and development of new resources indicate responsiveness to employer and trainee needs. However, although its annual surveys include apprentices and their employers, they do not provide specific information about the effectiveness of the on and off-job training programmes in gaining skills. Survey methods aimed specifically at all participants in the training programmes would benefit the organisation’s ability to better understand the needs of employers and trainees at an operational level, which could be used to inform improvements to the training programmes, support for trainees and the ITO’s systems.
1.2 What is the value of the outcomes for employers and their trainees?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is Good.
In addition to the feedback from key stakeholders, NZ Flooring ITO uses feedback from qualifications and unit standard reviews (including the Targeted Review of Qualifications process), moderation, meetings and informal anecdotal evidence to understand the relevance and value of its programmes. In particular, the advice from field staff is critical to the ITO’s understanding of trainee achievement and areas for improvement and the value to individual companies and their trainees.
The increase in numbers enrolling in the training programmes following the recession, and those completing, is an indicator of the high value placed by employers and trainees on the qualifications. There is immediate value in being able to apply the skills on the job. The ITO, through its assessment practices, ensures that the link between training and commercial competence is maintained. Employers and apprentices regarded the integration of on and off-job training, and the checks before block course assessment, to be valuable in understanding progress and gaining skills. In the 2010 survey, 95 per cent of companies surveyed considered that the match between FloorNZ training and company skill needs to be excellent or very good.
Final Report
14
DecorateNZ in particular values the way in which the ITO managed the merger and assisted the continuing apprentices to complete their qualifications. Field staff employers and apprentices in this sector spoke highly of the improved training and resources. Demand for skilled workers for the Christchurch rebuild and in other regions increases the value of the qualifications. The ITO was granted an additional 124 Standard Training Measures for 2013.
Apprentices that were interviewed regarded their training as essential for work in the industry, were generally very satisfied with the resources and support from field staff, and looked forward to the provider block courses.
The annual industry confidence/training surveys provide the ITO with general information about the high value industry places on the outcomes of training and the match to the growing need for more skilled workers.
Self-assessment is a mix of formal and informal processes that work well for an ITO of this size. The ITO relies on attestations from industry, survey results and feedback from field staff to determine how well outcomes are valued. Although the training programmes and associated initiatives are new or revised and recently introduced, there would be benefits to the organisation from collating and analysing existing feedback from a variety of sources, particularly as it relates to trainees, to better understand how training structures and resources contribute to the value of training outcomes.
1.3 How well do trainees achieve?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is Good.
Flooring ITO currently has 1,283 trainees and funding of 638 Standard Training Measures. It monitors trainee progress in ways that reflect their enrolment status. As part of its self-assessment, the ITO has reviewed the levels of support needed for the different types of trainees, such as modern and conventional apprentices, Recognition of Current Competence candidates and trainees enrolled in the level 3 qualification. Different levels of training support and assessment practices are needed, depending on enrolment. The ITO’s Candidate Support Procedures are designed to guide field and support staff with regard to enrolment, monitoring and assessment and documentation.
Trainees are currently enrolled in three qualifications. Thirteen trainees are enrolled in the National Certificate in Flooring (Trade Supply), which is currently being trialled. Seventy-three trainees are enrolled in the Flooring Planning and Design strand of the flooring level 4 national certificate. The table below profiles the number of apprentices currently enrolled in the level 4 national certificate programmes.
Apprentices Floor NZ DecorateNZ Total no. of
Final Report
15
National Certificate in Flooring (Level 4)
National Certificate in Painting and Decorating (Level 4)
apprentices
Modern 137 303 440
Conventional 58 299 357
Recognition of Current Competence (Skills Recognition)
156
244
400
Total 351 846 1,197
* Data supplied for numbers as at 11 June 2013
TEC educational performance indicator data for 2012 shows that the overall rate of credit completion was 54 per cent and qualification completion, 15.5 per cent, lower than the 2011 results of 61 per cent and 38 per cent respectively. The ITO’s own data shows a much higher rate of qualification completion of 60 per cent for the same period. When the achievement of over-duration trainees is taken into account, the completion rates are said to be much higher, although no data was available. In 2012, 17 DecorateNZ and 60 FloorNZ trainees gained their national certificates. Credit achievement for Māori trainees in 2012 was 52 per cent, an improvement of 7 per cent on the 2011 results.
The ITO is well aware of the need to manage training within the nominal duration, (the expected completion date), especially as it is in the interests of employers and industry overall to have staff performing at the required skill level as soon as possible. Most of the reports supplied analysed data relating to overviews of numbers of trainees enrolled, completing or withdrawing. Given the fluctuations in numbers, the ITO needs to monitor this data weekly, quarterly and annually. Trends can be analysed by region and sector and are important for the ITO’s planning.
An analysis of expected completion date by cohorts showed that for FloorNZ, 64 per cent of trainees in 2011, and 55 per cent in 2012, completed. For DecorateNZ, the number of completions was lower for 2011-12, as 90 per cent of these apprentices transferred to the new qualification in 2012 which have an expected completion date of 2015. These reports adapt the TEC reporting criteria and are designed to take account of fluctuations in numbers and the impact of earlier transfers of DecorateNZ trainees from one qualification to another (recorded as withdrawals in the TEC data).
Formal results for trainees are reported to the ITO following assessments completed by the block course providers. Credit and qualification completion results for individual trainees therefore only become available to the ITO following the annual block course for each of three stages (of the apprenticeships).
In the first three months of training, trainees are internally assessed for two generic knowledge-based unit standards, and the results are entered in the trainee database. This introduces trainees to the training manual, helps get them started
Final Report
16
on completing the modules and the record of work, and provides the employer and ITO with valuable information about progress and any literacy and numeracy needs. The results are forwarded to field staff who follow up as needed.
The relatively high number of Recognition of Current Competence candidates result from the ITO’s plan, implemented in 2009 (FloorNZ) and 2011 (DecorateNZ), to involve more industry employees in qualification achievement. In particular, the Auckland region has had what the ITO describes as a ‘low training culture’. Some employers with no formal industry qualifications themselves were reluctant to train others. They felt that they either lacked the ability to train or the capacity because of business pressures. The ITO provides resources, assessment of skill gaps, training opportunities, mentoring and some regional seminars through the FloorNZ provider. To date, the results have been mixed, with the number of withdrawals higher than for apprentices in training. In April, an analysis of Recognition of Current Competence candidates showed that many were over-duration with no credit achievement. This impact’s negatively on the ITO’s overall achievement data.
In response, the ITO is putting a number of monitoring and support measures in place such as providing a more structured format for FloorNZ Recognition of Current Competence candidates similar to that provided for DecorateNZ candidates. Unit standards that cause evidence collection problems have been identified and are part of the review and revision of assessment resources. A current focus for field staff is the Recognition of Current Competence programme, and over-duration Recognition of Current Competence candidates will be given additional mentoring and until October 2013 to progress or be withdrawn. Five candidates have graduated this year, but with the focus on managing those in the over-duration category, the number and progress of other current enrolees have not yet been analysed from the database or field staff reports.
Field staff provide most of the information about trainee progress between block course assessments through weekly and quarterly reports using the trainee record of work and on-site observation, and the staff are most likely to know about training and assessment issues (including literacy), progress with training and readiness for assessment, and the reasons for over-duration and withdrawals. Their reports are recorded on the trainee database and are regularly reviewed by training and quality assurance staff; any trends are discussed with field staff in their quarterly meetings. As most training and assessment issues are dealt with by field staff on site, the records of company visits are not collated as the information has already been used.
The TEC assessment tool is used to assess literacy and numeracy on enrolment and the results are recorded on the trainee database. Field staff organise remedial learning where needed. Some providers also provide remedial literacy training, and progress is monitored on a quarterly basis.
Each year the ITO celebrates its graduates and publishes the names of its achievers in the DecorateNZ and FloorNZ newsletters. DecorateNZ and FloorNZ
Final Report
17
both offer two Outward Bound scholarships to apprentices graduating with a national certificate as an incentive and reward for completion.
Self-assessment and analysis of trainee achievement provides the ITO with good information needed for strategic and operational planning, such as predictions of numbers completing, the duration of training and the number of over-duration trainees. However, information about individual trainee progress within enrolments is harder to obtain and analyse. The ITO has good information about individual trainee progress independent of the formal block course assessment results from reports from field staff, but does not collate the information or have a system that tracks progressive ‘credit’ achievement based on workplace verification in between block courses, as the verifiers do not sign off completions of the unit standards and the credits cannot be reported until the final block course assessment. The ITO does not regularly obtain and analyse reports on trainee credit achievement by stages for different sectors or populations. The ITO is heavily reliant on field staff for information, and the organisation could benefit from field staff having access to an online recording and reporting system that maps workplace progress in a simple format. Currently, ITO quality assurance and field staff can only monitor progress through scanned field staff reports, or by viewing the trainee Record of Credit on site. Reports showing achievements by stages within enrolments would also help explain the discrepancies between the TEC’s and the ITO’s own data, highlight any problems for individual trainees, show trends, and provide the basis for improvements in monitoring, recording and analysis of trainee achievement. The ITO has signalled that it has plans for improving its systems for analysing trainee achievement, which include remote access for field staff to its trainee database.
1.4 How effective is the training arranged by the ITO?
The rating for performance in relation to this key evaluation question is Excellent.
The rating for capability in self-assessment for this key evaluation question is Good.
Most of the training is conducted on the job in company workplaces. Off-job training and assessment is arranged at four providers in the form of two to three-week block courses. The providers are contracted by the ITO and their delivery of block courses and performance is monitored by the training manager. FloorNZ apprentices are assessed at the Allied Trades Institute, a Christchurch-based PTE. DecorateNZ apprentices are trained and assessed at three polytechnics: WelTec, Unitec and Christchurch Polytechnic Institute of Technology.
NZ Flooring ITO has focused on curriculum design and training that leads to commercial competence, increased productivity and improved self-esteem for employees. In 2011, it led an initiative to standardise learning materials across pre-trade and in-trade learning, so that trainees have access to the same resources regardless of the starting point of their training. This provides a clear training structure for all trainees, their employers and off-job providers. It is supported by well-presented and relevant resources that can be used by all participants, and
Final Report
18
ensures that the training pathway is understood and accessible. The structure and resources integrate on and off-job learning and are designed to match average industry literacy and numeracy levels. Current research is underway to identify the literacy and numeracy needs of ‘typical’ workers in the sectors in order to improve the training and assessment resources.
Field staff are pivotal to the success of the on and off-job training programmes and are highly regarded by employers, trainees and providers. They enrol trainees, help analyse skills sets, explain the level of skill needed for competence, and monitor progress and facilitate training in situations where the employer does not have work in the sector. They work closely with the block course providers and assist with additional training needs that may result from block course training and assessment.
The ITO’s resources include a training manual consisting of modules for workplace tasks aligned to the unit standards developed in consultation with the providers and industry specialists. The Record of Work provides trainees with a progressive record of achievement of workplace tasks arranged in stages and matches the pre-block course completion requirements. Workplace verifiers sign off competence in ‘critical’ tasks and the completion of ‘desirable’ tasks, indicating the additional skills apprentices may have completed or are currently working on. Field staff ensure that apprentices are prepared for their block courses. A key process, Are You Ready, is undertaken with the apprentice, employer and workplace verifier/trainer, and the coordinator. The completed Record of Work for each stage is sent to the provider.
The ITO intends to review and clarify the workplace verification model and strengthen its links to provider assessment. Nevertheless, providers considered the resources for training to be essential to successful block course completion. In addition, the Are You Ready sign-off meant that the providers were familiar with trainee levels of competence prior to entry, that ‘readiness’ had improved over the years, and that they are better able to review theory and practice and work to fill the gaps. Apprentices said that block courses were important for gaining experience and for the introduction of new skills.
The few trainees who are either not ready for assessment or who need further training are referred to their coordinator who arranges the training with the employer. A trainee may then attend for the next block course for that particular stage, or be assessed by their coordinator. Those who are over the duration for completion are coached and mentored by field staff. Additional training is arranged for Recognition of Current Competence candidates where an initial assessment identifies skill gaps.
The ITO’s training structure works for its industries and is easily managed in partnership with the providers. Providers are very appreciative of the ‘very good’ communication with the ITO and the timely information about trainees received before the courses.
Final Report
19
Given the fluctuation in numbers in recent years due to the recession, field staff have had a significant role in tracking former trainees and arranging a continuation of their training where possible. This also has the important result of keeping the ITO updated on numbers in training and information about individual employer capacity for training. The support and mentoring of DecorateNZ apprentices and the reinstatement of off-job training at three polytechnics is regarded as contributing to the high success rate and retention of these apprentices following the merger.
Proposals to improve on-job training include Train the Trainer courses for coordinators and company trainers such as younger tradespeople, and recent graduates able to communicate in the ‘same language’ as apprentices, as well as study groups and courses within companies or by cluster.
As with other activities, much of the ITO’s self-assessment relies on the information flows from relationships and communication within industry, including regular meetings with field staff and providers, and monitoring of their reports. Many training and assessment initiatives are new, including the recent reinstatement of off-job training for DecorateNZ. An understanding of the effectiveness of both the on and off-job training would benefit from better use and analysis of the information the ITO already has from a variety of sources and from more formal survey methods.
1.5 What is the quality of the assessment being undertaken towards industry standards and qualifications?
The rating for performance in relation to this key evaluation question is Good.
The rating for capability in self-assessment for this key evaluation question is Good.
Assessment is based on a workplace verification model, a double sign-off by the employer/workplace verifier and field staff, and contributes to the final assessment completed at the provider block courses. The reliability of assessor judgements is enhanced by the small number of tutor-assessors involved. Assessment is also based on moderated resources newly developed by NZ Flooring ITO, and by providers. Processes include several different methods of evidence collection such as observation and verification of practical tasks, and written and verbal questioning of theoretical knowledge, and may involve industry specialists for the block course assessments.
Although credit and qualification completion results for individual trainees are only available to the ITO on an annual basis as trainees progress through each stage, the Record of Work is checked regularly by field staff to ensure that the trainee is recording the completion of critical tasks validated by the verifier. This is ongoing between block courses. As these records are not for completed unit standards, no information is forwarded to the ITO’s trainee database. Field staff deal with any problems regarding the use of the materials and literacy or workplace verification
Final Report
20
on site and record the situation and advice in their activity reports which can be used later for moderation and in meetings.
The evaluators noted the need to clarify the resources and accountability for workplace verification. While the Are You Ready resource for checking readiness for assessment has improved the overall training and assessment process, not all employers and their apprentices understood it or used it effectively. The ITO’s internal moderation process has highlighted the issues involved and procedures are being put in place to strengthen and clarify workplace verification overall. This is essential for valid assessment by providers, who may observe competence in practical tasks but are otherwise reliant on workplace records.
Some employers were unsure about the level of skill required of the trainee in relation to the task and workplace competence, and problems relating to the sufficiency of evidence have been identified by field staff. This was also commented on by a provider. As a result, additional detailed checklists for verifiers to use when signing off apprentice competence will be developed.
A review of assessment resources for Recognition of Current Competence candidates is a major project for the ITO this year. Moderation has identified the need to revise assessment tools for some standards and for the assessment methods and structure to be improved. Field staff members are trained assessors (to unit standard 4098 Use standards to assess candidates) and are responsible for the assessment of Recognition of Current Competence candidates and reassessment of other trainees. The ITO’s internal moderation system has been reviewed and will be improved to close the quality management loop. A planned process will be put in place to ensure that moderation results are analysed and used to inform the review process for unit standards and qualifications, and to improve assessment tools and practices. The ITO manages an effective external moderation system, and reports were available. A review of this system showed a sound process of engagement with the providers. Post-assessment moderation of the four contracted providers is conducted annually by a visit from the ITO quality assurance manager and an industry moderator. Postal moderation is conducted for providers of pre-trade courses.
Overall, moderation and quality assurance processes have enabled the ITO to identify the need to strengthen some assessment processes and identify gaps in the internal moderation system, as well as difficulties in the assessment of some standards and the need for changes in the tool, method or standard. This will enable the ITO to follow up variances in assessment, in order to promote consistency. Changes have been made to the ITO’s Consent and Moderation Requirements to clarify the moderation requirements of providers following the inclusion of the painting and decorating sector in the moderation system.
1.6 How well does the ITO's governance and management support the ITO to meet its statutory functions?
Final Report
21
The rating for performance in relation to this key evaluation question is Excellent.
The rating for capability in self-assessment for this key evaluation question is Good.
NZ Flooring ITO is well managed and staff members clearly understand their roles and responsibilities. The team is ably led by a chief executive who has an open, consultative and inclusive approach to management and is focused on building strong relationships with stakeholders. Communication and cooperation between the board and the chief executive ensures that the governance and management functions of the ITO work for the benefit of industry. Board members spoke highly of the chief executive’s ability to find and support the ‘right’ people for the job, and the value to them of regular reports and analyses. They also considered that the ITO has the capability to manage growth in training in response to demand and improved confidence in the construction industry.
The ITO’s 2013-2015 investment plan outlines how the requirements of industry and government (the Tertiary Education Strategy) for leadership, and standards and qualifications development and training will be met by the ITO. It highlights key initiatives for building stakeholder relationships, ways to facilitate learning and reach beyond industry to attract people to its training programmes, collaboration with other TEOs, and industry trends. Evidence provided indicates that much of this is underway through the ITO’s active membership of various groups such as BETA, with its aim to engage with tertiary and industry stakeholders in the Christchurch rebuild and provide pathways from senior secondary school into industry; and through its involvement in the Targeted Review of Qualifications. The ITO’s research and analysis of the industrial environment contributes to improving training systems and resources. The chief executive is active within a number of groups such as the ministerial committee on vocational pathways and the TEC group working on the reporting of learner achievement, and thus able to keep the board and industry informed about developments in areas that affect the ITO.
NZ Flooring ITO managed the merger with the painting and decorating sector of the former Creative Trades ITO well. The chief executive actively engaged in visiting painting and decorating employers and building relationships. Considerable investment was needed from early 2011 to manage continuing trainees, review and revise qualifications and build confidence within the industry. The board and management were aware that the integration of services for the sector needed to provide for increased training volume, trainee achievement, upgrading of training and assessment resources, and industry participation in the overall strategic goals of the ITO. A revised qualification, new resources, active monitoring and the reinstatement of off-job training have contributed to the success of trainees and the merger overall.
The ITO has a very active board. As noted, members are representative of all its sectors, and are experienced tradespeople who have been, or are, actively engaged in training. They are well informed through their professional associations and by the ITO management. Following the merger, the ITO adopted a new governance model based on a small governance group supported by two sector
Final Report
22
representative industry advisory groups. This replaced an earlier industry consultation model built around working groups. Each industry group has a representative on the board, with the main point of contact being senior management.
Operationally, the ITO’s field staff and the regional and Modern Apprenticeship coordinators are essential to the overall success of the training programmes. They are key contributors to trainee achievement and the relationship-building with all industry stakeholders. Through its continued support, monitoring and training of field staff, the ITO has demonstrated its commitment to a training and assessment model that is valued by all stakeholders.
Field staff are contracted to the ITO, and the business development manager monitors their achievement of the key performance indicators through weekly and quarterly activity reports, meetings, and by attending company visits in the regions to observe best practice. Field staff commented that they felt supported and valued and that they were able to make a contribution not only to individual trainee achievement but also to the business of the ITO as a whole. This year they attended mentoring and coaching workshops at a polytechnic, and the ITO is applying to Ako Aotearoa for funding to conduct research into the effectiveness of mentoring in workplace training (in association with the provider).
The ITO is well aware, through its research and analyses, of the effects of the economic recession and the merger on negative credit and qualification achievement as reported by the TEC’s educational performance indicators. While actively monitoring trainee achievement, improving training and assessment processes and working to reduce over-duration, the ITO is looking forward to improvements in reporting systems that may result from the TEC working group examining how a cohort-based reporting model may provide greater validity and accuracy in reporting learner achievement and completion.
The ITO is also aware of the age-band risks associated with the gap between older tradespeople and a much younger workforce that has resulted from earlier non-engagement in training. As noted, it tracks numbers enrolled over the past three years to predict completions and the number of skilled workers likely to enter the industry, and through its Recognition of Current Competence programme is attempting to lift the number of qualified staff who are not only recognised by the industry but who may later become trainers of new staff. The planned review of assessment and the Recognition of Current Competence programme will lead to improvements in assessment practice and resources and higher achievement rates for programme candidates, who represent a significant number of enrolments.
The ITO is aware of some gaps in self-assessment and is working to resolve them. For example, it is looking at ways to improve recording and reporting systems to enable better monitoring and analysis of trainee achievement within enrolments, and incorporating feedback from field staff. The introduction of an online reporting system and remote access to the trainee database would greatly assist the field staff, who currently manage a paper-based system and are dependent on reports of
Final Report
23
trainee progress from support staff. The capability of current systems may not be sufficient to meet the needs of the ITO following the predicted growth in numbers.
Overall, NZ Flooring ITO demonstrated an awareness of how well it is performing based on the formal and informal information flows and communication between staff, the board and employers. A high level of understanding of the ITO’s philosophy and direction by staff and industry stakeholders was evident and contributes to the effective management of its training programmes.
Final Report
24
Focus Areas This section reports significant findings in each focus area, not already covered in Part 1.
2.1 Focus area: Governance and management:
The rating in this focus area for ITO performance is Excellent.
The rating for capability in self-assessment for this focus area is Good.
2.2 Focus area: National Certificate in Flooring (Level 4)
The rating in this focus area for ITO performance is Good.
The rating for capability in self-assessment for this focus area is Good.
2.3 Focus area: National Certificate in Painting and Decorating (Level 4)
The rating in this focus area for ITO performance is Good.
The rating for capability in self-assessment for this focus area is Good.
.
Final Report
25
Recommendations In conjunction with planned reviews, NZ Flooring ITO should undertake:
• A review of the role and responsibilities of workplace verifiers, only some of whom have already been trained as verifiers, with regard to the recording of achievement within enrolments and prior to block courses.
• A review of the trainee database and systems for recording reporting and analysing trainee achievement across sectors, regions and within enrolments.
• A strengthening of survey methods to include specific feedback on training resources, mentoring and support, the usefulness of resources, and provider block courses.
Final Report
26
Appendix
Regulatory basis for external evaluation and review
In 2009 NZQA introduced the evaluative approach to quality assurance in the tertiary education sector, consisting of self-assessment and external evaluation and review. This is applied in the quality assurance of programme approvals and accreditations under sections 249 and 250 of the Education Act 1989, as well as for training schemes (section 251), consents to assess against standards (section 252) AND PTE registration under Part 18 of that Act.
The NZQA Board also published policies and guidelines for the conduct of external evaluation and review of ITOs on 27 May 2010. NZQA relies on ITOs to build the self-assessment and external evaluation and review model into their infrastructure. ITOs will be evaluated on their infrastructure and use of the model in the following ways:
• Advising Ministerial recognition of an ITO under the Industry Training Act, 1992
• Accepting arrangements for monitoring and assessing of industry training under section 10 of the Industry Training Act 1992
• Registering standards on the Directory of Assessment Standards
• Awarding consents for ITOs (or their assessors) to assess against standards on the Directory of Assessment Standards under section 252 of the Education Act 1989
• Applying programme approval criteria where ITOs are course owners.
External evaluation and review is also used by NZQA as a monitoring and evaluation tool, with the outcomes of these processes informing Tertiary Education Commission decisions relating to re-recognition of an ITO. In addition, external evaluation and review reports are one contributing piece of information in determining future funding decisions in relation to an investment plan agreed between an ITO and the Tertiary Education Commission. The understandings and expectations for the implementation of ITO quality assurance are set out in a protocol to the Memorandum of Understanding between NZQA and the Tertiary Education Commission.
This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the policies and criteria approved by the NZQA Board. The report identifies strengths and areas for improvement in terms of the ITO’s performance and capability in self-assessment.
External evaluation and review reports are public information and are available from the NZQA website (www.nzqa.govt.nz).
Final Report
27
Information relevant to the external evaluation and review process is summarised in the publication Policy and Guidelines for the Conduct of External Evaluation and Review. Two other documents explain how the process is applied to ITOs: EER Policy and Guidelines – ITO Supplement and Evaluation Indicators for Industry Training Organisations. These documents are available at: http://www.nzqa.govt.nz/providers-partners/registration-and-accreditation/external-evaluation-and-review/policy-and-guidelines-for-eer-ito/introduction/
