1

Final Report

Usability of the NZQF for Learners – an independent review

Prepared for

New Zealand Qualifications Authority

Mana Tohu Matauranga O Aotearoa

22 November 2013

2

Table&of&Contents&

Final&Report ............................................................................................................1!Key&Point&Summary ................................................................................................3!Background ............................................................................................................3!Approach&taken&to&the&Review................................................................................4!Findings ..................................................................................................................5!Where do learners get information about qualifications? ............................................ 6!How usable is the NZQF? ........................................................................................... 7!Barriers and enablers to using the NZQF.................................................................... 8!Conclusion............................................................................................................10!Appendix&One:&Participant&Learners&and&their&study .............................................12!Appendix&Two:&Figure&Two:&Ratings&by&Age...........................................................13!Appendix&Three:&Detailed&website&feedback* .......................................................14!

3

Key Point Summary This review was undertaken to independently assess the usability of the New Zealand Qualifications Framework (NZQF) for learners. It was conducted in September-October 2013 and involved focus groups with learners engaged in qualifications at levels 1-6 and a small number of stakeholder interviews.

The NZQA role as a provider of reliable, clear and credible information on qualifications is confirmed through this review.

• Learners rated the information they looked at on qualifications on the NZQF highly in terms of understanding the information and its usefulness.

• Learners appreciated: • that the qualifications they looked at were current; • that information on who provides the qualification was available; • that information on the education and employment pathways will be

available in the future as new qualification information is added.

Knowledge levels amongst learners however are low about the qualifications system, about qualifications in general and the information that is available on the NZQF. Learners were also less likely to rate ease of finding information on the website highly.

There are ways in which advisors and learners think the NZQA website could be improved in relation to search options, information provided and its look and feel for learners so that they can access the NZQF more easily. These aspects would however need to be considered in light of the wider purpose of the site and its multiple audiences.

Background The 2009 research on learners’ views on the certificates and diplomas on the NZ Register of Quality Assured Qualifications found the following in relation to the qualifications register1:

• Qualifications are primarily chosen on the basis of their accessibility for learners: cost, location and time

• Learners generally find information about qualifications from the internet and people who work at tertiary education organisations (TEOs). The openness, helpfulness and knowledge of the staff they talked to at the TEOs were often the deciding factor in qualification enrolment

1 Parker, S., Long, A., Baldwin, K., & Bourke N. (Dec 2009) Learners’ Experiences of Certificates and Diplomas on the New Zealand Register of Quality Assured Qualifications. New Zealand Qualifications Authority

4

• Learners want to know the following before enrolling in a qualification: content of the qualification (and that it will meet their needs); that it is quality assured by NZQA; that it is accessible to them

• Learners spent little time comparing different qualifications2.

The 2008-9 Targeted Review of Qualifications (TRoQ) further identified that the qualifications system was difficult for learners (and others) to understand because it:

• was not relevant to some employers and industry • was not user friendly, and the status of some qualifications was unclear e.g.,

the qualification may have expired, or no longer be offered • contained a large number of similar qualifications that made distinguishing

between qualifications and identifying education/career pathways difficult.

A wide range of changes was implemented in 2010, in order to streamline and simplify the qualifications system. These included:

• The establishment of the NZQF as a single location for all qualifications in order to make it easier to find information about qualifications

• Assigning a status to qualifications and changing search function settings so that the default search function setting is for current qualifications

• New requirements for listing qualifications on the NZQF also sought to prevent the duplication and proliferation of qualifications (progressive replacement of local and National qualifications with New Zealand qualifications at levels 1-6)

• Clearly defining qualification outcome statements on new qualifications.

In 2012 NZQA undertook to ‘demonstrate that the usability of the NZQF for learners compared with the NZ Register of Quality Assured Qualifications is improved by 2013 as evidenced through independent analysis.’ (NZQA Statement of Intent, 2012/13, p. 17).

Approach taken to the Review This independent review sought to answer the following questions:

1. When learners and/or their advisors access the NZQF how useful and usable do they find the information?

2. What are the barriers and enablers for learners and/or their advisors using the NZQF and the information?

The review scope focused strongly on learners and their experiences of understanding and using the NZQF at levels 1-6. The learners included those from priority groups, of youth, Pasifika and Māori.

2 The stated purpose of the research was to learn about how learners understand and navigate the New Zealand qualifications system, not use of the Register per se. Much of the report and questions asked concern learners broad experience of finding information about and gaining their qualifications.

5

Based on existing evidence of the previously identified issues a framework of usability dimensions was developed along with questionnaires for use with learners, key learner advisors and stakeholders. The NZQA also asked that the review consider the NZQF within the context of international education.

Fifty-four learners enrolled in qualifications from Levels 1-6 from ten organisations participated in focus groups for the review. (See Appendix One for details.) These comprised two secondary schools, six private training establishments and two polytechnics in the Wellington region. Focus group participants were selected by their organisations. A further four organisations were invited, but declined to participate in the review.

Given these numbers the review provides a snapshot of the experiences of learners. While the sample is small in comparison to the population, Guest, Bunce and Johnson (2006)3 found that when questions are straightforward in a narrow applied field that 12 interviews are usually sufficient to generate themes. Their conclusions were confirmed in our review process where we found by the last two-three focus groups we were not getting new themes.

Koha was given to participants and while their organisations had been informed this would be the case, most of them chose not to tell the learners about this prior to the group.

To provide context for the review, and to understand any additional ways others support learners with information about qualifications a number of additional informants were invited to contribute to the review4. These included:

• Learner advisors in five of the organisations where we met with learners • Learner advisors from an ITO and a polytechnic • Ministry of Education (International) and Education New Zealand with a focus

on international education • Careers New Zealand and the Tertiary Education Commission who have

been working on the provision of information for learners.

Findings This section starts with an outline of the complex context in which learners are getting information and making decisions about qualifications. Information about qualifications is just one aspect of the information that learners use as part of their decision-making process. The review findings are presented in terms of where learners get information about qualifications, how usable the NZQF is for providing this information and the barriers and enablers found to accessing and using the NZQF. 3 Guest, G., Bunce, A., and Johnson, L. (2006). ‘How Many Interviews are Enough? An Experiment with Data Saturation and Variability’, Field Methods, 18: 59–82. 4 A total of 15 informants were interviewed for the review. A further three organisations were asked to participate but declined.

6

A number of organisations have an interest in how learners may access information and what information they find most useful. A literature review conducted by the Tertiary Education Commission (TEC) (2012)5 found the decision-making process is complex and choices are determined by personal interest, intrinsic and extrinsic motivations, socio-economic factors, outside influencers such as family, friends, and trusted advisors (teachers, careers advisors, church groups for Pasifika, whānau, hapū and iwi for Māori), and provider websites. Parents are particularly strong influencers of learners moving from secondary to tertiary education.

However, while learners wanted information about what and where to study they were unlikely to actively seek it out. There were some who felt overwhelmed about the amount of information available from providers and felt they had no mechanism for judging how reliable and accurate the information is. In relation to qualifications the TEC review found learners wanted information about the expected outcomes of qualifications and qualification/career pathways. This is in line with the direction the NZQA is heading with education and employment pathways information provided on qualifications on the NZQF.

Where do learners get information about qualifications? This review found similar decision-making processes by the learners as the TEC literature review. Learners were influenced by family, friends or trusted advisors, by word of mouth, by the reputation of the organisation and/or the information they provided.

• Members in my family. My father is in Corrections, my grandfather was in the army and in the government in Samoa and what I wanted to do was help people and I thought the best way to do that was to keep our community safe so I decided to become a police officer.

• I had been to an exhibition a long time ago … and then … people just kept coming across my path who were talking about [X organisation] and when I decided this was what I wanted to study I researched [other organisations] and decided on the one I felt was going to give me the most which was hands on experience and the ability to still get a qualification.

• I worked for builders and a few other tradesmen and they said that electrical probably was my thing and I was told to give it a go …

Some of the learners were also required by Work and Income New Zealand (WINZ) to undertake courses of study.

The international students were similar but they were also influenced by aspects relating to the International English Language Testing System (IELTS) requirements, the location (both having made a decision to come to New Zealand and to Wellington) and that an organisation is quality assured by NZQA, “Any college that is

5 Tertiary Education Commission. (2012). Tertiary information needs: Literature review. Unpublished Report.

7

related to NZQA … is not fake. They all have to be registered”. Key informants on international education also acknowledged the importance of quality assurance from the government, “This is really important to the international market. They want to hear that it’s government regulated and that you can move around the system and that it’s connected up.”

How usable is the NZQF? Learners in the focus groups were not familiar with the NZQA website or how to search for qualifications on the NZQF therefore they were shown how to access the qualifications page on the NZQA website and then asked to search for a subject area and level that interested them. They were then asked to select a particular qualification and asked to rate (1-5 scale with 1=low, 5=high) the information provided about the qualification. Their ratings are shown in Figure One below. Combining the ratings in the way we have enables NZQA to see ratings that can be considered “acceptable” (2-3) and ratings that can be consider “good” (4-5).

Figure One: Learner ratings (n=54)

As can be seen from the figure, highest ratings (those that rated at 4/5) were given to:

• the usefulness of the strategic purpose statement • the language used was easy to understand • the overall usefulness of the information • the information about where the qualification was offered.

The lowest ratings were given to:

• the ease of finding information • how useable the website was for learners like them • the likelihood of using the qualifications information in the future

0%! 20%! 40%! 60%! 80%! 100%!

Use!in!the!future!

Usability!for!learners!

Overall!usefulness!of!information!

Ease!of!language!

Where!to!do!quali@ication!useful!

Strategic!purpose!!statement!useful!

Enough!information!

Easy!to!@ind!

1!!2D3!!4D5!

8

Note this latter point about use in the future cannot be considered a criticism of the information per se as many learners commented that they were unlikely to use it again as they were not going to be undertaking further study or that they had already decided on a provider so would go directly to that website, “[I would go] directly to the source. I’m going to [X] so I would go there and find out.”

There is little difference in the ratings given by the different age groups (See Appendix Two), however there is an indication that the older students found the language easier to understand than the younger students. The youngest students gave higher ratings than the other students to there being enough information to understand what the qualification is, whether and where it is available.

As the ratings show the majority of students had a positive reaction to the information provided and this came through in focus group discussion after the learners had looked at the information on the NZQF.

• If I knew that this qualification search thing was on the site I would have been on there before. I didn’t know you could search through this for quals because I used Google. But this is way better. But even if I jumped on this site I wouldn’t know where to look for it.

A career’s advisor was also somewhat surprised about the information available and wondered why people in career advisory roles had not been informed about what was available. S/he thought it would be useful for NZQA to run workshops on the NZQF and the qualifications.

Barriers and enablers to using the NZQF As stated above none of the learners had used the NZQA website to access qualifications information on the NZQF. Those that had used the NZQA website did so to check their Record of Achievement and those at school also used it to search for National Certificate of Educational Achievement (NCEA) exemplars. Most of them had sourced information about qualifications through provider websites. Therefore there was an overall lack of familiarity with the site that ranged from not knowing it existed through to knowing there was information for learners about qualifications on the site.

• I didn’t even know they had a website. • I find it rubbish. • I have no idea how to use it to be honest. • All you know about NZQA is to check how many credits you have … Maybe if

we actually knew more about it in high school. • I just type in Google because it’s hard to find them from the home page.

Given these findings the learners were somewhat surprised when they found the information on qualifications, “It seems simple enough to use … [but] no one knows that stuff is on there or there’s no information to get people to use it. Some of the learners also thought that there is the potential for it to be used by in families, “It’s

9

somewhere where your family could go, sit down with their child and discuss, search the quals.”

While, as the ratings show, the learners found the information useful and usable there were a number of factors that prevented them from giving higher ratings to usability. These fell into four main areas: look and feel of the NZQA website, navigation, the information provided, and hyperlinks. Appendix Three provides a detailed analysis of the feedback specifically relating to website features.

Look and feel of the NZQA website

The look and feel of the website was not seen as very welcoming for learners, “It could be more visually appealing. It hasn’t changed in several years that I have been using it. Learners thought that is was not really a site for them. “I think that NZQA is more targeted to teachers, not students. It looks bland because it is targeting one demographic like tutors, academics, … it’s giving me options that I don’t even understand.”

Navigation of the website

In part the issues learners had with site navigation arise from their lack of knowledge about the system e.g., few understood qualification levels and the concept of progression through the NZQF, “I have no clue what the categories were for the levels and stuff.” This hampered their ability to easily access relevant qualifications or make sense of a list of different qualifications.

For some even finding where to start searching qualifications was difficult as the search function key could be sitting within text that requires the user to scroll down (e.g., when entering through ‘certificates and diplomas’ page). As learners have a strong focus on the location of a provider, a number suggested that an additional ‘location’ search field would assist them.

The information on the qualifications

The most commonly commented on information that learners and advisors would like to see concerned information on the location of the provider delivering the qualification, fees information, the range of ways a qualification is delivered e.g., by distance, and information that is not currently available (but will be) concerning education and employment pathways. International students were particularly interested in being able to search by location, “If we are looking to study in Wellington, much better to have search by the city, where we can see [X] and other colleges. Put the city would be much better.” For international education purposes, information on whether an international student can enrol in the qualification would also be desirable.

In addition greater consistency in the free text fields would be helpful, particularly to ensure that the field contains concise text that is comparable across qualifications. We noticed that those who had to “open” the strategic purpose statement for more information were likely to lose interest.

10

Hyperlinks to provider organisations

When a learner clicks on a provider hyperlink on NZQF they expect to be directed to the most relevant place on the provider’s website, such as the qualification itself or enrolment information, not the homepage of the institution or the provider information on the NZQA site. “If it was more defined to the actual link it would be good.”

Conclusion The original purpose for this review was to ascertain whether learners experience with using the NZQF has improved relative to the experience of learners using the NZ Register of Quality Assured Qualifications. That is, were the same issues apparent in 2013 as those identified in 2009-10 when decisions were taken to make significant improvements to the qualifications system and information about qualifications? We can confidently say that the issues identified in 2009 have largely been resolved for learners:

• there is now a single site where qualifications can be found • the default search setting for qualifications result in current qualifications

being listed first • learners can easily find the overall purpose of the qualification (strategic

purpose statement) • new qualifications are beginning to include information on qualification

outcomes; graduate profile, education and employment pathway information.

There are some barriers remaining for learners accessing information on qualifications on the NZQF (outlined above), although these need to be considered in light of the broader purpose of the NZQF and the information paths that potential enrolees may take when deciding on a qualification at levels 1-6.

The NZQA has as a primary purpose the role for quality assuring all non-university tertiary education organisations, and approval of all qualifications developed by these organisations. As such, the NZQF is the vehicle that enables and supports the provision of high-quality education pathways and enhances confidence in the quality and international comparability of New Zealand qualifications. Given this authoritative role the information that NZQA provides through the NZQF it will not necessarily meet all of the learners’ information needs in relation to study and careers.

Reliable and credible information about qualifications is essential to ensure that learners know about all their possible study options. Therefore NZQA need to utilise additional ways to disseminate information about the NZQF. This is being achieved through, for example, existing information feeds to Careers New Zealand for New Zealand learners, given younger people are more familiar with this site. There is also potential for a similar information feed to Education New Zealand to better service the international education market. Delivering high quality information through sites

11

that are more geared to particular learner audiences could greatly enhance the usability of the information.

The second key finding from this review concerns the low level of knowledge amongst learners about the NZQF, meaning that they know little about how the qualification system works, about qualifications in general and the information available on qualifications. A number of learners and advisors had suggestions about improving this situation. Younger learners tend to use their school websites to track their Record of Achievement and some were unaware information was also held on the NZQA site. Some of these learners suggested it would be useful if they were introduced to what was available on the NZQA site while they were still at school. School careers advisors also commented they would find updates and professional support for use of the NZQF (as they receive from Careers New Zealand) useful in profiling what’s available on the NZQF and prompt their greater use of the site. Overall however, we conclude the NZQA as the authoritative provider of information on NZ qualifications does this for a range of purposes and for a range of audiences, including learners. We suggest the provision of this information to learners needs to continue to be facilitated through sites that may better meet learner needs, such as Careers New Zealand and that options for delivering similar information through sites such as Education New Zealand would be worth further exploration. Taking this approach would reach wider audiences of learners and increase the usability of the information.

12

Appendix One: Participant Learners and their study

Gender Male 63% (34); Female 37% (20)

Age 17 or younger 18.5% (10)

18-20 37% (20)

21-25 31.5% (17)

26+ 13% (7)

Ethnicity Pasifika 11.1% (6)

Māori 11.1% (6)

Asian 13% (7)

NZ European 64.8% (35)

Level of Study Level 1-3 46.3% (25)

Level 4 25.9% (14)

Level 5-6 29.6% (16)

Area of Study Business 31.5% (17)

NCEA 20.4% (11)

Arts / Broadcasting 20.4% (11)

Trades 16.7% (9)

Tourism / Hospitality 11% (6)

Appendix Two: Figure Two: Ratings by Age

0%#10%#20%#30%#40%#50%#60%#70%#80%#90%#100%#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

17#and#younger#(n=10)#

18920#(n=20)#

21925#(n=17)#

26+#(n=7)#

Ease#of#>inding#quali>ication#

Enough#information#to#understand#what#the#quali>ication#is,#whether#and#where#it#is#available#

Usefulness#of#strategic#purpose#

statement#

Usefulness#of#information#about#

where#quali>ication#is#delivered#

Ease#of#the#language#

Overall#usefulness#of#information#

Usability#for#learners#

Likelihood#of#using#site#in#the#

future#

495#

293#

1#

Appendix Three: Detailed website feedback*

Dimension Feedback

Search fields: • Location of campus where qualification delivered would be very useful to learners

Information fields: • Short, consistent, easily understood data fields met learner needs well; longer fields lost learner interest

• Information also sought on: o Fees; fees free, fee ‘bands’ as minimum o The ways a qualification is delivered e.g., by

distance o For international learners – confirm

international students are able to enrol in the qualification

• Data in the new fields (most of which are currently not available) will be of benefit to learners e.g., education pathways, employment pathways

Navigation: • Learners could start a search from a range of places on the site: from the search function on the NZQA home page using a keyword a very large lists of qualifications tended to be generated that learners struggled to know how to sort or process

• The further into the site learners went e.g., from the certificates and diplomas page in the ‘Qualifications & Standards’ section, this helped the searcher narrow the results

• The ‘search qualifications’ points can be buried in text in the centre of the page however and are hard to find (e.g., on certificates and diplomas page)

• Search terms that give no results e.g., ceramics, sculpture, left learners unsure where to search

• Hampered by learners’ lack of knowledge about the qualifications system e.g., few know what the levels are on the framework

Hyperlinks: • Ideally clicking on a provider link on NZQF would take a learner to the most relevant place on the provider’s website, such as the qualification itself or enrolment information, not the homepage of the institution or the provider information on the NZQA site

Free text fields: • Need to ensure text is concise and structured similarly across qualifications

* This is a compilation of feedback from learners derived from focus groups conducted for this review and learner feedback to Careers New Zealand and TEC for similar projects. The themes were very similar from each of these sources, with some being more detailed in particular areas.