impact of prior experience (sample)
TRANSCRIPT
-
8/13/2019 Impact of Prior Experience (Sample)
1/97
TitleThe impact of prior experience on teachers' perceptionsof standards-referenced assessment
Author(s) Ng, Ka-ming;
Citation
Issue Date 2007
URL http://hdl.handle.net/10722/51294
RightsThe author retains all proprietary rights, (such as patentrights) and the right to use in future works.
-
8/13/2019 Impact of Prior Experience (Sample)
2/97
The impact of prior experience on teachers perceptions of
standards-referenced assessment
By
NG Ka-ming
(U. No.: 1991160744)
August 2007
-
8/13/2019 Impact of Prior Experience (Sample)
3/97
i
Acknowledgments
I am most grateful to my academic advisor, Dr. Philip Stimpson. His
professional guidance and continual support is of vital importance to my research.
Indeed, I have learnt a lot from him regarding research, teaching and other personal
matters. I would like to express my sincere gratitude to Mr. Gwyn Edward, Dr. Pang
Ming Fai and Prof. Mark Bray who served on my coursework. What I learned from
them over the years would fill many more pages than this dissertation. Also thanks to
Ms Gigi Lui for her constructive suggestions and kind words of encouragement.
-
8/13/2019 Impact of Prior Experience (Sample)
4/97
ii
Declaration
I hereby declare that this dissertation represents my own work and that it has
not been previously submitted to this University or any other institution in application
for admission to a degree, diploma or other qualifications.
_______________________
August 30, 2007
-
8/13/2019 Impact of Prior Experience (Sample)
5/97
iii
Abstract
Whilst many studies investigating teachers concern have been carried out to foster
curriculum change in the past, this paper examines teachers views of the recent
assessment reform, the adoption of standards-referenced approach, in an attempt to
ascertain what concerns appear prevalent as the hindrance to the change. By using
Concern Based Adoption Model 253 teachers stages of concern are collected and
analysed, and five interviews are conducted to consolidate the current beliefs about
standards-referenced assessment in practice. This study finds that the hindrances to
the educational change are the poor quality of teacher support, the obstructive
environment for the dissemination of information about SRA, the untimely
implementation, perceived heavy workload, and stultifying school management and
policy. Finally, this paper explores the good practice in educational change.
-
8/13/2019 Impact of Prior Experience (Sample)
6/97
iv
Contents
Acknowledgements .................................................................................. i
Declaration ............................................................................................ ii
Abstract .................................................................................................. iii
Table of Contents .................................................................................... iv
Chapter 1: Introduction
1.1 Background .................................................................................... 1
1.2 Adoption of standards-referenced assessment by teachers .............. 4
1.3 Teachers concerns about the innovation of SRA ............................ 6
1.4 The aim of the study ....................................................................... 8
1.5 Research questions ......................................................................... 8
1.6 The structure of the dissertation ...................................................... 9
Chapter 2: Literature Review
2.1 Introduction ................................................................................... 10
2.2 Standards Referencing .................................................................... 10
2.3 Curriculum Change ........................................................................ 19
2.4 Teachers perception and Concerns Based Adoption Model ............ 22
2.5 Conclusion ..................................................................................... 25
Chapter 3: Methodology
3.1 Introduction ................................................................................... 26
3.2 CBAM approach ............................................................................ 27
3.3 Research design ............................................................................. 28
3.4 The operationalisation of Stages of Concern Questionnaire .............. 31
3.5 Follow-up interviews ..................................................................... 37
-
8/13/2019 Impact of Prior Experience (Sample)
7/97
v
Chapter 4: Findings
4.1 Introduction ................................................................................... 42
4.2 Demographics of the study ............................................................. 42
4.3 Stages of Concern derived from the questionnaire survey ............... 44
4.4 The interview findings ................................................................... 47
4.5 The impact of prior experience ....................................................... 55
4.6 The hindrances to the change ......................................................... 60
4.7 Conclusion ..................................................................................... 64
Chapter 5: Conclusions
5.1 Introduction ................................................................................... 65
5.2 Concerns of teachers about the innovation (Research Question 1) .. 65
5.3 Factors influencing variation in concern (Research Question 2) ...... 66
5.4 Hindrances to the change (Research Question 3) ............................ 69
5.5 Limitations ..................................................................................... 71
5.6 Implications of findings for practice ............................................... 74
5.7 Implications for research ................................................................ 78
5.8 Concluding Remarks ...................................................................... 79
Appendix A (Stages of concern questionnaire) .................................... 80
References ............................................................................................. 83
-
8/13/2019 Impact of Prior Experience (Sample)
8/97
1
CHAPTER 1
Introduction
1.1 Background
The Hong Kong public assessment system is undergoing fundamental conceptual reform
including a move to standards referencing in which students are assessed against stated
standards or criteria. In the current school leaving public examinations at 17+ and 19+
years, the Hong Kong Certificate of Education Examination (HKCEE) and Hong Kong
Advanced Level Examination (HKALE), a norm-referenced grading methodology is
adopted for the majority of subjects (Choi, 1999) but this is to be changed to a
standards-referenced system in 2009 (EMB, 2005). This has followed views expressed in
1993 by the Education and Manpower Branch of the Hong Kong Special Administrative
Region (HKSAR) government that norm-referencing is inappropriate in a school system
to be introduced dedicated to the development of individual potential as opposed to
selecting an elite minority (Biggs, 2003, p.173). This study is concerned with teachers
perceptions of the change.
Under the current grading system within the two public examinations, the
norm-referenced design is such that only a set proportion of candidates can obtain a high
grade; a proportion must also fail although these proportions may vary as the examination
paper levels of difficulty vary slightly from year to year. For example, in the 2006
HKALE, Use of English was the most numerically popular subject, but only 13.8% of the
candidature were able to obtain a C grade or above within a system of six coarse grades
(A, B, C, D, E and F) (HKEAA, 2006) and 0.6% of the candidature obtained an A grade.
This norm-referenced approach, which the Hong Kong Examinations and Assessment
Authority (HKEAA) uses, more or less yields similar figures across the years. This
-
8/13/2019 Impact of Prior Experience (Sample)
9/97
2
situation, in which a large portion of candidates obtain lower grades as having achieved
nothing, was accepted by the Hong Kong society because the HKALE largely served
selection (Choi, 1999). This was acceptable in an environment where opportunities were
strongly limited. This is less so now. Before 1986, less than 4% of the 17-20 age cohort
enrolled tertiary institutions in Hong Kong (UGC, 2005b) and the HKALE, at that time
offered a sufficient platform for universities to select elite students. However, throughout
the 1980s, there was a growth of tertiary institutions and first-degree places available as
the University Grant Committee (UGC) (the University and Polytechnic Grants
Committee during 1980s) supported increased funding for higher education. Growth in
the 1990s was such that the Government intended to reach a tertiary participation rate of
18% with 14,500 (UGC, 1993). Thus, by 2004-05, there were 14,585 first-year students
in UGC-funded tertiary institutions (UGC, 2005a) representing a four-fold increase over
the previous 20 years. Almost 50% of the HKALE candidates found a place in a tertiary
institution in 2005. The need for a highly selective examination design became less
crucial than the situation two or three decades earlier.
If a norm-referenced system was now less required for picking out elites, it was
equally inappropriate for the low achievers. Every year, there are around 6,000 candidates
of the HKCEE obtaining a zero grade point, which means they have no subjects at E
grade or above. They are labelled as zero-mark students. The existing grading
methodology limits the grade distribution in each subject and cannot be changed
significantly even though the entire candidature performs well overall in the examination.
This system is discouraging many students who may go away with the misperception that
they have achieved nothing. Zero-mark does not mean achieving nothing at all. In fact,
they have something but at a lower level than the others.
The HKEAA initiated a study called the Review of Public Examination System
-
8/13/2019 Impact of Prior Experience (Sample)
10/97
3
(ROPES) in 1998. The ROPES (1998) noted that public examinations no longer serve
merely the need of an elitist cohort. In 2003, the Review of the Academic Structure for
Senior Secondary Education and Interface with Higher Education (RASIH) report
(Education Commission, 2003) concluded that the existing grading mechanism of public
examinations falls short of meeting the current expectations of community. The function
of the examinations has gradually moved from selection to certification. At a similar time,
the Education Commission in Hong Kong, an advisory body for the Government,
proposed general education reforms and suggested new mechanisms for assessing student
performance that embraced three functions: facilitating teaching and learning,
certification, and selection. It further emphasised that public examinations needed to
serve certification as well as selection. Therefore, reform of public examinations was
proposed and subsequently adopted (EMB, 2005). The reform proposal suggested the
adoption of a standards-referenced approach in public examinations arguing that this
would improve the grading mechanism.
A standards-referenced approach became a blueprint for assessment reform and a
new single qualification at Senior Secondary Education level, the Hong Kong Diploma of
Secondary Education (HKDSE), combines the HKCEE and the HKALE. The new system
would adopt a standards-referenced approach for reporting results. It would involve five
descriptor levels for each standard although to placate universities, Level 5, the highest,
was to adopt a norm-referenced approach in which Level 5** and Level 5* represent the
top 1% and the next 3% of the candidature (EMB, 2005, pp. 9-10). The HKDSE is thus
not a purely standards-referenced examination as in some other examination boards.
Patently, it is a hybrid approach. Nevertheless, feedback from public consultations reveals
that this approach is not rejected but not always fully accepted by teachers and other
stakeholders.
-
8/13/2019 Impact of Prior Experience (Sample)
11/97
4
1.2 Adoption of Standards-Referenced Assessment by Teachers
Hong Kong has been undergoing a raft of education reforms in decades. Most education
reforms in Hong Kong were brought in because of political and economical issues and
had substantially ameliorated the education system (Sweeting & Morris, 1993). Some
education reforms delineated the influence of the handover to Chinese sovereignty in
1997. The reforms in the transitional period to Chinese sovereignty (1984-1997) were
introduced at a rapid rate and seemed to be symbolic. The post-handover government has
inherited the problems in the education system and the symbolic reforms; the subsequent
reforms are then forceful (Morris & Scott, 2003). Some reforms (for example, nine-year
free and compulsory education) are successful, but some reforms such as Target-Oriented
Curriculum have encountered many encumbrances that many palliatives are embodied in
the implementation (Lam, 2003).
Reform means to make an improvement, especially by changing a person's
behaviour or the structure of something. Recent curriculum and assessment reforms in
Hong Kong purport to make a change for the better and their implementations are actions
to improve the existing conditions (Education Commission, 2000). The New Senior
Secondary Education aims to bring significant and long term benefits to Hong Kong
(EMB, 2004) while the HKDSE in the new academic structure aims to serve selection and
report student achievement explicitly. On the one hand, some local scholars posit that the
political system and internal constraints hinder education reforms in the present
environment (Morris & Scott, 2003, p.83). It is hard to forecast the success of the new
academic structure. On the other hand, after experiencing the recent policies for the two
major curriculum reforms, Target-Oriented Curriculum and School-Based Curriculum,
which had been compiled in 1990s, different stakeholders should have a thought on the
new academic structure and the corresponding new assessment framework
-
8/13/2019 Impact of Prior Experience (Sample)
12/97
5
Standards-Referenced Assessment (SRA).
Some teachers have some experiences of SRA whereas others have little. For
example, some large-scale assessment programmes based on standards-referenced
frameworks have been introduced in recent times. The Territory-wide System Assessment
(TSA) was gradually introduced from 2004, first in Primary 3, the next year in Primary 3
and Primary 6, and continued to Secondary 3 in 2006. It adopts a standards-referenced
approach to assess students performance against professionally defined competences in
Chinese, English and Mathematics. Secondary teachers of Chinese, English and
Mathematics have gained some knowledge about SRA through their involvement with
TSA in which all Secondary 3 students have to participate. While other teachers do not
have this opportunity to gain SRA experience, the change in 2009 may seem more
threatening to them. Their concern about the adoption of SRA in public examinations
varies.
These differences will be compounded in the languages as a standards-referenced
system has also been implemented in the HKCE English language and Chinese language
curricula from 2005. With effect from the 2007 HKCEE, a standards-referenced
framework will be adopted and the performance of candidates will be reported with
reference to a set of five defined levels of performance (HKEAA, 2005). Descriptors
illustrating the expected performance of students at each level have been disseminated to
teachers and students and this will have enhanced understanding of SRA by language
teachers. Moreover, all teachers involved with the two language curricula will have to
attend a teacher training course to support and carry out SRA. Having more exposures to
SRA than non-language teachers at HKCEE, the language teachers may be more
competent, confident, and less concerned at the adoption of the new curricula in 2009.
Regarding SRA, secondary teachers of Chinese language and English language
-
8/13/2019 Impact of Prior Experience (Sample)
13/97
6
are not only practitioners, but also users. In 2001, the EMB set up the Language
Proficiency Assessment for Teachers (LPAT), which is a five-level standards-referenced
assessment. It purports to raise the teacher professionalism and ensure the teaching
quality. It also safeguards against the public criticism of the falling language standards on
graduated students. Teachers who teach English language or Putonghua should reach
Level 3 or above in the LPAT or complete the equivalent teaching training course before
the school year in September 2006. The teachers should well understand the five defined
levels of performance and check their qualification against the standards, in order to
continue their language teaching career.
In the same way, teachers acquaintance with SRA is not limited to these
assessments and examinations. The EMB collaborating with the HKEAA had carried out
three large-scale consultation campaigns of the New Senior Secondary Education Reform
for principals, teachers and the public during 2004 - 2006. All teachers were entitled to
participate in these campaigns together with various ancillary forums and seminars
discussing curriculum change as well as the new assessment model, the
standards-referenced approach.
1.3 Teachers concerns about the innovation of SRA
The concerns or attitudes stakeholders of education reforms have about a change are an
important dimension in the change process (Hal, Wallace and Dossett, 1973). Teachers
views and concerns about the feasibility of SRA would vary. In the 2007 HKCE English
language and Chinese language examinations, teachers perceive the SRA differently. The
results of the public consultations regarding the innovation of SRA were positive and
teachers lauded the adoption of SRA. Nevertheless, some teachers publicly voice out the
difficulties they have encountered in teaching halfway through the first year of
-
8/13/2019 Impact of Prior Experience (Sample)
14/97
7
implementation of the curricula and decry the new assessment framework that comport
themselves with the disavowal of the new assessment practice. Thus, they are worried
about the innovation of the SRA.
Responses to these concerns come from several directions. The innovation of SRA
implies a change in the teaching practice that teachers have to cope with not only the new
grading methodology, but also the new mode of public assessment. The introduction of
School-Based Assessment (SBA) in the two language examinations leads to a new
assessment practice in school. Teachers perceive that SBA is an add-on workload to their
current practice, and they have dissenting views on this matter although Government
officials explained it in several teachers seminars.
Moreover, teachers can perceive no significant benefits, but an additional
workload to themselves and students at the beginning of the implementation. They are
likely to have a concern and to resist the change. It seems that more time for preparation
can help lower the resistance and an appropriate timeline of implementation is needed for
teachers to well prepare the change.
Furthermore, the former HKCE English language examination had two streams,
Syllabus A and Syllabus B. Students in Syllabus A were easier to obtain a good grade than
that in Syllabus B as the standard requirement in Syllabus A is lower than that in Syllabus
B. In the new HKCE English language examination, there is only one combined syllabus.
Teachers who were used to mainly teach Syllabus A have a concern about the fairness and
appropriateness of this change to their students, the weaker cohort, who may suffer a
disadvantage from the competition with the other. The examination results and their
reporting method become the spotlight of the reform.
In addition, a high population for awarding Level 5 is set in the new grading
structure, that is more or less the same as the existing population for awarding Grades A,
-
8/13/2019 Impact of Prior Experience (Sample)
15/97
8
B and C, meant to address and recognise the good performance of students. Teachers
remain sceptical about the new grading method which might evolve a portrait of grade
inflation, compared with the previous, long-life grading method. In the same way, the
new grading method might yield an unstable distribution of levels in the results over the
years and people with limited assessment literacy on SRA might think that it is not
reliable too. Thus, teachers may query the effectiveness of the new grading method and
have a concern about its reliability in the future years.
1.4 The Aim of the Study
Teachers, it can be argued, have varying stages of concern about the change of the public
examinations from a norm-referenced approach to a standards-referenced approach
because of their prior experience related to the issue. The aim of this study is to
investigate teachers perceptions of the change in the grading system of the public
examination from a norm-referenced approach to a standards-referenced approach using,
as a framework, Hall et al.s (1977) Stages of Concern (SoC) Model. This model on
stages of concern is to be discussed more fully in Chapter 2. The perceived advantages
and the potential drawbacks of a standards-referenced approach are examined in this
study.
Such an analysis is significant because it highlights the perceptions of the teachers
who are adopting and must ultimately implement SRA. It illuminates potential problems
that may occur in the process of this change and the possible directions continuing
professional development of teachers should take.
1.5 Research Questions
Given the overall aim to examine teachers views on the adoption of SRA, three research
-
8/13/2019 Impact of Prior Experience (Sample)
16/97
9
questions are formulated namely:
(1) What are the stages of concern of teachers about the innovation of SRA?
(2) What factors influence the variation?
(3) What concerns appear prevalent as the hindrance to the change?
These questions are examined through a small scale survey with follow-up
interviews involving both language and non-language secondary school teachers. Details
of the methodology are given in Chapter 3.
1.6 The Structure of the Dissertation
This dissertation is divided into four further chapters. Chapter 1 introduces the study and
provides an overview of the background. Chapter 2 reviews literature on SRA and on
issues of implementation and adoption of change. Chapter 3 describes and justifies the
methodology adopted. Chapter 4 sets out the findings with regard to teachers views on
the change and the stages of concern. In conclusion, Chapter 5 discusses the findings in
the light of the literature in Chapter 2 and offers suggestions to stakeholders on the
adoption of new assessment practices.
-
8/13/2019 Impact of Prior Experience (Sample)
17/97
10
CHAPTER 2
Standards Referencing and Assessment Change:
A Literature Review
2.1 Introduction
The focus of this study, as noted in Chapter 1, is on the perceptions and reactions of
teachers to the introduction of standards-referenced assessment within the public
examination system in the Hong Kong secondary school sector. It is necessary, therefore,
to examine what is understood by standards-referenced assessment.
Standards-referenced assessment is an innovation in the mainstream of the Hong
Kong school context and its adoption raises questions about factors that enhance or limit
the introduction of change within the curriculum. This leads us to the curriculum change
literature. In conclusion, the chapter summarises the key points from the review of these
two areas that are of significance to the study.
2.2 Standards Referencing
Standards referencing is gradually adopted in recent decades. Its influence to the
evolution of assessment framework is important to educational change. The nature and
emergence of standards referencing are discussed as follows.
2.2.1 Nature
All assessment activities, including public examinations, are designed for some
reasons and set in particular situations and/or specific cohorts of students. The purpose
defines the design. Public examinations are important and are major activities in many
education systems in the world. They have a long history and were first established, it is
claimed, during the period of the Sui emperors (589-618) in the school entrance and civil
-
8/13/2019 Impact of Prior Experience (Sample)
18/97
11
service examinations in China (Miyazaki, 1976). At that time, public examinations were
instituted for recruiting government officials. The activity was a winnowing process and
the function of the public examination was to select elites in what was seen as a fair and
objective way so as to remove favoritism and employ the best in a meritocracy. Given that
selection was the prime purpose, the system sought to rank individuals by performance
with respect to the norm of the assessed group. Today, this would be called
norm-referenced assessment.
With a long history, norm referencing is well known for its use and application
and has been applied to the assessment of many subject areas. Norm referencing scores
represent a concise summary of students achievement relative to other students in the
cohort that can be used directly and immediately. When selection of elite is needed, this
traditional method offers a simple interpretation of scores in which the ranking can
facilitate the selection.
In context of language assessment, for example, Brown and Hudson (2002) define
a norm-referenced test as:
Any test that is primarily designed to disperse the performances of students in a
normal distribution based on their general abilities, or proficiencies, for purposes
of categorizing the students into levels or comparing students performances to the
performances of the others who formed the normative group. (p.2)
Being specialised in language teaching and applied linguistics and implicitly accepting
that the function of assessment was to rank, they elaborated that norm-referenced tests are
appropriate for assessing abstracted language ability traits representing a broader, more
differentiated content range (Millman & Greene, 1989, p.341). The broader the range the
more possible it is to discriminate high and low ranking students.
Nitko (2001, p.3) suggested a slightly different viewpoint. He described the use of
-
8/13/2019 Impact of Prior Experience (Sample)
19/97
12
a norm-referencing framework as interpreting students assessment performance by
comparing it to the performance of a well-defined group of other students who had also
taken the same assessment. The focus of his notion was to identify the relative strengths
and weaknesses of students in their cohort, rather than individuals achievements. Or it is
more like making a relative decision (Brown & Hudson, 2002, p.2) in the assessment
result.
Whenever assessment takes place, a referencing framework is needed (Nitko,
2001) to interpret students performance in assessments in general and in public
examinations in particular and, as noted, the form of referencing must match the purpose
of assessment. Brown and Hudson (2002) suggest four different types of focus for
classification: norm-referencing, criterion-referencing, domain-referencing and
objectives-referencing. Domain-referencing and objectives-referencing are, however,
variations of criterion-referencing and Brown and Hudson note that much confusion has
also been generated by the different ways that criterion-referenced has been used, as well
as by the alternative terms that have evolved for different types of tests closely related to
criterion-referenced tests (p.3). In the same way, Nitko (1980) distinguishes the many
varieties of criterion-referencing assessment by revealing various conceptualisations it
can take.
Sadler (2005) has conceded that the meaning of criterion is broad in scope.
However, an assessment criterion is a statement of the basis against which the assessment
is made. A standard is a form of criterion. Criterion referencing as originally conceived
was based within a measurement paradigm and stressed that criteria should be specified
only where appropriately reliable measurement was possible. This produced narrow
criteria. In a learning environment, broader criteria were adopted but this caused
confusion as to the meaning of the term criterion. Hence the notion of standards was
-
8/13/2019 Impact of Prior Experience (Sample)
20/97
13
developed as a more broadly based concept. This is discussed more fully in the next
section. At the same time as considering the general definition of standard, Sadler (1987)
discerned a major shift in orientation from criterion-referencing per se to
standards-referencing so as to realise the aspirations of criterion-referencing (Sadler,
2005).
Norm-referencing, criterion-referencing and standards-referencing are now widely
used in many public examination systems in the world, but in recent years there has been
a shift away from the first. Nitko (2001) has commented that norm-referencing is not
enough to fully interpret scores, noting that:
Norm-referencing provides information helpful for your relative interpretations
of scores, but frequently these are not enough. Scores that reflect relative
achievement such as rank order, for example, may be helpful in picking the best
readers, or in sectioning a class into better, good, and poor readers. However, to
plan appropriate instruction, eventually you need to know the kinds of reading
performances each student can do and the particular types of difficulties each
student is experiencing. (p.3)
He signified that interpretation of scores requires more information for effective selection
and ranking of students and defining students level of achievement. He, however, was
not the first to articulate such defects in norm referencing, but he extended Robert
Glasers thinking documented in his seminal paper in 1963. Glaser (1963) introduced the
phrase criterion-referenced measurement for evaluating students performance. Over
the years, criterion-referenced assessment and criterion-referencing have come to be
used to represent his ideas and these phrases have now become integral to the
nomenclature of assessment.
Glaser (1963) pointed out that norm-referenced assessment merely revealed the
-
8/13/2019 Impact of Prior Experience (Sample)
21/97
14
relative performance, but he sought to emphasise what information could be obtained
from a norm-referenced score commenting that:
Along such a continuum of attainment, a students score on a criterion-referenced
measure provides explicit information as to what the individual can or cannot do.
Criterion-referenced measures indicate the content of the behavioral repertory, and
the correspondence between what an individual does and the underlying
continuum of achievement. Measures which assess student achievement in terms
of a criterion standard thus provide information as to the degree of competence
attained by a particular student which is independent of reference to the
performance of others. (pp. 519-520)
He argued that criterion-referencing was the only referencing system for all assessments
and viewed norm-referencing as a variation of criterion-referencing in which criteria were
set in relation to the mean score.
Popham and Husek (1969) also enumerated the differences between
norm-referencing and criterion-referencing. They define a criterion-referenced test as that
used to ascertain an individuals status with respect to some criterion, i.e. performance
standard (p.2). Thus, a criterion-referenced framework is not simply a change of scoring
practice or a different interpretation of scores. It is qualitatively a different perspective.
Glaser and Nitko (1971) agreed with Popham and Husek (1969) that such a change could
not be made by simple inspection of a particular instrument but was linked to teaching.
They emphasised the domain of instructionally relevant tasks in which assessments take
place. In other words, these scholars inclined to the view that the domain of knowledge
regarding instructional objectives should also be considered in any particular assessment
framework.
At the same time, Garvin (1971) introduced the idea of subject-matter areas to
-
8/13/2019 Impact of Prior Experience (Sample)
22/97
15
the criterion-referenced framework that subject matters might affect the application of this
framework because performance criteria could not be set in some conditions. Popham
(1973) added his idea on writing instructional objectives for teachers that
There is, however, another dimension to objective writing, a dimension that
further aids the teacher in planning and evaluating his instruction. It involves
establishing performance standards, that is specifying prior to instruction the
minimal levels of pupil achievement. (p. 3)
Hambleton and Sireci (1997, p. 383) summarised the situation writing that students test
scores were often interpreted with respect to the performance of other students, rather
than with respect to their attainment of desired educational objectives in the basic skills
testing movement from the 1970s and 1980s. Teachers and students often desired
diagnostic information for improving performance in such a way as to link to mastery of
important outcomes of instruction (often called objectives). For this reason, criterion
referencing increasingly became important.
2.2.2 The emergence of standards-referencing
As noted, traditionally, assessment practices sought to comply with a simple,
direct goal of selection, a goal which stakeholders including teachers, parents and
students accepted with alacrity. However, in recent years, criticisms (Glaser, 1963;
Popham, 1978; Hambleton, 1996; Nitko, 2001) have emerged. Traditional assessment
practices have been seen a narrow and narrowing in terms of learning as well as providing
only limited information on what has been achieved. The term educational testing has
consequently been replaced by educational assessment (Hambleton, 1996, p.899) and
the term assessment has been broadly interpreted as to include any judgment (or
appraisal, or evaluation) of a students work or performance made by a teacher or other
competent person, whether for purposes of improvement or certification. (Sadler, 1987,
-
8/13/2019 Impact of Prior Experience (Sample)
23/97
16
p.191) The focus has shifted to the instructional objectives within teaching and learning.
There has been what has been called a paradigm shift from the dominance of
measurement for selection (Gipps, 1992) to assessment for learning and, where
assessment is of learning as in public examinations, to ensure the assessment supports
learning as far as possible.
A criterion-referenced framework, as indicated earlier, is argued to be able to
fulfill such aims for assessment and overcome the aforementioned issue. Scholars and
educators have been keen to try it out and many developed countries have changed and
adopted a criterion-referenced framework in their public examinations. For example, in
the National Curriculum for England and Wales formulated in 1988, assessment criteria
were set out for the illustration of students attainment targets. It has also been adopted in
Australia, New Zealand, South Africa and in North America.
Sainsbury and Sizmur (1998) studied the meaning of criterion-referencing in the
UK National Curriculum approach and pointed out that:
Criterion-referenced assessments, in an educational context, aim to give
information about valued educational outcomes. They aim to tell us how well the
pupils have learned what they have been taught. They are typically about
cognitive outcomes, with understanding, knowledge and skill as central elements.
This understanding, knowledge and skill is often described in fairly abstract and
general terms. (p.3)
Notably, they were of the opinion that such an educational construct was abstract, general
and complex. In general, drawing criterion-referenced inferences from test scores means
claiming links from one extreme to another. (Sainsbury & Sizmur, 1998, p.3) This
educational construct, the criterion-referenced framework, failed in practice as Sainsbury
and Sizmur (1998) felt that there were difficulties of teacher workload, expressed in
-
8/13/2019 Impact of Prior Experience (Sample)
24/97
17
various ways over the years and culminating in a full-scale boycott in 1993. The
difficulties were identified as related to the complexity of the original, statement of
attainment based, assessment system (Dearing, 1993).
Earlier, in 1987, Royce Sadler highlighted the difficulty of adopting
criterion-referencing in public assessment and raised some concerns about its general
applicability, though many methods had been proposed for dealing with scores and grades
in large scale examinations (Berk, 1986). He clarified the terms criterion and
standard. Criterion-referenced assessment is deliberately constructed to yield
measurements that are directly interpretable in terms of what was specified performance
standards. (Glaser & Nitko, 1971, p.653) Glass (1978), however, interpreted the term
criterion in a different way and shifted its meaning to an arbitrary, minimal level of
competence.
Sadler (1987, p.195) suggested that:
Originally, a criterion referred to a characteristic or dimension of performance,
but since about the time of Glaser (1963), a criterion has been used to mean the
particular score that is taken to designate competence or mastery. The criterion in
criterion-referenced testing is invariably a numerical cut-off. According to the
above definitions, therefore, it would be better called a standard.
He (1987) proposed an influential assessment framework which he termed
standards-referencing for the certification of students complied with both quality control
and the due recognition of achievements. He also contended that practical considerations
should be in the context when dealing with standards (pp.196-197).
According to Sadler (1987), in order to forge strong links between grades and
actual achievement, observation of performance is of paramount important in
standards-referenced assessment. The original National Curriculum for England and
-
8/13/2019 Impact of Prior Experience (Sample)
25/97
18
Wales, which adopted the criterion-referencing, was then improved by introducing a
better judgment mechanism, after the original scheme led to a boycott in schools.
Sainsbury and Sizmur (1998) wrote that the level descriptions which were proposed in the
improvement in 1995 replaced the simple statements of attainment; the mode of use of the
level description was to form a best fit judgment. Although the current National
Curriculum for England and Wales is claimed to be criterion-referencing framework, the
change in practice after the instance of boycott reveals the shadowy side of a
standards-referencing framework defined by Sadler (1987). Others, however, have
described it as no more than a weak version of criterion-referenced assessment (Gipps,
1994, p.95).
In addition, Gipps (1994, p.85) alleged that:
A major problem for criterion-referenced assessment is aggregation, i.e. the
collapsing of the detailed performance profile for each individual into a single
report figure, or grade. Aggregating detailed assessment information into a crude
single grade compromises the information offered by the assessment. A final
summative aggregated grade does not help the pupil or employer: it obscures more
than it clarifies.
She raised queries about the validity and reliability of criterion-referenced assessment.
She felt that strict criterion-referenced assessment was an unmanageable and undesirable
educational assessment framework (p.96). She also commented that weak versions of
criterion-referenced assessment or standards-referenced assessment defined by Sadler
(1987), that were being pursued in the USA and in Australia, had huge problems in terms
of comparability across assessments.
Standards-referenced assessment thus has many potential benefits over a norm
referenced system but as a system for public examinations is noted to have conceptual
-
8/13/2019 Impact of Prior Experience (Sample)
26/97
19
and practical difficulties. It is perhaps not surprising that adoption creates issues of
curriculum change which are now discussed in the next section.
2.3 Curriculum Change
In this section, we review literature on curriculum change and on understandings of the
challenges involved in the process.
2.3.1 Conceptions of curriculum change innovation or adaptation?
Assessment change is a part of curriculum change. Change can be referred to an
update of existing elements or replacement by a new entry. Fullan (2001) avowed that
curriculum change should embrace everything new to stakeholders including knowledge,
teaching strategies, and beliefs. Stenhouse (1975) concurred arguing that knowledge and
teaching strategies were the key elements in curriculum change. He did not mention
assessment per se, but it may be viewed as a part of teaching and learning. He (p.169) was
concerned about the views and identity of teachers during change, and suggested updates
of these elements that were more easily accommodated than introducing new but
disturbing elements to teachers.
In the context of this study, this highlights a potential problem as standards
referencing marks a radical departure in practice rather than an evolution. Cuban (1979)
concurred and suggested that curriculum change should try to improve the existing
elements and perform incremental change rather than fundamental change (Cuban, 1997).
The implementation is also a noteworthy consideration to curriculum change.
On the one hand, curriculum change is needed when the current practice has some
recognisable weaknesses and does not fit the present environment. Curriculum change
provides a new setting or platform for innovation to grow. On the other hand, curriculum
change is introduced to bring about social change and society perceives the curriculum
-
8/13/2019 Impact of Prior Experience (Sample)
27/97
20
change as being necessary. However, Morris and Scott (2003) have discussed some
instances of curriculum change in Hong Kong and indicated that policy makers have
proposed curriculum changes for economic, social, and political reasons, and that
proposals often become rhetorical slogans rather than agendas for change when the
primary reason is not related to pedagogical perspective.
2.3.2 Curriculum change strategies
Assessment change in the public examination context is often led by government
or quasi-government organisations. Curriculum change as an educational process has
been studied for well over 40 years. Scholars like Schn, Havelock, Hargreaves, Bennis,
Benne and Chin have developed influential models on change strategies that may be
relevant to assessment change issues being considered in this study. Their models have
one thing in common. They all appreciate the top-down strategy in different parlance.
Schn (1973), Havelock (1973), Hargreaves (1994), and Bennis, Benne and Chin (1985)
respectively use centre periphery model, Research, Development and Diffusion model,
cultural interruption model and empirical-rational model to describe a similar idea that
leadership is centralised and change is disseminated to the periphery or inferiors by
reasoning.
Nevertheless, those scholars emphasise different approaches. Schns ideas (1973)
on curriculum change focus on diffusion of innovation. He comments that
The loss of the stable state means that our society and all of its institutions are in
continuous processes of transformation. We cannot expect new stable states that
will endure for our own lifetimes. (p. 28)
He asserts that a continuous process of transformation will be taken place and the stable
state or centre will then be lost. He therefore develops proliferation of centres and
shifting centres models to echo the phenomenon of the transformation.
-
8/13/2019 Impact of Prior Experience (Sample)
28/97
21
Havelock (1969) was originally concerned about the dissemination of knowledge
emphasising the characteristics of the change agent and explaining how change occurs.
He incorporated the idea of diffusion of innovation into the notion of the dissemination of
knowledge, and develops the concept of linkage (Havelock, 1973), social interaction
and problem solving. In the social interaction model, a five-stage cycle of awareness,
interest, evaluation, trial and adoption is codified to identify the strategy of change. The
concept of linkage focuses on the users of innovation as problem solvers. He argued that
problem solvers, as opposed to observers, will utterly and proactively involve themselves
in developing the innovation concerned.
Hargreaves (1994) focuses the strategy on changing teachers in a postmodern age.
He postulates that teachers have to adapt to a new school reality when a new culture
arises. Generally, it is named as cultural interruption or classified as a notion of
interrupting the existing culture. He also develops structural reinforcement model to
describe a dictatorial strategy. Sometimes a structural redefinition is needed to tackle the
alienation and marginalisation of teachers (Hargreaves, 1994, p.242). Bennis et al. (1985),
however, developed an empirical-rational strategy to illustrate the top-down approach
where reasons and cohorts interest are encompassed.
2.3.3 Teachers as change agents
Teachers, in the past, were expected to replicate predefined instructional tasks in
school. However, in the recent decades, education reforms are evolved and there is a shift
in teachers role. Many curriculum scholars (Cuban, 1988; Fullan, 2001; Hall & Hord,
1987; Oliver, 1977; Rudduck, 1991; Stenhouse, 1979) concern teachers as the major
players in education reform. They concur that teacher is the key role in curriculum change
despite the aforementioned top-down curriculum change strategies, and describe teachers
as change agents in a sense that facilitate the process of change.
-
8/13/2019 Impact of Prior Experience (Sample)
29/97
22
Cuban (1988) emphasised the vision of change, positing that teachers should share
the vision so as to lead to a success in curriculum change. Stenhouse (1979) focused on
the way from the teachers point of view and stated the axiom: No curriculum
development without teacher development. He confirmed the importance of teaching
involvement and the development should make teachers feel a sense of ownership of the
curriculum. Oliver (1977) also agreed that successful curriculum change should involve
teacher participation. Hall and Hord (1987) stressed that teachers not only have the
ownership and participate in the change, but also are able to moderate the change or be
facilitators of the change. Rudduck (1991), however, argued that ownership is fragile and
is difficult to be measured, though he articulated the importance of teacher beliefs on
changes in practice.
In brief, the involvement of teachers is highly regarded as an essential trait in
education reform. Their concerns about and perceptions of and their actions and reactions
to a curriculum change suggested with standards-referenced assessment are thus crucial
and are discussed in the next section.
2.4 Teachers perception and Concerns Based Adoption Model
Understanding teachers concerns and their level of use is vital in any issue of
implementation as is considered here in the case of assessment framework. In regard to
standards referencing, there seems relatively little research related to teachers perception
of SRA. However, in 2003, Ayres, Beechey and McCormick investigated teachers
perceptions of the impact of the New NSW Higher School Certificate in New South
Wales, Australia including the change of assessment framework to using standards
referencing. They designed a three stage longitudinal study within a theoretical
framework derived from established theories of motivation and occupational stress. They
-
8/13/2019 Impact of Prior Experience (Sample)
30/97
23
found that teachers perception helped them analyse the issues of curriculum change
including speed of implementation, workload and teaching strategies.
Over the decades, there have been various major, prominent approaches to
researching teachers concerns and reactions namely Action Research, Concerns Based
Adoption Model (CBAM), Curriculum Alignment, and Comprehensive School Reform
Programs (Marsh & Willis, 2003, p.249). The CBAM is a widely applied theory and
methodology for studying the process of implementing education change by teachers
(Anderson, 1997) and is a focus in this study.
Table 2.1 Typical expressions of concern about an innovation
Stages of Concern
6 Refocusing
5 CollaborationImpact
4 Consequence
Task 3 Management
2 PersonalSelf
1 Informational
Unconcerned 0 Awareness
The concept of CBAM was germinated in the work of Frances Fuller in 1969.
Meanwhile, others in response to the innovation focus approach to educational change
evolved (George, Hall & Stiegelbauer, 2006). CBAM is composed of Stages of Concern,
Levels of Use, and Innovation Configuration (Hall & Hord, 1987). In particular, seven
Stages of Concern about an innovation categorised in three groups are identified and a
corresponding 35-item questionnaire, namely Stages of Concern Questionnaire (SoCQ),
is developed by Hall, George, and Rutherford (1977). SoCQ can be used as a diagnostic
-
8/13/2019 Impact of Prior Experience (Sample)
31/97
24
tool to assess teachers perception in relation to the adoption of educational change. The
seven Stages of Concern about an innovation and their three corresponding groups are
shown in Table 2.1.
In some reviews of CBAM (Anderson, 1997; Cheung, Hattie, & Ng, 2001),
scholars agreed that SoCQ, as one diagnostic dimension of CBAM, is a possible tool for
measuring teacher concerns about a change in curriculum or instruction. However, they
reported some critical notes and even suggested modifications to SoCQ to cater for the
need in their research. Anderson (1997, p.334) noted that CBAM theory idealises the
Stages of Concern as a developmental progression in which teachers implementing a
change have concerns of varying intensity across all seven stages at different points in the
change process. He concluded that the resolution of early stage concerns does not
necessarily lead to the arousal of later stage concerns about the impact of those practices
on students.
Moreover, Cheung et al.(2001) found an imperfection in SoCQ from a statistical
point of view. They modified the original SoCQ and constructed a 24-item questionnaire
to measure teacher concerns clustered in five sequential stages (Cheung, 2001). George et
al.(2006) also comment the weakness of SoCQ noting that
Development of higher-level concerns cannot simply be engineered by an outside
agent. Holding concerns and changing concerns is a dynamic of the individual.
Providing affective experiences and cognitive resources in a timely manner
certainly can supply the grist for the emergence and resolution of concerns,
thereby facilitating the development of higher-level concerns. There is no
guarantee, however, that the emergence of higher-stage concerns will follow the
reduction of lower-stage concerns.
To sum up, there are criticisms to the use of CBAM yet many studies have
-
8/13/2019 Impact of Prior Experience (Sample)
32/97
25
demonstrated how effective CBAM can be to recognise the inevitable presence of
concerns within individuals and to extend a helping hand to assist in copying with and
resolving those concerns (George et al., 2006).
2.5 Conclusion
This chapter enumerates the reasons and the potential benefits, found by academics, of
the adoption of standards-referenced assessment in many education systems including
Hong Kongs. To implement or to execute the innovation in a curriculum change, there
are a number of possible top-down curriculum change strategies that could be applied,
and teachers play as change agents in the process, as magnified in different perspectives.
Concerns of teachers can then be measured through the CBAM to codify the teachers
perception in a system way that corresponding, appropriate policies can be made or
revamped to avoid any hindrances to the change, which will be discussed in the chapters
followed.
-
8/13/2019 Impact of Prior Experience (Sample)
33/97
26
CHAPTER 3: METHODOLOGY
3.1 Introduction
This chapter discusses the methodology used in the evaluation of teachers attitudes to the
introduction of new assessment approaches. As noted in Chapter 1, concern is an
important dimension when working with individuals involved in a change process
(George et al., 2006, p. 1). In the change process involved with a shift in the public
examinations from a norm-referenced approach to a standards-referenced approach,
teachers are the practitioners who play an important role to the success of the reform.
Teachers with or without prior experience may have various kinds and degrees of concern
about the adoption and implementation of standards-referenced assessment. The aim of
this research, as noted in Chapter 1, was to investigate the impact of prior experience on
the concerns of secondary school teachers about the innovation of SRA. Are there
different stages of concern regarding SRA between those to whom, this is new as opposed
to those who have been working with it for a few years? The three key research questions
are:
(1) What are the stages of concern of teachers about the innovation of SRA?
(2) What factors influence variation in concern?
(3) What concerns appear prevalent in hindering change?
To carry out a logical and reliable study, the Concerns-Based Adoption Model
(CBAM) was adopted as a framework for collecting and analyzing teacher concerns
through questionnaire. This chapter describes and explains how the CBAM was
operationalised as the framework in this case. In particular, it discusses issues in the use
of interview as a data collection tool and how the data collected was analysed.
-
8/13/2019 Impact of Prior Experience (Sample)
34/97
27
3.2 CBAM Approach
The CBAM is a diagnostic model for analyzing an education change in schools which
primarily focuses on the concerns of teachers (Hall, Wallace, and Dossett, 1973). It
comprises three elements: Stages of Concern (SoC), Levels of Use (LoU), and Innovation
Configuration (IC). SoC is used to evaluate the concerns of teachers when they are
going to implement an innovation, while LoU is used to find out what teachers have done
in the implementation. IC is used to help understand the difference in ways that teachers
modify and implement innovations. These three components, as three different
dimensions, reflect a diagnosis of an education change and a means to research change.
Table 3.1 Typical expressions of concern about an innovation
Stages of Concern Expressions of Concern
6 I have some ideas about something that would work even better.
5I would like to coordinate my effort with others, to maximise
the innovations effect.Impact
4 How is my use affecting my students?
Task 3 I seem to be spending all my time getting materials ready.
2 How will using it affect me?Self
1 I would like to know more about it.
Unconcerned 0 I am not concerned about it.
This research focused on Stages of Concern (SoC), one of the diagnostic
dimensions of CBAM, to understand teacher concerns. Since the 1970s, SoC has been
used in many studies. The concepts and knowledge that it reflects are well known in the
literature. It acts as a common language for researchers, policymakers, and users to talk
about the implementation of innovations (Snyder, Bolin, and Zumwalt, 1992). Particularly,
having a primarily descriptive and predictive approach, SoC is suitable for research on the
innovations about understanding curriculum implementation (Anderson, 1997). Typical
-
8/13/2019 Impact of Prior Experience (Sample)
35/97
28
expressions of concern about an innovation, suggested by George et al. (2006, p. 4), are
shown in Table 3.1.
The CBAM assumes that teachers undergo similar concerns in a change process in
such a way that teachers, starting at an unconcerned stage, experience a sequence of
changes in concerns about self, task, and impact. These teachers move their concerns
from personal issues to teaching, and then students themselves. This unidirectional
progression of teacher concerns about a change may be possible, but not necessary at all
time (Leithwood and Montgomery, 1982). Some researchers (e.g. Bailey and Palsha, 1992;
Cheung et al., 2001), on the other hand, point out the universal, fixed, discrete stages of
concern. They allege that SoC is not always appropriate to illustrate the teacher
concerns without the consideration of the authenticity in different scenarios. It leads to a
question of potential validity. For this reason, these researchers have refined and revised
SoC for its use in their specific situations.
No matter SoC is modified or not, it has been widely used in many seminal studies
and this demonstrates an effective methodology to collect and describe teachers concerns
clearly. The model, however, may suggest directions that can be pursued to find
interpretation. But it does not directly lead to the explanation to the findings. To put it
simply, the model cannot directly explain why teachers have a certain stage of concern,
and yield the reasons behind this. Hence, other research tools like interview adopted in
this study are required for this purpose.
3.3 Research Design
A naturalistic, qualitative case study design was adopted in this research. Such a design
has several hallmarks. It can provide rich and vivid descriptions of the events relevant
to the case and blends the descriptions of events with analysis of them (Hitchcock and
-
8/13/2019 Impact of Prior Experience (Sample)
36/97
29
Hughes, 1995), as presented in this study. The conceptual framework was developed
and shown in Figure 3.1. It illustrated that data are collected from teachers with or
without prior experience of SRA and classified into teachers perception of change and
perceived hindrances regarding the assessment change or the innovation of SRA. It also
manifested that reasons were deduced and analysed for further understanding with the
change and developing implication for practice.
Figure 3.1 Conceptual framework
There were two phases in data collection. In the first phrase, the CBAM Stages
of Concern Questionnaire (SoCQ) were used as a quantitative method to assess teacher
concerns. In the second phrase, with the preliminary results of the teacher concerns
about the innovation of SRA, five interviews were conducted to seek interpretation of
teacher concerns, and to strengthen and delineate a wider, in-depth scope of research
findings.
Perceived
hindrances
Perception
of change
Teachers
With prior experience
of SRA
Without prior
experience of SRA
Implications
for practice
Reasons
Reasons
-
8/13/2019 Impact of Prior Experience (Sample)
37/97
30
In the first phase, the SoCQ (described in the next section) was distributed to
teachers. The questionnaires were firstly classified into two cohorts: language and
non-language teachers and then by teaching experience. Data analysis on the SoCQ was
based on the SoCQ Quick Scoring Device of George et al. (2006, p. 86) and used to
calculate percentile scores for plotting Stages of Concern profiles. From this, variation
of stages of concern between different cohorts could be assessed, where the researcher
could gain a great deal of insight, not only into the types of concern that were most
intense and least intense, but also into the affective stance that the respondents were
taking towards the standards-referenced assessment (George et al., 2006, p.37).
After the completion of this survey, five interviews were conducted in the second
phase. The five interviewees were secondary school teachers, three of whom were
language teachers. Before starting the interviews, they had to complete a SoCQ to
delineate their individual Stages of Concern profile.
The researcher arranged and met five teachers for the data collection of the
questionnaires and interviews in two months. He mainly employed the interview guide
approach suggested by Patton (1980, p. 206) at first. Nine semi-structured questions, as
shown below, were designed for the teacher interviews.
1. What are your greatest concerns about the standards-referencedassessment?
2. What type of support do you think you need from EMB/HKEAA in orderto be successful?
3. What type of support do you think you need from school/mentor in orderto be successful?
4. What type of support did you receive that was most helpful in assessmentreform?
5. How does your experience compare to your expectations prior to teaching?6. Do you see the standards referencing adopted in the examination as a loss
or an improvement? Why?
-
8/13/2019 Impact of Prior Experience (Sample)
38/97
31
7. What keep you away from learning more about standards-referencedassessment?
8. What would help you most to continue to learn about the use ofstandards-referenced assessment?
9. Is there anything else you would like to say about the standards-referencedassessment?
The questions covered the related topics and issues which were constructed in advance,
and the researcher decided the sequence and working of the questions in the course of the
interviews, so as to collect interviewees responses and opinions accordingly.
When the data collection of the SoCQ was completed, Stages of Concern profiles
for different cohorts and individual interviewees were studied and collated with the
analysis of the qualitative data collected from the interviews. Through the quantitative
and qualitative analysis, it was hoped that the findings could apply theories to fit the
situation, following advice suggested by Lincoln and Guba (1985) to maintain the
credibility of the research.
3.4 The Operationalisation of Stages of Concern Questionnaire
This section describes how the data of the questionnaires was collected in the study. It
includes the design of the questionnaire, the selection of participants, the administration
procedures, and the presentation of results.
3.4.1 The Stages of Concern Questionnaire
The researcher adopted the Stages of Concern Questionnaire (SoCQ) developed
by Hall et al. (1977) but with the changes for this particular study as set out below. The
questionnaire (see Appendix A) was organised in two parts: the introductory page and two
pages of items. The introductory page stated the purpose of the questionnaire and
explained how to complete the items. A small number of questions were added to collect
the personal information of respondents, these being necessary to uncover the teaching
-
8/13/2019 Impact of Prior Experience (Sample)
39/97
32
experience of respondents which arguably was an influential factor. This facilitated data
analysis and the impact of prior experience on the standards-referenced assessment.
Respondents noted their teaching experience in English/Chinese language teaching and/or
other subjects. Language teachers were defined as teachers who taught English language
or Chinese language class in Form Four or above, because this was where
standards-referenced assessment had been introduced. Five option boxes with different
ranges of years of teaching experience were provided: nil, 0-2 years, 2-4 years, 4-6
years, and 6 years above. Six years of teaching experience was decided as a
maximum as teachers with six years of teaching experience were regarded as experienced
teachers because the secondary teachers who taught Form Four class or above were in the
Graduate Master/Mistress level, and in practice, they generally have at least five years of
teaching experience and have undergone professional training to get promotion to the
Senior Graduate Master/Mistress level. This meant they gained sufficient teaching
experience as an experienced teacher to take up more important school duties such as
panel chairperson and discipline master/mistress. In addition, optional name and email
were requested in the questionnaire.
On the next two pages, 35 items were included. The order and the wording of
these items were preserved from the original SoCQ (Hall et al., 1977) to avoid the risks
pertaining to reliability and validity (George et al., 2006, p.25). However, for better
understanding as suggested by George et al. (2006, p.25), the original word in the items,
such as innovation, was replaced by the more specific reference to standards-referenced
assessment, which was a familiar term to and a closer bonding with the respondents.
For example, Question 1 in the SoCQ was
I am concerned about students attitudes toward the standards-referenced
assessment.
-
8/13/2019 Impact of Prior Experience (Sample)
40/97
33
with the replacement of innovation by standards-referenced assessment.
Respondents circled each item on a 0-7 Likert scale according to how true the
item seemed to them at the moment they completed the questionnaires. The 0 as the
smallest value on the scale represented that the item was completely irrelevant to the
respondents, while the 7, as the extreme value on the scale, represented that the item
was strongly a truism to the respondent.
In the questionnaire, there were no open-ended questions. The reasons were
twofold. First, there were problems of data handling (Cohen, Manion and Morrison, 2000,
p.255). On the one hand, while data collected from open-ended questions are word-based,
respondents might write down their expressions in different ways so that it would be
difficult for the researcher to make comparisons between respondents. On the other hand,
it was questionable how to aggregate such qualitative data into somehow discrete,
confined quantitative data as the results generated from the existing 35 items. Second,
with a better interpretation of the questions to which respondents answered unclearly or
incompletely, the researcher should carry out follow-up activities such as sending
follow-up letters to maximise the responses from hundreds of respondents (Cohen et al.,
2000, p.263). However, this might not be possible to accomplish in a timely fashion.
3.4.2 The selection of respondents
This research focused on the assessment change in the new language curricula in
Form Four and Form Five levels. The target respondents were teachers who had
experience on teaching senior classes in school. Questionnaires were distributed to
twenty-two teachers in different secondary schools who completed them and attempted to
redistribute the questionnaires equally to languages and non-language teachers in their
schools. The initial twenty-two teachers were chosen in the study because they were
working closely with the researcher and the researcher could easily approach them. This
-
8/13/2019 Impact of Prior Experience (Sample)
41/97
34
represented a convenience sample, which is a possible sampling strategy for a case study
(Cohen et al., 2000, p.103) like this research. At the same time, however, the schools
were of different types including government school, aided school, caput school, and
private school. Beside themselves, they asked close colleagues to fill in the questionnaires.
It was very important that the respondents were encouraged, not coerced into completing
a questionnaire if reliable and valid responses were to be obtained. With the liaison
between the researcher and the twenty-two teachers, respondents were given appropriate
care and incentives to accomplish the questionnaires so as to maximise the response rate
following advice in Cohen et al.(2000, p.263). In total, about 450 questionnaires were
distributed through the twenty-two teachers in the entire exercise.
3.4.3 The administration procedures
The twenty-two participating teachers were provided with a detailed briefing as to
the purpose of the exercise and the researcher asked for their help on a voluntary base.
The researcher tried his best to explain the helpers the nature of the research so that they
could distribute the questionnaires to the right persons. He also explained the helpers their
involvement in this exercise in order to avoid any misunderstanding regarding ethical
issues such as the sensitivity of the questionnaire and data privacy (Cohen et al., 2000, pp.
245-246). Moreover, he closely communicated with the teacher helpers by email and
phone to answer their queries on matters arising for minimal misinterpretation.
After respondents completed the questionnaires, some teacher helpers put the
questionnaires into a return envelope provided by the researcher and sent it out by postal
mail. The others directly passed the questionnaires to the researcher by hand. Of 450
questionnaire copies, 259 questionnaires were completed and returned in June 2006.
The return rate was satisfactory as more than 50 per cent response (58%) to the
questionnaire was received (Cohen et al., 2000, p. 246).
-
8/13/2019 Impact of Prior Experience (Sample)
42/97
35
The SoCQ was well developed for decades by scholars. Its validity and
reliability relied on such a way that the 35 items were inter-correlated, meaning to the
equal essence of each item. In this research, missing data in SoCQ was defined as no
response to any items including the first one on the introductory page or multiple
responses to any one of the 35 items. Before the data entry process, the researcher
decided to void any questionnaire with missing data. It was because out of 35 items in a
SoCQ, no matter how many correct items were there, this kind of questionnaire had
outlived its usefulness and would affect the validity and reliability of the data analysis
(Cohen et al., 2000, pp. 245-246). When the data in the questionnaires were being entered
into a database program, it was found that some items in a small portion of the collected
questionnaires were completed by either double circles or no circles. After the data
entry process, six questionnaires were found to be voided. Hence, 253 valid
questionnaires were compiled in this study and 135 of them were filled in by language
teachers, who taught Chinese language or English language in Form Four class or above
in their schools.
The database program based on a database software package for data processing
was developed by the researcher. It was used to manage the item data of the
questionnaires and to administer the data analysis. Most of the data were values of the
35 items in the SoCQ ranged from 0 to 7. The program was designed with check sum,
type check, and range check to minimise data input errors. When data were input into
the program, the program could automatically and instantly detect typing errors and it did
not accept entries with missing data or incorrect data such as letters, symbols, and
numbers 8 and 9, by prompting the user to enter the data again.
Besides, a spreadsheet file consisted of preset formulae and macros which was
developed by George et al.(2006) was used for data analysis. Based on the SoCQ
-
8/13/2019 Impact of Prior Experience (Sample)
43/97
36
Quick Scoring Device (George et al.,2006, p. 86), the worksheets in the file calculated
raw score totals and percentile scores. Subsequently, those percentile scores were
compiled for plotting Stages of Concern profiles. The Stages of Concern profiles for
different cohorts were plotted accordingly and were represented by the mean percentile
scores of the individuals in a group (George et al., 2006, p.34). The cohorts included
language teachers, non-language teachers, experienced language teachers, inexperienced
language teachers, experienced non-language teachers, inexperienced non-language
teachers, and the whole group. The profiles were used as a tool to generate an overview
for diagnosing the factors involved in the variation of teacher concerns.
3.4.4 The presentation of results
The data collected from the SoCQ was interpreted and translated into meaningful
Stages of Concern profiles by the spreadsheet file as mentioned in the previous section.
Figure 3.2 An example of Stages of Concern profile
0 1 2 3 4 5 6
0
10
20
30
40
50
60
70
80
90
100
Awaren
ess
Inform
ation
al
Pers
onal
Man
agem
ent
Cons
eque
nce
Collabo
ration
Refocu
sing
Stages of Concern
RelativeIntensity
Figure 3.2 is an example of Stages of Concern profile showing the relative intensity of the
subject in each of the stages of concern. A Stages of Concern profile was a
-
8/13/2019 Impact of Prior Experience (Sample)
44/97
37
two-dimensional graph with Relative Intensity ranging from 0 to 100 against the Stages
of Concern: Awareness (0), Informational (1), Personal (2), Management (3),
Consequence (4), Collaboration (5), and Refocusing (6).
3.5 Follow-up Interviews
This section describes how the data of the interviews was collected in the study. It
composes of the purposes of the interviews, the formats of the interview questions, the
selection of interviewees, the administration procedures, and the presentation of results.
3.5.1 Purposes of interview
Interview is a powerful tool in academic research. It is not simply a unidirectional
way of collecting data as questionnaire. It enables an interaction between interviewers
and interviewees to discuss their interpretations of the subjects they concern. As Cohen et
al. (2000, p.268) have indicated that interview not only serves as a means of gathering
information having direct bearing on the research objectives. Interview can be used to
suggest hypotheses and in conjunction with other methods in a research undertaking.
In the exercise of interviewing teachers in this research, the interview guide
approach was used. As Patton has said (1980, p.206), the interview guide approach
represents a systematic way to collect comprehensive data for each interviewee in which
logical gaps in data can be anticipated and closed. This exercise purported to sample
teachers views so that the researcher could triangulate it with the result of questionnaire,
and seek to develop hypotheses for answering the research questions.
3.5.2 Format of interview questions
The research objectives were translated into semi-structured, open-ended
questions for interviewing the five teachers. Most of the questions were asking for the
interviewees personal experiences and feelings. For example, Question 7 is,
-
8/13/2019 Impact of Prior Experience (Sample)
45/97
38
What keep you away from learning more about standards-referenced
assessment?
Interviewees could give the answers of different aspects. Ostensibly the reasons that kept
interviewees away from learning more about standards-referenced assessment might be
something related to their personal professional development or obstacles they faced in
school in terms of students or other issues. This open-ended question allowed
interviewees a lot of flexibility to express their views. This kind of question can help the
research learn about interviewees understanding in greater details and truly assess what
they really believe. It may result in unexpected or unanticipated answers which may
suggest unthought-of hypotheses (Cohen et al., 2000, p.275).
The sequence of the interview questions was arranged in a way that easier and less
threatening, non-controversial questions are addressed earlier in the interview for putting
respondents at their ease (Patton, 1980, pp.210-211). That was why the first four
questions were what-type questions, the easiest one as warm-up questions, while
how-type and why-type questions were put forth later.
The first interview was planned to have dual functions. Besides collecting data of
the interviewee, it was treated as a trial run to benefit the rest of the interviews. This trial
run purported to fine-tune the original set of questions and the way that the interviewer
asked questions and managed the interviews, so that the interview questions could
adequately reflect what the research objectives were expected to be done.
3.5.3 Characteristics of interviewees
Similar to the said operationalisation of Stages of Concern Questionnaire, the
focus of the interviews was on the assessment change in the new language curricula in
Form Four and Form Five levels. The criteria to select target respondents were the same.
The interviewees should have experiences on teaching senior classes in school.
-
8/13/2019 Impact of Prior Experience (Sample)
46/97
39
As a result, it was planned to select and invite five acquaintances of the researcher
to participate in the interview part of this study to collect different views on the
innovation of standards-referenced assessment. They were invited not because it was a
convenient or casual selection, but because they basically fulfilled the preset criteria.
However the researcher had confidence to conduct the interviews with them in a way that
an appropriate atmosphere could be established for them being situated for good
interaction and communication (Cohen et al., 2000, p279). The five interviewees had also
participated in helping the distribution and collection of the SoCQ. So, they had some
basic knowledge about the research as well as the interviews beforehand. It facilitated the
conduct of the interviews indeed.
Table 3.1 Teaching backgrounds of interviewees
Interview Name Subjects taught Teaching experience
Senior form: Geography1 Rachel
Junior form: English language12 years
Senior form: English language2 Karen
Junior form: English language8 years
Senior form: English language3 Ida
Junior form: Chinese language5 years
Senior form: Geography, computer4 Lee
Junior form: Geography, computer25 years
Senior form: Computer5 Alfred
Junior form: Biology, computer20 years
The five interviewees were three females and two males ranging in age from 28 to
47 years old and with teaching experience from 5 years to 25 years. Three of them were
-
8/13/2019 Impact of Prior Experience (Sample)
47/97
40
specialised in teaching language subjects. They were secondary school teachers from
different teaching backgrounds and schools. Their schools were government-subsidised
except that one of them, Alfreds school, was a private one. In this study, anonymous
names were used to denote their identity as shown in Table 3.1.
3.5.4 The administration procedures
The whole interview exercise completed in six weeks. Since the five interviewees
were Cantonese speaking, for the sake of better interaction and communication, all the
interviews were conducted in Cantonese. During the interviews, questions were asked
in similar wordings, but with supplementary elaboration to interviewees when needed to
understanding of the questions in their context. The interviewer asked them some extra
open-ended questions to cover some important and salient topics that were inadvertently
omitted (Patton, 1980, p.206) and this offered opportunities for the interviewees to
provide in-depth qualitative feedback.
The first interview served as a trial run to sharp the interview questions to suit the
research objectives for the subsequent interviews. First, as Rachel was available, the
researcher carried out the first interview with her and spent much time to accomplish this
protracted interview. Having received the responses and suggestions by Rachel, some
questions were modified or omitted in order to collect more specified, befitting data in
outline form from the interviewees.
Nine preset, semi-structure questions were used in the five interviews. All
interviewees were well informed the purpose of this study beforehand. They focused on
the theme of the study to respond questions during the interviews. The researcher
transcribed their verbal responses as well as non-verbal responses including their factual
expressions and tones emanated through the interviews in such a way that richer data
could be collected. The researcher also provided feedback to the interviews to confirm
-
8/13/2019 Impact of Prior Experience (Sample)
48/97
41
the validity of the transcription.
Moreover, the interviews were not merely a data collection exercise. All
interviewees were acquaintances of the researcher and the interviews were carried out
during coffee breaks, lunches, and dinners. The exercise was deemed to be social
encounters and the transcription was not simply a record of data. Other valuable
information emanated from non-verbal communication was recorded as well. Serving
interviews as social encounters helped avoid massive data loss, distortion and the
reduction of complexity (Cohen et al., 2003, p.281).
Next, the transcribed data were counterchecked by interviewees to minimise the
transcription errors and misinterpretation by the interviewer. Because of these, the
validity of this study could be enhanced in different ways. Mainly, the collected data
could accurately describe the situation and internal validity could be preserved.
3.5.5 Presentation of results
The interviewer tried to write down all the important verbal responses from the
interviewees including what was being said to answer the interview questions. Moreover,
their tone of voice to the questions was interpreted and transcribed by the interviewer.
Once the data had been transcribed from the interviews, the researcher employed
some tactics suggested by Miles and Huberman (1994) to generate meaning from the data.
Among the data, he counted the frequencies of occurrence of ideas, classified them into
categories, identified relations between variables, built a logical chain of evidence, and
made conceptual coherence to explain the situation.
-
8/13/2019 Impact of Prior Experience (Sample)
49/97
42
CHAPTER 4: FINDINGS
4.1 Introduction
This Chapter sets out the findings from the questionnaire survey and the follow-up
interviews. By using the CBAM approach as elucidated in Chapter 3, the stages of
concern of teachers about the innovation of Standards-Refere