The assessment of bilingual pupils: observations from recent Welsh experiences

Dylan V. Jones Department of Education

University of Wales Aberystwyth, Ceredigion, Wales E-mail: dqj@aber.ac.uk

Paper presented at the British Educational Research Association Annual Conference

(September 11-14 1997: University of York)

Abstract

The first part of this paper provides a brief description of the current extent and nature of bilingual education in Wales. It considers the two main models of bilingual schools currently to be found in Wales as well as some common bilingual teaching and learning practices to be found within them. It is within this context that the paper goes on to raise a number of issues concerning the assessment of bilingual pupils. Although the paper makes particular reference to the Welsh experience and the assessment of mathematics in particular, it is envisaged that the issues raised are of interest to delegates who may be concerned or involved with bilingual education elsewhere.KEYWORDS: Wales, bilingual, assessment

Introduction

Whether bilingual education in Wales is judged on the basis of the quality of the education which is provided or the rise in the number of pupils involved, any reasonable analysis must conclude that it is a story of considerable achievement. From a situation where Welsh was traditionally considered to be a language for poets and peasants and where English was the language 'to get on in the world', the last half century and in particular the last 25 years have seen a revolution in both thinking and practice. Not only has Welsh become fashionable and marketable within Wales, but the last few years have witnessed a language revival which has seen the number of young Welsh speakers, in certain parts of Wales, rise for the first time this century (Aitchison and Carter, 1994). A range of events including the launching of the Welsh television channel - Sianel Pedwar Cymru (S4C), the introduction of the National Curriculum and the inclusion of Welsh as a core or foundation subject in all schools in Wales, and more recently the implementation of the Welsh Language Act 1993 have certainly contributed towards enhancing the status of the Welsh language. Although undoubtedly influenced and fuelled by such an improved image, however, the modest language revival amongst young children in the South East of Wales in particular must clearly be mainly attributed to the growth in Welsh-medium education. In other parts of Wales, Welsh-medium or bilingual education is providing

an increasing number of pupils with the opportunity to study all aspects of the curriculum through the medium of Welsh.

The increase in the number of pupils sitting the General Certificate of Secondary Education (GCSE) through the medium of Welsh provides a striking illustration of the growth in the number of pupils now learning through the medium of Welsh (Figure 1).

Between 1972 and 1987 the graph plots the number of Welsh-medium candidates sitting the Ordinary (O) level and the Certificate of Secondary Education (CSE).From 1988 these examinations were replaced by the combined General Certificate of Secondary Education (GCSE).Graph is based on Welsh Office Statistics (Welsh Office, 1985, 1991, 1996)

Figure 1

Bilingual education in Wales is currently provided by two main types of bilingual schools. They are commonly referred to as natural bilingual schools and designated bilingual schools or Ysgolion Cymraeg.

Natural bilingual schools are mainly to be found in Welsh heartland areas where there exists a significant proportion of native Welsh speakers. These schools have traditionally served the local area and have, during recent years, increasingly offered pupils a wider bilingual experience. Ysgolion Cymraeg have been established in the more Anglicised areas of Wales and usually offer pupils a bilingual alternative to their local, predominantly, English speaking, school (see Baker, 1985,1990).

For many pupils who attend natural bilingual schools, Welsh is their first language and English is learnt as a second language at school. English is also assimilated, informally, through the pupils' continuous exposure to English on television and radio and through interactions with family and friends etc. Although some pupils may be more fluent in Welsh than English, even at 16+, the great majority of pupils are reasonably fluent in English by the time they leave secondary school.

Pupils who attend Ysgolion Cymraeg often come from English speaking homes and learn Welsh mainly within the educational environment. Given the very Anglicised nature of the communities which some designated schools serve, pupils who attend these schools learn Welsh mainly whilst at school. Although many of these pupils may study most subjects through the medium of Welsh, English remains the language they generally use outside school and is the language which they may well feel more comfortable with.

Most lessons in Ysgolion Cymraeg are usually in Welsh and although English technical terms may be referred to in subjects such as mathematics and science, pupils often read, write and discuss most subjects, predominantly if not entirely in Welsh. In the natural bilingual schools teaching and learning may often be very similar to that which happens in Ysgolion Cymraeg. At other times, because of the presence of non-Welsh speakers, lessons in subjects such as mathematics and science may be completely bilingual (Jones, 1994).

Linguistic factors which appear to merit particular attention when considering the assessments of English/Welsh bilingual pupils are that :-

The language used for assessment purposes is often not the pupils' first or home language. This may be particularly significant in the case of assessments taken at the end of Key Stage Two (11-year-olds). Some pupils understanding of Welsh may be limited, especially when applied to unfamiliar situations. In other cases, pupils may not yet be totally fluent in English.

Pupils experience a range of bilingual teaching approaches. Some pupils are familiar with discussing some subjects in both English and Welsh whilst others do so primarily in one language. Pupils may or may nor be familiar with technical terms in both languages.

Having provided a very brief flavour of bilingual education in Wales the paper now goes on to examine a number of issues in relation to the assessment of bilingual pupils in Wales.

Creating Matched Assessments

Creating matched assessments for use across different cultures and languages is not a new challenge and there is considerable international literature on this topic (see for example Hambleton, 1993: Brislin,1986; Poortinga, 1995). Since the introduction of the National Curriculum for England and Wales, the statutory testing of pupils achievement at the ages of 7,11 and 14 has seen the use of 'the same' tests throughout England and Wales. Making sure that the English and Welsh versions of the test are the same or equivalent is however not a simple task. Test development agencies over recent years have developed a process which is commonly referred to parallel development which involves a consideration of both English and Welsh versions of test papers from the outset. By doing so it is possible to avoid a number of problems which may otherwise arise.

Four particular problem areas relate to the use of :-

i) Unfamiliar wordsOne of the main challenges in translating from one language to another is not only the equivalence but the familiarity of vocabulary. Brislin (1986) for example refers to the fact that the Japanese have no single word for 'brother' but always make the distinction between 'older brother' and 'younger brother'. Wiliam (1994) refers to the English word 'brown' which until recently did nor have a Welsh equivalent

Such problems, are however, not likely to be encountered often in translating from English to Welsh. What is much more likely is that the Welsh equivalent is less familiar than the original English word.

Whilst a process of parallel development or de-centering (Brislin,1970) helps ensure that words are equivalent in both language versions of the tests, it is not as easy to ensure that they are equally familiar.

The Welsh for 'apples',afalau is a familiar Welsh word whilst the Welsh for 'pears', gellyg is not. A question involving apples could therefore be argued to be preferable to one which refers to 'pears'. A group of teachers identified the translations of 'mini buses', 'tray' and 'tracing paper' ( bysiau bach, hambwrdd and papur dargopio) in the 1995 Key Stage 2 tests as words which were potentially more unfamiliar than their English equivalents. They were not, however, considered to pose any real problems.

ii) Technical Terms.Similar problems can arise with the use of technical terms. Akagi (1991) refers to an international test item which asked "What changes the voltage of electric power supply?" and goes on to offer four answer choices, "alternator", "battery", "rectifier" and "transformer". Akagi explains that

"the correct answer choice is transformer. However "transformer" in Chinese characters means "change voltage apparatus". Therefore, this item when translated into Japanese, measures reading ability of ideograms rather than scientific knowledge, which was measured in the English version of the test item". Some Welsh terms can also be helpful in that they provide pupils with a greater insight into their meaning than their English equivalents. 'Eighteen' for example is commonly translated as un deg ag wyth or deunaw which when literally translated become 'one ten and eight' and 'two nines' respectively. The 1995 Key Stage 2 tests had a question which referred to a 'quadrilateral' with another illustrating a 'plan view'. The Welsh for 'quadrilateral' is pedrochr which literally translates to 'foursides'. 'Plan view' becomes uwcholwg which is 'above view'. Clearly, such words could possibly assist Welsh language candidates.

Wiliam (1994) refers to instances where the recognised Welsh term can fail to cue the necessary 'scientific' register. The scientific terms for 'velocity' is cyflymder . In every day spoken Welsh, cyflymder would be understood to mean 'speed'. In scientific terms the scalar quantity 'speed' is quite different to the vector quantity 'velocity' and therefore a pupils who sees the word cyflymder could be cued to think of 'speed' as opposed to the appropriate quantity 'velocity'. In contrast, Jones (1993)

identified a particular case where the Welsh term for 'similar' may have helped to cue the correct 'mathematical register'. A 1992 GCSE question involving 'similar triangles' produced significantly different responses from a matched sample of English and Welsh candidates. The Welsh equivalent to the mathematical term 'similar' is cyflun but is not a word used outside Mathematics. When encountered by pupils in an examination question it may have contributed towards cueing the necessary mathematical response. English candidates of comparable overall achievement who scored significantly poorer marks on the particular question, may have missed the mathematical significance of the word 'similar' ( Figure 2 ).

An investigation to see if there was any evidence that pupils were underachieving mathematically because of linguistic problems?

Paper 2 GCSE Summer 1992

112 Welsh Scripts, 112 English Scripts (matched)A significant difference in the responses to one question involving the need to use mathematical significance of the 'similarity' of two triangles :-

Bydd tri triongl sy'n gyflun i'r tri uchod, ABC, JKL a PQR.....Three triangles, similar to the triangles, ABC,JKL and PQR above.....Mean marks awarded:- Welsh 1.22 : English 0.69A very significant difference!See Jones (1993)

Figure2

Technical terminology in a particular language can clearly at times be helpful. On other occasions terms may be misleading or confusing. Whilst the range of technical terminology for any particular subject is limited compared to the more general

vocabulary one is likely to find on a test paper, the use of technical terminology can clearly influence such cross-cultural assessments and should be carefully considered.

iii) Dialects. The existence of different regional Welsh dialects, which have much greater differences than are found in English dialects, means that some English words will have more than one common Welsh equivalents. There have been several instances during the development of Welsh version of national curriculum assessments where dialect problems arose.

A 1994 Key Stage 3 sample paper had a question involving 'milk'. Milk is commonly known as llefrith in North Wales and llaeth in the South. The translator opted for llaeth but a question to do with 'tea' or 'cheese' would have avoided the need to choose one word in preference to the other. An example where an element of parallel test development had helped to avoid a potential problem was cited by a representative for the 1996 KS2 mathematics assessment development agency. One of the questions at one stage in its development had 'onions' as one of the options for 'extra pizza toppings'. Since, however, pupils in North Wales would tend to use the word nionod for 'onions' whilst those in South Wales would use winwns, 'extra onions' was replaced by 'extra tomatoes' which causes no such dialect problems teachers (Evaluation of Key Stage 2 Assessments in England and Wales, 1996)

Possible problems created by different dialects can clearly pose problems to test developers who may not consider themselves as such to be involved in a cross-linguistic exercise. A degree of parallel development may also be helpful in such circumstances.

iv) Unfamiliar ContextsWelsh is a minority language with currently just over 500,000 speakers, most of whom understand English. The use of Welsh in advertising, business, the tourist industry etc. is often limited and as a result particular care must be taken when considering so called 'real contexts' for questions. Cooper (1992,1994) raises important questions about the use of 'real life' questions which have been set over recent years. With particular reference to Bernstein's work ( 1990) he suggests that middle class pupils are better able to understand what is required from 'real life' problems than children from 'lower' social classes. He provides examples of test questions which have a real context but where pupils need to 'bracket out their knowledge of it' in order to succeed. See Figure 3.

Figure 3

The marking scheme expected candidates to divide 269 by 14 and rounded up the answer to 20. In real life it is very unlikely that exactly 14 people would enter the lift each time but the answer scheme makes no allowance for this. As Cooper writes 'the child has to ignore the reference to the 'real world' in order to succeed'.

Consider now the example in Figure 4 from the 1993 Key Stage 3 tests.

Cooper would probably argue that there are several ways of interpreting the question and that a perfectly acceptable answer could refer to how much popcorn the purchaser might actually want. The mark scheme made no allowance for such a response and so would not credit pupils for answering a 'real' problem with a reasonable, everyday solution.

When such questions are presented in Welsh however, there is the added dimension that the question is not actually 'real' in the first place since you would be very privileged indeed to spot a Welsh popcorn poster. Cooper raises some fundamental questions and his misgivings are even more marked if one sets questions which relate to contexts which are not even encountered in Welsh!

Experience over recent years has clearly demonstrated that parallel development can help avoid such problems and is the approach clearly advocated by ACAC (ACAC,1997). Welsh-medium versions of National Curriculum tests over recent years have generally been well received. Neither the WJEC or ACAC, however, appear to use or commission any form of statistical methods to test for any possible differential item functioning (DIF) and possible item bias in relation to English / Welsh versions of test / examination papers ( Ellis, 1991; Camilli and Shepard, 1994). Given the considerable amount of money invested in the development and evaluation of National Curriculum assessments during recent years, such an examination would appear to be worth pursuing.

A more flexible bilingual approach?

Pupils siting GCSE (at 16+) or Advanced Level (A level at 18+) are required to choose either a Welsh or English versions of the examination papers. Before 1994 teachers could do no more than provide pupils with alternative words for any terms which may have been unfamiliar to pupils.

In a 1992 survey (Jones, 1994) teachers were asked whether or not they considered such an unilingual policy to be fair. Thirty three of the eighty one teachers who responded felt that it wasnít. Teachers who taught bilingual classes in some natural bilingual schools felt strongly that pupils in such classes should be allowed to 'use their bilingualism' under examination conditions and be allowed access to examination papers in both languages.

Since the summer of 1994 the WJEC has changed its policy and now, as with National Curriculum tests, allows candidates access to both versions of examination papers if the teacher feels that this may benefit the candidate. Pupils are, however, still required to complete either a Welsh or English script.

One consequence of such a policy is that it encourages the practice of requiring bilingual pupils to study some subjects in English and others in Welsh. The concept that a bilingual pupil is one who can for example discuss geography in Welsh and mathematics in English is now, however, being increasingly seen as a narrow interpretation of bilingualism. A school in Gwynedd has recently developed a language policy whereby all pupils experience some subjects bilingually. The mathematics department for example alternates the language of instruction according to a scheme of work which ensures that all pupils encounter all topics in both languages by the end of Year 9 (14 year olds). The first two lessons on Fractions for example may be in English whilst the following two lessons, on the same topic, may be through the medium of Welsh. At the end of Year 9, however, pupils are guided to study each subject through the medium of either English or Welsh in preparation for external examinations at 16+. Even then, however, like many other bilingual classrooms in Wales, they will experience mathematics lessons which switch from one language to another. Although they may well be familiar with both Welsh and English mathematical terms etc., the examination system currently requires them to choose one language in preference to the other.

If one of the main aims of bilingual education is to produce fully balanced bilinguals then are current approaches to teaching, learning and assessment the most appropriate to achieve those aims or is there a need to think again? Would a bilingual examination process pave the way for more flexibility approaches to teaching and learning? When does a bilingual pupil benefit from having available to him or her both language versions of an examination paper? Would a bilingual paper be preferable to two separate versions? If so, how should it be laid out? These question would appear to merit further investigation.

Conclusion

To prove that Welsh-medium education provides education of a superior quality to comparable English-medium schools is not easy and even to claim as such is clearly contentious. There is no doubt, however, that Welsh-medium schools have gained from their reputation, rightly or wrongly that they provide a better standard of education. Parents must have good reasons to send their children often for many miles an past a number of other schools to have their pupils educated through a language of which they, as parents, have no understanding. The belief that they 'are better' is clearly widespread ( Campbell and Packer, 1991). In recent government school performance tables Welsh-medium schools have constantly featured among the best in Wales and recent evidence suggests that even having accounted for possible socio-economic differences, Welsh-medium schools are doing 'rather well'. (Bellin et al,1996). Credible suggestion have been put forward why they 'are better' (Reynolds and Bellin, 1996). From a position of relative strength there is now, however, a need to subjected current practice to closer scrutiny. This paper suggests that there are particular problems in using one language to assess a bilingual's attainment in a particular curriculum area and that difficulties are often more subtle than they may at first appear. This paper also suggests that more flexible bilingual approaches to teaching, learning and assessment more generally is a particular area which merits further exploration.

Welsh education has recently gained greater autonomy than ever before (Jones,1994) and this appears set to increase still further. Given the recent reorganisation of local government, the implementation of The Welsh Language Act and the recent white paper on Welsh education, the stage is set for a new era in Welsh education. Whether the people of Wales vote for further autonomy or not on the 18th September, there is a clear need to develop a national sense of direction for bilingual education. Accepted principles need to be questioned as part of a strategy which builds on the achievements of the past and helps ensure an even brighter bilingual future for the pupils of Wales.

References

ACAC (1997) A report to the secretary of state for Wales on the second phase of the assessment arrangements review (Cardiff, Curriculum and Assessment Authority for Wales)

Aitchison J. and Carter H. (1994) A Geography of the Welsh Language 1961-1991 (Cardiff, University of Wales Press)

Akagi A. (1991) Difficulties in adapting test items for cross-cultural studies, in Bulletin of the International Test Commission, 18, pp.65-71

Baker, C. (1985) Aspects of Bilingualism in Wales ( Clevedon, Multilingual Matters)

Baker, C. (1990) The growth of bilingual education in Wales, in: Evans, W.G. (Ed.) Perspectives on a Century of Secondary Education in Wales ( Aberystwyth, Canolfan Astudiaethau Addysg)

Bellin, W., Farrell, S., Higgs, G. and White, S. (1996) A strategy for using census information in comparison of school performance. The Welsh Journal of Education, Vol. 5, No. 2, 3-25

Bernstein, B. (1990) The Structure of Pedagogic Discourse (London, Routledge)

Brislin R.W. (1970) Back-translation for cross-cultural research. Journal of cross-cultural psychology, 1, 185-216

Brislin R.W. (1986) The Wording and Translation of Research Instruments, in: W.J. Lonner and W.B. Berry (Eds.) Field Methods in cross-cultural Research (California, Sage)

Camilli G. and Shepard L., A. (1994) Methods for identifying biased test items (London, Sage)

Campbell, C. and Packer, A. (1991) Cymhellion Rhieni Di-Gymraeg Dros Ddewis Addysg Gymraeg I'w Plant, The Welsh Journal of Education, Vol. 3, no.1, 27-33

Cooper, B. (1992) Testing National Curriculum Mathematics: some critical comments on the treatment of 'real' contexts for Mathematics, The Curriculum Journal, Vol. 3, No. 3, pp.231-244.

Cooper, B. (1994) Authentic Testing in Mathematics ? The boundary between everyday and mathematical knowledge in National Curriculum testing in English Schools, Assessment in Education, Vol.1, No. 2, pp.143 -166

Ellis, B.B. (1991) Item response theory: a tool for assessing the equivalence of translated tests, in Bulletin of the International Test Commission, 18, pp.33-51.

Hambleton R.K.(1993) Translating achievement tests for use in cross-national studies, European Journal of Psychological Assessment, 9(1), pp. 57-68

Jones, D.V. (1993), Words With a Similar Meaning, Mathematics Teaching, 145, pp. 14-15.

Jones, D.V. (1994), Mathemateg Trwy Gyfrwng y Gymraeg; Trefniadaeth, Adnoddau a Barn Athrawon, Aberystwyth Education Papers ( Aberystwyth, Education Department )

Jones G.E. (1994) 'Which Nation's Curriculum ? - The Case of Wales'. The Curriculum Journal 5(1): 5-16 Poortinga, Y (1995) Uses of tests across cultures, in : T. Oakland and R. K. Hambleton, (Eds.) International perspectives on academic assessment (Kluwer, London)

Poortinga, Y. and van de Vijver, F.J.R. (1991) Culture-free measurement in the history of cross-cultural psychology, Bulletin of the International Test Commission, 18, 72-87.

Reynolds, D and Bellin, W. (1996) Welsh-medium Schools; Why they are Better. Agenda, Summer 1996. Cardiff: The Institute of Welsh Affairs.

Welsh Office (1985-1996) Statistics of Education in Wales. (Welsh Office, Cardiff)

Wiliam,D. (1994) Creating matched National Curriculum assessments in English and Welsh : test translation and parallel development, The Curriculum Journal5(1), pp. 17-29