ENHANCING TRAINING AND ENHANCING TRAINING AND TESTING EFFECTIVENESS TESTING EFFECTIVENESS

THROUGH LESSONS THROUGH LESSONS LEARNEDLEARNED

Emilia Nesheva, Ministry of Defense, Bulgaria

Greta Keremidchieva, Rakovski National Defense Academy, Bulgaria

MINISTRY OF DEFENSEREPUBLIC OF BULGARIA

ENGLISH LANGUAGE TRAINING ENGLISH LANGUAGE TRAINING AND TESTING – OFFICIAL AND TESTING – OFFICIAL

POLICY OF THE MODPOLICY OF THE MOD

BASED ON:BASED ON: DOCUMENTATIONDOCUMENTATION TRAINING PROGRAMSTRAINING PROGRAMS QUALIFIED TEACHERS – “SOFT POWER”QUALIFIED TEACHERS – “SOFT POWER” BOOKS AND EQUIPMENT – “HARDWARE”BOOKS AND EQUIPMENT – “HARDWARE”

REGULATING DOCUMENTSREGULATING DOCUMENTS

1. 1. Order of the Minister of DefenseOrder of the Minister of Defense №130 №130// 21.02.2011 21.02.2011 regarding regarding Strategy for Strategy for development of the English language development of the English language training systemtraining system

2. 2. Order of the Minister of DefenseOrder of the Minister of Defense №760/04.10.2011 №760/04.10.2011 regarding regarding selection of selection of candidatescandidates, , preliminary testing and preliminary testing and conducting English courses conducting English courses

3. 3. RegulationRegulation №Н-19/25.05.2010 №Н-19/25.05.2010 regarding regarding planning and development of language planning and development of language coursescourses..

LANGUAGE TRAINING CENTERSLANGUAGE TRAINING CENTERS

RAKOVSKI NATIONAL DEFENSE ACADEMY – SOFIARAKOVSKI NATIONAL DEFENSE ACADEMY – SOFIA

NATIONAL MILITARY UNIVERSITY – VELIKO TARNOVONATIONAL MILITARY UNIVERSITY – VELIKO TARNOVO

LANGUAGE TRAINING CENTER – SHUMEN - part of LANGUAGE TRAINING CENTER – SHUMEN - part of the the Individual Training and Education Programme Individual Training and Education Programme – – ITEPITEP,, NATO NATO Tier 1 Tier 1 Smart Defence 1.7 Project Smart Defence 1.7 Project

NAVAL MILITARY ACADEMY – VARNANAVAL MILITARY ACADEMY – VARNA

MILITARY UNITSMILITARY UNITS

ENGLISH LANGUAGE TRAINING IN ENGLISH LANGUAGE TRAINING IN RAKOVSKI NATIONAL DEFENSE RAKOVSKI NATIONAL DEFENSE

ACADEMYACADEMY

Training of officers in the Master’s degree Training of officers in the Master’s degree programprogram

Training of military personnel and civilians Training of military personnel and civilians in General and Specialized English in General and Specialized English coursescourses

Target audience in Rakovski National Defense Academy

Officers in the Master’s Program

Entry – now 1111; next year (2222)

Exit – now 2222; next year (3 2+3 2)

Officers, NCO’s and civilians in English language courses

NEW

Training program in Rakovski National Defense Academy

The Master’s Program starts with a preliminary block of 360 hours intensive English training prior to their first semester of military studies

Advantage: some military subjects could be studied in English

NEW

General and Specialized English language training

Course Length Books

ALC

Entry requirements

Expected exit outcome

Level I 18 weeks 1-12 - 45 points

ALCPT

Level II 18 weeks 13-24 45 points 1+ 1 1+ 1

Level III 15 weeks 25-30 1+ 1 1+ 1 2 2 2 2

Level IV 15 weeks 31-34 2 2 2 2 3 2+ 3 2+

Specialized courses

2-8 weeks 2 2 2 2 -

NEW

ASSESSMENT AND CONTROLASSESSMENT AND CONTROL

Preliminary TestingPreliminary Testing

((ALCPT before LevelALCPT before Level ІІ) ІІ) Entry TestsEntry Tests Book quizzes, Progress testsBook quizzes, Progress tests Final TestFinal Test, , ALCPT, STANAG 6001ALCPT, STANAG 6001

FACULTY MEMBERSFACULTY MEMBERS

Teachers – 10 University graduates withTeachers – 10 University graduates with

specializations in different countries and specializations in different countries and institutions; experienced; participants in institutions; experienced; participants in national & international conferencesnational & international conferences

Test Team members – 5 University Test Team members – 5 University graduates; trained; design, pilot & graduates; trained; design, pilot & administer STANAG Tests nation-wideadminister STANAG Tests nation-wide

BOOKS & EQUIPMENTBOOKS & EQUIPMENT

ALC Books 1 - 34ALC Books 1 - 34 Specialized textbooks – Campaign, Specialized textbooks – Campaign,

Military English, Army, Air Force, Navy, Military English, Army, Air Force, Navy, etc.etc.

Language Labs Language Labs Self-Access CenterSelf-Access Center Audio & multimedia programsAudio & multimedia programs

FINANCIAL RESOURCESFINANCIAL RESOURCES

MOD program provides financial MOD program provides financial resources for the accomplishment of Force resources for the accomplishment of Force GoalGoal Е 0356 Е 0356;;

for improvement of infrastructure and for improvement of infrastructure and learning technologies;learning technologies;

for conferences and seminars.for conferences and seminars.

Enhancing Enhancing Testing Effectiveness Testing Effectiveness

through through Computer TestingComputer Testing

Action PlanAction PlanMinisterial Order Ministerial Order № 857№ 857//1010.11.2011.11.2011

till 1 March, 2012 – approbation of the computer-assisted till 1 March, 2012 – approbation of the computer-assisted “writing” module;“writing” module;

till 1 July, 2012 – analysis of the results;till 1 July, 2012 – analysis of the results; since exam session Autumn 2012 – optional computer-assisted since exam session Autumn 2012 – optional computer-assisted

“writing” module; “writing” module; till 1 March, 2013 – approbation of computer-assisted modules till 1 March, 2013 – approbation of computer-assisted modules

for “reading” and “listening”;for “reading” and “listening”; till 1 July, 2013 – analysis of the results;till 1 July, 2013 – analysis of the results; since exam session Autumn 2013 – optional computer-assisted since exam session Autumn 2013 – optional computer-assisted

“writing”, “reading”, and “listening” modules;“writing”, “reading”, and “listening” modules; since exam session Autumn 2014 – all candidates are to be since exam session Autumn 2014 – all candidates are to be

tested on computers for the modules “writing”, “reading”, and tested on computers for the modules “writing”, “reading”, and “listening”.“listening”.

End of pen-and-paper STANAG 6001 tests.End of pen-and-paper STANAG 6001 tests.

Approbation Approbation

of the computer-assisted “writing” moduleof the computer-assisted “writing” module

In February 2012, two In February 2012, two groups of volunteers – groups of volunteers – military and civilian military and civilian personnel sat for the personnel sat for the approbation of the writing approbation of the writing test. The only requirement test. The only requirement for taking part in the for taking part in the approbation was to have a approbation was to have a STANAG 6001 certificate STANAG 6001 certificate issued during the exam issued during the exam session “Autumn 2011”.session “Autumn 2011”.

Comparability of handwritten and computer-Comparability of handwritten and computer-based writing assessments – some findings based writing assessments – some findings

in the specialized literaturein the specialized literature

Questionnaires (excerpt)Questionnaires (excerpt)

Did you find the font of the tasks friendly Did you find the font of the tasks friendly enough?enough?

Did you meet any technical difficulties Did you meet any technical difficulties during the computer-assisted “writing” during the computer-assisted “writing” test?test?

Do you think that your speed of typing is Do you think that your speed of typing is decisive for your success at the decisive for your success at the computer-assisted “writing” test?computer-assisted “writing” test?

Recommendations Recommendations from the Questionnairesfrom the Questionnaires

1.1. The font of typing to be Times New Roman 12 The font of typing to be Times New Roman 12 oror 1414..

2.2. Candidates to be able to go back to the Candidates to be able to go back to the previous task for revising or correcting previous task for revising or correcting mistakes.mistakes.

3.3. Word count to be available.Word count to be available.4.4. Tabulation for indicating a new paragraph to Tabulation for indicating a new paragraph to

be available.be available.5.5. Five minutes for a spell check to be added to Five minutes for a spell check to be added to

the time allotted for the writing test.the time allotted for the writing test.

Actions TakenActions Taken

The commercial company that had The commercial company that had developed the testing software was tasked developed the testing software was tasked to correct the faults noticed during the to correct the faults noticed during the approbation and to add the functions approbation and to add the functions recommended in the questionnaires.recommended in the questionnaires.

The STANAG testing team developed The STANAG testing team developed detailed administrative instructions for the detailed administrative instructions for the computer-assisted writing test.computer-assisted writing test.

Approbation ResultsApprobation Results

55%

12%

33%

sustained level

higher level

lower level

AnalysisAnalysis

Special attention was paid to the candidates who Special attention was paid to the candidates who attained higher or lower STANAG 6001 level at attained higher or lower STANAG 6001 level at the computer-assisted writing test. Analysis of the computer-assisted writing test. Analysis of the possible reasons:the possible reasons:

- was the rating of their pen-and-paper test was the rating of their pen-and-paper test subject to discussions? how many raters had subject to discussions? how many raters had assessed the papers?assessed the papers?

- additional English language studies?additional English language studies?- different “pre-writing” habits?different “pre-writing” habits?- rater effects in scoring handwritten and rater effects in scoring handwritten and

computer-based writing assessments? computer-based writing assessments?

LESSONS LEARNTLESSONS LEARNT(possible scoring biases)(possible scoring biases)

Typed essays generally appear to raters Typed essays generally appear to raters shorter than identical handwritten responses.shorter than identical handwritten responses.

Errors made are more evident when typed.Errors made are more evident when typed.

LESSONS LEARNT (cont.)LESSONS LEARNT (cont.)(the importance of the Interface)(the importance of the Interface)

Traditional text editing tools – cut, copy, and Traditional text editing tools – cut, copy, and paste – need to be designed to support less paste – need to be designed to support less experienced computer users.experienced computer users.

Space management – “word count” and Space management – “word count” and “print preview” should be available.“print preview” should be available.

Questions?Questions?

In this presentation have been used In this presentation have been used quotations by: quotations by:

Breland, Lee, &Muraki, 2005 Breland, Lee, &Muraki, 2005 Bridgeman & Cooper, 1998;Bridgeman & Cooper, 1998; Davis, Strain-Seymour, Lin, & Kong, 2008;Davis, Strain-Seymour, Lin, & Kong, 2008; Horkay et al., 2006;Horkay et al., 2006; Pearson, 2010; Pearson, 2010; Russell &Plati, 2001; Russell &Plati, 2001; Russell & Haney, 1997;Russell & Haney, 1997; Way & Fitzpatrick, 2006. Way & Fitzpatrick, 2006.