validation of the grammatical carefulness scale using a discourse completion task and a reading and...
TRANSCRIPT
Validation of the Grammatical Carefulness Scale Using a
Discourse Completion Task and a Reading and Underlining Task
November 22, 2014 84th LET Chubu
Shizuoka University
The handout is available from…
The handout is available from…
Yu TAMURAGraduate School, Nagoya Univ.
[email protected] KUSANAGI
Graduate School, Nagoya Univ.JSPS Research Fellow [email protected]
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Introduction• What is grammatical carefulness(GC)?
• Why is it needed?
• How can it be used?
Introduction• What is grammatical carefulness (GC)?
• a construct referring to learners’ characteristics
• personal traits reflecting learner’s behaviour and belief in language use
• explains learners’ variance • reflects highly controlled, cautious, and
analytical language use
Introduction• Why is it needed?
• To capture the variance of L2 learners’ inconsistent performance
• To get more information about individual differences focusing on grammatical performance
• To take into account speed accuracy trade-off (Dennis & Evans, 1996; Goldhammer & Kroehne, 2014; van der Linden, 2007, 2009)
Introduction• speed accuracy trade-off
• consistent within a learner • linked to other learners’ factors (e.g., psychological,
behavioural, and meta-cognitive)
Introduction• How can it be used?
• as covariance in psycholinguistic experimental research
• as independent variables in interventional experiments
• in investigating relationship between individual differences and L2 learners’ performance
• as a diagnostic assessment tool in educational settings
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Background• Grammatical Carefulness Scale (GCS)
• A questionnaire developed by Kusanagi et al. (2014) • 14 items (3 sub-scales) • 7 point Likert scale • written in Japanese • Some validation has been done (Kusanagi et al., 2014)
• factorial validity • content validity • criterion-based validity
Background• Grammatical Carefulness Scale (GCS)
• 3 sub-scales • Lexical-Syntactic Carefulness (LSC) (k = 4)
• e.g., I always notice the wrong use of the words.
• Pragmatic Carefulness (PC) (k = 5) • e.g., I’m likely to think about inconsistent expressions.
• Phonological Carefulness (PHC) (k = 5) • e.g., I’m always careful about correct pronunciation.
Background• Grammatical Carefulness Scale (GCS)
• factorial validity • both exploratory and confirmatory • high reliability (α = .90)
• content validity • judgments by linguistic experts
• criterion-based validity • correlated to analytical belief (Tanaka & Ellis, 2003) more strongly
than empirical belief • correlated to the accuracy score of GJTs • correlated to the accuracy score of C-test and its time to completion
GCS is validated!
Validation never ends.
Background• Grammatical Carefulness Scale (GCS)
• factorial validity • both exploratory and confirmatory • high reliability (α = .90)
• content validity • judgments by linguistic experts
• criterion-based validity • correlated to analytical belief (Tanaka & Ellis, 2003) more strongly
than empirical belief • correlated to the accuracy score of GJTs • correlated to the accuracy score of C-test and its time to completion
Background• Grammatical Carefulness Scale (GCS)
• factorial validity • both exploratory and confirmatory • high reliability (α = .90)
• content validity • judgments by linguistic experts
• criterion-based validity• correlated to analytical belief (Tanaka & Ellis, 2003) more strongly
than empirical belief • correlated to the accuracy score of GJTs • correlated to the accuracy score of C-test and its time to completion
Background• Grammatical Carefulness Scale (GCS)
• 3 sub-scales • Lexical-Syntactic Carefulness (LSC) (k = 4)
• e.g., I always notice the wrong use of the words.
• Pragmatic Carefulness (PC) (k = 5) • e.g., I’m likely to think about inconsistent expressions.
• Phonological Carefulness (PHC) (k = 5) • e.g., I’m always careful about correct pronunciation.
Background• Grammatical Carefulness Scale (GCS)
• 3 sub-scales • Lexical-Syntactic Carefulness (LSC) (k = 4)
• e.g., I always notice the wrong use of the words.
• Pragmatic Carefulness (PC) (k = 5) • e.g., I’m likely to think about inconsistent expressions.
• Phonological Carefulness (PHC) (k = 5) • e.g., I’m always careful about correct pronunciation.
Background• RQ
• Do the scores of the tasks focusing either on LSC or PC correlate with the LSC and PC measured by GCS?
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
The Present Study• Participants (N = 63)
• undergraduate students (3 university) and graduate students (1 university)
• 78% of the participants were female
undergrad grad
A B C D
n 20 27 8 8
The Present Study• Tasks
• LSC • Reading and Underlining Task (RUT)
• PC • Discourse Completion Task (DCT)
The Present Study• Reading and Underlining Task (RUT)
• often used as a measure of learner’s noticing toward formal properties of TL (e.g., Izumi & Bigelow, 2000; Izumi, Bigelow, Fujiwara, & Fearnow, 1999; Uggen, 2012)
• requiring to read a short passage and underline ungrammatical parts of the sentences.
The Present Study• Reading and Underlining Task (RUT)
• passage (153 words) • from internet sources (http://www.eslyes.com) • Flesch-Kincaid Reading Ease 94.252 • Flesch-Kincaid Grade Level 1.558 • revised and controlled vocabularies
(JACET8000 Level1-3) • Glossary was given.
The Present Study• Reading and Underlining Task (RUT)
• 9 errors • inflection errors
• was killing/ *kill her; She cried/ *cryed; She stood/ *standing up
• misspelling • walked/ *wolked; could/ *cuold; always/ *alweys
• wrong use of words • Whenever she was stressed out, the stress went straight to her/
*his stomach. • Almost an hour later/ *ago,…. • I feel much/ *more better.
The Present Study• Tasks
• LSC • Reading and Underlining Task (RUT)
• PC • Discourse Completion Task (DCT)
The Present Study• Tasks
• LSC • Reading and Underlining Task (RUT)
• PC • Discourse Completion Task (DCT)
The Present Study• Discourse Completion Task (DCT)
• used to tap into learners’ intuition about how they behave in certain speech acts (e.g., Bardovi-Harlig & Dornyei, 1998; Olshtain & Blum-Kulka, 1985)
• can be both spoken and written • can be both multiple-choice or free
production
The Present Study• Discourse Completion Task (DCT)
• based on Blum-Kulka, House, and Kasper (1989) • 2 speech acts (K = 12)
• request (k =8)
• apology (k = 8)
• various dimensions of social power and social distance
• multiple-choice • situation and instructions are given in Japanese
The Present Study• Discourse Completion Task (DCT)
• example • 見知らぬ街へ出かけたメアリー。駐車禁止場所とは知らずに車を停めてしまいました。すると警察官が寄ってきて…
Policeman: ( )
Mary: Oh, I’m sorry. I didn’t know that. I’ll move the car right now.
a. Would you please move your car from here?
b. How about moving the car from here?
c. Could you move the car from here?
d. Do you want to move the car from here?
e. Move your car.
The Present Study• Discourse Completion Task (DCT)
• example • 見知らぬ街へ出かけたメアリー。駐車禁止場所とは知らずに車を停めてしまいました。すると警察官が寄ってきて…
Policeman: ( )
Mary: Oh, I’m sorry. I didn’t know that. I’ll move the car right now.
a. Would you please move your car from here?
b. How about moving the car from here?
c. Could you move the car from here?
d. Do you want to move the car from here?
e. Move your car.
The Present Study• Reliability issue
• Reading and Underlining Task (RUT)
• Some items were excluded
N K M SD Cronbach α 95%CI
60 9 .55 .21 .50 [.28, .72]
The Present Study• Reliability issue
• Reading and Underlining Task (RUT)
• Excluded items: was killing/*kill, her/*his, later/*ago
N K M SD Cronbach α 95%CI
60 6 .49 .27 .58 [.28, .72]
The Present Study• Reliability issue
• Discourse Completion Task (DCT)
• Some items were excluded.
N K M SD Cronbach α 95%CI
61 12 .56 .18 .48 [.26, .69]
The Present Study• Reliability issue
• Discourse Completion Task (DCT)
• Excluded items: Item2, 9, 11, 5, 8
N K M SD Cronbach α 95%CI
61 7 .56 .23 .56 [.35, .77]
The Present Study• Procedure
• Two English language teachers working for a college
• Two tasks were administered as part of classroom activities
• Graduate students were recruited individually
The Present Study• Analysis
• 10 students missed one of the two tasks → excluded from the analysis • 3 students showed extremely low
products of deviation → excluded from the analysis (outlier) • In total, 50 students were included in
the final analysis.
The Present Study• Analysis
• Correlation between the score of the tasks and GCs
→ Pearson product-moment correlation coefficient → Partical Correlation Coefficient
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Results
N M SD skew kurtosis SE α
RUT 50 .50 .32 .56 -1.01 -.04 .58
LSC 50 3.24 1.24 .31 -.20 .18 .87
PC 50 3.18 1.20 .18 -.05 .17 .85
DCT 50 .54 .20 -.10 -.46 .03 .56
Descriptive Statistics of the Two tasks, LSC, and PC
Partical Correlation Coefficient
RUT LSC PC DCT
RUT
LSC [.013, .526]
PC [-.083, .453] [.686, .888]
DCT [-.232, .324] [.002, .518] [-.062, .470]
Interval Estimation of Population Correlation Coefficient (95%CI)
RUT LSC PC DCT
RUT
LSC .408
PC .285 .942
DCT .088 .401 .319
Adjusting Correction for Attenuation of Correlation
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Discussion• RUT and LSC
• Pearson’s r →weakly correlated • Partical Correlation Coefficient →weakly correlated • Interval Estimation of Population Correlation Coefficient →◯ • Adjusting Correction for Attenuation of Correlation →moderately correlated
Discussion• RUT and LSC
• Pearson’s r →weakly correlated • Partical Correlation Coefficient →weakly correlated • Interval Estimation of Population Correlation Coefficient →◯ • Adjusting Correction for Attenuation of Correlation →moderately correlated
• The students who showed high LSC were likely to notice lexical and grammatical errors
Discussion• DCT and PC
• Pearson’s r →weakly correlated • Partical Correlation Coefficient →△ • Interval Estimation of Population Correlation Coefficient →△ • Adjusting Correction for Attenuation of Correlation →weakly correlated
Discussion• DCT and PC
• Pearson’s r →weakly correlated • Partical Correlation Coefficient →△ • Interval Estimation of Population Correlation Coefficient →△ • Adjusting Correction for Attenuation of Correlation →weakly correlated
because LSC and PC showed high correlation
Discussion• The results seemed to confirm the validity of
GCS • difficult to investigate the correlation
between L2 learners’ performance and the subscales of GC
→ GCS showed high reliability
Partical Correlation Coefficient
Limitations• Reliability of the two tasks • Underrepresentation • What about the correlation between
phonological carefulness and learners’ performance?
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Overview• Introduction• Background• The Present Study• Results• Discussion• Conclusion
Conclusion• The score of LSC predicted the score of
RUT • The score of PC partly predicted the score
of DCT • Validation never ends
ReferenceBardovi-Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? pragmatic versus grammatical
awareness in instructed L2 Learning. TESOL Quarterly, 32, 233–259. doi:10.2307/3587583
Blum-Kulka, S., House, J., & Kasper, G. (1985). Investigating cross-cultural pragmatics: An Introductory Overview. In S. Blum-Kulka, J. House, & G. Kasper (Eds.), Cross-cultural pragmatics: Requests and apologies (pp. 1–33). Norwood, NJ: Ablex Publisher Corporation.
Dennis, I., & Evans, J. St. B. T. (1996). The speed-error trade-off problem in psychometric testing. British Journal of Psychology, 87, 105–129.
Goldhammer, F., & Kroehne, U. (2014). Controlling individuals’ time spent on task in speeded performance measures: Experimental time limits, posterior time limits, and response time Modelling. Applied Psychological Measurement, 38, 255–267.
Izumi, S., & Bigelow, M. (2000). Does output promote noticing and second language acquisition? TESOL Quarterly, 34, 239–278.
Izumi, S., Bigelow, M., Fujiwara, M., & Fearnow, S. (1999). Testing the output hypothesis: Effects of output on noticing and second language lcquisition. Studies in Second Language Acquisition, 21, 421–452.
Kusanagi, K., Fukuta, J., Kawaguchi, Y., Tamura, Y., Goto, A., Kurita, A., & Murota, D. (2014). Gaikokugo ni okeru bumpouteki shinchou sei syakudo no kaihatsu [Development of grammatical carefulness in English as a foreign language]. 40th Annual Conference of Japan Society of English Language Education. Tokushima University, Japan.
Olshtain, E., & Blum-Kulka, S. (1985). Degree of approximation: Nonnative reactions to native speech act behaviour. In S. M. Gass & C. G. Madden (Eds.), Input in second language acquisition (pp. 303–325). Newbury House.
Tanaka, K., & Ellis, R. (2003). Study abroad, language proficiency, and learner beliefs about language learning. JALT journal, 25, 63-85. Uggen, M. S. (2012). Reinvestigating the Noticing Function of Output. Language Learning, 62, 506–540. doi:10.1111/j.
1467-9922.2012.00693.x
van der Linden, W. J. (2007). A hierarchical framework for modelling speed and accuracy on test items. Psychometrika, 73, 287–308. van der Linden, W. J. (2009). Conceptual issues in response-time modelling. Journal of Educational Measurement, 46, 247–272.