The Effect of Repetition on Acceptability and Confidence Judgments of Linguistic Tokens

Elliot Schwartz

The Effect of Repetition on Acceptability and Confidence Judgments of Linguistic Tokens

What is generative linguistics?

● Syntax: the structure of words and phrases within a language○ “Colorless green ideas sleep furiously”

● Grammar: the underlying syntactic rules of a language○ Speakers have have a tacit knowledge of their mental

grammars

How do linguists learn about mental grammars?

The Effect of Repetition on Acceptability and Confidence Judgments of Linguistic Tokens

Acceptability judgments

● Native speaker are presented with sentence and asked to judge whether they find it natural and consider it something they could say under the appropriate circumstances

(1) The cat sat on the mat (2) * Cat the mat the on sat

What’s the problem?

● Competence: a speaker’s abstract, underlying knowledge of their language

● Performance: the speaker’s use of that knowledge in concrete situations

Extra-grammatical factors

Humans are limited capacity processors; acceptability judgments reflect performance cannot be taken as a direct indication of grammatical competence.

○ Expertise (Culbertson & Gross, 2009; Dąbrowska, 2010)○ Working memory (Casasanto, Hofmeister, & Sag, 2010; Gibson & James,

1999)○ Context (Warner & Glass, 1987)

The impact of such factors cannot be determined a priori

Formal

● Solicited from participants

● Minimal linguistic experience

● Large number of sentence tokens

● Large sample size

Informal

● Self-solicited● Linguistic expertise● Small number of

sentence tokens● Small sample size

The Effect of Repetition on Acceptability and Confidence Judgments of Linguistic Tokens

Syntactic Satiation

● Phenomenon in which repeatedly judging an ungrammatical structure leads to an increase in acceptability

● Frequently reported by linguists but mixed empirical results

The Effect of Repetition on Acceptability and Confidence Judgments of Linguistic Tokens

Mechanisms for Satiation

● Repeated judgment decrease the ability to process syntactic features Acceptability may increase in ungrammatical structures because...○ Features causing grammatical violations can no longer be distinguished○ A Decrease in processing ability may decrease one’s confidence in their

ability judge.

How do we distinguish these possibilities? Introduce a second rating scale!

The Effect of Repetition on Acceptability and Confidence Judgments of Linguistic Tokens

Hypotheses

1. Ungrammatical structures will show an increase in acceptability after repeated exposure

2. Ungrammatical structures will show a decrease in confidence after repeated exposure

3. For ungrammatical structures, increases in acceptability will correlate with decreases in confidence

Method

● Participants: 30 Carleton College students, native English speakers, minimal linguistic experience

● Procedure: Participants made judgments of both acceptability and confidence for eight syntactic structures (four grammatical, four ungrammatical)○ Total of 56 sentences presented in seven blocks○ Judgments on a 7-point Likert Scale (1: “Completely unacceptable,” 7:

“Completely acceptable”)○ Presented online using Qualtrics Software

Materials● Whether Island (ungrammatical)

○ Situation: Julia asked whether snakes lay eggs.○ * What did Julia ask whether snakes lay?

● Adjunct Island (ungrammatical)○ Situation: Susan walked out of the theater during the intermission.○ * What did Susan walk out of the theater during?

● Matrix Subject (grammatical)○ Situation: Jeremy believes everyone should be able to attend college.○ * Who believes everyone should be able to attend college?

Hypothesis 1 :Ungrammatical structures will show an increase in acceptability after repeated exposure

○ Only one structure (adjunct islands) showed a significant increase in acceptability.

○ All four ungrammatical structures trended toward increases in acceptability

Hypothesis 2: Ungrammatical structures will show a decrease in confidence after repeated exposure

○ No structures showed significant decreases in confidence

○ All four ungrammatical structures trended toward decreases in confidence

Hypothesis 3: For ungrammatical structures, increases in acceptability will correlate with decreases in confidence

○ I found a small (r = -.20) but significant correlation between increases in acceptability and decreases in confidence

Implications

● Satiation effects do not invalidate informal methodology (at least in the short term)

● More work is needed to identify the particular structures subject to satiation

● The situation may be more worrisome in languages other than English

Thanks

● My primary reader: Kathie Galotti● My second readers: Roy Elveton and Cherlon Ussery● Dustin Chacón● My Fellow Majors: Ahmed Abdirahman, Morgan Ross, and Valerie Umscheid● My Participants● Pam Groves-Gaggioli● My friends and family

ReferencesAlmeida, D. (2014). Subliminal wh-islands in Brazilian Portuguese and the consequences for syntactic theory. Revista da ABRALIN, 13, 55-93.Casasanto, L. S., Hofmeister, P., & Sag, I. (2010). Understanding acceptability judgments: Additivity and working memory effects. Proceedings of the Annual Meeting of the Cognitive Science Society, 32, 224-229. Ceci, S. J., & Roazzi, A. (1994). The effects of context on cognition: Postcards from Brazil. In Stenberg, R. J. & Wagner, R. K. (Eds.), Mind in context: Interactionist perspectives on human intelligence, (74-101). Cambridge University Press.Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge: M.I.T. Press.Chomsky, N. (1975). The logical structure of linguistic theory. New York: Plenum.Culbertson, J., & Gross, S. (2009). Are linguists better subjects? The British Journal for the Philosophy of Science, 60, 721-736.Culicover, P., & Jackendoff, R. (2010). Quantitative methods alone are not enough: Response to Gibson and Fedorenko. Trends in Cognitive Sciences, 14, 234–235.Crawford, J. (2012) Using syntactic satiation to investigate subject islands. In Proceedings of the 29th West Coast Conference on Formal Linguistics, (38-45). Somerville, MA: Cascadilla Proceedings Project.Dąbrowska, E. (2010). Naive v. expert intuitions: An empirical study of acceptability judgments. The Linguistic Review, 27, 1-23.Edelman, S., & Christiansen, M. H. (2003). How seriously should we take minimalist syntax? Trends in Cognitive Sciences, 7, 60–61.Esposito, N. J. & Pelton, L. H. (1971). Review of the measurement of semantic satiation. Psychological Bulletin, 75, 330–346.Francom, J. (2009). Experimental syntax: Exploring the effect of repeated exposure to anomalous syntactic structure - evidence from rating and reading tasks (Doctoral dissertation). University of Arizona.Gibson, E., & Fedorenko, E. (2010). Weak quantitative standards in linguistics research. Trends in Cognitive Sciences, 14, 233–234.Gibson, E., & Fedorenko, E. (2013). The need for quantitative methods in syntax and semantics research. Language and Cognitive Processes, 28, 88-124.Gibson, E., Piantadosi, S. T., & Fedorenko, E. (2013). Quantitative methods in syntax/semantics research: A response to Sprouse and Almeida (2013). Language and Cognitive Processes, 28, 229-240.

References continuedGibson, E., & James, T. (1999). Memory limitations and structural forgetting: The perception of complex ungrammatical sentences as grammatical. Language and Cognitive Processes, 14, 225–248.Goodall, Grant. (2011). Syntactic satiation and the inversion effect in English and Spanish Wh-questions. Syntax: A Journal of Theoretical, Experimental and Interdisciplinary Research, 14, 29-47.Goodluck, H., & Rochemont, M. (1992). Island constraints: an introduction. In Goodluck, H., & Rochemont, M. (Eds.), Island constraints (1-33). Springer: Dordrecht.Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92, 373-385.Hiramatsu, K. (2000) Accessing linguistic competence: Evidence from children’s and adults’ acceptability judgments (Doctoral dissertation). University of Connecticut.IBM Corp. (2017). IBM SPSS Version 25 [Computer software]. Armonk, NY: IBM Corp.Levelt, W., van Gent, J., Haans, A., & Meijers, A. (1977). Grammaticality, paraphrase, and imagery. In Greenbaum, S. (ed.), Acceptability in language, 87–101. The Hague: Mouton.Linzen, T., & Oseki, Y. (2018). The reliability of acceptability judgments across languages. Glossa: a journal of general linguistics, 3, 100.Marantz, A. (2005). Generative linguistics within the cognitive neuroscience of language. The Linguistic Review, 22, 429–445.Nagata, Hiroshi. (1987a). Change in the modulus of judgmental scale: An inadequate explanation for the repetition effect in judgments of grammaticality. Perceptual and Motor Skills, 65, 907–910.Nagata, Hiroshi. (1987b). Long-term effect of repetition on judgments of grammaticality. Perceptual and Motor Skills, 65, 295–299.Nagata, Hiroshi. (1988). The relativity of linguistic intuition: The effect of repetition on grammaticality judgments. Journal of Psycholinguistic Research, 17, 1–17.Phillips, C. (2010). Should we impeach armchair linguists? Japanese/Korean Linguistics, 17, 49–64.

References continuedPhillips, C., & Lasnik, H. (2003). Linguistics and empirical evidence: Reply to Edelman and Christiansen. Trends in Cognitive Sciences, 7, 61–62.Richardson, J. (1996). Working memory and human cognition. New York: Oxford University Press.Qualtrics. (2018). Qualtrics [Computer software]. Provo, UT: Qualtrics.Ross, J. (1986). Infinite syntax. Norwood, N.J.: ABLEX.Schutze, C. T. (1996). The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. Chicago: University of Chicago Press.Snyder, W. (2000). An experimental investigation of syntactic satiation effects. Linguistic Inquiry, 31, 575-582.Sprouse, J. (2009). Revisiting satiation: Evidence for an equalization response strategy. Linguistic Inquiry, 40, 329-341.Sprouse, J. & Almeida, D. (2012). Assessing the reliability of textbook data in syntax: Adger's Core Syntax. Journal of Linguistics, 48, 609-652.Sprouse, J., Schütze, C. T., & Almeida, C. (2013). A comparison of informal and formal acceptability judgments using a random sample from linguistic inquiry 2001-2010. Lingua, 134, 219-248.Warner, J. & Glass, A. L. (1987). Context and distance-to-disambiguation effects in ambiguity resolution: Evidence from grammaticality judgments of garden path sentences. Journal of Memory and Language, 26, 714–738.Zervakis, J., & Mazuka, R. (2013). Effect of repeated evaluation and repeated exposure on acceptability ratings of sentences. Journal of Psycholinguistic Research, 42, 505-525

Questions?