et al. (2014) found stronger NS agreements with all their items showing at least 50% agreement among NS, and they assigned 2 points for test taker responses that were chosen by the largest NS group and 1 point for responses chosen by the next 2 largest groups, provided they were at least 10% of the NS sample. This scoring approach tried to take into account NS preference but the point distribution is essentially a tester decision with little empirical basis.
Finally, tests of sociopragmatics have often been designed contrastively for a pair of languages, for example, native Japanese speakers learning English (Hudson et al., 1995), native English speakers learning Japanese (Yamashita, 1996), native English speakers learning Korean (Ahn, 2005), or native Chinese speakers learning English (Liu, 2006). This necessarily lowers the practicality of tests, as well as the likelihood that they will eventually become part of large‐scale international test batteries (like TOEFL or IELTS). Roever (2005) did not limit his test taker population to a specific L1, and used differential item functioning to show that there were some L1 effects but that they were generally minor (Roever, 2007), indicating that limiting pragmatics tests to a specific population is not a necessity.
Conclusion
Tests of L2 pragmatics have seen a great deal of development and focused research in the last two decades. They offer a promising addition to the traditional language tests, which tend to focus on grammar, vocabulary, and skills. However, they pose significant challenges for test design if a complex construct like pragmatics is to be assessed comprehensively and reliably. There is still a great deal of research required.
SEE ALSO: Assessment of Speaking; Paired and Group Oral Assessment
References
1 Ahn, R. C. (2005). Five measures of interlanguage pragmatics in KFL (Korean as a foreign language) learners (Unpublished PhD thesis). University of Hawai'i at Manoa.
2 Austin, J. L. (1962). How to do things with words. Oxford, England: Oxford University Press.
3 Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford, England: Oxford University Press.
4 Bardovi‐Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? Pragmatic versus grammatical awareness in instructed L2 learning. TESOL Quarterly, 32, 233–62.
5 Bardovi‐Harlig, K., & Hartford, B. (1993). Learning the rules of academic talk: A longitudinal study of pragmatic development. Studies in Second Language Acquisition, 15, 279–304.
6 Bouton, L. F. (1988). A cross‐cultural study of ability to interpret implicatures in English. World Englishes, 17, 183–96.
7 Bouton, L. F. (1994). Conversational implicature in the second language: Learned slowly when not deliberately taught. Journal of Pragmatics, 22, 157–67.
8 Bouton, L. F. (1999). Developing non‐native speaker skills in interpreting conversational implicatures in English: Explicit teaching can ease the process. In E. Hinkel (Ed.), Culture in second language teaching and learning (pp. 47–70). Cambridge, England: Cambridge University Press.
9 Brown, J. D. (2001). Six types of pragmatics tests in two different contexts. In K. Rose & G. Kasper (Eds.), Pragmatics in language teaching (pp. 301–25). New York, NY: Cambridge University Press.
10 Brown, J. D. (2008). Raters, functions, item types and the dependability of L2 pragmatics tests. In E. Alcón Soler & A. Martínez‐Flor (Eds.), Investigating pragmatics in foreign language learning, teaching and testing (pp. 224–48). Clevedon, England: Multilingual Matters.
11 Brown, J. D., & Ahn, R. C. (2011). Variables that affect the dependability of L2 pragmatics tests. Journal of Pragmatics, 43(1), 198–217.
12 Brown, P., & Levinson, S. D. (1987). Politeness: Some universals in language usage. Cambridge, England: Cambridge University Press.
13 Byram, M. (1997). Teaching and assessing intercultural communicative competence. Philadelphia, PA: Multilingual Matters.
14 Canale, M. (1983). From communicative competence to communicative language pedagogy. In J. Richards & R. Schmidt (Eds.), Language and communication (pp. 2–27). London, England: Longman.
15 Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1, 1–47.
16 Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an argument‐based approach to validity make a difference? Educational Measurement: Issues and Practice, 29(1), 3–13.
17 Clift, R. (2016). Conversation analysis. Cambridge, England: Cambridge University Press.
18 Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge, England: Cambridge University Press.
19 Crystal, D. (1997). A dictionary of linguistics and phonetics. Oxford, England: Blackwell.
20 Ewald, J. D. (2012). “Can you tell me how to get there?”: Naturally occurring versus role‐play data in direction giving. Pragmatics, 22(1), 79–102.
21 Galaczi, E. (2014). Interactional competence across proficiency levels: How do learners manage interaction in paired speaking tests? Applied Linguistics, 35(5), 553–74.
22 Galaczi, E., & Taylor, L. (2018). Interactional competence: Conceptualisations, operationalisations, and outstanding questions. Language Assessment Quarterly, 15(3), 219–36.
23 Garfinkel, H. (1967). Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice Hall.
24 Golato, A. (2003). Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics, 24(1), 90–121.
25 Grabowski, K. (2009). Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking (Unpublished doctoral dissertation). Teachers College, Columbia University, New York.
26 Grabowski, K. (2013). Investigating the construct validity of a role‐play test designed to measure grammatical and pragmatic knowledge at multiple proficiency levels. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 149–71). New York, NY: Palgrave MacMillan.
27 Grice, H. P. (1975). Logic and conversation. In P. Cole & J. Morgan (Eds.), Syntax and semantics (Vol. 3, pp. 41–58). New York, NY: Academic Press.
28 Hall, J. K., & Pekarek Doehler, S. (2011). L2 interactional competence and development. In J. K. Hall, J. Hellermann, & S. Pekarek Doehler (Eds.), L2 interactional competence and development (pp. 1–15). Bristol, England: Multilingual Matters.
29 Heritage, J. (1984). Garfinkel and ethnomethodology. Cambridge, England: Polity.
30 Hudson, T., Detmer, E., & Brown, J. D. (1992). A framework for testing cross‐cultural pragmatics (Technical report #2). Honolulu: University of Hawai'i, Second Language Teaching and Curriculum Center.
31 Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing prototypic measures of cross‐cultural pragmatics (Technical report #7). Honolulu: University of Hawai'i, Second Language Teaching and Curriculum Center.
32 Ikeda, N. (2017). Measuring L2 oral pragmatic abilities for use in
