Mizzuki Research Featured Image

Content Validity of Japanese Language Proficiency Test

Executive summary

Led by Mizuki Mazzotta, a professor at Georgia State University, this study aims to investigate the content validity of the gap-filling (or rational deletion cloze) test included in the Japanese Language Proficiency Test (JLPT), a high stake standardized test offered by the Ministry of Education. Though JLPT contains gap-filling (or rational deletion cloze) test as a part of reading comprehension section, whether cloze tests measure only local linguistic knowledge or global level comprehension ability has not been understood clearly. The present study attempts to clarify this issue by qualitatively examining test-takers’ retrospective verbal reports.

Project Description

This study aims to investigate the content validity of the gap-filling (or rational deletion cloze) test that is a part of the reading comprehension section of the Japanese Language Proficiency Test (JLPT), a high stake standardized test offered by the Ministry of Education. Though JLPT contains gap-filling (or rational deletion cloze) test as a part of reading comprehension section, whether cloze tests measure only local linguistic knowledge or global level comprehension ability has not been understood clearly (Yamashita, 2003). Studies such as Chihara et al. (1977), Bachman (1985), Jonz (1990), and Chavez-Oller et al. (1994) showed that cloze test measured text-level processing ability. However, there are also studies which demonstrated that what cloze tests measure is local syntactic ability (e.g., Alderson, 1979; 1980; Shanahan et al., 1982). Conflicting results of previous studies may be explained partly by the fact that the information carried by a cloze item varies (e.g., Alderson, 1980; Bachman, 1982: 1985). Yamashita (2003:268) points out that to restore words like prepositions and idioms, only local linguistic knowledge is necessary while anaphora or conjunctions cannot be restored without textual understanding. For the purpose of revealing the difference between each cloze item, Bachman (1985) classified cloze items into four types: “(1) within clause; (2) across clause, within sentence; (3) across sentence, within text; (4) extra-textual” (modified from Yamashita 2003, p. 268). Although Bachman’s (1985) is useful, Yamashita (2003) argues that it is not guaranteed that cloze items hypothesized to measure test-takers’ text-level understanding do indeed measure such skill. To clarify this issue, Yamashita investigated EFL learners’ perspectives by examining their verbal reports while taking the gap-filling tests. Her qualitative analysis of the verbal protocol showed that test-takers used text-level information. This study continues this line of research by investigating the validity of a gap-filling test in Japanese, a language that has been under investigated. Research questions are as follows:
1. Do JLPT cloze items require text-level understanding according to the cloze item classification framework developed by Bachman (1985)?
2. What is the source of information that test-takers use to answer gap-filling test items? Do they use text-level information?

Method

Participants:Ten non-native speakers of Japanese will participate in this study.
Materials: This study will examine actual JLPT used in the past which is published as an official study guide. Levels N4 –N1 contain one set of gap-filling questions. Total of 3 sets of questions from Level N4 – N2 will be examined.
Procedure: Participants will take gap-filling test individually with the researcher. Immediately after a participant answers a question, researcher will ask the participant to say what was happening in his/her mind while answering the test questions.
Analysis: Using Bachman’s (1985) classification of cloze item types as a framework, the researcher will analyze if cloze items in JLPT (Levels N4-L2) require text-level understanding. The audio-recorded think-aloud protocol data will be analyzed qualitatively using Yamashita’s (2003) categorization framework.

Expected Outcomes and Impact

This project will directly influence Japanese language study in an academic environment. The results obtained from this study will illustrate the gap-filling test taking process of test takers. Such information is valuable because it will enable Japanese language teachers better prepare their students for JLPT. When the project is published, it may indirectly influence the publisher of JLPT to improve the content validity of its gap-filling tests.

Relevance to NFLRC Mission

This project is relevant to NFLRC mission because through investigating the content validity of JLPT, it aims to reveal the nonnative speakers’ test taking process. Dissemination of the results will inform Japanese language teachers on efficient test preparation methods.

Relevance to the Research Areas of the Proposed LRC at GSU

This project is relevant to the LRC at GSU because it provides significant focus on the needs of Japanese teachers and learners, a language that has been taught at GSU.

Post-Project Evaluation Plan and Criteria

The LRC at GSU will conduct an external evaluation of the following two objectives of this project. Student survey will be the primary measurement instrument used in the evaluation of Objective 1. Every year, the researcher will conduct a survey among students of Japanese to find out the number of students who obtained the JLPT certificate. Objective 2 will be evaluated by the successful publication of the project.
Objective 1: This project will increase the number of Japanese learners who receive the JLPT certificate of various levels.
Objective 2: This project will disseminate the information regarding the content validity of the JLPT reading section through publishing its result in a scholarly journal with an aim to contribute to the improvement of the content validity of the JLPT reading comprehension section.