Testing has long been acknowledged as an essential part of language teaching and learning and plays an important role to evaluate the language proficiency. However, the construction of a test is both time-consuming and labor-intensive. With the progress of information technology, computer-assisted item generation (CAIG) has been received considerable attention in the area of computer assisted language learning (CALL). In this thesis, we introduce a system to automatically generate multiple-choice cloze items, targeting at testing learner’s proficiency on collocation usages. The system utilizes natural language processing (NLP) techniques to extract collocations from a large corpus such as British National Corpus in advance. Then, verb-noun collocations within a given text are identified, and verb collocates are selected and blanked out automatically. Finally, for each blanked-out collocates, three choices are automatically generated to distract the test taker. The evaluation results indicate that the proposed method performs satisfactorily, and the automatic generated items are comparable to human-prepared items.

語言測驗一直以來被公認為語言教學與學習必要的一部分，在評量語言能力時也很重要。然而，出題卻是非常耗費時間與勞力的工作。隨著電腦科技的進步，電腦輔助產生試題的研究(Computer Assisted Item Generation,CAIG) 逐漸在電腦輔助語言教學(Computer Assisted Language Learning, CALL)領域中受到關注。在本論文針對學習者對於搭配詞用法的測驗，發展了一個自動產生克漏字選擇題系統。此系統利用自然語言處理技術，預先從英國國家語料庫擷取搭配詞做為參考資料，以供在使用者輸入的文章中偵測出動詞名詞結構搭配詞。而被偵測出動詞名詞結構搭配詞組中動詞部分，在文章之中將自動被挖空形成為答案，另外也自動產生誘答選項來混淆學生。實驗評估結果顯示本方法的效果令人滿意，且自動產生的選項也相當於一般人工出題選項。我們期望此線上克漏字選擇題產生系統不僅可以輔助教師編製試題，亦能提供給學習者自我學習與練習之用。

Alderson, J.C., Clapham, C. & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.
Anderson, J.C. (2005). Diagnosing foreign language language proficiency. London:Contiuum.
Bonk, W. J. (2000). Testing ESL learners’ knowledge of collocations. (ERIC Document Reproduction Service No. ED 442309)
Brown, J.C., Friahkoff, G.A., & Eskenazi, M. (2005). Automatic Question Generation for Vocabulary Assessment. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Proceeding, 819-826. Vancouver, Canada.
Carey, L.M. (2001) Measuring and Evaluating School Learning (3rd Version). Boston: Allyn and Bacon.
Chan, T. P. (2004). Effects of CALL approaches on EFL college students’ learning of verb-noun collocations. Unpublished master’s thesis, National Tsing Hua University,Taiwan, R.O.C.
Chan, T. P. & Chang, Y. C. (2004) An Online Miscollocation Detection Aid for Chinese EFL Writers. Paper presented at the Thirteenth International Symposium and Book Fair on English Teaching, ETA, Taipei.
Chang, Y. C., Chen, T. P., & Chang, J. S. (2004) An Automatic Collocation Writing Assistant for Taiwanese EFL Learners: using corpora for language teaching and learning based on NLP Technology. Paper presented in EuroCALL 2004, European Association for Computer Assisted Language Learning, Vienna.
Chang, Y. C., Chang, J. S., Chen, H. J., & Liou, H. C. (2008) An automatic collocation writing assistant for Taiwanese EFL learners: a case of corpus-based NLP technology. Computer Assisted Language Learning, 21(3), 283–299.
Chapelle, C. (1998). Construct definition and validity inquiry in SLA research. In L. F. Bachman & A. D. Cohen (Eds.), Second language acquisition and language testing interfaces, (pp. 32-70). Cambridge: Cambridge University Press.
Chen. C.Y, Liou, H.C. & Chang, J. S. (2006). FAST – An automatic generation system for grammar Tests. In Proceedings of the COLING/ACL Interactive Presentation Sessions, 1-4. Sydney.
Chen, M.H. (2008). A Study of the English Collocation Competence of College Students in Taiwan. Unpublished master’s thesis, National Taiwan University of Science and Technolgy, Taiwan.
Chklovski, T. & Pantel, P. (2004). VerbOcean: Mining the Web for Fine-Grained Semantic Verb Relations. In Proceedings of Conference on Empirical Methods in Natural Language Processing. Barcelona, Spain.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
Coniam, D. (1997). A Preliminary Inquiry into Using Corpus Word Frequency Data in the Automatic Generation of English Cloze Tests. CALICO Journal, 14, 15-33.
Coniam, D. (1998). From Text to Test, Automatically – An Evaluation of a Computer Cloze Test Generator. Hong Kong Journal of Applied Linguistics, 3(1). 41-60.
ETS Taiwan Representative (Eds.). (2007). New TOEIC office Test-Preparation Guide(Bilingual version). Educational Testing Service.
Gao, Z.M. (2000). AWETS: An Automatic Web-Based English Testing System. In Proceedings of the 8th Conference on Computers on Education/International Conference on Computer-Assisted Instruction, 25-31.
Goto, T., Kojiri, T. Watanabe, T., Iwata, T. & Yamada, T. (2009). An Automatic generation of multiple-choice cloze questions based on statistical learning. In Proceedings of the 17th International Conference on Computer in Education.
Henriksen, B. (1999). Three dimensions of vocabulary development. Studies in Second Language Acquisition, 21, 303-317.
Hoshino, A. & Nakagawa, H. (2007). Assisting cloze test making with a web application. In Proceedings of the Society For Information Technology and Teacher Education International Conference, 2807-2814.
Irvine, S.H. & Kyllonen, P.C. (Eds.). (2002). Item Generation for testing development. Lawrence Erlbaum Associates.
Jian, J. Y., Chang, Y. C., & Chang, J. S. (2004) Collocational Translation Memory Extraction Based on Statistical Linguistic Information, Paper presented in ROCLING 2004, Conference on Computational Linguistics and Speech Processing, Taipei.
Kuder, G. F. & Richardson, M. W. (1937). The theory of estimation of test reliability. Psychometrika, 2(3), 151-160.
Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.
Lin, Y. P. (2001). The Effects of collocation instruction on English vocabulary development of senior high students in Taiwan. Unpublished master’s thesis, National Kaohsiung Normal University, Taiwan.
Linn, R.L. & Gronlund, N.E. (2000). Measurement and assessment in teaching (8th ed.). Upper Saddle River, NJ: Prentice Hall.
Liu, C.L., Wang, C.H., & Gao, Z.M. (2005). Using Lexical constraints to enhance the quality of Multiple-choice Cloze Items. International Journal of Computer Linguistics and Chinese Language Processing. 10(3), 303-328.
Liu, L. E. (2002). A corpus-based lexical semantic investigation of verb-noun miscollocations in Taiwan learners’ English. Unpublished master’s thesis, Tamkang University, Taipei, January.
Lord, F.M. (1952). The Relationship of the Reliability of Multiple-Choice Test to the Distribution of Item Difficulties, Psychometrika, 18, 181-194.
Miller, G. A. (1995). WordNet: A Lexical Database for English. Communications of the ACM (11).
Mitkov, R. & Ha, L.A. (2003) Computer-Aided Generation of Multiple-Choice Tests. In Proceedings of the HLT-NAACL 2003 Workshop on Building Educational Applications Using Natural Language Processing, 17-22.
Mitkov, R., Ha, L. A. & Karamanis, N. (2006) A computer-aided environment for generating multiple-choice test items. Natural Language Engineering. 12(2). 177-194. Cambridge University press.
Mitkov, R., Ha, L.A., Varga, A. & Rello, L. (2009). Semantic Similarity of Distractors in Multiple-Choice Tests: Extrinsic Evaluation. In Proceedings of EACL 2009 Workshop on GEometrical Models of Natural Language Semantics, 49-56. Athens, Greece.
Nation, I. S. P. (1990). Teaching and learning vocabulary. New York: Newbury House.
Nation, I. S. P. (2001). Learning vocabulary in another language. Cambridge: Cambridge Press.
Nesselhauf, N. (2003). The use of collocations by advanced learners of English and some implications for teaching. Applied Linguistics, 24, 223-242.
Richards, J.C. (1976). The role of vocabulary teaching. TESOL Quartly, 10, 77-89.
Qian, D.D. (1999). Assessing its role in adults’ reading comprehension in English as a second language. Canadian Modern Language Review, 56, 282-308.
Shei, C. C. (2001). Follow You! : An automatic language lesson generation system. Computer Assisted Language Learning, 14(2), 129-144.
Sim, J. & Wright C. C. (2005). The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, (85), 257–268.
Smith, S. Kilgarriff, A., Gong, W.L., Sommer, S. & Wu, G.Z. (2009). Automatic cloze generation for English Proficiency Testing. LTTC International Conference on English Language Teaching and Testing. Taipei.
Sumita, E., Sugaya,F. & Yamamoto, S.(2005). Measuring non-native speakers’ proficiency of English by using a test with automatically generated fill-in-blank questions. In Proceedings of the 2nd workshop on Building Educational Application Using NLP, 61-68.
Tang, Y. T. (2004). A Study of the collocation errors in the oral and written production of the college students in Taiwan. Unpublished master’s thesis, National Taiwan Normal University, Taiwan.
Thompson, B., & Levitov, J.E. (1985). Using microcomputers to score and evaluate test items. Collegiate Microcomputer, 3, 163-168.
Wang, H. C. (2005). The relationship between EFL learners’ depth of vocabulary knowledgeand oral collocational errors. Unpublished master’s thesis, National Kaohsiung Normal University, Taiwan.