30/04 — Anaïs Tack — Predicting the difficulty of words for L2 readers: An empirical investigation into meaning-based and personalized word-level readability assessment with lexicon, statistical, and machine learning methods.
With vocabulary being one of the core aspects of successful reading comprehension in the foreign language (L2), a critical issue in educational technology development is to research methods that can ensure, for a given learner, the readability of the material at the word level. In the area of computational linguistics in particular, a recent number of studies have therefore developed various heuristics and models for predicting lexical difficulty in reading. However, there are two restrictions in the current methodology that might pose a limit on achieving more accurate and learner-tailored diagnostics. On the one hand, there is the issue of contextualization. Given that lexical difficulty is often estimated from measurements and rankings of words appearing in isolation, these estimates might not accurately inform us about the difficulty involved in reading words in context. On the other hand, there is the issue of personalization. Seeing that recent advances in machine learning require the availability of sizable data, studies often resort to aggregated and crowdsourced annotations to optimize the data collection process, which leads to a loss in valuable information on the variability in difficulty among learners. The aim of this presentation is therefore to look into the ways in which we could address these two limitations.
In the first part, I will give a systematic scoping review of previous studies that examined lexical competence in L2 reading and which predicted the effect of various factors on the construct measured as a dependent variable. The synthesis, which covers 125 publications and spans almost 50 years of research, aims to be methodological in nature in that it provides an overview of the types of measurements and predictors investigated to date. After having defined the scope of research for lexical competence in general, I will then zoom in on the construct of lexical difficulty in particular and briefly review how recent benchmarks have furthered the development of automated methods.
In the second part, I will compare two types of empirical measures of lexical difficulty. First, I will look at a priori knowledge of difficulty which can be drawn from reading material attested in textbooks and readers graded along the CEFR (Common European Framework of Reference) scale. As a follow-up on previous work on the use of CEFR-graded word frequencies for French L2 (Tack, François, Ligozat, & Fairon, 2016a, 2016b), I will investigate the added value of word-sense disambiguation (WSD) through the development of a similar resource for Dutch L2, viz. NT2Lex (Tack, François, Desmet, & Fairon, 2018). In particular, I will look at the link between WSD and semantic complexity measures such as hypernymy and contrast the distribution of cognates in the French and Dutch lexicons. Next, in order to account for a number of limitations in using this lexicon-based approach, the focus will be shifted towards using a posteriori knowledge of difficulty as measured in self-paced reading. Since the construct of difficulty can be defined and operationalized in various ways, I will concentrate on the use of subjective judgments to understand what triggers learners of French to notice difficulty while reading.
In the final part, I will discuss the use of statistical and machine learning methods to learn to predict diffculty from the previously collected learner data. In particular, two types of predictive analyses will be discussed. On the one hand, I will make use of Hierarchical Generalized Linear Mixed Models (HGLMMs) to integrate randomness at the subject level into a number of fixed effects selected from more than 200 features of lexical complexity. The results indicate that a small set of features are sufficient to explain the majority of the variance (.61 ≤ R2 ≤ .86). On the other hand, the results also corroborate findings that most of the variance in the complexity features can be accounted for by state-of-the-art 300-dimensional word embeddings. Enhancing deep neural networks with both contextualization and personalization significantly betters the discriminative
power as well as the correlation with learner judgments.
Date : Jeudi 30 avril 2020
14h00-15h00
Lieu : Conférence Teams