Demonstrating proficiency in a foreign language can be crucial for the opportunity to study, work, or even obtain a residence permit in another country. Language tests can therefore have a major impact on people’s lives.
Today, several organizations responsible for standardized language tests already use automated assessment systems. However, the use of AI raises questions about fairness, transparency, and reliability.
In his dissertation, Ricardo Muñoz Sánchez examines how fair automated language assessments are, focusing on systems that evaluate texts written by people learning a foreign language.
“When language tests can have such significant consequences for individuals, we must be able to trust that the systems used are both reliable and fair,” says Ricardo Muñoz Sánchez.
What do AI systems actually assess?
The dissertation combines research on language models for the automated classification of essays with studies of bias and fairness in AI systems.
Ricardo has investigated how well the systems work, which linguistic features they focus on when evaluating texts, and whether personal information that students include in their essays can influence the results.
– I wanted to investigate not only how well the systems evaluate texts written by people learning a new language, but also what they base their evaluations on and whether information that should not be relevant can influence the results.
The research shows that the systems primarily focus on the linguistic features of the texts. At the same time, they still make too many errors to be considered sufficiently reliable in situations where the results can have major consequences for the individual.
Too Many Errors When the Stakes Are High
One of the dissertation’s main conclusions is that automated essay-grading systems are not yet reliable enough to be used in contexts where the results can be of decisive importance.
“These systems have potential, but they still make too many errors. When the result of a language test can affect someone’s ability to study, work, or live in another country, that is a serious problem.”
The results highlight the importance of critically evaluating automated assessment systems before they are implemented in contexts where reliability and fairness are crucial.
The dissertation also raises a broader question about the increasing use of AI in education: under what conditions, if any, can automated language assessment be considered fair?