LLMs for language teaching
On the application of Large Language Models for language teaching and assessment technology
- Identifier
- arXiv:2307.08393
- Published
- Jul 17, 2023
- Authors
- 15
- Type
- Paper
- Area
- Domain-specific specialization
In one paragraph
A review of where large language models fit in language teaching and assessment. They make content generation newly practical, but need careful prompting and editing. For essay grading and grammatical error correction they do not yet beat the state of the art on standard metrics, and established linguistic features still matter.
- QWK for GPT-3.5 alone scoring TOEFL11 essays (cited result)
- .388
- QWK for GPT-3.5 alone scoring TOEFL11 essays (cited result)
- QWK for GPT-3.5 combined with 45 linguistic features
- .605
- QWK for GPT-3.5 combined with 45 linguistic features
FIG. 02: Mechanism
How it works
Generate teaching content
An expert writes a prompt with the target genre and key content points. GPT-3 generates reading texts for B2-level multiple-choice questions, experts accept or reject them, and question writers edit them.
Calibrate item difficulty
Asked how hard a question is for learners at different CEFR levels, ChatGPT partly tells levels apart. It sometimes rated a question harder for C1 learners than for A1 learners.
Score essays
In cited work on 12,000 TOEFL11 essays, GPT-3.5 alone reached a quadratic weighted kappa of .388. Linguistic features beat it, and combining both gave .605. The authors propose chain-of-thought prompts to explain scores.
Give feedback on errors
On CoNLL-2014, JFLEG and BEA-2019, GPT models score below existing GEC systems because they over-correct. Human raters preferred their more fluent output, a style the standard metrics do not reward.
Weigh the risks
Bias towards English, representational and allocational bias, false information, privacy and climate cost. The authors ask for evidence that LLMs beat existing models, help learners and disadvantage no group.
FIG. 03: Interactive figure
What the LLM does, what a human still does
Switch between research areas to see where LLMs help and where they need checking, then turn off review steps, change a scorer's error or change a feedback style.
Loading interactive demo…
FIG. 04: Notes
Summary
The paper looks at how large language models such as PaLM and GPT-4 could be built into AI-driven language teaching and assessment systems. It covers:
- content creation, including a human-in-the-loop study that uses GPT-3 to write B2-level reading texts
- calibrating the difficulty of assessment items
- automated essay scoring, where established linguistic features still add value on top of GPT
- feedback to learners, including grammatical error correction and feedback comment generation
- risks and ethics: bias, false information, privacy, over-reliance and environmental cost
The authors conclude that LLMs are promising but that the best systems may still need human intervention, linguistic features, or both. They describe LLMs as an evolutionary step in neural network models rather than a revolution.
Reference
@inproceedings{caines2023application,
title={On the application of Large Language Models for language teaching and assessment technology},
author={Andrew Caines and Luca Benedetto and Shiva Taslimipoor and Christopher Davis and Yuan Gao and {\O}istein Andersen and Zheng Yuan and Mark Elliott and Russell Moore and Christopher Bryant and Marek Rei and Helen Yannakoudakis and Andrew Mullooly and Diane Nicholls and Paula Buttery},
booktitle={Proceedings of the Workshop on Empowering Education with LLMs at AIED 2023},
year={2023},
eprint={2307.08393},
archivePrefix={arXiv},
primaryClass={cs.CL}
}