LLMs for language teaching

On the application of Large Language Models for language teaching and assessment technology

Andrew Caines, Luca Benedetto, Shiva Taslimipoor, Christopher Davis, Yuan Gao, Øistein Andersen, Zheng Yuan, Mark Elliott, Russell Moore, Christopher Bryant, Marek Rei, Helen Yannakoudakis, Andrew Mullooly, Diane Nicholls, Paula Buttery

FIG. 01 Generate teaching content → Calibrate item difficulty → Score essays → Give feedback on errors → Weigh the risks
Identifier
arXiv:2307.08393
Published
Jul 17, 2023
Authors
15
Type
Paper
Area
Domain-specific specialization

In one paragraph

A review of where large language models fit in language teaching and assessment. They make content generation newly practical, but need careful prompting and editing. For essay grading and grammatical error correction they do not yet beat the state of the art on standard metrics, and established linguistic features still matter.

QWK for GPT-3.5 alone scoring TOEFL11 essays (cited result)
.388
QWK for GPT-3.5 alone scoring TOEFL11 essays (cited result)
QWK for GPT-3.5 combined with 45 linguistic features
.605
QWK for GPT-3.5 combined with 45 linguistic features

FIG. 02: Mechanism

How it works

  1. Generate teaching content

    An expert writes a prompt with the target genre and key content points. GPT-3 generates reading texts for B2-level multiple-choice questions, experts accept or reject them, and question writers edit them.

  2. Calibrate item difficulty

    Asked how hard a question is for learners at different CEFR levels, ChatGPT partly tells levels apart. It sometimes rated a question harder for C1 learners than for A1 learners.

  3. Score essays

    In cited work on 12,000 TOEFL11 essays, GPT-3.5 alone reached a quadratic weighted kappa of .388. Linguistic features beat it, and combining both gave .605. The authors propose chain-of-thought prompts to explain scores.

  4. Give feedback on errors

    On CoNLL-2014, JFLEG and BEA-2019, GPT models score below existing GEC systems because they over-correct. Human raters preferred their more fluent output, a style the standard metrics do not reward.

  5. Weigh the risks

    Bias towards English, representational and allocational bias, false information, privacy and climate cost. The authors ask for evidence that LLMs beat existing models, help learners and disadvantage no group.

FIG. 03: Interactive figure

What the LLM does, what a human still does

Switch between research areas to see where LLMs help and where they need checking, then turn off review steps, change a scorer's error or change a feedback style.

Loading interactive demo…

FIG. 04: Notes

Summary

The paper looks at how large language models such as PaLM and GPT-4 could be built into AI-driven language teaching and assessment systems. It covers:

  • content creation, including a human-in-the-loop study that uses GPT-3 to write B2-level reading texts
  • calibrating the difficulty of assessment items
  • automated essay scoring, where established linguistic features still add value on top of GPT
  • feedback to learners, including grammatical error correction and feedback comment generation
  • risks and ethics: bias, false information, privacy, over-reliance and environmental cost

The authors conclude that LLMs are promising but that the best systems may still need human intervention, linguistic features, or both. They describe LLMs as an evolutionary step in neural network models rather than a revolution.

Reference

Cite this work
@inproceedings{caines2023application,
  title={On the application of Large Language Models for language teaching and assessment technology},
  author={Andrew Caines and Luca Benedetto and Shiva Taslimipoor and Christopher Davis and Yuan Gao and {\O}istein Andersen and Zheng Yuan and Mark Elliott and Russell Moore and Christopher Bryant and Marek Rei and Helen Yannakoudakis and Andrew Mullooly and Diane Nicholls and Paula Buttery},
  booktitle={Proceedings of the Workshop on Empowering Education with LLMs at AIED 2023},
  year={2023},
  eprint={2307.08393},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}