Prompting LLMs for GEC

Prompting open-source and commercial language models for grammatical error correction of English learner text

Christopher Davis, Andrew Caines, Øistein Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, Paula Buttery

FIG. 01 Pick models and benchmarks → Filter the prompts → Add few-shot examples → Score against references → Break it down by error type
Identifier
arXiv:2401.07702
Published
Jan 15, 2024
Authors
9
Type
Paper
Area
Practical evaluations

In one paragraph

Seven open-source and three commercial LLMs, including InstructPalmyra-20B, are prompted to correct English learner text on four benchmarks. They fall well short of supervised systems on minimal-edit benchmarks and come close on JFLEG, which rewards fluency. Open-source models match or beat the commercial ones on minimal edits, and zero-shot prompts often do as well as few-shot.

best LLM F0.5 on FCE test (FLAN-T5), vs 0.626 supervised
0.463
best LLM F0.5 on FCE test (FLAN-T5), vs 0.626 supervised
CoNLL-14 F0.5: Stable Beluga 2 ties GPT-3.5 Turbo, vs 0.696 supervised
0.572
CoNLL-14 F0.5: Stable Beluga 2 ties GPT-3.5 Turbo, vs 0.696 supervised
JFLEG GLEU for GPT-3.5 Turbo, vs 0.647 supervised
0.625
JFLEG GLEU for GPT-3.5 Turbo, vs 0.647 supervised
models do best zero-shot on FCE dev
6 of 10
models do best zero-shot on FCE dev

FIG. 02: Mechanism

How it works

  1. Pick models and benchmarks

    Seven instruction-tuned open-source models from 7B to 70B parameters, plus Command, GPT-3.5 Turbo and GPT-4, are tested on FCE, CoNLL-14 and W&I+LOCNESS, which use minimal-edit references, and JFLEG, which uses fluency rewrites.

  2. Filter the prompts

    Eleven zero-shot prompts are tried on 2,000 sentences sampled from FCE and W&I+LOCNESS training data. The seven best go forward. Two give the model a role, an English teacher or a GEC tool, and one is the prompt from Coyne et al. (2023).

  3. Add few-shot examples

    The three best prompts get 1 to 4 fixed learner examples, the last of which needs no correction, giving 12 few-shot prompts. Each model's best prompt on the dev set is then run on the test set.

  4. Score against references

    FCE and W&I+LOCNESS are scored with ERRANT F0.5, CoNLL-14 with the M2 scorer and JFLEG with GLEU. Results are compared with the best single supervised system for each test set.

  5. Break it down by error type

    On W&I+LOCNESS dev, the models handle spelling, missing determiners, agreement and noun number well, and struggle with open-class noun and verb replacements, which need larger lexical changes.

FIG. 03: Interactive figure

Same prompts, two kinds of reference

Switch between minimal-edit and fluency benchmarks to see where prompted LLMs land against supervised systems, then pick a model to read the exact prompt it was given and add few-shot examples.

Loading interactive demo…

FIG. 04: Notes

Summary

Earlier work found that GPT models write fluent corrections but score poorly on grammatical error correction (GEC) benchmarks that reward minimal edits. This paper tests whether careful prompting fixes that, and whether open-source models can compete with commercial ones.

  • Supervised systems still win on minimal edits. The best prompted model reaches 0.463 F0.5 on FCE test and 0.572 on CoNLL-14, against 0.626 and 0.696 for the best supervised systems. On JFLEG, GPT-3.5 Turbo reaches 0.625 GLEU against 0.647.
  • Open-source models hold their own. FLAN-T5 is the best model on FCE test, and Stable Beluga 2 ties GPT-3.5 Turbo on CoNLL-14. GPT-4, run zero-shot on the dev sets only, scores highest on every dev set.
  • Few-shot is not reliably better. On FCE dev, six of ten models do best with a zero-shot prompt. Few-shot helps most models on W&I+LOCNESS dev, and results are mixed on JFLEG.
  • Role prompts work well. The best prompts either cast the model as an English teacher or a GEC tool, or are Coyne et al.’s prompt, which the authors suggest as a strong baseline.
  • Some error types stay hard. Corrections close to the original form, such as spelling and noun number, are handled well. Lexical and phrasal replacements are not.

Reference

Cite this work
@inproceedings{davis2024prompting,
  title={Prompting open-source and commercial language models for grammatical error correction of English learner text},
  author={Christopher Davis and Andrew Caines and {\O}istein Andersen and Shiva Taslimipoor and Helen Yannakoudakis and Zheng Yuan and Christopher Bryant and Marek Rei and Paula Buttery},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2024},
  year={2024},
  eprint={2401.07702},
  archivePrefix={arXiv}
}