Prompting LLMs for GEC
Prompting open-source and commercial language models for grammatical error correction of English learner text
- Identifier
- arXiv:2401.07702
- Published
- Jan 15, 2024
- Authors
- 9
- Type
- Paper
- Area
- Practical evaluations
In one paragraph
Seven open-source and three commercial LLMs, including InstructPalmyra-20B, are prompted to correct English learner text on four benchmarks. They fall well short of supervised systems on minimal-edit benchmarks and come close on JFLEG, which rewards fluency. Open-source models match or beat the commercial ones on minimal edits, and zero-shot prompts often do as well as few-shot.
- best LLM F0.5 on FCE test (FLAN-T5), vs 0.626 supervised
- 0.463
- best LLM F0.5 on FCE test (FLAN-T5), vs 0.626 supervised
- CoNLL-14 F0.5: Stable Beluga 2 ties GPT-3.5 Turbo, vs 0.696 supervised
- 0.572
- CoNLL-14 F0.5: Stable Beluga 2 ties GPT-3.5 Turbo, vs 0.696 supervised
- JFLEG GLEU for GPT-3.5 Turbo, vs 0.647 supervised
- 0.625
- JFLEG GLEU for GPT-3.5 Turbo, vs 0.647 supervised
- models do best zero-shot on FCE dev
- 6 of 10
- models do best zero-shot on FCE dev
FIG. 02: Mechanism
How it works
Pick models and benchmarks
Seven instruction-tuned open-source models from 7B to 70B parameters, plus Command, GPT-3.5 Turbo and GPT-4, are tested on FCE, CoNLL-14 and W&I+LOCNESS, which use minimal-edit references, and JFLEG, which uses fluency rewrites.
Filter the prompts
Eleven zero-shot prompts are tried on 2,000 sentences sampled from FCE and W&I+LOCNESS training data. The seven best go forward. Two give the model a role, an English teacher or a GEC tool, and one is the prompt from Coyne et al. (2023).
Add few-shot examples
The three best prompts get 1 to 4 fixed learner examples, the last of which needs no correction, giving 12 few-shot prompts. Each model's best prompt on the dev set is then run on the test set.
Score against references
FCE and W&I+LOCNESS are scored with ERRANT F0.5, CoNLL-14 with the M2 scorer and JFLEG with GLEU. Results are compared with the best single supervised system for each test set.
Break it down by error type
On W&I+LOCNESS dev, the models handle spelling, missing determiners, agreement and noun number well, and struggle with open-class noun and verb replacements, which need larger lexical changes.
FIG. 03: Interactive figure
Same prompts, two kinds of reference
Switch between minimal-edit and fluency benchmarks to see where prompted LLMs land against supervised systems, then pick a model to read the exact prompt it was given and add few-shot examples.
Loading interactive demo…
FIG. 04: Notes
Summary
Earlier work found that GPT models write fluent corrections but score poorly on grammatical error correction (GEC) benchmarks that reward minimal edits. This paper tests whether careful prompting fixes that, and whether open-source models can compete with commercial ones.
- Supervised systems still win on minimal edits. The best prompted model reaches 0.463 F0.5 on FCE test and 0.572 on CoNLL-14, against 0.626 and 0.696 for the best supervised systems. On JFLEG, GPT-3.5 Turbo reaches 0.625 GLEU against 0.647.
- Open-source models hold their own. FLAN-T5 is the best model on FCE test, and Stable Beluga 2 ties GPT-3.5 Turbo on CoNLL-14. GPT-4, run zero-shot on the dev sets only, scores highest on every dev set.
- Few-shot is not reliably better. On FCE dev, six of ten models do best with a zero-shot prompt. Few-shot helps most models on W&I+LOCNESS dev, and results are mixed on JFLEG.
- Role prompts work well. The best prompts either cast the model as an English teacher or a GEC tool, or are Coyne et al.’s prompt, which the authors suggest as a strong baseline.
- Some error types stay hard. Corrections close to the original form, such as spelling and noun number, are handled well. Lexical and phrasal replacements are not.
Reference
@inproceedings{davis2024prompting,
title={Prompting open-source and commercial language models for grammatical error correction of English learner text},
author={Christopher Davis and Andrew Caines and {\O}istein Andersen and Shiva Taslimipoor and Helen Yannakoudakis and Zheng Yuan and Christopher Bryant and Marek Rei and Paula Buttery},
booktitle={Findings of the Association for Computational Linguistics: ACL 2024},
year={2024},
eprint={2401.07702},
archivePrefix={arXiv}
}