Code-switched GEC
Grammatical Error Correction for Code-Switched Sentences by Learners of English
- Identifier
- arXiv:2404.12489
- Published
- Apr 18, 2024
- Authors
- 5
- Type
- Paper
- Area
- Enterprise-optimized models
In one paragraph
Learners often mix English with their first language, and GEC systems trained on English alone miss errors in those sentences. The paper builds synthetic code-switched training data by translating spans of existing GEC corpora, compares six ways to pick the span, and finds that switching a single noun works best: +1.57 F0.5 on average across English-Chinese, English-Korean and English-Japanese test sets, with no loss on monolingual English.
- mean F0.5 gain across 5 code-switched test sets
- +1.57
- mean F0.5 gain across 5 code-switched test sets
- BEA-19 Dev F0.5 of the EN-ZH model, vs 51.55 baseline
- 52.42
- BEA-19 Dev F0.5 of the EN-ZH model, vs 51.55 baseline
- natural code-switched Lang-8 test sentences (ZH, KO, JA)
- 5,773
- natural code-switched Lang-8 test sentences (ZH, KO, JA)
FIG. 02: Mechanism
How it works
Start from a GEC pair
Take a learner sentence and its correction from an existing English GEC corpus such as W&I+LOCNESS, FCE, NUCLE or Lang-8.
Select a span
Pick part of the corrected sentence with one of six methods: random tokens up to a 20% code-switch ratio, one contiguous run, a random parsed phrase, the phrase closest to the target length, the phrase overlapping the fewest errors, or a single noun.
Translate it
The span is translated with Google Translate and tokenised with Jieba, Komoran or Nagisa for Chinese, Korean or Japanese.
Reapply the errors
The learner's original errors are put back into the code-switched sentence. Errors that overlap the translated span are discarded.
Train and test
GECToR with an XLM-RoBERTa encoder is trained on the synthetic data and tested on new code-switched test sets drawn from Lang-8, including a re-annotated subset. Switching one noun at all three training stages gave the best results.
FIG. 03: Interactive figure
Pick a span, translate it, keep the errors
Choose a sentence, a language pair and a span-selection method to see which words get translated and which learner errors survive, then compare each method's reported F0.5.
Loading interactive demo…
FIG. 04: Notes
Summary
Code-switching, using more than one language in a single sentence, is common among language learners. Most grammatical error correction (GEC) systems are trained on monolingual English and do not expect non-English words. In the paper’s example, a system restores the missing verb in “But the pay a little low” but not once “pay” is written in Korean.
There was no annotated code-switched GEC data to train or test on, so the paper:
- generates synthetic training data by translating spans of the corrected side of existing GEC corpora and then reapplying the original errors
- compares six span-selection methods, from random tokens to linguistically motivated choices such as whole phrases or single nouns
- builds test sets of 201 English-Chinese, 764 English-Korean and 4,808 English-Japanese code-switched sentences from Lang-8, plus 376 re-annotated by bilingual annotators
Switching a single noun, the most common pattern in natural code-switching, was the most reliable method. Applied to all three GECToR training stages, it raised F0.5 by 1.57 on average across the five code-switched test sets. Scores on the monolingual BEA-19 development set rose slightly rather than fell. Models trained on one language pair also improved on the others, and the EN-KO model edged out the EN-JA model on Japanese by 0.10 F0.5.
Reference
@inproceedings{chan2024grammatical,
title={Grammatical Error Correction for Code-Switched Sentences by Learners of English},
author={Kelvin Wey Han Chan and Christopher Bryant and Li Nguyen and Andrew Caines and Zheng Yuan},
booktitle={Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
year={2024},
eprint={2404.12489},
archivePrefix={arXiv}
}