Code-switched GEC

Grammatical Error Correction for Code-Switched Sentences by Learners of English

Kelvin Wey Han Chan, Christopher Bryant, Li Nguyen, Andrew Caines, Zheng Yuan

FIG. 01 Start from a GEC pair → Select a span → Translate it → Reapply the errors → Train and test
Identifier
arXiv:2404.12489
Published
Apr 18, 2024
Authors
5
Type
Paper
Area
Enterprise-optimized models

In one paragraph

Learners often mix English with their first language, and GEC systems trained on English alone miss errors in those sentences. The paper builds synthetic code-switched training data by translating spans of existing GEC corpora, compares six ways to pick the span, and finds that switching a single noun works best: +1.57 F0.5 on average across English-Chinese, English-Korean and English-Japanese test sets, with no loss on monolingual English.

mean F0.5 gain across 5 code-switched test sets
+1.57
mean F0.5 gain across 5 code-switched test sets
BEA-19 Dev F0.5 of the EN-ZH model, vs 51.55 baseline
52.42
BEA-19 Dev F0.5 of the EN-ZH model, vs 51.55 baseline
natural code-switched Lang-8 test sentences (ZH, KO, JA)
5,773
natural code-switched Lang-8 test sentences (ZH, KO, JA)

FIG. 02: Mechanism

How it works

  1. Start from a GEC pair

    Take a learner sentence and its correction from an existing English GEC corpus such as W&I+LOCNESS, FCE, NUCLE or Lang-8.

  2. Select a span

    Pick part of the corrected sentence with one of six methods: random tokens up to a 20% code-switch ratio, one contiguous run, a random parsed phrase, the phrase closest to the target length, the phrase overlapping the fewest errors, or a single noun.

  3. Translate it

    The span is translated with Google Translate and tokenised with Jieba, Komoran or Nagisa for Chinese, Korean or Japanese.

  4. Reapply the errors

    The learner's original errors are put back into the code-switched sentence. Errors that overlap the translated span are discarded.

  5. Train and test

    GECToR with an XLM-RoBERTa encoder is trained on the synthetic data and tested on new code-switched test sets drawn from Lang-8, including a re-annotated subset. Switching one noun at all three training stages gave the best results.

FIG. 03: Interactive figure

Pick a span, translate it, keep the errors

Choose a sentence, a language pair and a span-selection method to see which words get translated and which learner errors survive, then compare each method's reported F0.5.

Loading interactive demo…

FIG. 04: Notes

Summary

Code-switching, using more than one language in a single sentence, is common among language learners. Most grammatical error correction (GEC) systems are trained on monolingual English and do not expect non-English words. In the paper’s example, a system restores the missing verb in “But the pay a little low” but not once “pay” is written in Korean.

There was no annotated code-switched GEC data to train or test on, so the paper:

  • generates synthetic training data by translating spans of the corrected side of existing GEC corpora and then reapplying the original errors
  • compares six span-selection methods, from random tokens to linguistically motivated choices such as whole phrases or single nouns
  • builds test sets of 201 English-Chinese, 764 English-Korean and 4,808 English-Japanese code-switched sentences from Lang-8, plus 376 re-annotated by bilingual annotators

Switching a single noun, the most common pattern in natural code-switching, was the most reliable method. Applied to all three GECToR training stages, it raised F0.5 by 1.57 on average across the five code-switched test sets. Scores on the monolingual BEA-19 development set rose slightly rather than fell. Models trained on one language pair also improved on the others, and the EN-KO model edged out the EN-JA model on Japanese by 0.10 F0.5.

Reference

Cite this work
@inproceedings{chan2024grammatical,
  title={Grammatical Error Correction for Code-Switched Sentences by Learners of English},
  author={Kelvin Wey Han Chan and Christopher Bryant and Li Nguyen and Andrew Caines and Zheng Yuan},
  booktitle={Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year={2024},
  eprint={2404.12489},
  archivePrefix={arXiv}
}