Palmyra-Med

Palmyra-Med: Instruction-Based Fine-Tuning of LLMs Enhancing Medical Domain Performance

FIG. 01 Start from general models → Build a medical instruction set → Fine-tune on the instructions → Evaluate on both benchmarks
Identifier
WRITER Research
Published
Jul 3, 2023
Authors
WRITER
Type
Paper
Area
Domain-specific specialization

In one paragraph

Palmyra 20B and 40B models fine-tuned on a custom 200,000-example medical instruction dataset. The 40B model reaches 81.1% on PubMedQA and 72.4% on MedQA.

PubMedQA (Palmyra-Med-40B)
81.1%
PubMedQA (Palmyra-Med-40B)
MedQA (Palmyra-Med-40B)
72.4%
MedQA (Palmyra-Med-40B)
medical instruction examples
200K
medical instruction examples

FIG. 02: Mechanism

How it works

  1. Start from general models

    Palmyra-20B and Palmyra-40B are general-purpose base models. Models like these are often unreliable on medical questions.

  2. Build a medical instruction set

    About 200,000 examples are drawn from PubMedQA (biomedical research questions) and MedQA (USMLE-style exam questions), each pairing an instruction and input with the expected answer.

  3. Fine-tune on the instructions

    Both models are trained on the set with AdamW, a warmup-decay learning-rate schedule and bf16 precision across multiple GPUs.

  4. Evaluate on both benchmarks

    Accuracy is measured on PubMedQA and MedQA. The 40B model scores 81.1% and 72.4%, the 20B model 75.6% and 44.6%.

FIG. 03: Interactive figure

Same question, before and after tuning

Switch instruction tuning on and off to see a base model continue the text while the tuned model answers, then compare the reported 20B and 40B scores.

Loading interactive demo…

FIG. 04: Notes

Summary

General-purpose models are often unreliable on medical questions. Palmyra-Med fine-tunes Palmyra-20B and Palmyra-40B on a custom dataset of 200,000 medical instructions drawn from PubMedQA (biomedical research questions) and MedQA (USMLE-style exam questions). Training used AdamW, a warmup-decay learning-rate schedule and bf16 precision across multiple GPUs.

Results

  • Palmyra-Med-20B: 75.6% on PubMedQA and 44.6% on MedQA.
  • Palmyra-Med-40B: 81.1% on PubMedQA and 72.4% on MedQA.

Both models outperform their base models and earlier domain-pretrained LLMs.

Reference