Palmyra-Med
Palmyra-Med: Instruction-Based Fine-Tuning of LLMs Enhancing Medical Domain Performance
- Identifier
- WRITER Research
- Published
- Jul 3, 2023
- Authors
- WRITER
- Type
- Paper
- Area
- Domain-specific specialization
In one paragraph
Palmyra 20B and 40B models fine-tuned on a custom 200,000-example medical instruction dataset. The 40B model reaches 81.1% on PubMedQA and 72.4% on MedQA.
- PubMedQA (Palmyra-Med-40B)
- 81.1%
- PubMedQA (Palmyra-Med-40B)
- MedQA (Palmyra-Med-40B)
- 72.4%
- MedQA (Palmyra-Med-40B)
- medical instruction examples
- 200K
- medical instruction examples
FIG. 02: Mechanism
How it works
Start from general models
Palmyra-20B and Palmyra-40B are general-purpose base models. Models like these are often unreliable on medical questions.
Build a medical instruction set
About 200,000 examples are drawn from PubMedQA (biomedical research questions) and MedQA (USMLE-style exam questions), each pairing an instruction and input with the expected answer.
Fine-tune on the instructions
Both models are trained on the set with AdamW, a warmup-decay learning-rate schedule and bf16 precision across multiple GPUs.
Evaluate on both benchmarks
Accuracy is measured on PubMedQA and MedQA. The 40B model scores 81.1% and 72.4%, the 20B model 75.6% and 44.6%.
FIG. 03: Interactive figure
Same question, before and after tuning
Switch instruction tuning on and off to see a base model continue the text while the tuned model answers, then compare the reported 20B and 40B scores.
Loading interactive demo…
FIG. 04: Notes
Summary
General-purpose models are often unreliable on medical questions. Palmyra-Med fine-tunes Palmyra-20B and Palmyra-40B on a custom dataset of 200,000 medical instructions drawn from PubMedQA (biomedical research questions) and MedQA (USMLE-style exam questions). Training used AdamW, a warmup-decay learning-rate schedule and bf16 precision across multiple GPUs.
Results
- Palmyra-Med-20B: 75.6% on PubMedQA and 44.6% on MedQA.
- Palmyra-Med-40B: 81.1% on PubMedQA and 72.4% on MedQA.
Both models outperform their base models and earlier domain-pretrained LLMs.
Reference