This repository provides codes and models of the Multi-Task Learning on BERT for Biomedical Text Mining.
The package is based on mt-dnn
.
The pre-trained MT-BlueBERT weights, vocab, and config files can be downloaded from:
The benchmark datasets can be downloaded from https://github.com/ncbi-nlp/BLUE_Benchmark
- python3.6
- install requirements
pip install -r requirements.txt
Please refer to download BLUE_Benchmark: https://github.com/ncbi-nlp/BLUE_Benchmark
bash ncbi_scripts/blue_prepro.sh
bash ncbi_scripts/run_blue_mt_dnn.sh
bash ncbi_scripts/run_blue_fine_tune.sh
python ncbi_scripts/convert_tf_to_pt.py --tf_checkpoint_root $SRC_ROOT --pytorch_checkpoint_path $DEST --encoder_type 1```
Peng Y, Chen Q, Lu Z. An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining. In Proceedings of the 2020 Workshop on Biomedical Natural Language Processing (BioNLP 2020). 2020.
@InProceedings{peng2019transfer,
author = {Yifan Peng and Qingyu Chen and Zhiyong Lu},
title = {An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining},
booktitle = {Proceedings of the 2020 Workshop on Biomedical Natural Language Processing (BioNLP 2020)},
year = {2020},
}
This work was supported by the Intramural Research Programs of the National Institutes of Health, National Library of Medicine. This work was supported by the National Library of Medicine of the National Institutes of Health under award number K99LM013001-01.
We are also grateful to the authors of BERT and mt-dnn to make the data and codes publicly available.
This tool shows the results of research conducted in the Computational Biology Branch, NLM/NCBI. The information produced on this website is not intended for direct diagnostic use or medical decision-making without review and oversight by a clinical professional. Individuals should not change their health behavior solely on the basis of information produced on this website. NIH does not independently verify the validity or utility of the information produced by this tool. If you have questions about the information produced on this website, please see a health care professional. More information about NLM/NCBI's disclaimer policy is available.