Finetune flag #540
Finetune flag #540
Conversation
HaokunLiu
requested review from
iftenney,
sleepinyourhat and
W4ngatang
as code owners
There was a problem hiding this comment.
This looks reasonable to me, but I'd prefer that @pruksmhc or @W4ngatang be the main reviewers on this. It might help to add a comment to defaults clarifying what counts as a pretrained model—this is for GPT/BERT/ELMo, not for smaller models that we train here.
|
Yep this looks good, thanks for tackling this Haokun! Do some runs to test that this works. |
Does it look better now? |
Tested, looks reasonable. |
config/defaults.conf
Outdated
| @@ -70,8 +70,11 @@ do_full_eval = 1 // Evaluate the model on the tasks on target_tasks. | |||
|
|
|||
| // Related configuration | |||
| load_model = 1 // If true, restore from checkpoint when starting do_pretrain. No impact on do_target_task_training. | |||
| transfer_paradigm = "frozen" // How to do learning on top of the pretrained model | |||
| transfer_paradigm = "frozen" // How to do learning on top of the task-independently pretrained model | |||
| // Note: this option only works for models like ELMO/GPT/BERT (models that was pretrained on large amount of data) | |||
There was a problem hiding this comment.
why wouldn't these apply if we're using LSTMs or some non pre-trained models?
There was a problem hiding this comment.
you are right, I don't what I was thinking.
|
Now there are merge conflicts—dang. Mind fixing those, then we can merge? |
* deprive bert_fine_tune and openai_transformer_fine_tune * update config and readme * add explaination to transfer_paradigm * fix the comment
Removed redundant bert_fine_tune and openai_transformer_fine_tune flags. transfer_paradigm now controls whether those parameters in bert and gpt are trainable.
#505