-
Notifications
You must be signed in to change notification settings - Fork 297
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding Masked Language Modelling (#1030)
* misc run scripts * sbatch * sweep scripts * update * qa * update * update * update * update * update * sb file * moving update_metrics to outside scope of dataparallel * fixing micro_avg calculation * undo debugging * Fixing tests, moving update_metrics out of other tasks * remove extraneous change * MLM task * Added MLM task * update * fix multiple choice dataparallel forward * update * add _mask_id to transformers * Update * MLM update * adding update_metrics abstraction * delete update_metrics_ notation * fixed wrong index problem * removed unrelated files * removed unrelated files * removed unrelated files * fix PEP8 * Fixed get_pretained_lm_head for BERT and ALBERT * spelling check * black formatting * fixing tests * bug fix * Adding batch_size constraints to multi-GPU setting * adding documentation * adding batch size test * black correct version * Fixing batch size assertion * generalize batch size assertion for more than 2 GPU setting * reducing label loops in code * fixing span forward * Fixing span prediction forward for multi-GPU * fix commonsenseQA forward * MLM * adding function documentation * resolving nits, fixing seq_gen forward * remove nit * fixing batch_size assert and SpanPrediction task * Remove debugging * Fix batch size mismatch multi-GPU test * Fix order of assert checking for batch size mismatch * mlm training * update * sbatch * update * data parallel * update data parallel stuffs * using sequencelabel, using 1 paragraph per example * update label mapping * adding exmaples-porportion-mixing * changing dataloader to work with wikitext103 * weight sampling * add early stopping only onb one task * commit * Cleaning up code * Removing unecessarily tracked git folders * Removing unnecesary changes * revert README * revert README.md again * Making more general for Transformer-based embedders * torch.uint8 -> torch.bool * Fixing indexing issues * get rid of unecessary changes * black cleanup * update * Prevent updating update_metrics twice in one step * update * update * add base_roberta * update * reverting CCG edit added for debugging * refactor defaults.conf * black formatting * merge * removed SOP task and mlm_manual_scaling * Fixing label namespace vocabulary creation, mergeing from master * Deleting MLM weight * black formatting * Adding early_stopping_method to defaults.conf * Fixing MLM with preprocessed wikitext103 * Deleting intermediate class hierarchy for MLM * Correcting black * LanguageModelingTask -> AutoregressiveModelingTask * code style * fixing MaskedLanguageModelTask * Fixing typo * Fixing label namespace * extracting out masking portion * Revert "extracting out masking portion" This reverts commit f21165c. * Code cleanup * Adding tests for early_stpping_method * Adding pretrain_stop_metric * Reverting get_data_iter * Reverting to get_data_iter * Fixing get_pretrained_lm_head for all embedder types * Extracting out MLM probability masking * Move dynamic masking function to Task for easier testing * Adding unit tests for MLM * Adding change to MLM forward function to expose more intermediate steps for testing * Fixing code style * Adding more detailed instructions of how to generate Wikipedia data * Adding rest of MLM data generation code * Black style and remove comment * black style * updating repro code for MLM data Co-authored-by: phu-pmh <[email protected]> Co-authored-by: Haokun Liu <[email protected]> Co-authored-by: pruksmhc <[email protected]> Co-authored-by: DeepLearning VM <[email protected]>
- Loading branch information
1 parent
c975afa
commit c87a86b
Showing
15 changed files
with
545 additions
and
39 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
// Base config file for mlm experiments wit roberta | ||
include "defaults.conf" | ||
|
||
early_stopping_method=auto // Early stopping method. Options: task_name to only do early stopping based | ||
// on a specific task, 'auto': use the macro_avg | ||
|
||
// Multi-task Training | ||
weighting_method = proportional // Weighting method for task sampling, relative to the number of | ||
// training examples in each task: | ||
// Options: uniform, power_<power>, softmax_<temp> | ||
// proportional, proportional_log_batch, and | ||
// proportional_log_example (plus the less-useful inverse, | ||
// inverse_log_example, and inverse_log_batch). | ||
// Additionally, we include the T5 method of examples-proportional-mixing. | ||
// See relevant source code for details. | ||
scaling_method = uniform // Method for scaling loss: | ||
// Options: uniform, max_power_<power>, max_proportional, | ||
// max_proportional_log, max_inverse, max_inverse_log | ||
// max_epoch_<E1_E2_..._En> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.