Skip to content

Latest commit

 

History

History

megatron

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
data
data
 
 
fp16_deprecated
fp16_deprecated
 
 
fused_kernels
fused_kernels
 
 
model
model
 
 
mpu
mpu
 
 
optimizer
optimizer
 
 
text_generation
text_generation
 
 
tokenizer
tokenizer
 
 
README.md
README.md
 
 
__init__.py
__init__.py
 
 
arguments.py
arguments.py
 
 
checkpointing.py
checkpointing.py
 
 
dist_signal_handler.py
dist_signal_handler.py
 
 
global_vars.py
global_vars.py
 
 
indexer.py
indexer.py
 
 
initialize.py
initialize.py
 
 
learning_rates.py
learning_rates.py
 
 
memory.py
memory.py
 
 
microbatches.py
microbatches.py
 
 
p2p_communication.py
p2p_communication.py
 
 
preprocess_data.py
preprocess_data.py
 
 
schedules.py
schedules.py
 
 
text_generation_server.py
text_generation_server.py
 
 
training.py
training.py
 
 
utils.py
utils.py
 
 

README.md

Megatron-LM

Readme

Original Megatron-LM readme

Installation

cd ./data
make

Data Preprocessing

docs from Megatron-LM

changes in preprocess_data.py:

  • preprocess_data.py script is moved to megatron folder
  • supports tokenizers from HuggingFace Transformers
  • input can be a folder with multiple json/jsonl files

example usage with HF Tokenizer:

python preprocess_data.py \
       --input ./train \
       --output-prefix ./train \
       --dataset-impl mmap \
       --tokenizer-type HFTokenizer \
       --tokenizer-name-or-path bert-base-uncased \
       --split-sentences --workers 8