Fix issue#85

README.md

@@ -20,6 +20,13 @@ FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensibl
 
 The code is partially based on [GLM](https://github.com/THUDM/GLM), [Transformers](https://github.com/huggingface/transformers), [timm](https://github.com/rwightman/pytorch-image-models) and [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/Megatron-LM).
 
+## News
+- [29 Aug 2022] release v1.3.0, Added CLIP module and redesigned tokenizer apis in [#81](https://github.com/FlagAI-Open/FlagAI/pull/81)
+- [21 Jul 2022] release v1.2.0, ViTs are supported in [#71](https://github.com/FlagAI-Open/FlagAI/pull/71)
+- [29 Jun 2022] release v1.1.0, support OPTs downloading and inference/finetuning [#63](https://github.com/FlagAI-Open/FlagAI/pull/63)
+- [17 May 2022] made our first contribution in [#1](https://github.com/FlagAI-Open/FlagAI/pull/1)
+
+--------------------------------------------------------------------------------
 
 <!-- toc -->
 

README_zh.md

@@ -20,6 +20,13 @@
 
 本项目的部分代码基于 [GLM](https://github.com/THUDM/GLM),[Transformers](https://github.com/huggingface/transformers)，[timm](https://github.com/rwightman/pytorch-image-models) 和 [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/Megatron-LM).
 
+## 动态
+- [29 Aug 2022] release v1.3.0, Added CLIP module and redesigned tokenizer apis in [#81](https://github.com/FlagAI-Open/FlagAI/pull/81)
+- [21 Jul 2022] release v1.2.0, ViTs are supported in [#71](https://github.com/FlagAI-Open/FlagAI/pull/71)
+- [29 Jun 2022] release v1.1.0, support OPTs downloading and inference/finetuning [#63](https://github.com/FlagAI-Open/FlagAI/pull/63)
+- [17 May 2022] made our first contribution in [#1](https://github.com/FlagAI-Open/FlagAI/pull/1)
+
+--------------------------------------------------------------------------------
 <!-- toc -->
 
 - [安装](#安装)

doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md

@@ -37,17 +37,16 @@ GLM 对下游任务进行微调，并将它们重新定义为空白填充生成
 ```python
 import torch
 from flagai.model.glm_model import GLMModel
-from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.tokenizer import Tokenizer
 from flagai.model.predictor.predictor import Predictor
 if __name__ == "__main__":
     """Main training program."""
     print('Generate Samples') 
-    tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model',
-                                    add_block_symbols=True,
-                                    add_task_mask=True,
-                                    add_decoder_mask=False,
-                                    fix_command_token=False)
-    model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False)
+    model_name = 'GLM-large-ch'
+    model = GLMModel.from_pretrain(model_name=model_name,
+                                   download_path="./state_dict/")
+    tokenizer = Tokenizer.from_pretrained(model_name)
+    tokenizer = Tokenizer.from_pretrained(model_name, only_download_config=False)
     model.cuda(torch.cuda.current_device())
     predictor = Predictor(model, tokenizer)
     # question-answering
@@ -60,17 +59,14 @@ if __name__ == "__main__":
 ```python
 import torch
 from flagai.model.glm_model import GLMModel
-from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.tokenizer import Tokenizer
 from flagai.model.predictor.predictor import Predictor
 if __name__ == "__main__":
     """Main training program."""
     print('Generate Samples') 
-    tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model',
-                                    add_block_symbols=True,
-                                    add_task_mask=True,
-                                    add_decoder_mask=False,
-                                    fix_command_token=False)
-    model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False)
+    model_name = 'GLM-large-ch'
+    tokenizer = Tokenizer.from_pretrained(model_name)
+    model = GLMModel.from_pretrain(model_name=model_name, only_download_config=False)
     model.cuda(torch.cuda.current_device())
     predictor = Predictor(model, tokenizer)
     # question-answering
@@ -88,12 +84,9 @@ from flagai.model.predictor.predictor import Predictor
 if __name__ == "__main__":
     """Main training program."""
     print('Generate Samples') 
-    tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model',
-                                    add_block_symbols=True,
-                                    add_task_mask=True,
-                                    add_decoder_mask=False,
-                                    fix_command_token=False)
-    model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False)
+    model_name = 'GLM-large-ch'
+    tokenizer = Tokenizer.from_pretrained(model_name)
+    model = GLMModel.from_pretrain(model_name=model_name, only_download_config=False)
     model.cuda(torch.cuda.current_device())
     predictor = Predictor(model, tokenizer)
     # question-answering

doc_zh/TUTORIAL_1_TOKENIZER.md

@@ -9,25 +9,15 @@
 
 值得注意的是，不同的分词器可以有不同的文本分割方式，并且有不同的词表文件,  相关算法的介绍可以在 [这里](tokenization.md) 查看。
 
-目前我们支持下列七个分词器:
-
-| 分词器                          | 语言  |
-|------------------------------|-----|
-| GLMLargeEnWordPieceTokenizer | 英文  |
-| GLMLargeChTokenizer          | 中文  |
-| GLM10bENBPETokenizer         | 英文  |
-| T5BPETokenizer               | 中文  |
-| ROBERTATokenizer             | 中文  |
-| BertWordPieceTokenizer       | 中文  |
-| CPMTokenizer                 | 中文  |
-
 
 ## 加载分词器
 ```python
-from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-tokenizer = GLMLargeEnWordPieceTokenizer()
+from flagai.data.tokenizer import Tokenizer
+model_name = "GLM-large-ch"
+tokenizer = Tokenizer.from_pretrained(model_name)
 ```
-在这一步里，模型仓库中的词表文件将被自动下载到`cache_dir`参数中指定的路径。默认设置为分词器文件下的 ./vocab 目录。
+在这一步里，模型仓库中的词表文件将被自动下载到`cache_dir`参数中指定的路径。默认设置为 `./checkpoints/{model_name}` 目录。
+
 
 ## 应用分词器
 让我们使用一个分词器将原始文本编码成数字序列，然后将数字序列恢复成原始文本：
@@ -54,29 +44,3 @@ class T5BPETokenizer(Tokenizer):
                                                             cache_dir=cache_dir)
         self.text_tokenizer.max_len = int(1e12)
 ```
-
-### 3. 自定义分词器的接口
-如果Hugging Face里的分词器不能满足您的需求，那么需要先准备好一份词表，然后手动实现下列函数的功能：
-
-```python
-def EncodeAsIds(self, text: str, process_fn=None):
-    """输入文本 => 一个token序号列表"""
-
-def EncodeAsTokens(self, text: str, process_fn=None):
-    """输入文本 => 一个token列表"""
-
-def IdToToken(self, Id: int):
-    """Token序号 => token"""
-
-def TokenToId(self, token: str):
-    """Token => token序号"""
-    return self.text_tokenizer._convert_token_to_id(token)
-
-def DecodeIds(self, Ids: list[int]):
-    """一个token序号列表 => 对应的文本"""
-    return self.DecodeTokens([self.IdToToken(id) for id in Ids])
-
-def DecodeTokens(self, tokens: list[str]):
-    """一个token列表 => 对应的文本"""
-    return self.text_tokenizer.convert_tokens_to_string(tokens)
-```

doc_zh/TUTORIAL_2_DATASET.md

@@ -33,7 +33,7 @@
 ### 分类任务应用代码
 ```python
 import torch
-from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
+from flagai.data.tokenizer import Tokenizer 
 from flagai.data.dataset import SuperGlueDataset
 from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSuperglueStrategy
@@ -42,7 +42,7 @@ from flagai.data.dataset import ConstructSuperglueStrategy
 cl_args = CollateArguments()
 
 # 创建分词器
-tokenizer = GLMLargeEnWordPieceTokenizer()
+tokenizer = Tokenizer.from_pretrained("GLM-large-en")
 
 # 初步读取并处理数据集
 dataset = SuperGlueDataset(task_name='cb',
@@ -368,22 +368,15 @@ class ExamplePVP(PVP):
 ```
 ### 预训练的任务处理实例代码
 ```python
-from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.tokenizer import Tokenizer 
 from flagai.data.dataset import BlockDataset
 from flagai.data.dataset.block.data_utils import split_ds, get_dataset_lazy, add_args
 from flagai.test_utils import PretrainDatasetArguments
 
-tokenizer = GLMLargeChTokenizer(add_block_symbols=True,
-                                add_task_mask=True,
-                                add_decoder_mask=False,
-                                fix_command_token=True)
 
 ds_args = PretrainDatasetArguments()
 
-tokenizer = GLMLargeChTokenizer(fix_command_token=True,
-                                add_block_symbols=True,
-                                add_task_mask=True,
-                                add_decoder_mask=False)
+tokenizer = Tokenizer.from_pretrained("GLM-large-ch")
 
 ds_args = add_args(ds_args, tokenizer)
 
@@ -432,18 +425,15 @@ datasets = create_dataset(tokenizer, should_split=True)
 ```python 
 import torch
 from flagai.data.dataset import Seq2SeqDataset
-from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
+from flagai.data.tokenizer import Tokenizer
 from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSeq2seqStrategy
 
 # 得到默认参数
 cl_args = Seq2SeqCollateArguments()
 
 # 创建分词器
-tokenizer = GLMLargeChTokenizer(add_block_symbols=True,
-                       TUTORIAL_4_DATASET.md         add_task_mask=False,
-                                add_decoder_mask=False,
-                                fix_command_token=False)
+tokenizer = Tokenizer.from_pretrained("GLM-large-ch")
 
 # 初步读取并处理数据集
 dataset = Seq2SeqDataset(task_name='cmrc',

doc_zh/TUTORIAL_4_TRAINER.md

@@ -13,7 +13,7 @@
     - [deepspeed](#deepspeed)
     - [pytorchDDP](#pytorchddp)
     - [deepspeed + megatron-lm](#deepspeed--megatron-lm)
-- [EnvTrainer](#EnvTrainer)
+- [EnvTrainer](#envtrainer)
 
 Trainer 类提供了API用于多种并行框架的训练。API 支持在多个 GPU上使用Pytorch DDP/Deepspeed进行分布式训练，同时支持Megatron-LM+Deepspeed的混合并行分布式训练，同时也通过 NVIDIA Apex 实现混合精度。
 ## 入门
@@ -335,6 +335,7 @@ trainer = MyTrainer(
 )
 ```
 
+
 # EnvTrainer
 
 为了更容易的输入参数，我们提供了EnvTrainer代替原来的Trainer
@@ -385,9 +386,10 @@ python train.py --test1=1
 ```
 更多的例子可以查看 :
 
-1. [vit-env-trainer](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/vit_cifar100/train_env_trainer.py)
+1. [vit-env-trainer](https://github.com/BAAI-Open/FlagAI/tree/master/examples/vit_cifar100/train_env_trainer.py)
+
+2. [glm-title-generation-env-trainer](https://github.com/BAAI-Open/FlagAI/tree/master/examples/glm_title_generation/train_env_trainer.py)
 
-2. [glm-title-generation-env-trainer](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/glm_title_generation/train_env_trainer.py)
 
 # 使用 pytorchDDP launcher 或 deepspeed launcher 运行
 如果你使用多个GPU来训练模型，你可以直接运行train.py来调用FlagAI训练器中的启动器。
@@ -402,4 +404,4 @@ python -m torch.distributed.launch --nproc_per_node 2 --nnodes 1 --node_rank 0 -
 ### deepspeed
 ```commandline
 python -m deepspeed.launcher.launch  --master_addr=172.31.125.121 --master_port=17500 train.py --not_call_launch
-```
+```

docs/TUTORIAL_11_GLM_BLANK_FILLING_QA.md

@@ -47,17 +47,16 @@ filling task
 ```python
 import torch
 from flagai.model.glm_model import GLMModel
-from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.tokenizer import Tokenizer
 from flagai.model.predictor.predictor import Predictor
 if __name__ == "__main__":
     """Main training program."""
     print('Generate Samples') 
-    tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model',
-                                    add_block_symbols=True,
-                                    add_task_mask=True,
-                                    add_decoder_mask=False,
-                                    fix_command_token=False)
-    model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False)
+    model_name = 'GLM-large-ch'
+    model = GLMModel.from_pretrain(model_name=model_name,
+                                   download_path="./state_dict/")
+    tokenizer = Tokenizer.from_pretrained(model_name)
+    tokenizer = Tokenizer.from_pretrained("GLM-large-ch", only_download_config=False)
     model.cuda(torch.cuda.current_device())
     predictor = Predictor(model, tokenizer)
     # question-answering
@@ -71,17 +70,14 @@ Similar to BERT, GLM can predict masked tokens as
 ```python
 import torch
 from flagai.model.glm_model import GLMModel
-from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.tokenizer import Tokenizer
 from flagai.model.predictor.predictor import Predictor
 if __name__ == "__main__":
     """Main training program."""
     print('Generate Samples') 
-    tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model',
-                                    add_block_symbols=True,
-                                    add_task_mask=True,
-                                    add_decoder_mask=False,
-                                    fix_command_token=False)
-    model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False)
+    model_name = 'GLM-large-ch'
+    tokenizer = Tokenizer.from_pretrained(model_name)
+    model = GLMModel.from_pretrain(model_name=model_name, only_download_config=False)
     model.cuda(torch.cuda.current_device())
     predictor = Predictor(model, tokenizer)
     # question-answering
@@ -94,17 +90,14 @@ and predict masked sentences as
 ```python
 import torch
 from flagai.model.glm_model import GLMModel
-from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.tokenizer import Tokenizer
 from flagai.model.predictor.predictor import Predictor
 if __name__ == "__main__":
     """Main training program."""
     print('Generate Samples') 
-    tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model',
-                                    add_block_symbols=True,
-                                    add_task_mask=True,
-                                    add_decoder_mask=False,
-                                    fix_command_token=False)
-    model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False)
+    model_name = 'GLM-large-ch'
+    tokenizer = Tokenizer.from_pretrained(model_name)
+    model = GLMModel.from_pretrain(model_name=model_name, only_download_config=False)
     model.cuda(torch.cuda.current_device())
     predictor = Predictor(model, tokenizer)
     # question-answering

docs/TUTORIAL_1_TOKENIZER.md

@@ -17,29 +17,16 @@ and have different vocabulary files.
 
 [//]: # (An introduction to those algorithms can be viewed [here](tokenization.md).)
 
-Our projects currently support six tokenizers
-as listed below:
-
-| Tokenizer                    | Language |
-|------------------------------|----------|
-| GLMLargeEnWordPieceTokenizer | English  |
-| GLMLargeChTokenizer          | Chinese  |
-| GLM10bENBPETokenizer         | English  |
-| T5BPETokenizer               | Chinese  |
-| ROBERTATokenizer             | Chinese  |
-| BertWordPieceTokenizer       | Chinese  | 
-| CPMTokenizer                 | Chinese  |
-
 
 
 
 ## Loading a tokenizer
 ```python
-from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-
-tokenizer = GLMLargeEnWordPieceTokenizer()       # Load tokenizer
+from flagai.data.tokenizer import Tokenizer
+model_name = "GLM-large-en"
+tokenizer = Tokenizer.from_pretrained(model_name) # Load tokenizer 
 ```
-At this step, the vocab files from Modelhub will be automatically downloaded to the path specified in `cache_dir` parameter. It is set to `./vocab` directory under the tokenizer file in default.  
+At this step, the vocab files from Modelhub will be automatically downloaded to the path specified in `cache_dir` parameter. It is set to `./checkpoints/{model_name}` directory in default.  
 
 ## Applying a tokenizer
 The tokenizer can be used to encode text to a list of token IDs, as well as decoding the token IDs to the original text. 
@@ -68,28 +55,3 @@ class T5BPETokenizer(Tokenizer):
         self.text_tokenizer.max_len = int(1e12)
 ```
 
-### 3. Define Tokenizer APIs (without huggingface)
-If huggingface tokenizers are not used, you need to implement the following class functions by your own.
-
-```python
-def EncodeAsIds(self, text: str, process_fn=None):
-    """Input text string => a list of token ids"""
-
-def EncodeAsTokens(self, text: str, process_fn=None):
-    """Input text string => a list of tokens"""
-
-def IdToToken(self, Id: int):
-    """Token id => token"""
-
-def TokenToId(self, token: str):
-    """Token => token id"""
-    return self.text_tokenizer._convert_token_to_id(token)
-
-def DecodeIds(self, Ids: list[int]):
-    """A list of token ids => recovered text string"""
-    return self.DecodeTokens([self.IdToToken(id) for id in Ids])
-
-def DecodeTokens(self, tokens: list[str]):
-    """A list of tokens => recovered text string"""
-    return self.text_tokenizer.convert_tokens_to_string(tokens)
-```