Add feature pyramid network (FPN) and deformable FPN support

msracver · Dec 26, 2017 · 7703085 · 7703085
1 parent 3b1622e
commit 7703085
Show file tree

Hide file tree

Showing 32 changed files with 6,544 additions and 19 deletions.
diff --git a/README.md b/README.md
@@ -3,7 +3,9 @@
 
 The major contributors of this repository include [Yuwen Xiong](https://github.com/Orpine), [Haozhi Qi](https://github.com/Oh233), [Guodong Zhang](https://github.com/gd-zhang), [Yi Li](https://github.com/liyi14), [Jifeng Dai](https://github.com/daijifeng001), [Bin Xiao](https://github.com/leoxiaobin), [Han Hu](https://github.com/ancientmooner) and  [Yichen Wei](https://github.com/YichenWei).
 
-**[A third-party improvement](https://github.com/bharatsingh430/Deformable-ConvNets) of Deformable R-FCN + Soft NMS, best single-model performance on COCO detection**
+**We released training/testing code and pre-trained models of Deformable FPN, which is the foundation of our COCO detection 2017 entry.** Slides at [COCO 2017 workshop](http://presentations.cocodataset.org/COCO17-Detect-MSRA.pdf).
+
+**[A third-party improvement](https://github.com/bharatsingh430/Deformable-ConvNets) of Deformable R-FCN + Soft NMS**
 
 
 ## Introduction
@@ -65,7 +67,12 @@ If you find Deformable ConvNets useful in your research, please consider citing:
 | <sub>Faster R-CNN (2fc), ResNet-v1-101 </sub>           | <sub>coco trainval</sub> | <sub>coco test-dev</sub> | 30.3 | 52.1    |   31.4  | 9.9  | 32.2  | 47.4  | 
 | <sub>Deformable Faster R-CNN (2fc), </br>ResNet-v1-101</sub> | <sub>coco trainval</sub> | <sub>coco test-dev</sub> | 35.0 | 55.0    | 38.3    | 14.3  | 37.7  | 52.0  |
 
-
+|                                 | <sub>training data</sub> | <sub>testing data</sub>  | <sub>mAP</sub>  | <sub>[email protected]</sub> | <sub>[email protected]</sub>| <sub>mAP@S</sub> | <sub>mAP@M</sub> | <sub>mAP@L</sub> |
+|---------------------------------|---------------|---------------|------|---------|---------|-------|-------|-------|
+| <sub> FPN+OHEM, ResNet-v1-101 </sub>            | <sub>coco trainval35k</sub> | <sub>coco minival</sub> | 37.8 | 60.8 | 41.0 | 22.0  | 41.5  | 49.8  | 
+| <sub>Deformable FPN + OHEM, ResNet-v1-101</sub> | <sub>coco trainval35k</sub> | <sub>coco minival</sub> | 41.2 | 63.5 | 45.5 | 24.3  | 44.9  | 54.4  |
+| <sub> FPN + OHEM + Soft NMS + multi-scale testing, </br>ResNet-v1-101 </sub>   | <sub>coco trainval35k</sub> | <sub>coco minival</sub> | 40.9 | 62.5    |   46.0  | 27.1  | 44.1  | 52.2  | 
+| <sub> Deformable FPN + OHEM + Soft NMS + multi-scale testing, ResNet-v1-101</sub> | <sub>coco trainval35k</sub> | <sub>coco minival</sub> | 44.4 | 65.5    | 50.2   | 30.8 | 47.3 | 56.4  |
 
 |                                   | training data              | testing data   | mIoU | time  |
 |-----------------------------------|----------------------------|----------------|------|-------|
@@ -105,6 +112,8 @@ git clone https://github.com/msracver/Deformable-ConvNets.git
 2. For Windows users, run ``cmd .\init.bat``. For Linux user, run `sh ./init.sh`. The scripts will build cython module automatically and create some folders.
 
 3. Install MXNet:
+
+	**Note: The MXNet's Custom Op cannot execute parallelly using multi-gpus after this [PR](https://github.com/apache/incubator-mxnet/pull/6928). We strontly suggest the user rollback to version [MXNet@(commit 998378a)](https://github.com/dmlc/mxnet/tree/998378a) for training (following Section 3.2 - 3.5).**
 
 	***Quick start***
 
@@ -113,39 +122,28 @@ git clone https://github.com/msracver/Deformable-ConvNets.git
 	pip install -r requirements.txt
 	```
 	If there is no other error message, MXNet should be installed successfully. 
-	If you want to build MXNet from source, please follow 3.2 - 3.7.
 
 	***Build from source (alternative way)***
-
-	3.2 Checkout this repo to [Deformable-ConvNets@(commit ecc484f)](https://github.com/msracver/Deformable-ConvNets/tree/ecc484f) by
-	```
-	cd $(DCN_ROOT)
-	git checkout ecc484f
-	```
 
-	3.3 Clone MXNet and checkout to [MXNet@(commit 62ecb60)](https://github.com/dmlc/mxnet/tree/62ecb60) by
+	3.2 Clone MXNet and checkout to [MXNet@(commit 998378a)](https://github.com/dmlc/mxnet/tree/998378a) by
 	```
 	git clone --recursive https://github.com/dmlc/mxnet.git
-	git checkout 62ecb60
+	git checkout 998378a
 	git submodule update
 	```
-	3.4 Copy operators in `$(DCN_ROOT)/rfcn/operator_cxx` or `$(DCN_ROOT)/faster_rcnn/operator_cxx` to `$(YOUR_MXNET_FOLDER)/src/operator/contrib` by
-	```
-	cp -r $(DCN_ROOT)/rfcn/operator_cxx/* $(MXNET_ROOT)/src/operator/contrib/
-	```
-	3.5 Compile MXNet
+	3.3 Compile MXNet
 	```
 	cd ${MXNET_ROOT}
 	make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1
 	```
-	3.6 Install the MXNet Python binding by
+	3.4 Install the MXNet Python binding by
 
-	***Note: If you will actively switch between different versions of MXNet, please follow 3.7 instead of 3.6***
+	***Note: If you will actively switch between different versions of MXNet, please follow 3.5 instead of 3.4***
 	```
 	cd python
 	sudo python setup.py install
 	```
-	3.7 For advanced users, you may put your Python packge into `./external/mxnet/$(YOUR_MXNET_PACKAGE)`, and modify `MXNET_VERSION` in `./experiments/rfcn/cfgs/*.yaml` to `$(YOUR_MXNET_PACKAGE)`. Thus you can switch among different versions of MXNet quickly.
+	3.5 For advanced users, you may put your Python packge into `./external/mxnet/$(YOUR_MXNET_PACKAGE)`, and modify `MXNET_VERSION` in `./experiments/rfcn/cfgs/*.yaml` to `$(YOUR_MXNET_PACKAGE)`. Thus you can switch among different versions of MXNet quickly.
 
 4. For Deeplab, we use the argumented VOC 2012 dataset. The argumented annotations are provided by [SBD](http://home.bharathh.info/pubs/codes/SBD/download.html) dataset. For convenience, we provide the converted PNG annotations and the lists of train/val images, please download them from [OneDrive](https://1drv.ms/u/s!Am-5JzdW2XHzhqMRhVImMI1jRrsxDg).
 
@@ -159,6 +157,8 @@ We provide trained deformable convnet models, including the deformable R-FCN & F
 	```
 	./model/rfcn_dcn_coco-0000.params
 	./model/rfcn_coco-0000.params
+	./model/fpn_dcn_coco-0000.params
+	./model/fpn_coco-0000.params
 	./model/rcnn_dcn_coco-0000.params
 	./model/rcnn_coco-0000.params
 	./model/deeplab_dcn_cityscapes-0000.params

diff --git a/experiments/fpn/cfgs/resnet_v1_101_coco_trainval_fpn_dcn_end2end_ohem.yaml b/experiments/fpn/cfgs/resnet_v1_101_coco_trainval_fpn_dcn_end2end_ohem.yaml
@@ -0,0 +1,163 @@
+---
+MXNET_VERSION: "mxnet"
+output_path: "./output/fpn/coco"
+symbol: resnet_v1_101_fpn_dcn_rcnn
+gpus: '0,1,2,3,4,5,6,7'
+CLASS_AGNOSTIC: false
+SCALES:
+- 800
+- 1280
+# TEST_SCALES: [[800, 1280]] # single scale testing
+TEST_SCALES: [[480, 800], [576, 900], [688, 1100], [800, 1200], [1200, 1600], [1400, 2000]] # multi-scale testing
+default:
+  frequent: 100
+  kvstore: device
+network:
+  pretrained: "./model/pretrained_model/resnet_v1_101"
+  pretrained_epoch: 0
+  PIXEL_MEANS:
+  - 103.06
+  - 115.90
+  - 123.15
+  IMAGE_STRIDE: 32
+  RCNN_FEAT_STRIDE: 16
+  RPN_FEAT_STRIDE:
+  - 4
+  - 8
+  - 16
+  - 32
+  - 64
+  FIXED_PARAMS:
+  - conv1
+  - bn_conv1
+  - res2
+  - bn2
+  - gamma
+  - beta
+  FIXED_PARAMS_SHARED:
+  - conv1
+  - bn_conv1
+  - res2
+  - bn2
+  - res3
+  - bn3
+  - res4
+  - bn4
+  - gamma
+  - beta
+  ANCHOR_RATIOS:
+  - 0.5
+  - 1
+  - 2
+  ANCHOR_SCALES:
+  - 8
+  NUM_ANCHORS: 3
+dataset:
+  NUM_CLASSES: 81
+  dataset: coco
+  dataset_path: "./data/coco"
+  image_set: train2014+valminusminival2014
+  # image_set: minival2014
+  root_path: "./data"
+  test_image_set: minival2014
+  proposal: rpn
+TRAIN:
+  lr: 0.01
+  warmup_lr: 0.001
+  warmup_step: 250
+  warmup: true
+  # lr: 0.000001
+  lr_step: '4,6'
+  wd: 0.0001
+  begin_epoch: 0
+  end_epoch: 7
+  model_prefix: 'fpn_coco'
+  # whether resume training
+  RESUME: false
+  # whether flip image
+  FLIP: true
+  # whether shuffle image
+  SHUFFLE: true
+  # whether use OHEM
+  ENABLE_OHEM: true
+  # size of images for each device, 2 for rcnn, 1 for rpn and e2e
+  BATCH_IMAGES: 1
+  # e2e changes behavior of anchor loader and metric
+  END2END: true
+  # group images with similar aspect ratio
+  ASPECT_GROUPING: true
+  # R-CNN
+  # rcnn rois batch size
+  BATCH_ROIS: -1
+  BATCH_ROIS_OHEM: 512
+  # rcnn rois sampling params
+  FG_FRACTION: 0.25
+  FG_THRESH: 0.5
+  BG_THRESH_HI: 0.5
+  BG_THRESH_LO: 0.0
+  # rcnn bounding box regression params
+  BBOX_REGRESSION_THRESH: 0.5
+  BBOX_WEIGHTS:
+  - 1.0
+  - 1.0
+  - 1.0
+  - 1.0
+  # RPN anchor loader
+  # rpn anchors batch size
+  RPN_BATCH_SIZE: 256
+  # rpn anchors sampling params
+  RPN_FG_FRACTION: 0.5
+  RPN_POSITIVE_OVERLAP: 0.7
+  RPN_NEGATIVE_OVERLAP: 0.3
+  RPN_CLOBBER_POSITIVES: false
+  # rpn bounding box regression params
+  RPN_BBOX_WEIGHTS:
+  - 1.0
+  - 1.0
+  - 1.0
+  - 1.0
+  RPN_POSITIVE_WEIGHT: -1.0
+  # used for end2end training
+  # RPN proposal
+  CXX_PROPOSAL: false
+  RPN_NMS_THRESH: 0.7
+  RPN_PRE_NMS_TOP_N: 12000
+  RPN_POST_NMS_TOP_N: 2000
+  RPN_MIN_SIZE: 0
+  # approximate bounding box regression
+  BBOX_NORMALIZATION_PRECOMPUTED: true
+  BBOX_MEANS:
+  - 0.0
+  - 0.0
+  - 0.0
+  - 0.0
+  BBOX_STDS:
+  - 0.1
+  - 0.1
+  - 0.2
+  - 0.2
+TEST:
+  # use rpn to generate proposal
+  HAS_RPN: true
+  # size of images for each device
+  BATCH_IMAGES: 1
+  # RPN proposal
+  CXX_PROPOSAL: false
+  RPN_NMS_THRESH: 0.7
+  RPN_PRE_NMS_TOP_N: 12000
+  RPN_POST_NMS_TOP_N: 2000
+  RPN_MIN_SIZE: 0
+  # RPN generate proposal
+  PROPOSAL_NMS_THRESH: 0.7
+  PROPOSAL_PRE_NMS_TOP_N: 20000
+  PROPOSAL_POST_NMS_TOP_N: 2000
+  PROPOSAL_MIN_SIZE: 0
+  # RCNN nms
+  NMS: 0.3
+  USE_SOFTNMS: true
+  SOFTNMS_THRESH: 0.6
+  test_epoch: 7
+  max_per_image: 100
+  # soft nms
+  USE_SOFTNMS: true
+  SOFTNMS_THRESH: 0.6