mindspore/model_zoo/official/cv/simple_pose

• Contents
• Description
• Model Architecture
• Dataset
• Features
• • Mixed Precision
• Environment Requirements
• Quick Start
• • Dataset Preparation
  • Model Checkpoints
  • Running
• Script Description
• • Script and Sample Code
  • Script Parameters
  • Training Process
  • • Training
    • • Running on Ascend
    • Distributed Training
    • • Running on Ascend
  • Evaluation Process
  • • Running on Ascend
• Model Description
• • Performance
  • • SimplePoseNet on COCO2017 with detector
    • • Performance parameters
      • Note
• Description of Random Situation
• ModelZoo Homepage

Contents

• Description
• Model Architecture
• Dataset
• Features
  • Mixed Precision
• Environment Requirements
• Quick Start
  • Dataset Preparation
  • Model Checkpoints
  • Running
• Script Description
  • Script and Sample Code
  • Script Parameters
  • Training Process
    • Training
    • Distributed Training
  • Evaluation Process
• Model Description
  • Performance
• Description of Random Situation
• ModelZoo Homepage

Description

SimplePoseNet is a convolution-based neural network for the task of human pose estimation and tracking. It provides baseline methods that are surprisingly simple and effective, thus helpful for inspiring and evaluating new ideas for the field. State-of-the-art results are achieved on challenging benchmarks. More detail about this model can be found in:

B. Xiao, H. Wu, and Y. Wei, “Simple baselines for human pose estimation and tracking,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 472–487.

This repository contains a Mindspore implementation of SimplePoseNet based upon Microsoft's original Pytorch implementation (https://github.com/microsoft/human-pose-estimation.pytorch). The training and validating scripts are also included, and the evaluation results are shown in the Performance section.

Model Architecture

The overall network architecture of SimplePoseNet is shown below:

Link

Dataset

Note that you can run the scripts based on the dataset mentioned in original paper or widely used in relevant domain/network architecture. In the following sections, we will introduce how to run the scripts using the related dataset below.

Dataset used: COCO2017

• Dataset size:
  • Train: 19G, 118,287 images
  • Test: 788MB, 5,000 images
• Data format: JPG images
  • Note: Data will be processed in src/dataset.py
• Person detection result for validation: Detection result provided by author in the repository

Features

Mixed Precision

The mixed precision training method accelerates the deep learning neural network training process by using both the single-precision and half-precision data formats, and maintains the network precision achieved by the single-precision training at the same time. Mixed precision training can accelerate the computation process, reduce memory usage, and enable a larger model or batch size to be trained on specific hardware. For FP16 operators, if the input data type is FP32, the backend of MindSpore will automatically handle it with reduced precision. Users could check the reduced-precision operators by enabling INFO log and then searching ‘reduce precision’.

Environment Requirements

To run the python scripts in the repository, you need to prepare the environment as follow:

• Hardware
  • Prepare hardware environment with Ascend. If you want to try Ascend, please send the application form to ascend@huawei.com. Once approved, you can get the resources.
• Python and dependencies
  • python 3.7
  • mindspore 1.0.1
  • easydict 1.9
  • opencv-python 4.3.0.36
  • pycocotools 2.0
• For more information, please check the resources below：
  • MindSpore tutorials
  • MindSpore API

Quick Start

Dataset Preparation

SimplePoseNet use COCO2017 dataset to train and validate in this repository. Download the dataset from official website. You can place the dataset anywhere and tell the scripts where it is by modifying the DATASET.ROOT setting in configuration file src/config.py. For more information about the configuration file, please refer to Script Parameters.

You also need the person detection result of COCO val2017 to reproduce the multi-person pose estimation results, as mentioned in Dataset. Please checkout the author's repository, download and extract them under <ROOT>/experiments/, and make them look like this:

└─ <ROOT>
 └─ experiments
   └─ COCO_val2017_detections_AP_H_56_person.json

Model Checkpoints

Before you start your training process, you need to obtain mindspore imagenet pretrained models. The model weight file can be obtained by running the Resnet training script in official model zoo. We also provide a pretrained model that can be used to train SimplePoseNet directly in GoogleDrive. The model file should be placed under <ROOT>/models/ like this:

└─ <ROOT>
 └─ models
  └─resnet50.ckpt

Running

To train the model, run the shell script scripts/train_standalone.sh with the format below:

sh scripts/train_standalone.sh [device_id] [ckpt_path_to_save]

To validate the model, change the settings in src/config.py to the path of the model you want to validate. For example:

config.TEST.MODEL_FILE='results/xxxx.ckpt'

Then, run the shell script scripts/eval.sh with the format below:

sh scripts/eval.sh [device_id]

Script Description

Script and Sample Code

The structure of the files in this repository is shown below.

└─ mindspore-simpleposenet
 ├─ scripts
 │ ├─ eval.sh                 // launch ascend standalone evaluation
 │ ├─ train_distributed.sh    // launch ascend distributed training
 │ └─ train_standalone.sh     // launch ascend standalone training
 ├─ src
 │ ├─utils
 │ │ ├─ transform.py          // utils about image transformation
 │ │ └─ nms.py                // utils about nms
 │ ├─evaluate
 │ │ └─ coco_eval.py          // evaluate result by coco
 │ ├─ config.py               // network and running config
 │ ├─ dataset.py              // dataset processor and provider
 │ ├─ model.py                // SimplePoseNet implementation
 │ ├─ network_define.py       // define loss
 │ └─ predict.py              // predict keypoints from heatmaps
 ├─ eval.py                   // evaluation script
 ├─ param_convert.py          // model parameters convertion script
 ├─ train.py                  // training script
 └─ README.md                 // descriptions about this repository

Script Parameters

Configurations for both training and evaluation are set in src/config.py. All the settings are shown following.

• config for SimplePoseNet on COCO2017 dataset:

# pose_resnet related params
POSE_RESNET.HEATMAP_SIZE = [48, 64]                # heatmap size
POSE_RESNET.SIGMA = 2                              # Gaussian hyperparameter in heatmap generation
POSE_RESNET.FINAL_CONV_KERNEL = 1                  # final convolution kernel size
POSE_RESNET.DECONV_WITH_BIAS = False               # deconvolution bias
POSE_RESNET.NUM_DECONV_LAYERS = 3                  # the number of deconvolution layers
POSE_RESNET.NUM_DECONV_FILTERS = [256, 256, 256]   # the filter size of deconvolution layers
POSE_RESNET.NUM_DECONV_KERNELS = [4, 4, 4]         # kernel size of deconvolution layers
POSE_RESNET.NUM_LAYERS = 50                        # number of layers(for resnet)
# common params for NETWORK
config.MODEL.NAME = 'pose_resnet'                  # model name
config.MODEL.INIT_WEIGHTS = True                   # init model weights by resnet
config.MODEL.PRETRAINED = './models/resnet50.ckpt' # pretrained model
config.MODEL.NUM_JOINTS = 17                       # the number of keypoints
config.MODEL.IMAGE_SIZE = [192, 256]               # image size
# dataset
config.DATASET.ROOT = '/data/coco2017/'            # coco2017 dataset root
config.DATASET.TEST_SET = 'val2017'                # folder name of test set
config.DATASET.TRAIN_SET = 'train2017'             # folder name of train set
# data augmentation
config.DATASET.FLIP = True                         # random flip
config.DATASET.ROT_FACTOR = 40                     # random rotation
config.DATASET.SCALE_FACTOR = 0.3                  # random scale
# for train
config.TRAIN.BATCH_SIZE = 64                       # batch size
config.TRAIN.BEGIN_EPOCH = 0                       # begin epoch
config.TRAIN.END_EPOCH = 140                       # end epoch
config.TRAIN.LR = 0.001                            # initial learning rate
config.TRAIN.LR_FACTOR = 0.1                       # learning rate reduce factor
config.TRAIN.LR_STEP = [90,120]                    # step to reduce lr
# test
config.TEST.BATCH_SIZE = 32                        # batch size
config.TEST.FLIP_TEST = True                       # flip test
config.TEST.POST_PROCESS = True                    # post process
config.TEST.SHIFT_HEATMAP = True                   # shift heatmap
config.TEST.USE_GT_BBOX = False                    # use groundtruth bbox
config.TEST.MODEL_FILE = ''                        # model file to test
# detect bbox file
config.TEST.COCO_BBOX_FILE = 'experiments/COCO_val2017_detections_AP_H_56_person.json'
# nms
config.TEST.OKS_THRE = 0.9                         # oks threshold
config.TEST.IN_VIS_THRE = 0.2                      # visible threshold
config.TEST.BBOX_THRE = 1.0                        # bbox threshold
config.TEST.IMAGE_THRE = 0.0                       # image threshold
config.TEST.NMS_THRE = 1.0                         # nms threshold

Training Process

Training

Running on Ascend

Run scripts/train_standalone.sh to train the model standalone. The usage of the script is:

sh scripts/train_standalone.sh [device_id] [ckpt_path_to_save]

For example, you can run the shell command below to launch the training procedure.

sh scripts/train_standalone.sh 0 results/standalone/

The script will run training in the background, you can view the results through the file train_log[X].txt as follows:

loading parse...
batch size :128
loading dataset from /data/coco2017/train2017
loaded 149813 records from coco dataset.
loading pretrained model ./models/resnet50.ckpt
start training, epoch size = 140
epoch: 1 step: 1170, loss is 0.000699
Epoch time: 492271.194, per step time: 420.745
epoch: 2 step: 1170, loss is 0.000586
Epoch time: 456265.617, per step time: 389.971
...

The model checkpoint will be saved into [ckpt_path_to_save].

Distributed Training

Running on Ascend

Run scripts/train_distributed.sh to train the model distributed. The usage of the script is:

sh scripts/train_distributed.sh [rank_table] [ckpt_path_to_save] [device_number]

For example, you can run the shell command below to launch the distributed training procedure.

sh scripts/train_distributed.sh /home/rank_table.json results/distributed/ 4

The above shell script will run distribute training in the background. You can view the results through the file train_parallel[X]/log.txt as follows:

loading parse...
batch size :64
loading dataset from /data/coco2017/train2017
loaded 149813 records from coco dataset.
loading pretrained model ./models/resnet50.ckpt
start training, epoch size = 140
epoch: 1 step: 585, loss is 0.0007944
Epoch time: 236219.684, per step time: 403.794
epoch: 2 step: 585, loss is 0.000617
Epoch time: 164792.001, per step time: 281.696
...

The model checkpoint will be saved into [ckpt_path_to_save].

Evaluation Process

Running on Ascend

Change the settings in src/config.py to the path of the model you want to validate. For example:

config.TEST.MODEL_FILE='results/xxxx.ckpt'

Then, run scripts/eval.sh to evaluate the model with one Ascend processor. The usage of the script is:

sh scripts/eval.sh [device_id]

For example, you can run the shell command below to launch the validation procedure.

sh scripts/eval.sh 0

The above shell command will run validation procedure in the background. You can view the results through the file eval_log[X].txt. The result will be achieved as follows:

use flip test: True
loading model ckpt from results/distributed/sim-140_1170.ckpt
loading dataset from /data/coco2017/val2017
loading bbox file from experiments/COCO_val2017_detections_AP_H_56_person.json
Total boxes: 104125
1024 samples validated in 18.133189916610718 seconds
2048 samples validated in 4.724390745162964 seconds
...

Model Description

Performance

SimplePoseNet on COCO2017 with detector

Performance parameters

Parameters	Standalone	Distributed
Model Version	SimplePoseNet	SimplePoseNet
Resource	Ascend 910	4 Ascend 910 cards
Uploaded Date	12/18/2020 (month/day/year)	12/18/2020 (month/day/year)
MindSpore Version	1.1.0	1.1.0
Dataset	COCO2017	COCO2017
Training Parameters	epoch=140, batch_size=128	epoch=140, batch_size=64
Optimizer	Adam	Adam
Loss Function	Mean Squared Error	Mean Squared Error
Outputs	heatmap	heatmap
Train Performance	mAP: 70.4	mAP: 70.4
Speed	1pc: 389.915 ms/step	4pc: 281.356 ms/step

Note

• Flip test is used.
• Person detector has person AP of 56.4 on COCO val2017 dataset.
• The dataset preprocessing and general training configurations are shown in Script Parameters section.

Description of Random Situation

In src/dataset.py, we set the seed inside “create_dataset" function. We also use random seed in src/model.py to initial network weights.

ModelZoo Homepage

Please check the official homepage.