9.4. AI Benchmark User Guide¶

D-Robotics’s X5 AI Benchmark Sample Package contains the most frequently used performance and accuracy evaluation samples of classification, detection, segmentation, optical-flow, tracking estimation, lidar multitask, bev and depth estimation models. In model performance evaluation samples, developers are able, not only to evaluate the single frame latency, but also to evaluate the bpu core latency using multi-thread scheduling. The pre-build source code, executable programs and evaluation scripts in the AI Benchmark Sample Package allow developers to experience the samples, and develop their own applications, which makes development easier.

9.4.1. About the Deliverables¶

The AI Benchmark sample package is located in the samples/ai_benchmark/ path of the horizon_x5_open_explorer release package and consists the following main contents:

NO.	NAME	DESCRIPTIONS
1	code	This folder contains sample source code and compilation scripts.
2	x5	Dev board operating environment of the AI Benchmark Sample Package.

9.4.1.1. Sample Code Package¶

Note

The on-board model needs to be obtained first by executing the resolve_ai_benchmark_ptq.sh and the resolve_ai_benchmark_qat.sh in the samples/ai_toolchain/model_zoo/runtime/ai_benchmark directory of the OE package respectively.

Directory of the sample code package is shown as below:

ai_benchmark/code/                    # sample source code
├── build_ptq_x5.sh
├── build_qat_x5.sh
├── CMakeLists.txt
├── deps_gcc11.3                      # third party dependencies
├── include                           # source code header files
├── README.md
└── src                               # sample source code

ai_benchmark/x5                       # Sample package runtime environment
├── ptq                               # PTQ (post-training quantization) model samples
│   ├── data                          # accuracy evaluation dataset
│   ├── mini_data                     # performance evaluations dataset
│   ├── model                         # PTQ (post-training quantization) solution nv12 model
│   │   ├── README.md
│   │   └── runtime -> ../../../../model_zoo/runtime/ai_benchmark/ptq   # Soft link to the model in the OE package, the board environment you need to specify the model path
│   ├── README.md
│   ├── script                        # execution script
│   │   ├── aarch64                   # executable files generated by the compilation and dependencies
│   │   ├── classification            # samples classification models
│   │   ├── config                    # configuration files of model inference
│   │   ├── detection                 # detection model samples
│   │   ├── segmentation              # segmentation model samples
│   │   ├── env.sh                    # basic environment script
│   │   └── README.md
│   └── tools                         # accuracy evaluation tools
└── qat                               # QAT training model samples
    ├── data                          # model accuracy evaluation dataset
    ├── mini_data                     # model performance evaluations dataset
    ├── model                         # QAT scheme nv12 model
    │   ├── README.md
    │   └── runtime -> ../../../../model_zoo/runtime/ai_benchmark/qat    # Soft link to the model in the OE package, the board environment you need to specify the model path
    ├── README.md
    ├── script                        # execution script
    │   ├── aarch64                   # executable files generated by the compilation and dependencies
    │   ├── bev                       # bev model samples
    │   ├── classification            # samples classification models
    │   ├── config                    # model inference profile
    │   ├── detection                 # detection model samples
    │   ├── disparity_pred            # disparity pred model samples
    │   ├── multitask                 # multi task model samples
    │   ├── opticalflow               # optical flow model samples
    │   ├── segmentation              # segmentation model samples
    │   ├── tracking                  # tracking model samples
    │   ├── traj_pred                 # traj pred model samples
    │   ├── env.sh                    # basic environment scripts
    │   └── README.md
    └── tools

The code directory contains the source code of the evaluation program, used to evaluate model performance and accuracy.
The x5 directory contains various pre-compiled application programs and evaluation scripts, used to evaluate the accuracy and performance of different models in D-Robotics’s BPU (Brain Processing Unit).
The build_ptq_x5.sh script is the PTQ real program one-click compilation script.
The build_qat_x5.sh script is the QAT real program one-click compilation script.
The deps_gcc11.3 sub-directory contains dependencies required by sample codes, including:
```
appsdk  gflags  glog  nlohmann  opencv  rapidjson
```

9.4.1.2. Sample Models¶

Model releases for the AI Benchmark sample package include PTQ model and QAT model releases:

You can get the model_zoo of the PTQ model by executing the script resolve_ai_benchmark_ptq.sh of the samples/ai_toolchain/model_zoo/runtime/ai_benchmark/ path.
You can get the model_zoo of the QAT model by executing the script resolve_ai_benchmark_qat.sh of the samples/ai_toolchain/model_zoo/runtime/ai_benchmark/ path.

Among them, which contain the commonly used classification, detection, segmentation and optical flow prediction models, and the naming rules of the models is {model_name}_{backbone}_{input_size}_{input_type}.

Both the PTQ and QAT models in model_zoo are compiled by the original model.

For the PTQ original model details, you can refer to the section PTQ Conversion Samples Guide. For the QAT original model details, you can refer to the section Quantized Awareness Training (QAT).

The performance data of the model in the AI Benchmark sample package can be found in the section Benchmark of Model Performance.

9.4.1.3. Public Datasets¶

The dataset will be used in the sample, you can download the corresponding dataset in section Dataset Download. If you have any questions during the data preparation process, please visit the D-Robotics Developer Community.

9.4.2. Development Environment¶

Before using the AI Benchmark sample package, you need to ensure that the development board environment and compilation environment are available:

Prepare the Development Board

1.After getting the development board, upgrade the system image file to the version recommended by the sample package.

2.Make sure the remote connection between local dev machine and the dev board.
Prepare the compilation environment

Install the arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu cross-compilation tool in current environment, and then run the build_ptq_x5.sh script and the build_qat_x5 in the code folder to compile executable programs in real machine. The executable programs and corresponding dependencies will be copied into the aarch64 sub-folders of the x5/ptq/script and x5/qat/script folders automatically.

Note

The cross-compilation tool specified by the build_ptq_x5.sh and the build_qat_x5 script is located in the /opt folder. If you want to install it into some other locations, please modify the script.
```
export CC=/opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc
export CXX=/opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-g++
```

9.4.3. How to Use¶

9.4.3.1. Evaluation Scripts¶

Evaluation sample scripts are in the script and tools folders.

The script folder contains the scripts used for evaluating frequently used classification, detection, segmentation, optical-flow, tracking estimation, lidar multitask, bev and depth estimation models in dev board. There are three scripts under each model:
- The fps.sh script implements FPS statistics (multi-threading scheduling. You can freely specify number of threads as needed) .
- The latency.sh implements statistics of single-frame latency (one thread, single-frame).
- The accuracy.sh script is used for evaluating model accuracy.

script:

├── aarch64                                    # executable files generated by the compilation and dependencies
│   ├── bin
│   └── lib
├── env.sh                                     # base config
├── config
│   ├── model
│   │   ├── data_name_list                     # image name config
│   │   └── input_init                         # model input config
│   ├── preprocess
│   │   └── centerpoint_preprocess_5dim.json   # preprocess config
│   └── reference_points                       # reference points config
│   │   ├── bev_gkt_mixvargenet_multitask_nuscenes
│   │   └── ...
├── detection                                  # detection models
│   ├── fcos_efficientnetb0_mscoco                    # there are other models in this directory, just use this model directory as a reference
│   │   ├── accuracy.sh
│   │   ├── fps.sh
│   │   ├── latency.sh
│   │   ├── workflow_accuracy.json             # accuracy config file
│   │   ├── workflow_fps.json                  # fps config file
│   │   └── workflow_latency.json              # latency config file
│   └──...
├── ...
└── README.md

The (PTQ)tools folder contains the precision calculation scripts under python_tools, which used for accuracy evaluation.

python_tools
  └── accuracy_tools
      ├── cityscapes_metric.py
      ├── cls_eval.py
      ├── coco_metric.py
      ├── config.py
      ├── coco_det_eval.py
      ├── parsing_eval.py
      └── voc_det_eval.py
      └── voc_metric.py

The (QAT)tools folder contains pre-processing scripts and precision calculation scripts, which used for QAT model accuracy evaluation.

tools/
  ├── eval_preprocess
  │     ├── util
  │     ├── ......
  │     └── voc.py
  ├── python_tools
  │     └── accuracy_tools
  │         ├── argoverse_util
  │         ├── nuscenes_metric_pro
  │         ├── whl_package
  │         ├── bev_eval.py
  │         ├── ......
  │         └── yolov3_eval.py
  └── README.md

Attention

Run the following commands before the evaluation and copy the ptq (or the qat) directory to the dev board.

scp -r samples/ai_benchmark/x5/ptq root@192.168.1.1:/userdata/ptq/
scp -r samples/ai_benchmark/x5/qat root@192.168.1.1:/userdata/qat/

9.4.3.2. Parameters of json Configuration File¶

This section we provide you the brief introduction of the configuration for the workflow_fps.json, workflow_latency.json and workflow_accuracy.json. The configuration can be simply divided into input_config, output_config and the workflow configuration.

Note

The configuration parameters given below are the general configuration, some sample models will have additional configuration due to the model specificity, please refer to the sample model json file for details.

9.4.3.2.1. input_config¶

Parameter name	Description	Involved json files
input_type	Specify the input data format, support image or bin file.	fps.json, latency.json and accuracy.json
height	Specify the input data height.	fps.json, latency.json and accuracy.json
width	Specify the input data width.	fps.json, latency.json and accuracy.json
data_type	Specify the input data type. Supported type can refer to hbDNNDataType section. The corresponding data types are sorted from top to bottom, corresponding to the numbers 0,1,2… For example, HB_DNN_IMG_TYPE_Y corresponds to the 0, HB_DNN_IMG_TYPE_NV12 corresponds to the 1, HB_DNN_IMG_TYPE_NV12_SEPARATE corresponds to the 2…	fps.json, latency.json and accuracy.json
image_list_file	Specify the path of the lst file of the preprocessing dataset.	fps.json, latency.json and accuracy.json
need_pre_load	Specify whether to read the dataset using the preload method.	fps.json, latency.json and accuracy.json
limit	Specify the threshold for the difference between the amount of input data being processed and has been processed, which is used to control the processing threads for the input data.	fps.json, latency.json and accuracy.json
need_loop	Specify whether to use cyclic read data for evaluation.	fps.json, latency.json and accuracy.json
max_cache	Specify the number of preloaded data. Attention: When this parameter takes effect, the image will be pre-processed and read into memory. To ensure the stable running of your application, do not set too large a value, we recommend that you set a value of no more than 30.	fps.json, latency.json and accuracy.json

9.4.3.2.2. output_config¶

Parameter name	Description	Involved json files
output_type	Specify the output data format.	fps.json, latency.json and accuracy.json
in_order	Specify whether to output in order.	fps.json, latency.json and accuracy.json
enable_view_output	Specify whether to visualize the output.	fps.json and latency.json
image_list_enable	When visualizing, set to true to save the output as the image type.	fps.json and latency.json
view_output_dir	Specify the path of the visualization result output file.	fps.json and latency.json
eval_enable	Specify whether to evaluate the accuracy.	accuracy.json
output_file	Specify the model output result file.	accuracy.json

9.4.3.2.3. workflow Configuration¶

Model inference configurations:

Parameter name

Description

Involved json files

method_type

Specify the model inference method, which here needs to be configured as InferMethod.

fps.json, latency.json and accuracy.json

method_config

Specify the model inference parameters.

core: specify the inference core id.
model_file: specify the model file.

fps.json, latency.json and accuracy.json

Note

core x5 has only one bpu core, which can only be set to 0 (core any) or 1 (core 0).

Post-processing configurations:

Parameter name	Description	Involved json files
thread_count	Specify the post-processing thread count, in range `1-8`.	fps.json, latency.json and accuracy.json
method_type	Specify the post-processing method.	fps.json, latency.json and accuracy.json
method_config	Specify the post-processing parameters.	fps.json, latency.json and accuracy.json

9.4.3.3. Performance Evaluation¶

Performance evaluation is divided into latency and fps.

9.4.3.3.1. How to Use Performance Evaluation Scripts¶

To evaluate the latency:

In the directory of the to-be-evaluated model, run sh latency.sh to evaluate single frame latency, as shown below:

I0419 02:35:07.041095 39124 output_plugin.cc:80]  Infer latency:  [avg:  13.124ms,  max:  13.946ms,  min:  13.048ms], Post process latency: [avg:  3.584ms,  max:  3.650ms,  min:  3.498ms].

Note

infer denotes the time consumption of model inference.
Post process denotes the time consumption of post-processing.

fps:

This function uses multi-threaded concurrency and is designed to allow the model to reach the ultimate performance on BPU. Due to the multi-thread concurrency and data sampling, the frame rate value will be low during the start-up phase, then the frame rate will increase and gradually stabilize, with the frame rate fluctuating within 0.5%.. To test the frame rate, go to the model directory and run sh fps.sh, as shown below.

I0419 02:35:00.044417 39094 output_plugin.cc:109]  Throughput: 1129.39fps      # model frame rate

9.4.3.3.2. About Command-line Parameters¶

The fps.sh script is shown as below:

#!/bin/sh

source ../../env.sh
export SHOW_FPS_LOG=1
export STAT_CYCLE=100                             # specify environment variable, fps log printing frequency

${app} \
  --config_file=workflow_fps.json \
  --log_level=1

The latency.sh script is shown as below:

#!/bin/sh

source ../../env.sh
export SHOW_LATENCY_LOG=1                         # specify environment variable, print latency level log
export STAT_CYCLE=50                              # specify environment variable, latency log printing frequency

${app} \
  --config_file=workflow_latency.json \
  --log_level=1

9.4.3.3.3. Result Visualization¶

If you want to see the effect of a single inference of the model, you can modify workflow_latency.json and re-run the latency.sh script to generate the display effect in the output_dir directory.

Attention

When the display effect is generated, the script will run slowly due to the dump effect. Only the latency.sh script dump is supported.

The Visual operation steps are as follows:

Modify the latency configuration file

"output_config": {
  "output_type": "image",
  "enable_view_output": true,                         # turn on visualization
  "view_output_dir": "./output_dir",                  # visualization result output path
  "image_list_enable": true,
  "in_order": false
}

Execute the latency.sh script

sh latency.sh

Attention

The visualization of the bev model needs to specify the scene information and the path of the homography matrix. The homography matrix is used for the conversion of the camera perspective and the bird’s-eye view. Different scenes have their own homography matrices.

The workflow_latency.json configuration file of the bev model is recommended to be modified as follows:

"output_config": {
  "output_type": "image",
  "enable_view_output": true,                            # turn on visualization
  "view_output_dir": "./output_dir",                     # visualization result output path
  "bev_ego2img_info": [
    "../../config/visible/bev/scenes.json",              # scene information of input file
    "../../config/visible/bev/boston.bin",               # homography matrix of the boston scene
    "../../config/visible/bev/singapore.bin"             # homography matrix of the singapore scene
  ],
  "image_list_enable": true,
  "in_order": false
}

The visualization results that can be achieved by different types of models are different, refer to the table below:

model category	visualization
classification
detection 2d
detection 3d
segmentation
keypoint
lane line
optical flow
lidar
lidar multitask
bev
traj_pred
disparity_pred

Attention

If you need to visualize images other than minidata during trajectory prediction visualization, you need to configure additional road information and trajectory information files in minidata/argoverse1/visualization. You can use the densent_process.py preprocessing script to generate configuration files, and set –is-gen-visual-config parameter to true.

9.4.3.4. Model Accuracy Evaluation¶

Take the following 5 steps to perform the model evaluation:

1.data pre-process.

2.data mounting.

3.lst file generation.

4.model inference.

5.model accuracy computing.

9.4.3.4.1. Data Pre-processing¶

The following section will provide the description of the PTQ and QAT model data preprocessing methods.

PTQ Model Data Pre-processing:

To the PTQ model data pre-processing, run the hb_eval_preprocess tool in x86 to pre-process data. The so-called pre-processing refers to the special processing operations before images are fed into the model. For example: resize, crop and padding, etc. The tool is integrated into the horizon_tc_ui tool and it will be available after the tool is installed using the install script. After the raw dataset is pre-processed by the tool, the corresponding pre-processed binary file .bin file set of the model will be generated. You can directly run hb_eval_preprocess --help for help.

Tip

About the hb_eval_preprocess tool command line parameters and usage, you can type hb_eval_preprocess -h, or see the hb_eval_preprocess Tool in the PTQ tools guide.

The datasets corresponding to each model in the sample package are described in detail below, as well as the pre-processing operations for the corresponding datasets.

The datasets used for PTQ models include the following:

Dataset

Description

VOC

Dataset

The VOC dataset is used for evaluating the ssd_mobilenetv1 model.

1.The dataset download and reference download structure can be found in section Preparing Datasets. The sample mainly use the val.txt file under the Main folder, the source images under the JPEGImages folder and the annotations under the Annotations folder.

2.Dataset preprocessing methods:

hb_eval_preprocess -m ssd_mobilenetv1 -i VOCdevkit/VOC2012/JPEGImages \

                   -v VOCdevkit/VOC2012/ImageSets/Main/val.txt -o ./pre_ssd_mobilenetv1

COCO

Dataset

The COCO dataset is used for evaluating the centernet_resnet101, detr_efficientnetb3_mscoco, detr_resnet50_mscoco, yolov2_darknet19, yolov3_darknet53, yolov3_vargdarknet, yolov5x, preq_qat_fcos_efficientnetb0, preq_qat_fcos_efficientnetb2 and preq_qat_fcos_efficientnetb3.

1.The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the instances_val2017.json annotation file under the annotations folder images under the images folder.

2.Dataset preprocessing methods:

hb_eval_preprocess -m model_name -i coco/coco_val2017/images -o ./pre_model_name

ImageNet

Dataset

The ImageNet dataset is used for evaluating the efficientnasnet_m, efficientnasnet_s, efficientnet_lite0, efficientnet_lite1, efficientnet_lite2, efficientnet_lite3, efficientnet_lite4, googlenet, mobilenetv1, mobilenetv2, resnet18 and vargconvnet models.

1.The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the annotation file val.txt and the source images in the val directory.

2.Dataset preprocessing methods:

hb_eval_preprocess -m model_name -i imagenet/val -o ./pre_model_name

Cityscapes

Dataset

The Cityscapes dataset is used for evaluating the deeplabv3plus_efficientnetb0, deeplabv3plus_efficientnetm1, deeplabv3plus_efficientnetm2 and fastscnn_efficientnetb0 models.

1.The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the annotation files under the ./gtFine/val folder and the source images in the ./leftImg8bit/val folder.

2.Dataset preprocessing methods:

hb_eval_preprocess -m model_name -i cityscapes/leftImg8bit/val -o ./pre_model_name

QAT Model Data Pre-processing:

The QAT model data pre-process needs to execute the preprocess scripts in ai_benchmark_x5/x5/qat/tools/eval_preprocess in the x86 environment. The datasets corresponding to each model in the sample package are described in detail as below, as well as the pre-processing operations for the corresponding datasets.

Tip

Before use, please modify the dataset path and save path in the script to make the script run properly.

Dataset

Description

ImageNet

Dataset

The ImageNet dataset is used for evaluating the mixvargenet_imagenet, mobilenetv1_imagenet, mobilenetv2_imagenet, resnet50_imagenet, horizon_swin_transformer_imagenet and vargnetv2_imagenet QAT models.

Dataset preprocessing methods:

python3 imagenet.py --image-path=./standard_imagenet/val/ --save-path=./pre_model_name

VOC

Dataset

The VOC dataset is used for evaluating the yolo_mobilenetv1_voc model.

Dataset preprocessing methods:

python3 voc.py --image-path=./VOCdevkit/VOC2012/JPEGImages/ --save-path=./pre_yolov3_mobilenetv1

COCO

Dataset

The COCO dataset is used for evaluating the fcos_efficientnetb0_mscoco and retinanet_vargnetv2_fpn_mscoco QAT models.

Dataset preprocessing methods:

fcos_efficientnetb0_mscoco

python3 fcos_process.py --image-path=./mscoco/images/val2017/ \

                        --label-path=./mscoco/images/annotations/instances_val2017.json \

                        --save-path=./pre_fcos_efficientnetb0

retinanet_vargnetv2_fpn_mscoco

python3 retinanet_process.py --image-path=./mscoco/images/val2017/ \

                             --label-path=./mscoco/images/annotations/instances_val2017.json \

                             --save-path=./pre_retinanet

Cityscapes

Dataset

The Cityscapes dataset is used for evaluating the unet_mobilenetv1_cityscapes model, it doesn’t require preprocessing to use validation set data directly.

FlyingChairs

Dataset

The FlyingChairs dataset is used for evaluating the pwcnet_pwcnetneck_flyingchairs model.

1.You can download this dataset from the official website: FlyingChairs dataset official website download address. The sample mainly uses the data and the ./FlyingChairs_train_val.txt annotation file under the FlyingChairs_release/data folder. We recommend that you unzip the downloaded dataset into the following structure. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── FlyingChairs_release
│   └── data
│       ├── 00001_img1.ppm
│       ├── 00001_img2.ppm
│       └── 00001_flow.ppm
├── FlyingChairs_train_val.txt

Among them:

{id}_img1.ppm and {id}_img1.ppm are an image pair, the image width size is 512, the height size is 384; {id} is serial numbers from 00001 to 22872, and each image pair is labeled {id}_flow.flo.
The FlyingChairs_train_val.txt file is used to divide the training set and the validation set. The label value of 2 indicates the validation set.

2.Dataset preprocessing methods:

python3 pwcnet_process.py --input-path=./flyingchairs/FlyingChairs_release/data/ \

                          --val-file=./flyingchairs/FlyingChairs_train_val.txt \

                          --output-path=./pre_pwcnet_opticalflow

Kitti3D

Dataset

The Kitti3D dataset is used for evaluating the pointpillars_kitti_car model.

1.You can download this dataset from the official website: Kitti3D dataset official website download address. We recommend you download the following zip files. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── kitti3d
    ├── data_object_calib.zip        # camera calibration matrices of object data set
    ├── data_object_image_2.zip      # left color images of object data set
    ├── data_object_label_2.zip      # taining labels of object data set
    └── data_object_veloodyne.zip    # velodyne point cloud

We recommend that you unzip the downloaded dataset into the following structure.

.
├── kitti3d_origin
    ├── ImageSets
    │   ├── test.txt
    │   ├── train.txt
    │   ├── trainval.txt
    │   └── val.txt
    ├── testing
    │   ├── calib
    │   ├── image_2
    │   └── velodyne
    └── training
        ├── calib
        ├── image_2
        ├── label_2
        └── velodyne

2.Dataset preprocessing methods:

python3 pointpillars_process.py --data-path=./kitti3d_origin  --save-path=./pre_kitti3d  --height=1 --width=150000

Culane

Dataset

The Culane dataset is used for evaluating the ganet_mixvargenet_culane model.

1.You can download this dataset from the official website: Culane dataset official website download address. We recommend you download the following zip files. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── culane
    ├── annotations_new.tar.gz
    ├── driver_23_30frame.tar.gz
    ├── driver_37_30frame.tar.gz
    ├── driver_100_30frame.tar.gz
    ├── driver_161_90frame.tar.gz
    ├── driver_182_30frame.tar.gz
    ├── driver_193_90frame.tar.gz
    ├── laneseg_label_w16.tar.gz
    └── list.tar.gz

In which, the annotations_new.tar.gz needs to be unpacked last in order to make corrections to the original annotations file. We recommend that you unzip the downloaded dataset into the following structure.

.
├── culane                     # root directory
    ├── driver_23_30frame      # datasets and annotations
    │   ├── 05151640_0419.MP4  # a section of the dataset, containing each frame of the picture
    │   │   ├──00000.jpg       # source images
    │   │   ├──00000.lines.txt # annotation file where each line gives the x,y coordinates of the lane marking keypoints
    │   ......
    ├── driver_37_30frame
    ├── driver_100_30frame
    ├── driver_161_90frame
    ├── driver_182_30frame
    ├── driver_193_90frame
    ├── laneseg_label_w16      # lane segment labels
    └── list                   # train, validate, test list

2.Dataset preprocessing methods:

python3 ganet_process.py --image-path=./culane  --save-path=./pre_culane

Nuscenes

Dataset

The Nuscenes dataset is used for evaluating the fcos3d_efficientnetb0_nuscenes, centerpoint_pointpillar_nuscenes, centerpoint_mixvargnet_multitask_nuscenes, bev_gkt_mixvargenet_multitask_nuscenes, bev_lss_efficientnetb0_multitask_nuscenes, bev_ipm_efficientnetb0_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, detr3d_efficientnetb3_nuscenes models.

1.You can download this dataset from the official website: Nuscenes dataset official website download address. We recommend you download the following zip files. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── Nuscenes
    ├── nuScenes-map-expansion-v1.3.zip
    ├── nuScenes-map-expansion-v1.2.zip
    ├── nuScenes-map-expansion-v1.1.zip
    ├── nuScenes-map-expansion-v1.0.zip
    ├── v1.0-mini.tar
    ├── v1.0-test_blobs.tar
    ├── v1.0-test_meta.tar
    ├── v1.0-trainval01_blobs.tar
    ├── v1.0-trainval02_blobs.tar
    ├── v1.0-trainval03_blobs.tar
    ├── v1.0-trainval04_blobs.tar
    ├── v1.0-trainval05_blobs.tar
    ├── v1.0-trainval06_blobs.tar
    ├── v1.0-trainval07_blobs.tar
    ├── v1.0-trainval08_blobs.tar
    ├── v1.0-trainval09_blobs.tar
    ├── v1.0-trainval10_blobs.tar
    └── v1.0-trainval_meta.tar

For the lidar multitask model, you have to download the lidar segmentation tag lidarseg from the official website as well, and follow the tutorial on the official nuscenes website to update the v1.0-trainval. We recommend that you unzip the downloaded dataset into the following structure.

.
├── Nuscenes
    ├── can_bus
    ├── lidarseg
    ├── maps
    ├── nuscenes
    │   └── meta
    │       ├── maps
    │       ├── v1.0-mini
    │       └── v1.0-trainval
    ├── samples
    ├── sweeps
    ├── v1.0-mini
    └── v1.0-trainval

2.Dataset preprocessing methods:

Attention:

In addition to generating pre-processed images, the fcos3d_process.py will also process the internal parameters of the camera to be used, and generate the corresponding camera internal reference configuration files.
In addition to generating preprocessed data, centerpoint_preprocess.py, the bev_preprocess.py and lidar_preprocess.py will also generate a val_gt_infos.pkl file under the preprocessed data path for precision calculation.
The bev_preprocess.py needs to specify the name of the model through –model, the options are bev_gkt_mixvargenet_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, bev_ipm_efficientnetb0_multitask_nuscenes, bev_lss_efficientnetb0_multitask_nuscenes, detr3d_efficientnetb3_nuscenes

The reference commands are as follows:

fcos3d_efficientnetb0_nuscenes

python3 fcos3d_process.py --src-data-dir=./Nuscenes \

                          --file-path=../../script/config/model/data_name_list/nuscenes_names.txt \

                          --save-path=./processed_fcos3d_images

centerpoint_pointpillar_nuscenes

python3 centerpoint_preprocess.py --data-path=./Nuscenes --save-path=./nuscenes_lidar_val

bev model

python3 bev_preprocess.py --model=model_name --data-path=./Nuscenes --meta-path=./Nuscenes/meta \
                          --reference-path=../../script/config/reference_points --save-path=./nuscenes_bev_val

centerpoint_mixvargnet_multitask_nuscenes

python3 lidar_preprocess.py --data-path=./Nuscenes --save-path=./nuscenes_lidar_val

Mot17

Dataset

The Mot17 dataset is used for evaluating the motr_efficientnetb3_mot17 model.

1.You can download this dataset from the official website: Mot17 dataset official website download address. We recommend that you unzip the downloaded dataset into the following structure. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── valdata                     # root directory
    ├── gt_val
    │   ├── MOT17-02-SDP
    │   ├── MOT17-04-SDP
    │   ├── MOT17-05-SDP
    │   ├── MOT17-09-SDP
    │   ├── MOT17-10-SDP
    │   ├── MOT17-11-SDP
    │   ├── MOT17-13-SDP
    ├── images
    │   └── train
    │       ├── MOT17-04-SDP
    │       ├── MOT17-05-SDP
    │       ├── MOT17-09-SDP
    │       ├── MOT17-10-SDP
    │       ├── MOT17-11-SDP
    │       ├── MOT17-13-SDP
    └── mot17.val

2.Dataset preprocessing methods:

python3 motr_process.py --image-path=./valdata/images/train --save-path=./processed_motr

Carfusion

Dataset

The Carfusion dataset is used for evaluating the keypoint_efficientnetb0_carfusion model.

1.You can download this dataset from the official website: Carfusion dataset official website download address. We recommend that you unzip the downloaded dataset into the following structure. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── carfusion                     # root directory
    ├── train
    └── test

2.Dataset preprocessing methods:

# First generate the data required for evaluation(If you are using this dataset for the first time,
you must use the following script to generate the data required for the evaluation)

python3 gen_carfusion_data.py --src-data-path=carfusion --out-dir=cropped_data --num-workers 2

The directory after executing the first script is as follows:

.
├── cropped_data                     # root directory
    ├── test
    └── simple_anno

Ensure that the data root address is at the same level as the cropped_data, and then run the following command:

python3 keypoints_preprocess.py --data-root=./ \

                                --label-path=cropped_data/simple_anno/keypoints_test.json \

                                --save-path=./processed_carfusion

Argoverse1

Dataset

The Argoverse1 dataset is used for evaluating the densetnt_vectornet_argoverse1 model.

1.You can download this dataset from the official website: Argoverse1 dataset official website download address. We recommend that you unzip the downloaded dataset into the following structure. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── arogverse-1                     # root directory
    ├── map_files
    └── val

2.Dataset preprocessing methods:

The densetnt_process.py generates the appropriate review meta file under src-path in addition to the preprocessed input. The evaluation only needs to use the two parameters --src-path and --dst-path, and there is no need to pay attention to other parameters.

python3 densetnt_process.py --src-path=arogverse-1 --dst-path=processed_arogverse1

SceneFlow

Dataset

The SceneFlow dataset is used for evaluating the stereonetplus_mixvargenet_sceneflow model.

1.You can download this dataset from the official website: SceneFlow dataset official website download address. We recommend that you unzip the downloaded dataset into the following structure. Please contact D-Robotics if you encounter any problems during data preparation.

.
├── SceneFlow                     # root directory
    ├── FlyingThings3D
    │   ├── disparity
    │   ├── frames_finalpass
    └── SceneFlow_finalpass_test.txt

2.Dataset preprocessing methods:

In addition to generating preprocessed data, the stereonet_preprocess.py, bev_preprocess.py and lidar_preprocess.py will also generate a val_gt_infos.pkl file under the preprocessed data path for precision calculation.

python3 stereonet_preprocess.py --data-path=SceneFlow/ \

                                --data-list=SceneFlow/SceneFlow_finalpass_test.txt \

                                --save-path=sceneflow_val

9.4.3.4.2. Model Mounting¶

Because datasets are huge, it is recommended to mount them for dev board to load, rather than to copy them into the dev board.

Server PC terminal (root permission is required):

Attention

A root permission is required to run this command.

1.Edit one line into /etc/exports: /nfs *(insecure,rw,sync,all_squash,anonuid=1000,anongid=1000,no_subtree_check). Wherein, /nfs denotes mounting path of local machine, it can be replaced by user-specified directory.

2.Run exportfs -a -r to bring /etc/exports into effect.

Board terminal:

1.Create the directory to be mounted: mkdir -p /mnt.

2.Mount: mount -t nfs {PC terminal IP}:/nfs /mnt -o nolock.

Mount the /nfs folder at PC terminal to the /mnt folder in dev board. In this way, mount the folder in which contains preprocessed folder to dev board and create a soft link of /data folder in the /ptq or /qat folder (at the same directory level as /script) in dev board.

9.4.3.4.3. Lst File Generation¶

The running process of precision calculation script in the sample is:

According to the value of image_list_file in workflow_accuracy.json , find the lst file of the corresponding data set.
Load each preprocessing file according to the path information of preprocessing file stored in lst file, and then perform the inference.

Therefore, after generating the preprocessing file, you need to generate the corresponding LST file, and write the path of each preprocessing file into the lst file, which is related to the storage location of the data set at the board end. Here, we recommend that its storage location shall be the same level as the ./data/dataset_name/pre_model_name folder.

The structure of the PTQ pre-processed data set is as follows:

|── ptq
|   |── data
|   |   |── cityscapes
|   |   |   |── pre_deeplabv3plus_efficientnetb0
|   |   |   |   |── xxxx.bin                            # pre-processed binary file
|   |   |   |   |── ....
|   |   |   |── pre_deeplabv3plus_efficientnetb0.lst    # lst file: record the path of each preprocessing file
|   |   |   |── ....
|   |   |── ....
|   |── model
|   |   |── ...
|   |── script
|   |   |── ...

The structure of the QAT pre-processed data set is as follows:

|── qat
|   |── data
|   |   |── carfusion
|   |   |   |── pre_keypoints
|   |   |   |   |── xxxx                   # pre-processed data
|   |   |   |   |── ....
|   |   |   |── pre_carfusion.lst          # lst file: record the path of each preprocessing file
|   |   |── ....
|   |── model
|   |   |── ...
|   |── script
|   |   |── ...

The corresponding LST file is generated by reference as follows:

Except for the Densetnt_vectornet_argoverse1, Bev, Motr_efficientnetb3_mot17, and Stereonetplus_mixvargenet_sceneflow models, the reference generation method of the lst files for the other models:

find ../../../data/coco/pre_centernet_resnet101 -name "*bin*" > ../../../data/coco/pre_centernet_resnet101.lst

Note

The parameters after -name need to be adjusted according to the format of the preprocessed dataset, such as bin, png.

The path stored in the generated lst file is a relative path: ../../../data/coco/pre_centernet_resnet101/ , which can match the workflow_accuracy.json default configuration path.

If you need to change the storage location of the preprocessing dataset, you need to ensure that the corresponding lst file can be used by workflow_accuracy.json, Secondly, it is necessary to ensure that the program can read the corresponding preprocessing file according to the path information in lst file.

For the Densetnt_vectornet_argoverse1, Bev, Motr_efficientnetb3_mot17, and Stereonetplus_mixvargenet_sceneflow models, the reference generation method of the lst files:

Model name	Reference generation method and description of lst file
Densetnt	Reference generation method: sh generate_acc_lst.sh The path stored in the generated `lst` file is a relative path: `../../../data/argoverse1/densetnt/` , which can match the `workflow_accuracy.json` default configuration path.
Bev	Take the bev_ipm_efficientnetb0_multitask_nuscenes as an example. This model has two types of input: images and reference points. The input image and reference point of the same frame have the same name. In order to ensure that the input corresponds, you need to add `sort` to sort by name when executing the `find` command. Reference generation method: find ../../../data/nuscenes_bev/images -name "bin" \| sort > ../../../data/nuscenes_bev/images.lst find ../../../data/nuscenes_bev/reference_points0 -name "bin" \| sort > ../../../data/nuscenes_bev/reference_points0.lst In addition to images and reference points, detr3d_efficientnetb3_nuscenes also have coords, masks and position embedding inputs. The method of generating lst is as follows: find ../../../data/nuscenes_bev/coords0 -name "bin" \| sort > ../../../data/nuscenes_bev/coords0.lst find ../../../data/nuscenes_bev/coords1 -name "bin" \| sort > ../../../data/nuscenes_bev/coords1.lst find ../../../data/nuscenes_bev/coords2 -name "bin" \| sort > ../../../data/nuscenes_bev/coords2.lst find ../../../data/nuscenes_bev/coords3 -name "bin" \| sort > ../../../data/nuscenes_bev/coords3.lst find ../../../data/nuscenes_bev/masks -name "bin" \| sort > ../../../data/nuscenes_bev/masks.lst Attention: For the bev model, the `reference_points` of detr3d_efficientnetb3_nuscenes are used for model post processing. You need to configure the correct path in `workflow_accuracy.json` to ensure that the program can read the corresponding reference points file. In addition, bev_ipm_4d_efficientnetb0_multitask_nuscenes is a timing model, which requires input order. Therefore, we provide a script `gen_file_list.sh` for generating lst files, the usage is as follows: sh gen_file_list.sh The path stored in the generated `lst` file is a relative path: `../../../data/nuscenes_bev/` , which can match the `workflow_accuracy.json` default configuration path. If you need to change the storage location of the preprocessing dataset, you need to ensure that the corresponding `lst` file can be used by `workflow_accuracy.json`. Secondly, it is necessary to ensure that the program can read the corresponding preprocessing file according to the path information in `lst` file.
Motr	Reference generation method: sh generate_acc_lst.sh The path stored in the generated `lst` file is a relative path: `../../../data/mot17/motr/` , which can match the `workflow_accuracy.json` default configuration path. If you need to change the storage location of the preprocessing dataset, you need to ensure that the corresponding `lst` file can be used by `workflow_accuracy.json`. Secondly, it is necessary to ensure that the program can read the corresponding preprocessing file according to the path information in `lst` file.
Stereonet_plus	Take the stereonetplus_mixvargenet_sceneflow as an example. The input left image and right the same frame have the same name. In order to ensure that the input corresponds, you need to add `sort` to sort by name when executing the `find` command. find ../../../data/sceneflow/left -name "png" \| sort > ../../../data/sceneflow/left.lst find ../../../data/sceneflow/right -name "png" \| sort > ../../../data/sceneflow/right.lst

9.4.3.4.4. Model Inference¶

The accuracy.sh script is shown as below:

#!/bin/sh

source ../../base_config.sh                      # load basic configurations
export SHOW_FPS_LOG=1                            # specify environment variable, print fps level log

${app} \                                         # executable program defined in the accuracy.sh script
  --config_file=workflow_accuracy.json \         # load workflow configuration file of accuracy evaluation
  --log_level=2                                  # specify log level

After the data has been mounted, log in dev board and run the accuracy.sh script in the centernet_resnet101 directory, as shown below:

root@x5dvb-hynix8G:/userdata/ptq/script/detection/centernet_resnet101# sh accuracy.sh
../../aarch64/bin/example --config_file=workflow_accuracy.json --log_level=2
...
I0419 03:14:51.158655 39555 infer_method.cc:107] Predict DoProcess finished.
I0419 03:14:51.187361 39556 ptq_centernet_post_process_method.cc:558] PTQCenternetPostProcessMethod DoProcess finished, predict result: [{"bbox":[-1.518860,71.691170,574.934631,638.294922],"prob":0.750647,"label":21,"class_name":"
I0118 14:02:43.636204 24782 ptq_centernet_post_process_method.cc:558] PTQCenternetPostProcessMethod DoProcess finished, predict result: [{"bbox":[3.432283,164.936249,157.480042,264.276825],"prob":0.544454,"label":62,"class_name":"
...

Inference results will be saved into the eval.log file dumped by dev board program.

9.4.3.4.5. Model Accuracy Computing¶

Attention

Please perform the accuracy calculation in docker environment or Linux environment.

Accuracy computing is presented in two cases: PTQ model accuracy computing and QAT model accuracy computing.

PTQ Model Accuracy Computing:

For the PTQ model, the model accuracy computing scripts are placed under the ptq/tools/python_tools/accuracy_tools folder, in which:

The cls_eval.py script is used for computing accuracy of classification models.
The coco_det_eval.py script is used for computing the accuracy of models evaluated using the COCO dataset.
The parsing_eval.py script is used for computing the accuracy of segmentation models evaluated using the Cityscapes dataset.
The voc_det_eval.py script is used for computing the accuracy of detection models using the VOC dataset.

Below we provide you with the description of the different types of PTQ model accuracy computing:

Model type

Description of the accuracy computing

Classification

Model

Method to compute the accuracy of those models using the CIFAR-10 and ImageNet datasets is shown as below:

python3 cls_eval.py --log_file=eval.log --gt_file=val.txt

In which:

log_file refers to inference result file of classification models.
gt_file refers to the annotation files of CIFAR-10 and ImageNet datasets.

Detection

Model

1.Method to compute the accuracy of those models using the COCO dataset is shown as below:

python3 coco_det_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json

In which:

eval_result_path refers to inference result file of detection models.
annotation_path refers to the annotation file of of the COCO dataset.

2.Method to compute the accuracy of those detection models using the VOC dataset is shown as below:

python3 voc_det_eval.py --eval_result_path=eval.log --annotation_path=../Annotations --val_txt_path=../val.txt

In which:

eval_result_path refers to the inference result file of detection models.
annotation_path refers to the annotation file of the VOC dataset.
val_txt_path refers to the val.txt file in the ImageSets/Main folder in the VOC dataset.

Segmentation

Model

Method to compute the accuracy of those segmentation models using the Cityscapes dataset is shown as below:

python3 parsing_eval.py --log_file=eval.log --gt_path=cityscapes/gtFine/val

In which:

log_file refers to the inference result file of segmentation models.
gt_path refers to the annotation file of the Cityscapes dataset.

QAT Model Accuracy Computing:

For the QAT model, the model accuracy computing scripts are placed under the qat/tools/python_tools/accuracy_tools folder, in which:

The bev_eval.py is used to calculate the accuracy of bev model.
The centerpoint_eval.py is used to calculate the accuracy of centerpoint lidar 3D model.
The cls_eval.py is used to calculate the accuracy of the classification model.
The densetnt_eval.py is used to calculate the accuracy of densetnt_vectornet_argoverse1 model.
The detr_eval.py is used to calculate the accuracy of detr model.
The fcos3d_eval.py is used to calculate the accuracy of fcos3d_efficientnetb0_nuscenes model.
The fcos_eval.py is used to calculate the accuracy of fcos model.
The ganet_eval.py is used to calculate the accuracy of ganet_mixvargenet_culane model.
The keypoints_eval.py is used to calculate the accuracy of keypoint_efficientnetb0_carfusion model.
The lidar_multitask_eval.py is used to calculate the accuracy of centerpoint_mixvargnet_multitask_nuscenes model.
The motr_eval.py is used to calculate the accuracy of motr_efficientnetb3_mot17 model.
The parsing_eval.py is used to calculate the accuracy of segmentation model.
The pointpillars_eval.py is used to calculate the accuracy of pointpillars_kitti_car model.
The pwcnet_eval.py is used to calculate the accuracy of opticalflow model pwcnet_pwcnetneck_flyingchairs.
The retinanet_eval.py is used to calculate the accuracy of retinanet_vargnetv2_fpn_mscoco model.
The yolov3_eval.py is used to calculate the accuracy of yolov3 model.
The stereonet_eval.py is used to calculate the accuracy of stereonetplus_mixvargenet_sceneflow model.

Below we provide you with the description of the different types of QAT model accuracy computing:

Model type

Description of the accuracy computing

Bev

Model

Method to compute the accuracy of those models using the Nuscenes dataset is shown as below:

python3 bev_eval.py --det_eval_path=bev_det_eval.log --seg_eval_path=bev_seg_eval.log \

                    --gt_files_path=./nuscenes_bev_val/val_gt_infos.pkl --meta_dir=./Nuscenes/meta/

# detr3d_efficientnetb3_nuscenes is bev detection model,
# and there is no need to specify --seg_eval_path

python3 bev_eval.py --det_eval_path=eval.log --gt_files_path=./nuscenes_bev_val/val_gt_infos.pkl \

                    --meta_dir=./Nuscenes/meta/

In which:

det_eval_path refers to detection inference result file of bev models.
seg_eval_path refers to segment inference result file of bev models.
gt_files_path refers to gt file generated by preprocessing the nuscenes dataset.
meta_dir refers to Nuscenes dataset meta data.

Classification

Model

Method to compute the accuracy of those models using the CIFAR-10 and ImageNet datasets is shown as below:

python3 cls_eval.py --log_file=eval.log --gt_file=val.txt

In which:

log_file refers to inference result file of classification models.
gt_file refers to the annotation files of CIFAR-10 and ImageNet datasets.

Detection

Model

1.Method to compute the accuracy of those models using the COCO dataset is shown as below:

python3 fcos_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json \

                     --image_path=./mscoco/images/val2017/

# The qat fcos model needs to add --is_qat=True

python3 fcos_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json \

                     --image_path=./mscoco/images/val2017/ --is_qat=True

In which:

eval_result_path refers to inference result file of detection model fcos.
annotation_path refers to the annotation file of of the COCO dataset.
image_path refers to COCO dataset source data.
is_qat refers to whether result evaluation of qat fcos model.

python3 retinanet_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json \

                          --image_path=./mscoco/images/val2017/

In which:

eval_result_path refers to inference result file of detection model retinanet.
annotation_path refers to the annotation file of of the COCO dataset.
image_path refers to COCO dataset source data.

python3 detr_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json \

                          --image_path=./mscoco/images/val2017/

In which:

eval_result_path refers to inference result file of detection model retinanet.
annotation_path refers to the annotation file of of the COCO dataset.
image_path refers to COCO dataset source data.

2.Method to compute the accuracy of those detection models using the VOC dataset is shown as below:

python3 yolov3_eval.py --eval_result_path=eval.log --annotation_path=../Annotations --val_txt_path=../val.txt \

                       --image_height=416  --image_width=416

In which:

eval_result_path refers to the inference result file of detection models.
annotation_path refers to the annotation file of the VOC dataset.
val_txt_path refers to the val.txt file in the …/ImageSets/Main folder in the VOC dataset.
image_height refers to the image height.
image_width refers to the image width.

3.Method to compute the accuracy of those detection models using the KITTI dataset is shown as below:

python3 pointpillars_eval.py --eval_result_path=eval.log --annotation_path=./val_gt_infos.pkl

In which:

eval_result_path refers to the inference result file of detection models.
annotation_path refers to the annotation file val_gt_infos.pkl of the KITTI dataset.

4.Method to compute the accuracy of those detection models using the Culane dataset is shown as below:

python3 ganet_eval.py --eval_path=eval.log --image_path=./culane

In which:

eval_result_path refers to the inference result file of detection models.
image_path refers to the Culane dataset.

5.Method to compute the accuracy of those models using the Nuscenes dataset is shown as below:

python3 fcos3d_eval.py --eval_result_path=eval.log --image_path=./Nuscenes

In which:

eval_result_path refers to inference result file of detection models.
image_path refers to Nuscenes dataset source data.

python3 centerpoint_eval.py --predict_result_path=eval.log --gt_files_path=./nuscenes_lidar_val/val_gt_infos.pkl \

                            --meta_dir=./Nuscenes/meta/

In which:

predict_result_path refers to inference result file of detection models.
gt_files_path refers to gt file generated by preprocessing the nuscenes dataset.
meta_dir refers to Nuscenes dataset meta data.

6.Method to compute the accuracy of those detection models using the Carfusion dataset is shown as below:

python3 keypoints_eval.py --anno_path=./processed_carfusion/processed_anno.json --eval_result_path=eval.log

In which:

anno_path refers to the processed_anno.json file generated by preprocessing.
eval_result_path refers to inference result file of detection models.

Segmentation

Model

Method to compute the accuracy of those segmentation models using the Cityscapes dataset is shown as below:

python3 parsing_eval.py --log_file=eval.log --gt_path=cityscapes/gtFine/val

In which:

log_file refers to the inference result file of segmentation models.
gt_path refers to the annotation file of the Cityscapes dataset.

Opticalflow

Model

Method to compute the accuracy of the pwcnet_pwcnetneck_flyingchairs model using the FlyingChairs dataset is shown as below:

python3 pwcnet_eval.py --log_file=eval.log --gt_path=./flyingchairs/FlyingChairs_release/data/ \

                       --val_file=./flyingchairs/FlyingChairs_train_val.txt

In which:

log_file refers to the inference result file of opticalflow estimation models.
val_file refers to the annotation file of the FlyingChairs dataset.
gt_path refers to FlyingChairs dataset source data.

Tracking

Model

Method to compute the accuracy of those models using the mot17 datasets is shown as below:

python3 motr_eval.py --eval_result_path=eval_log --gt_val_path=valdata/gt_val

In which:

eval_result_path refers to inference result file of tracking models.
gt_val_path refers to the annotation files of motr17 datasets.

Multitask

Model

Method to compute the accuracy of those models using the Nuscenes dataset is shown as below:

python3 lidar_multitask_eval.py --det_eval_path=det_eval.log --seg_eval_path=seg_eval.log \

                                --gt_files_path=./nuscenes_lidar_val/val_gt_infos.pkl --data_dir=./Nuscenes

In which:

det_eval_path refers to detection inference result file of models.
seg_eval_path refers to segment inference result file of models.
gt_files_path refers to gt file generated by preprocessing the nuscenes dataset.
data_dir refers to Nuscenes dataset.

Traj Pred

Model

Method to compute the accuracy of those models using the argoverse1t datasets is shown as below:

python3 densetnt_eval.py --eval_result_path=eval.log --meta_path=argoverse1/meta

In which:

eval_result_path refers to inference result file of traj pred models.
meta_path refers to the annotation file generated by the argoverse1 dataset, after using the preprocessing script, it is generated in the original dataset directory meta.

Disparity Pred

Model

Method to compute the accuracy of those models using the Sceneflow datasets is shown as below:

python3 stereonet_eval.py --log_file=eval.log --gt_file=val_gt_infos.pkl

In which:

log_file refers to inference result file of disparity pred models.
gt_file refers to gt file generated by preprocessing the Sceneflow dataset.

9.4.4. Model Integration¶

9.4.4.1. Pre-processing¶

Users can add model pre-processing as needed and deploy it to CPU , taking centerpoint_pointpillar_nuscenes as an example:

1.Add the preprocessing file qat_centerpoint_preprocess_method.cc and the header file qat_centerpoint_preprocess_method.h.

2.Add model preprocessing configuration file.

9.4.4.1.1. Add Pre-processing File¶

The pre-processing qat_centerpoint_preprocess_method.cc files are placed under the ai_benchmark/code/src/method/ folder. While the header file qat_centerpoint_preprocess_method.h files are placed under the ai_benchmark/code/include/method/ floder.

|── ai_benchmark
|  |── code                                                 # source code of samples
|  |  |── include
|  |  |  |── method                                         # add your header files into this folder
|  |  |  |  |── qat_centerpoint_preprocess_method.h
|  |  |  |  |── ...
|  |  |── src
|  |  |  |── method                                         # add your .cc preprocess files into this folder
|  |  |  |  |── qat_centerpoint_preprocess_method.cc
|  |  |  |  |── ...

9.4.4.1.2. Add Pre-processing Configuration File¶

|── ai_benchmark
|  |── x5/qat/script                                       # sample script folder
|  |  |── config
|  |  |  |── preprocess
|  |  |  |  |── centerpoint_preprocess_5dim.json           # pre-processing configuration file

9.4.4.1.3. To Evalute the Latency of Preprocess¶

Run sh latency.sh to evaluate single frame latency of preprocess, as shown below:

I0615 13:30:40.772293  3670 output_plugin.cc:91]  Pre process latency: [avg:  20.295ms,  max:  28.690ms,  min:  18.512ms], Infer latency:  [avg:  25.053ms,  max:  31.943ms,  min:  24.702ms], Post process latency: [avg:  52.760ms,  max:  54.099ms,  min:  51.992ms].

In which:

Pre process denotes the time consumption of pre-processing.
Infer denotes the time consumption of model inference.
Post process denotes the time consumption of post-processing.

9.4.4.2. Post-processing¶

Post-processing consists of 2 steps. Let’s take integration of CenterNet model as an example:

1.Add the post-processing file ptq_centernet_post_process_method.cc and the header file ptq_centernet_post_process_method.h.

2.Add a model execution script and a configuration file.

9.4.4.2.1. Add Post-processing File¶

Post-processing code file can reuse any post-processing files in the src/method directory. You only need to modify the InitFromJsonString function and the PostProcess function.

The InitFromJsonString function is used for loading the post-processing related parameters in the workflow.json. You can customize the corresponding input parameters. The PostProcess function is used for implementing post-processing logic.

The post-processing ptq_centernet_post_process_method.cc files are placed under the ai_benchmark/code/src/method/ folder. While the header files ptq_centernet_post_process_method.h are placed under the ai_benchmark/code/include/method/ folder.

|── ai_benchmark
|  |── code                                                 # source code of samples
|  |  |── include
|  |  |  |── method                                         # add your header files into this folder
|  |  |  |  |── ptq_centernet_post_process_method.h
|  |  |  |  |── ...
|  |  |── src
|  |  |  |── method                                         # add your .cc postprocess files into this folder
|  |  |  |  |── ptq_centernet_post_process_method.cc
|  |  |  |  |── ...

9.4.4.2.2. Add Model Execution and Configuration Files¶

Directory structure of scripts is shown as below:

The centerpoint_pointpillar_nuscenes model:

|── ai_benchmark
|  |── x5/qat/script                                       # sample script folder
|  |  |── detection
|  |  |  |── centerpoint_pointpillar_nuscenes
|  |  |  |  |── accuracy.sh                                # accuracy evaluation script
|  |  |  |  |── fps.sh                                     # performance evaluation script
|  |  |  |  |── latency.sh                                 # single-frame latency sample script
|  |  |  |  |── workflow_accuracy                          # accuracy configuration folder
|  |  |  |  |── workflow_fps.json                          # performance configuration file
|  |  |  |  |── workflow_latency.json                      # single-frame latency configuration file

The motr_efficientnetb3_mot17 model:

|── ai_benchmark
|  |── x5/ptq/script                                       # sample script folder
|  |  |── detection
|  |  |  |── centernet_resnet101
|  |  |  |  |── accuracy.sh                                # accuracy evaluation script
|  |  |  |  |── fps.sh                                     # performance evaluation script
|  |  |  |  |── latency.sh                                 # single-frame latency sample script
|  |  |  |  |── workflow_accuracy.json                     # accuracy configuration file
|  |  |  |  |── workflow_fps.json                          # performance configuration file
|  |  |  |  |── workflow_latency.json                      # single-frame latency configuration file

The models except for the centerpoint_pointpillar_nuscenes and motr_efficientnetb3_mot17:

|── ai_benchmark
|  |── x5/qat/script                                       # sample script folder
|  |  |── tracking
|  |  |  |── motr
|  |  |  |  |── accuracy.sh                                # accuracy evaluation script
|  |  |  |  |── fps.sh                                     # performance evaluation script
|  |  |  |  |── generate_acc_lst.sh                        # generate accuracy lst script
|  |  |  |  |── latency.sh                                 # single-frame latency sample script
|  |  |  |  |── workflow_accuracy                          # accuracy configuration folder
|  |  |  |  |── workflow_fps.json                          # performance configuration file
|  |  |  |  |── workflow_latency.json                      # single-frame latency configuration file

9.4.5. Helper Tools¶

9.4.5.1. Log¶

There are 2 types of logs: sample Log and DNN Log. Wherein, sample log refers to the log in the AI Benchmark Sample Package deliverables, while DNN log refers to the log in the embedded runtime library. Developers can specify logs as needed.

9.4.5.1.1. Sample Log¶

1.Log level.

Both glog and vlog are used in sample log and there are 4 customized log levels:

0: SYSTEM level, this log level is used for generating error information in sample code.
1: REPORT level, this log level is used for generating performance data in sample code.
2: DETAIL level, this log level is used for generating current system status in sample code.
3: DEBUG level, this log level is used for generating debugging information in sample code.

2.Set log levels.

Rules to set log levels: The default ranks of log level: DEBUG>DETAIL>REPORT>SYSTEM, the higher the level, the more logs will be output. That is, if you set a high level, the logs corresponding to your own level and the level below it will be output.

When running samples, specify the log_level parameter to set log levels. For example, if log_level=0, then SYSTEM log should be dumped; else if log_level=3, then DEBUG, DETAIL, REPORT and SYSTEM logs should be dumped.

9.4.5.1.2. Dnn Log¶

For the configuration of dnn logs, please read the Configuration Info section in the BPU SDK API DOC.

9.4.5.2. OP Time Consumption¶

9.4.5.2.1. General Descriptions¶

Use the HB_DNN_PROFILER_LOG_PATH environment variable to specify statistics of OP performance. Types and values of this environment variable are described as below:

HB_DNN_PROFILER_LOG_PATH=${path}: denotes the output path of OP node. After the program is executed, a profiler.log file should be generated.

9.4.5.2.2. Sample¶

Takeing mobilenetv1 as an example, as shown in the following code block: Start 1 threads to run the model at the same time, set export HB_DNN_PROFILER_LOG_PATH=. /, then the profiler.log file will output the performance data of the OPs.

The output information contains model_latency and task_latency. Wherein, model_latency contains the time consumption required to run each operator of the model; while task_latency contains the time consumption of each task of the model.

9.4.5.3. Dump Tool¶

Enable the HB_DNN_DUMP_PATH environment variable to dump the input and output of each node in inference process. The dump tool can check if there are consistency problems between simulator and real machine, i.e. Whether the output of the real machine and the simulator are exactly the same, given the same model and the same inputs.