6.3.4. Model board running application development instructions
6.3.4.1. Model Inference DNN API usage example description
Overview
This chapter introduces the specific usage of the horizon_runtime_sample sample package running on the model board. Developers can experience and use application development based on these examples to lower the development threshold.
The example package provides three examples:
Model Inference DNN API usage example.
Examples of special functions such as custom OP.
Miscellaneous examples of non-NV12 input models.
For details, please read below.
Tips:
Please refer to “Delivery Instructions” for obtaining the sample package of horizon_runtime_sample”.
Introduction to the sample code package structure
+---horizon_runtime_sample
|--code # sample source code
| |--00_quick_start # Quick start example, sample code for reading a single image for reasoning with mobilenetv1
| | |--CMakeLists.txt
| | |--CMakeLists_x86.txt
| | `--src
| |--01_api_tutorial # BPU SDK DNN API usage sample code
| | |--CMakeLists.txt
| | |--mem
| | |--model
| | |--roi_infer
| | `--tensor
| |--02_advanced_samples #Special function example
| | |--CMakeLists.txt
| | |--custom_identity
| | |--multi_input
| | |--multi_model_batch
| | `--nv12_batch
| |--03_misc # Miscellaneous Examples
| | |--CMakeLists.txt
| | |--lenet_gray
| | `--resnet_feature
| |--build_x5.sh # compile script on board aarch64
| |--build_x86.sh # PC side X86 compilation script
| |--CMakeLists.txt
| |--CMakeLists_x86.txt
| `--deps_gcc11.3 # Compile dependency library
| |--aarch64
| `--x86
|--README.md
`--x5
|--data # Preset data file
| |--cls_images
| |--custom_identity_data
| |--det_images
| |--dsl_data
| `--misc_data
|--model
| |--README.md
| `--runtime -> ../../../model_zoo/runtime/horizon_runtime_sample # The soft link points to the model in the OE package. The board-side running environment needs to specify the model path by itself
|--script # aarch64 sample run script
| |--00_quick_start
| |--01_api_tutorial
| |--02_advanced_samples
| |--03_misc
| `--README.md
`--script_x86 # x86 sample run script
|--00_quick_start
`--README.md
code: This directory is the source code of the example.
code/00_quick_start: Quick Start Example, based on the dnn API, using mobilenetv1 for single image model inference and result analysis.
code/01_api_tutorial: The dnn API uses instructional code, including four parts: mem, model, roi_inferand tensor.
code/02_advanced_samples: Examples of special features, including custom_identity, multi_input, multi_model_batchand nv12_batchfunctions.
ode/03_misc: Miscellaneous examples of non-NV12 input models.
code/build_x5.sh: The program compiles scripts with one click.
code/build_x86.sh: One-click scripts for x86 simulation environment.
code/deps_gcc11.3: The three-party dependencies required for the sample code. Users can replace or crop according to the actual situation when developing their own code programs.
x5: Sample running script, preset data and related models
Notice:
To run the private model on the board, please refer to the 00_quick_start/src/run_mobileNetV1_224x224.cc sample code flow for code rewriting. After the compilation is successful, you can test and verify it on the development board!
Environment Construction
Development Board Preparation
After getting the development board, please update the development version image to the latest version. Please refer to the System Update chapter content.
Ensure that local development machines and development boards can be connected remotely.
Compilation
Compilation requires the current environment to install the cross-compilation tool: arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu. Please use the development machine Docker image provided by the X5 algorithm toolchain to compile and use it directly. To obtain and use the Docker environment of the development machine, please read the chapter content of Environment Installation;
Depending on the development board you use, please use the build_x5.sh script in the horizon_runtime_sample/code directory to compile executable programs in the development board environment with one click. The executable programs and corresponding dependencies will be automatically copied to the aarch64 directory under the x5/script directory.
Remark:
The project specifies the path to the cross-compilation tool by obtaining the environment variable LINARO_GCC_ROOT. Before using it, users can check whether the local environment variable is the target cross-compilation tool.
If you need to specify the cross-compilation tool path, you can set the environment variable LINARO_GCC_ROOT, or directly modify the script build_x5.sh`, specify the variables CCandCXX`.
export CC=/opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc
export CXX=/opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-g++
Example usage
basic_samples example
The model inference example script is mainly in the x5/script and x5/script_x86 directories. The directory structure after compiling the program is as follows:
# X5 Use script information
├── data
│ ├── cls_images
│ │ ├── cat_cls.jpg
│ └── zebra_cls.jpg
│ ├── custom_identity_data
│ │ ├── input0.bin
│ │ └── input1.bin
│ ├── det_images
│ └── kite.jpg
│ ├── dsl_data
│ └── zebra_bgr.bin
│ └── misc_data
│ ├── 7.bin
│ └── np_0
├── model
│ ├── README.md
│ └── runtime -> ../../../model_zoo/runtime/horizon_runtime_sample
├── script # Compile and generate executable programs and dependency libraries
│ ├── 00_quick_start
│ │ ├── README.md
│ └── run_mobilenetV1.sh
│ ├── 01_api_tutorial
│ │ ├── README.md
│ │ ├── model.sh
│ │ ├── roi_infer.sh
│ │ ├── sys_mem.sh
│ └── tensor.sh
│ ├── 02_advanced_samples
│ │ ├── README.md
│ │ ├── plugin
│ │ │ └── custom_arm_op_custom_identity.sh
│ │ ├── run_multi_input.sh
│ │ ├── run_multi_model_batch.sh
│ │ └── run_nv12_batch.sh
│ ├── 03_misc
│ │ ├── README.md
│ │ ├── run_lenet.sh
│ │ └── run_resnet50_feature.sh
│ ├── README.md
│ └── aarch64
│ ├── bin
│ │ ├── model_example
│ │ ├── roi_infer
│ │ ├── run_custom_op
│ │ ├── run_lenet_gray
│ │ ├── run_mobileNetV1_224x224
│ │ ├── run_multi_model_batch
│ │ ├── run_resnet_feature
│ │ ├── sys_mem_example
│ │ └── tensor_example
│ └── lib
│ ├── libdnn.so
│ ├── libhbrt_bayes_aarch64.so
│ └── libopencv_world.so.3.4
└── script_x86
├── 00_quick_start
│ ├── README.md
│ └── run_mobilenetV1.sh
└── README.md
Remark::
The model folder contains the path of the model, where the
runtimefolder is a soft link and the link path is../../../model_zoo/runtime/horizon_runtime_sample, and you can directly find the model path in the delivery packageThe board operation environment needs to place the model in the
modelfolder
quick_start
In the 00_quick_start directory is a quick start example of model reasoning:
00_quick_start/
├── README.md
└── run_mobilenetV1.sh
run_mobilenetV1.sh: This script implements the example function of using the mobilenetv1 model to read a single image for inference. When using it, enter the 00_quick_start directory, and then directly execute sh run_mobilenetV1.sh, as shown in the following code block:
#!/bin/sh
root@x5dvb:/userdata/app/horizon/basic_samples/x5/script/00_quick_start# sh run_mobilenetV1.sh
../aarch64/bin/run_mobileNetV1_224x224 --model_file=../../model/runtime/mobilenetv1/mobilenetv1_224x224_nv12.bin --image_file=../../data/cls_images/zebra_cls.jpg --top_k=5
I0000 00:00:00.000000 10765 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
[A][DNN][packed_model.cpp:225][Model](2023-04-11,17:51:17.206.804) [HorizonRT] The model builder version = 1.15.0
I0411 17:51:17.244180 10765 run_mobileNetV1_224x224.cc:135] DNN runtime version: 1.17.2_(3.15.18 HBRT)
I0411 17:51:17.244376 10765 run_mobileNetV1_224x224.cc:252] input[0] name is data
I0411 17:51:17.244508 10765 run_mobileNetV1_224x224.cc:268] output[0] name is prob
I0411 17:51:17.260176 10765 run_mobileNetV1_224x224.cc:159] read image to tensor as nv12 success
I0411 17:51:17.262075 10765 run_mobileNetV1_224x224.cc:194] TOP 0 result id: 340
I0411 17:51:17.262118 10765 run_mobileNetV1_224x224.cc:194] TOP 1 result id: 292
I0411 17:51:17.262148 10765 run_mobileNetV1_224x224.cc:194] TOP 2 result id: 282
I0411 17:51:17.262177 10765 run_mobileNetV1_224x224.cc:194] TOP 3 result id: 83
I0411 17:51:17.262205 10765 run_mobileNetV1_224x224.cc:194] TOP 4 result id: 290
api_tutorial
An example in the 01_api_tutorial directory to describe how to use an embedded API. Its directory contains the following scripts:
├── model.sh
├── roi_infer.sh
├── sys_mem.sh
└── tensor.sh
model.sh: This script mainly implements the function of reading model information. When using it, go directly to the 01_api_tutorial directory, and then executesh model.shdirectly, as shown below:
#!/bin/sh
root@x5dvb-hynix8G:/userdata/horizon/x5/script/01_api_tutorial# sh model.sh
../aarch64/bin/model_example --model_file_list=../../model/runtime/mobilenetv1/mobilenetv1_224x224_nv12.bin
I0000 00:00:00.000000 10810 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
[A][DNN][packed_model.cpp:225][Model](2023-04-11,17:53:28.970.396) [HorizonRT] The model builder version = 1.15.0
I0411 17:53:29.007853 10810 model_example.cc:104] model count:1, model[0]: mobilenetv1_224x224_nv12
I0411 17:53:29.007939 10810 model_example.cc:112] hbDNNGetModelHandle [mobilenetv1_224x224_nv12] success!
I0411 17:53:29.008011 10810 model_example.cc:186] [mobilenetv1_224x224_nv12] Model Info: input num: 1, input[0] validShape: ( 1, 3, 224, 224 ), alignedShape: ( 1, 3, 224, 224 ), tensorType: 1, output num: 1, output[0] validShape: ( 1, 1000, 1, 1 ), alignedShape: ( 1, 1000, 1, 1 ), tensorType: 13
roi_infer.sh: This script mainly guides how to use thehbDNNRoiInferAPI. The function implemented by the sample code is to resize an image to the model input size, convert it to nv12 data, and give the roi box to perform model inference (infer). When using it, just go to the 01_api_tutorial directory and then executesh roi_infer.shdirectly.sys_mem.sh: This script mainly guides how to use thehbSysAllocMem,hbSysFlushMemandhbSysFreeMem. When using it, just go directly to the 01_api_tutorial directory and executesh sys_mem.sh.tensor.sh: This script mainly guides how to prepare the model input and output tensor. When using it, just go to the 01_api_tutorial directory and executesh tensor.sh, as shown below:
root@x5dvb-hynix8G:/userdata/horizon/x5/script/01_api_tutorial# sh tensor.sh
*****************************test_prepare_free_fn*************************************************
Tensor data type:0, Tensor layout: 2, shape:1x1x721x1836, aligned shape:1x1x721x1840
Tensor data type:1, Tensor layout: 2, shape:1x3x773x329, aligned shape:1x3x773x336
Tensor data type:2, Tensor layout: 2, shape:1x3x108x1297, aligned shape:1x3x108x1312
Tensor data type:5, Tensor layout: 2, shape:1x3x858x477, aligned shape:1x3x858x477
Tensor data type:5, Tensor layout: 0, shape:1x920x102x3, aligned shape:1x920x102x3
Tensor data type:4, Tensor layout: 2, shape:1x3x723x1486, aligned shape:1x3x723x1486
Tensor data type:4, Tensor layout: 0, shape:1x372x366x3, aligned shape:1x372x366x3
Tensor data type:3, Tensor layout: 2, shape:1x3x886x291, aligned shape:1x3x886x291
Tensor data type:3, Tensor layout: 0, shape:1x613x507x3, aligned shape:1x613x507x3
*****************************test_prepare_free_fn************************************************
*****************************test_info_fn********************************************************
Tensor data type:14, shape:1x1x1x3x2, stride:24x24x24x8x4, ndim: 5, data:
[[[[[0, 1], [2, 3], [4, 5]]]]]
Tensor data type:9, shape:3x3x1x2x1, stride:6x2x2x1x1, ndim: 5, data:
[[[[[0], [1]]], [[[2], [3]]], [[[4], [5]]]], [[[[6], [7]]], [[[8], [9]]],
[[[10], [11]]]], [[[[12], [13]]], [[[14], [15]]], [[[16], [17]]]]]
*****************************test_info_fn********************************************************
********************test_dequantize_fn***********************************************************
Tensor data type:8, shape:1x1x2x4, ndim: 4, quantiType: 2, quantizeAxis: 1,
quantizeValue: (0.1,), data: [[[[0, 1, 2, 3], [4, 5, 6, 7]]]],
dequantize data: [[[[0, 0.1, 0.2, 0.3], [0.4, 0.5, 0.6, 0.7]]]]
Tensor data type:8, shape:2x4x1x1, ndim: 4, quantiType: 2, quantizeAxis: 3,
quantizeValue: (0.1,),
data: [[[[0]], [[1]], [[2]], [[3]]], [[[4]], [[5]], [[6]], [[7]]]],
dequantize data: [[[[0]], [[0.1]], [[0.2]], [[0.3]]], [[[0.4]], [[0.5]], [[0.6]], [[0.7]]]]
********************test_dequantize_fn***********************************************************
advanced_samples
The example in the 02_advanced_samples directory is used to introduce the use of special functions of custom operators. Its directory contains the following scripts:
├── plugin
│ └── custom_arm_op_custom_identity.sh
├── README.md
├── run_multi_input.sh
├── run_multi_model_batch.sh
└── run_nv12_batch.sh
-custom_arm_op_custom_identity.sh: This script mainly implements the inference function of custom operator model,
When using it, enter the 02_advanced_samples directory, and then directly execute sh custom_arm_op_custom_identity.sh, as shown below:
root@x5dvb-hynix8G:/userdata/horizon/x5/script/02_advanced_samples# sh custom_arm_op_custom_identity.sh
../../aarch64/bin/run_custom_op --model_file=../../../model/runtime/custom_op/custom_op_featuremap.bin --input_file=../../../data/custom_identity_data/input0.bin,../../../data/custom_identity_data/input1.bin
I0000 00:00:00.000000 10841 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
I0411 17:55:59.928918 10841 main.cpp:212] hbDNNRegisterLayerCreator success
I0411 17:55:59.929064 10841 main.cpp:217] hbDNNRegisterLayerCreator success
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
[A][DNN][packed_model.cpp:225][Model](2023-04-11,17:56:00.667.991) [HorizonRT] The model builder version = 1.15.0
I0411 17:56:00.676071 10841 main.cpp:232] hbDNNGetModelNameList success
I0411 17:56:00.676204 10841 main.cpp:239] hbDNNGetModelHandle success
I0411 17:56:00.676276 10841 main.cpp:245] hbDNNGetInputCount success
file length: 602112
file length: 602112
I0411 17:56:00.687402 10841 main.cpp:268] hbDNNGetOutputCount success
I0411 17:56:00.687788 10841 main.cpp:297] hbDNNInfer success
I0411 17:56:00.695663 10841 main.cpp:302] task done
I0411 17:56:03.145243 10841 main.cpp:306] write output tensor
The first output data of the model is saved to the output0.txt file.
run_multi_input.sh: This script mainly implements multiple small model batch inference functions, When using it, enter the 02_advanced_samples directory, and then directly executesh run_multi_input.sh, as shown below:
root@x5dvb:/userdata/horizon/x5/script/02_advanced_samples# sh run_multi_input.sh
../aarch64/bin/run_multi_input --model_file=../../model/runtime/mobilenetv2/mobilenetv2_multi_224x224_gray.bin --image_file=../../data/cls_images/zebra_cls.jpg --top_k=5
I0000 00:00:00.000000 10893 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
[A][DNN][packed_model.cpp:225][Model](2023-04-11,17:57:03.277.375) [HorizonRT] The model builder version = 1.15.0
I0411 17:57:03.327527 10893 multi_input.cc:148] read image to tensor as bgr success
I0411 17:57:03.329546 10893 multi_input.cc:183] TOP 0 result id: 340
I0411 17:57:03.329598 10893 multi_input.cc:183] TOP 1 result id: 292
I0411 17:57:03.329628 10893 multi_input.cc:183] TOP 2 result id: 352
I0411 17:57:03.329656 10893 multi_input.cc:183] TOP 3 result id: 351
I0411 17:57:03.329684 10893 multi_input.cc:183] TOP 4 result id: 282
run_multi_model_batch.sh: This script mainly implements multiple small model batch inference functions. When using it, enter the 02_advanced_samples directory, and then directly executesh run_multi_model_batch.sh, as shown in the following code block:
root@x5dvb-hynix8G:/userdata/horizon/x5/script/02_advanced_samples# sh run_multi_model_batch.sh
../aarch64/bin/run_multi_model_batch --model_file=../../model/runtime/googlenet/googlenet_224x224_nv12.bin,../../model/runtime/mobilenetv2/mobilenetv2_224x224_nv12.bin --input_file=../../data/cls_images/zebra_cls.jpg,../../data/cls_images/zebra_cls.jpg
I0000 00:00:00.000000 10916 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
[A][DNN][packed_model.cpp:225][Model](2023-04-11,17:57:43.547.52) [HorizonRT] The model builder version = 1.15.0
[A][DNN][packed_model.cpp:225][Model](2023-04-11,17:57:51.811.477) [HorizonRT] The model builder version = 1.15.0
I0411 17:57:51.844280 10916 main.cpp:117] hbDNNInitializeFromFiles success
I0411 17:57:51.844388 10916 main.cpp:125] hbDNNGetModelNameList success
I0411 17:57:51.844424 10916 main.cpp:139] hbDNNGetModelHandle success
I0411 17:57:51.875140 10916 main.cpp:153] read image to nv12 success
I0411 17:57:51.875686 10916 main.cpp:170] prepare input tensor success
I0411 17:57:51.875875 10916 main.cpp:182] prepare output tensor success
I0411 17:57:51.876082 10916 main.cpp:216] infer success
I0411 17:57:51.878844 10916 main.cpp:221] task done
I0411 17:57:51.878948 10916 main.cpp:226] googlenet class result id: 340
I0411 17:57:51.879084 10916 main.cpp:230] mobilenetv2 class result id: 340
I0411 17:57:51.879177 10916 main.cpp:234] release task success
run_nv12_batch.sh: This script mainly implements the inference function of the batch model. Infer1 sets the address of each batch of the input tensor separately. Infer2 sets only one address, including all batches. When using it, enter the02_advanced_samplesdirectory, and then executesh run_nv12_batch.shdirectly, as shown in the following code block:
root@x5dvb:/userdata/horizon/x5/script/02_advanced_samples# sh run_nv12_batch.sh
../aarch64/bin/run_nv12_batch --model_file=../../model/runtime/googlenet/googlenet_4x224x224_nv12.bin --image_file=../../data/cls_images/zebra_cls.jpg,../../data/cls_images/cat_cls.jpg,../../data/cls_images/zebra_cls.jpg,../../data/cls_images/cat_cls.jpg --top_k=5
I0000 00:00:00.000000 21511 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
I0705 11:39:43.429180 21511 nv12_batch.cc:151] Infer1 start
I0705 11:39:43.488143 21511 nv12_batch.cc:166] read image to tensor as nv12 success
I0705 11:39:43.491156 21511 nv12_batch.cc:201] Batch[0]:
I0705 11:39:43.491211 21511 nv12_batch.cc:203] TOP 0 result id: 340
I0705 11:39:43.491240 21511 nv12_batch.cc:203] TOP 1 result id: 83
I0705 11:39:43.491266 21511 nv12_batch.cc:203] TOP 2 result id: 41
I0705 11:39:43.491298 21511 nv12_batch.cc:203] TOP 3 result id: 912
I0705 11:39:43.491324 21511 nv12_batch.cc:203] TOP 4 result id: 292
I0705 11:39:43.491348 21511 nv12_batch.cc:201] Batch[1]:
I0705 11:39:43.491374 21511 nv12_batch.cc:203] TOP 0 result id: 282
I0705 11:39:43.491398 21511 nv12_batch.cc:203] TOP 1 result id: 281
I0705 11:39:43.491422 21511 nv12_batch.cc:203] TOP 2 result id: 285
I0705 11:39:43.491447 21511 nv12_batch.cc:203] TOP 3 result id: 287
I0705 11:39:43.491472 21511 nv12_batch.cc:203] TOP 4 result id: 283
I0705 11:39:43.491497 21511 nv12_batch.cc:201] Batch[2]:
I0705 11:39:43.491514 21511 nv12_batch.cc:203] TOP 0 result id: 340
I0705 11:39:43.491539 21511 nv12_batch.cc:203] TOP 1 result id: 83
I0705 11:39:43.491564 21511 nv12_batch.cc:203] TOP 2 result id: 41
I0705 11:39:43.491587 21511 nv12_batch.cc:203] TOP 3 result id: 912
I0705 11:39:43.491612 21511 nv12_batch.cc:203] TOP 4 result id: 292
I0705 11:39:43.491637 21511 nv12_batch.cc:201] Batch[3]:
I0705 11:39:43.491662 21511 nv12_batch.cc:203] TOP 0 result id: 282
I0705 11:39:43.491685 21511 nv12_batch.cc:203] TOP 1 result id: 281
I0705 11:39:43.491710 21511 nv12_batch.cc:203] TOP 2 result id: 285
I0705 11:39:43.491734 21511 nv12_batch.cc:203] TOP 3 result id: 287
I0705 11:39:43.491760 21511 nv12_batch.cc:203] TOP 4 result id: 283
I0705 11:39:43.492235 21511 nv12_batch.cc:223] Infer1 end
I0705 11:39:43.492276 21511 nv12_batch.cc:228] Infer2 start
I0705 11:39:43.549713 21511 nv12_batch.cc:243] read image to tensor as nv12 success
I0705 11:39:43.552248 21511 nv12_batch.cc:278] Batch[0]:
I0705 11:39:43.552292 21511 nv12_batch.cc:280] TOP 0 result id: 340
I0705 11:39:43.552320 21511 nv12_batch.cc:280] TOP 1 result id: 83
I0705 11:39:43.552345 21511 nv12_batch.cc:280] TOP 2 result id: 41
I0705 11:39:43.552371 21511 nv12_batch.cc:280] TOP 3 result id: 912
I0705 11:39:43.552397 21511 nv12_batch.cc:280] TOP 4 result id: 292
I0705 11:39:43.552421 21511 nv12_batch.cc:278] Batch[1]:
I0705 11:39:43.552445 21511 nv12_batch.cc:280] TOP 0 result id: 282
I0705 11:39:43.552469 21511 nv12_batch.cc:280] TOP 1 result id: 281
I0705 11:39:43.552495 21511 nv12_batch.cc:280] TOP 2 result id: 285
I0705 11:39:43.552520 21511 nv12_batch.cc:280] TOP 3 result id: 287
I0705 11:39:43.552567 21511 nv12_batch.cc:280] TOP 4 result id: 283
I0705 11:39:43.552592 21511 nv12_batch.cc:278] Batch[2]:
I0705 11:39:43.552616 21511 nv12_batch.cc:280] TOP 0 result id: 340
I0705 11:39:43.552641 21511 nv12_batch.cc:280] TOP 1 result id: 83
I0705 11:39:43.552665 21511 nv12_batch.cc:280] TOP 2 result id: 41
I0705 11:39:43.552690 21511 nv12_batch.cc:280] TOP 3 result id: 912
I0705 11:39:43.552716 21511 nv12_batch.cc:280] TOP 4 result id: 292
I0705 11:39:43.552739 21511 nv12_batch.cc:278] Batch[3]:
I0705 11:39:43.552763 21511 nv12_batch.cc:280] TOP 0 result id: 282
I0705 11:39:43.552788 21511 nv12_batch.cc:280] TOP 1 result id: 281
I0705 11:39:43.552812 21511 nv12_batch.cc:280] TOP 2 result id: 285
I0705 11:39:43.552837 21511 nv12_batch.cc:280] TOP 3 result id: 287
I0705 11:39:43.552861 21511 nv12_batch.cc:280] TOP 4 result id: 283
I0705 11:39:43.553154 21511 nv12_batch.cc:300] Infer2 end
misc
Examples under the 03_misc directory to introduce the use of non-nv12 input models. Its directory contains the following scripts:
├── run_lenet.sh
└── run_resnet50_feature.sh
run_lenet.sh: This script mainly implements the lenet model inference function of Y data input, When using it, enter the 03_misc directory and then executesh run_lenet.shdirectly, as shown below:
root@x5dvb-hynix8G:/userdata/horizon/x5/script/03_misc# sh run_lenet.sh
../aarch64/bin/run_lenet_gray --model_file=../../model/runtime/lenet_gray/lenet_28x28_gray.bin --data_file=../../data/misc_data/7.bin --image_height=28 --image_width=28 --top_k=5
I0000 00:00:00.000000 10979 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
[A][DNN][packed_model.cpp:225][Model](2023-04-11,18:02:12.605.436) [HorizonRT] The model builder version = 1.15.0
I0411 18:02:12.613317 10979 run_lenet_gray.cc:128] hbDNNInitializeFromFiles success
I0411 18:02:12.613404 10979 run_lenet_gray.cc:136] hbDNNGetModelNameList success
I0411 18:02:12.613440 10979 run_lenet_gray.cc:143] hbDNNGetModelHandle success
I0411 18:02:12.614181 10979 run_lenet_gray.cc:159] prepare y tensor success
I0411 18:02:12.614310 10979 run_lenet_gray.cc:172] prepare tensor success
I0411 18:02:12.614503 10979 run_lenet_gray.cc:182] infer success
I0411 18:02:12.615538 10979 run_lenet_gray.cc:187] task done
[W][DNN][hb_sys.cpp:108][Mem](2023-04-11,18:02:12.615.583) memory is noncachable, ignore flush operation
I0411 18:02:12.615624 10979 run_lenet_gray.cc:192] task post process finished
I0411 18:02:12.615667 10979 run_lenet_gray.cc:198] TOP 0 result id: 7
I0411 18:02:12.615698 10979 run_lenet_gray.cc:198] TOP 1 result id: 9
I0411 18:02:12.615727 10979 run_lenet_gray.cc:198] TOP 2 result id: 3
I0411 18:02:12.615754 10979 run_lenet_gray.cc:198] TOP 3 result id: 4
I0411 18:02:12.615782 10979 run_lenet_gray.cc:198] TOP 4 result id: 2
run_resnet50_feature.sh: This script mainly implements the resnet50 model inference function for feature data input. The sample code quantizes and paddings the feature data to meet the input conditions of the model, and then inputs it to the model for inferring. When using it, enter the 03_misc directory, and then directly executesh run_resnet50_feature.sh, as shown below:
root@x5dvb-hynix8G:/userdata/horizon/x5/script/03_misc# sh run_resnet50_feature.sh
../aarch64/bin/run_resnet_feature --model_file=./resnet50_64x56x56_featuremap_modified.bin --data_file=../../data/misc_data/np_0 --top_k=5
I0000 00:00:00.000000 11024 vlog_is_on.cc:197] RAW: Set VLOG level for "*" to 3
[BPU_PLAT]BPU Platform Version(1.3.3)!
[HBRT] set log level as 0. version = 3.15.18.0
[DNN] Runtime version = 1.17.2_(3.15.18 HBRT)
[A][DNN][packed_model.cpp:225][Model](2023-04-11,18:03:30.317.594) [HorizonRT] The model builder version = 1.15.1
I0411 18:03:30.523054 11024 run_resnet_feature.cc:160] hbDNNInitializeFromFiles success
I0411 18:03:30.523152 11024 run_resnet_feature.cc:168] hbDNNGetModelNameList success
I0411 18:03:30.523188 11024 run_resnet_feature.cc:175] hbDNNGetModelHandle success
I0411 18:03:30.529860 11024 run_resnet_feature.cc:346] input data size: 802816; input valid size: 200704; input aligned size: 229376
I0411 18:03:30.536860 11024 run_resnet_feature.cc:357] tensor padding featuremap success
I0411 18:03:30.536912 11024 run_resnet_feature.cc:190] prepare feature tensor success
I0411 18:03:30.537052 11024 run_resnet_feature.cc:200] prepare tensor success
I0411 18:03:30.537197 11024 run_resnet_feature.cc:210] infer success
I0411 18:03:30.541096 11024 run_resnet_feature.cc:215] task done
[W][DNN][hb_sys.cpp:108][Mem](2023-04-11,18:03:30.541.149) memory is noncachable, ignore flush operation
I0411 18:03:30.541409 11024 run_resnet_feature.cc:220] task post process finished
I0411 18:03:30.541453 11024 run_resnet_feature.cc:226] TOP 0 result id: 74
I0411 18:03:30.541483 11024 run_resnet_feature.cc:226] TOP 1 result id: 815
I0411 18:03:30.541512 11024 run_resnet_feature.cc:226] TOP 2 result id: 73
I0411 18:03:30.541538 11024 run_resnet_feature.cc:226] TOP 3 result id: 78
I0411 18:03:30.541565 11024 run_resnet_feature.cc:226] TOP 4 result id: 72
Auxiliary Tools and Common Operations
Logs
This section mainly includes two parts: Example Logs and Model Inference DNN API Logs.
The example logs refer to application logs from the delivery package’s sample code; the model inference DNN API logs refer to logs within the embedded DNN library. Users can obtain different log information according to their specific needs.
Example Logs
Example logs primarily use the vlog feature from glog. In the basic_samples reference examples, all log content will be fully output.
Model Inference DNN API Logs
For configuration of model inference DNN API logs, please read the Configuration Information section in the Model Inference DNN API User Guide document.
6.3.4.2. Public Model Performance and Accuracy Evaluation Guide
Introduction
This section introduces the usage of the public model accuracy and performance evaluation ai_benchmark example package. The package includes source code, executable programs, and evaluation scripts. Developers can directly experience these examples on the X5 development board and use them for embedded application development, lowering the development barrier.
The example package provides performance and accuracy evaluation examples for common classification, detection, segmentation, optical flow, tracking estimation, radar multi-task, BEV, and depth estimation models. For detailed content, please read the following sections.
Tip:
To obtain the public model accuracy and performance evaluation ai_benchmark example package, refer to Delivery Package Description.
Release Package Description
Example Code Package Structure
ai_benchmark/code/ # Example source code directory
├── build_ptq_x5.sh
├── build_qat_x5.sh
├── CMakeLists.txt
├── deps_gcc11.3 # Third-party dependency libraries
├── include # Source header files
├── README.md
└── src # Example source code
ai_benchmark/x5 # Example package runtime environment
├── ptq # PTQ scheme model examples
│ ├── data # Model accuracy evaluation dataset
│ ├── mini_data # Model performance evaluation dataset
│ ├── model # PTQ scheme NV12 models
│ │ ├── README.md
│ │ └── runtime -> ../../../../model_zoo/runtime/ai_benchmark/ptq # Symbolic link pointing to models in the toolchain SDK package; model path must be manually specified in the board runtime environment
│ ├── README.md
│ ├── script # Execution scripts
│ │ ├── aarch64 # Compiled executables and dependencies
│ │ ├── classification # Classification model examples
│ │ ├── config # Model inference configuration files
│ │ ├── detection # Detection model examples
│ │ ├── segmentation # Segmentation model examples
│ │ ├── env.sh # Basic environment script
│ │ └── README.md
│ └── tools # Accuracy evaluation tools
└── qat # QAT scheme model examples
├── data # Model accuracy evaluation dataset
├── mini_data # Model performance evaluation dataset
├── model # QAT scheme NV12 models
│ ├── README.md
│ └── runtime -> ../../../../model_zoo/runtime/ai_benchmark/qat # Symbolic link pointing to models in the toolchain SDK package; model path must be manually specified in the board runtime environment
├── README.md
├── script # Execution scripts
│ ├── aarch64 # Compiled executables and dependencies
│ ├── bev # BEV model examples
│ ├── classification # Classification model examples
│ ├── config # Model inference configuration files
│ ├── detection # Detection model examples
│ ├── disparity_pred # Depth estimation model examples
│ ├── multitask # Multi-task model examples
│ ├── opticalflow # Optical flow model examples
│ ├── segmentation # Segmentation model examples
│ ├── tracking # Tracking model examples
│ ├── traj_pred # Trajectory prediction examples
│ ├── env.sh # Basic environment script
│ └── README.md
└── tools # Preprocessing and accuracy evaluation tools
code: This directory contains the source code of the evaluation program, used for model performance and accuracy evaluation.
x5: This directory provides pre-compiled applications and various evaluation scripts to test the performance and accuracy of multiple models running on the X5 BPU.
build_ptq_x5.sh: One-click compilation script for PTQ on-device programs.
build_qat_x5.sh: One-click compilation script for QAT on-device programs.
deps_gcc11.3: Dependencies required by the example code, mainly as follows:
appsdk gflags glog nlohmann opencv rapidjson
Example Models
The model release package includes both PTQ and QAT model releases:
PTQ model model_zoo can be obtained from the path:
horizon_model_convert_sample/01_common/model_zoo/runtime/ai_benchmark/ptq.QAT model model_zoo can be obtained from the path:
horizon_model_convert_sample/01_common/model_zoo/runtime/ai_benchmark/qat.
These include commonly used models such as classification, detection, segmentation, and optical flow prediction. The model naming convention is {model_name}{backbone}{input_size}_{input_type}.
Tip: To obtain the horizon_model_convert_sample model conversion example package, execute the following command on a Linux server: wget -c ftp://oeftp@sdk.d-robotics.cc/model_convert_sample/horizon_model_convert_sample.tar.xz --ftp-password=Oeftp~123$%.
Public Datasets
Datasets used in the evaluation examples mainly include VOC, COCO, ImageNet, Cityscapes, FlyingChairs, KITTI, CULane, Nuscenes, Mot17, Carfusion, Argoverse 1, and SceneFlow datasets.
Please download them in a Linux environment using the methods below:
| Dataset | Download Link | Download Structure |
|---|---|---|
| ImageNet | https://www.image-net.org/download.php | Please refer to the description in Data Preprocessing for download structure |
| COCO | https://cocodataset.org/ | Please refer to the description in Data Preprocessing for download structure |
| VOC | http://host.robots.ox.ac.uk/pascal/VOC/ | Download both 2007 and 2012 versions; refer to Data Preprocessing for download structure |
| Cityscapes | https://github.com/mcordts/cityscapesScripts | Please refer to the description in Data Preprocessing for download structure |
| CIFAR-10 | http://www.cs.toronto.edu/~kriz/cifar.html | Please refer to the description in Data Preprocessing for download structure |
| FlyingChairs | https://lmb.informatik.uni-freiburg.de/resources/datasets/FlyingChairs.en.html | Please refer to the description in Data Preprocessing for download structure |
| KITTI3D | https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d | Please refer to the description in Data Preprocessing for download structure |
| CULane | https://xingangpan.github.io/projects/CULane.html | Please refer to the description in Data Preprocessing for download structure |
| Nuscenes | https://www.nuscenes.org/nuscenes | Please refer to the description in Data Preprocessing for download structure |
| Mot17 | https://opendatalab.com/MOT17 | Please refer to the description in Data Preprocessing for download structure |
| Carfusion | http://www.cs.cmu.edu/~ILIM/projects/IM/CarFusion/cvpr2018/index.html | Please refer to the description in Data Preprocessing for download structure |
| Argoverse 1 | https://www.argoverse.org/av1.html | Please refer to the description in Data Preprocessing for download structure |
| SceneFlow | https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html | Please refer to the description in Data Preprocessing for download structure |
Environment Setup
Development Board Preparation
After receiving the development board, update the board image to the latest version. Refer to the Install System section for upgrade instructions.
Ensure that the local development machine and the development board can connect remotely.
Compilation Environment Preparation
Compilation requires the cross-compilation tool arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc to be installed in the current environment. Use the development machine Docker image provided by the X5 algorithm toolchain for direct compilation. For Docker environment acquisition and usage, refer to the Environment Installation section.
Use the build_ptq_x5.sh script under the code directory to compile the executable program for the development board environment in one click. The executable program and its dependencies will be automatically copied to the aarch64 directory under x5/ptq/script.
Note:
The cross-compilation toolchain path specified in the build_ptq_x5.sh script is under the /opt directory. If users have installed it elsewhere, they can manually modify the script.
export CC=/opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc
export CXX=/opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-g++
Evaluation Example Usage Guide
Evaluation example scripts are mainly located under the script and tools directories. The script directory contains evaluation scripts for running on the development board, including common classification, detection, and segmentation models. Under each model, there are three scripts, representing:
fps.sh: Uses multi-threaded scheduling to calculate FPS statistics. Users can freely set the number of threads according to their needs.
latency.sh: Calculates single-frame latency statistics (single thread, single frame).
accuracy.sh: Used for accuracy evaluation.
script:
├── aarch64 # Compiled executables and dependencies
│ ├── bin
│ └── lib
├── env.sh # Basic configuration
├── config
│ ├── model
│ │ ├── data_name_list # image_name configuration file
│ │ └── input_init # Model input configuration file
│ ├── preprocess
│ │ └── centerpoint_preprocess_5dim.json # Preprocessing configuration file
│ └── reference_points # Model reference point information
│ │ ├── bev_mt_gkt_mixvargenet_multitask_nuscenes
│ │ └── ...
├── detection # Detection models
│ ├── fcos_efficientnetb0_mscoco # Other models exist in this directory; this model directory is for reference only
│ │ ├── accuracy.sh
│ │ ├── fps.sh
│ │ ├── latency.sh
│ │ ├── workflow_accuracy.json # Accuracy configuration file
│ │ ├── workflow_fps.json # Performance configuration file
│ │ └── workflow_latency.json # Single-frame latency configuration file
│ └──...
├── ...
└── README.md
The ptq/tools directory mainly includes accuracy calculation scripts under python_tools, used for PTQ model accuracy evaluation.
python_tools
└── accuracy_tools
├── cityscapes_metric.py
├── cls_eval.py
├── coco_metric.py
├── config.py
├── coco_det_eval.py
├── parsing_eval.py
├── voc_det_eval.py
└── voc_metric.py
The qat/tools directory mainly includes preprocessing scripts and accuracy calculation scripts, used for QAT model accuracy evaluation.
tools/
├── eval_preprocess
│ ├── util
│ ├── ......
│ └── voc.py
├── python_tools
│ └── accuracy_tools
│ ├── argoverse_util
│ ├── nuscenes_metric_pro
│ ├── whl_package
│ ├── bev_eval.py
│ ├── ......
│ └── yolov3_eval.py
└── README.md
Note:
Before evaluation, execute the following commands to copy the ptq or qat directory to the development board, then copy model_zoo/runtime to the ptq/model or qat/model directory.
scp -r ai_toolchain_package/Ai_Toolchain_Package-release-vX.X.X-OE-vX.X.X/ai_benchmark/x5/ptq root@192.168.1.10:/userdata/ptq/
scp -r model_zoo/runtime root@192.168.1.10:/userdata/ptq/model/
scp -r ai_toolchain_package/Ai_Toolchain_Package-release-vX.X.X-OE-vX.X.X/ai_benchmark/x5/qat root@192.168.1.10:/userdata/ptq/
scp -r model_zoo/runtime root@192.168.1.10:/userdata/qat/model/
JSON Configuration File Parameter Information
This section briefly explains the configuration items in workflow_fps.json, workflow_latency.json, and workflow_accuracy.json, categorized by input configuration (input_config), output configuration (output_config), and workflow configuration.
Note: The parameter information below represents general configurations. Some example models may have additional configuration items due to their specific characteristics. Please refer to the specific JSON files of the example models for details.
Input Configuration (input_config)
| Parameter Name | Description | Related JSON Files |
|---|---|---|
| input_type | Set input data format, supporting image or bin files. | fps.json, latency.json, accuracy.json |
| height | Set input data height. | fps.json, latency.json, accuracy.json |
| width | Set input data width. | fps.json, latency.json, accuracy.json |
| data_type | Set input data type. Supported types can be found in hbDNNDataType. Data types are listed from top to bottom, corresponding to numbers 0, 1, 2... For example, HB_DNN_IMG_TYPE_Y corresponds to number 0, HB_DNN_IMG_TYPE_NV12 corresponds to number 1, HB_DNN_IMG_TYPE_NV12_SEPARATE corresponds to number 2... |
fps.json, latency.json, accuracy.json |
| image_list_file | Set the path to the preprocessed dataset lst file. | fps.json, latency.json, accuracy.json |
| need_pre_load | Set whether to use preloading for dataset reading. | fps.json, latency.json, accuracy.json |
| limit | Set the threshold for the difference between processed and unprocessed input data, used to control input data processing threads. | fps.json, latency.json, accuracy.json |
| need_loop | Set whether to loop through the dataset for evaluation. | fps.json, latency.json, accuracy.json |
| max_cache | Set the number of preloaded data items. Note: This parameter pre-processes images and loads them into memory. To ensure stable program operation, do not set this value too high. It is recommended to keep it under 30. | fps.json, latency.json, accuracy.json |
Output Configuration (output_config)
| Parameter Name | Description | Related JSON Files |
|---|---|---|
| output_type | Set output data type. | fps.json, latency.json, accuracy.json |
| in_order | Set whether outputs are in order. | fps.json, latency.json, accuracy.json |
| enable_view_output | Set whether to visualize output results. | fps.json, latency.json |
| image_list_enable | When visualizing, set to true to save output results as image files. | fps.json, latency.json |
| view_output_dir | Set the output file path for visualization results. | fps.json, latency.json |
| eval_enable | Set whether to evaluate accuracy. | accuracy.json |
| output_file | Set the model output result file. | accuracy.json |
Workflow Configuration
Model Inference Configuration:
| Parameter Name | Description | Related JSON Files |
|---|---|---|
| method_type | Set the model inference method; this should be configured as InferMethod. |
fps.json, latency.json, accuracy.json |
| method_config | Set model inference parameters. | fps.json, latency.json, accuracy.json |
| - core: Set the inference core ID. Note: X5 has only one BPU core, so this can only be set to 0 (core any) or 1 (core 0). | ||
| - model_file: Specify the model file. |
Post-processing Configuration:
| Parameter Name | Description | Related JSON Files |
|---|---|---|
| thread_count | Set the number of post-processing threads, valid range is 1-8. |
fps.json, latency.json, accuracy.json |
| method_type | Set the post-processing method. | fps.json, latency.json, accuracy.json |
| method_config | Set post-processing parameters. | fps.json, latency.json, accuracy.json |
Performance Evaluation
Performance evaluation includes two aspects: latency and FPS.
Evaluation Script Usage
Enter the directory of the model to be evaluated and execute sh latency.sh to test single-frame latency. As shown below:
I0419 02:35:07.041095 39124 output_plugin.cc:80] Infer latency: [avg: 13.124ms, max: 13.946ms, min: 13.048ms], Post process latency: [avg: 3.584ms, max: 3.650ms, min: 3.498ms].
Note:
inferindicates model inference time.Post processindicates post-processing time.
Enter the model directory and execute sh fps.sh to test the frame rate. As shown below:
I0419 02:35:00.044417 39094 output_plugin.cc:109] Throughput: 1129.39fps # Model frame rate
Note: This feature uses multi-threaded concurrency to maximize model performance on the BPU. Due to multi-threading and data sampling, the frame rate may be lower initially and then gradually stabilizes, with fluctuations controlled within 0.5%.
Command-line Parameter Explanation
The content of the fps.sh script is as follows:
#!/bin/sh
source ../../base_config.sh
export SHOW_FPS_LOG=1
export STAT_CYCLE=100 # Set environment variable, FPS statistics cycle
${app} \
--config_file=workflow_fps.json \
--log_level=1
The content of the latency.sh script is as follows:
#!/bin/sh
source ../../base_config.sh
export SHOW_LATENCY_LOG=1 # Set environment variable, print LATENCY level logs
export STAT_CYCLE=50 # Set environment variable, LATENCY statistics cycle
${app} \
--config_file=workflow_latency.json \
--log_level=1
Result Visualization
If you want to view the result of a single model inference, you can modify workflow_latency.json and re-run the latency.sh script to generate visual output in the output_dir directory.
Note: When generating visual output, the script will run slower due to dump overhead. Only latency.sh script supports dump.
Visualization Steps are as follows:
Modify the workflow_latency.json configuration file
"output_config": {
"output_type": "image",
"enable_view_output": true, # Enable visualization
"view_output_dir": "./output_dir", # Visualization output path
"image_list_enable": true,
"in_order": false
}
Run the latency.sh script
sh latency.sh
Note: BEV model visualization requires specifying scene information and the homography matrix path. The homography matrix is used for camera view to bird’s-eye view transformation, and different scenes have their own homography matrices.
The recommended configuration for the BEV model’s workflow_latency.json is as follows:
"output_config": {
"output_type": "image",
"enable_view_output": true, # Enable visualization
"view_output_dir": "./output_dir", # Visualization output path
"bev_ego2img_info": [
"../../config/visible/bev/scenes.json", # Input scene information
"../../config/visible/bev/boston.bin", # Homography matrix for Boston scene
"../../config/visible/bev/singapore.bin" # Homography matrix for Singapore scene
],
"image_list_enable": true,
"in_order": false
}
Different model types support different visualization effects, as shown in the table below:
Classification
2D Detection
3D Detection
Segmentation
Keypoints
Lane Detection
Optical Flow
Radar
Radar Multi-task
BEV
Trajectory Prediction
Depth Estimation
Note: When visualizing trajectory prediction, if images beyond minidata need to be visualized, road information and trajectory information files must be additionally configured into minidata/argoverse1/visualization. To generate the configuration file, use the densetnt_process.py preprocessing script and set the –is-gen-visual-config parameter to true.
Accuracy Evaluation
Model accuracy evaluation consists of five steps:
Data preprocessing.
Generating lst files.
Data mounting.
Model inference.
Accuracy calculation.
Data Preprocessing
For PTQ models: Data preprocessing must be performed on an x86 development machine using the hb_eval_preprocess tool to process the dataset.
Preprocessing refers to specific operations performed on image data before feeding it into the model, such as image resizing, cropping, and padding.
This tool is integrated into the model conversion and compilation environment on the development machine. After processing, the original dataset generates a set of preprocessed binary .bin files corresponding to the model.
Run hb_eval_preprocess --help to view the usage instructions.
Tips:
For command-line arguments of the
hb_eval_preprocesstool, enterhb_eval_preprocess -h, or refer to the hb_eval_preprocess Tool section in the PTQ quantization principles and procedures documentation.
PTQ Model Data Preprocessing:
Below we detail each dataset used by models in the example package and their corresponding preprocessing operations:
The datasets used by PTQ models include:
VOC Dataset: This dataset is mainly used for evaluating the ssd_mobilenetv1 model.
Its directory structure is as follows. In the example, theval.txtfile under theMainfolder, source images inJPEGImages, and annotation data inAnnotationsare primarily used:
.
└── VOCdevkit # Root directory
└── VOC2012 # Datasets from different years; only 2012 is downloaded here, but others such as 2007 also exist
├── Annotations # Stores XML files, corresponding one-to-one with images in JPEGImages, describing image content, etc.
├── ImageSets # Contains text files, each line holding an image name, optionally followed by ±1 indicating positive/negative samples
│ ├── Action
│ ├── Layout
│ ├── Main
│ └── Segmentation
├── JPEGImages # Stores source images
├── SegmentationClass # Stores images related to semantic segmentation
└── SegmentationObject # Stores images related to instance segmentation
Preprocess the dataset:
hb_eval_preprocess -m ssd_mobilenetv1 -i VOCdevkit/VOC2012/JPEGImages -v VOCdevkit/VOC2012/ImageSets/Main/val.txt -o ./pre_ssd_mobilenetv1
COCO Dataset: This dataset is primarily used for evaluating detection models such as centernet_resnet101, detr_efficientnetb3_mscoco, detr_resnet50_mscoco, yolov2_darknet19, yolov3_darknet53, yolov3_vargdarknet, yolov5x, preq_qat_fcos_efficientnetb0, preq_qat_fcos_efficientnetb2, and preq_qat_fcos_efficientnetb3.
Its directory structure is as follows. In the example, theinstances_val2017.jsonannotation file under theannotationsfolder and images underimagesare mainly used:
.
├── annotations # Stores annotation data
└── images # Stores source images
Preprocess the dataset:
hb_eval_preprocess -m model_name -i coco/coco_val2017/images -o ./pre_model_name
ImageNet Dataset: This dataset is primarily used for evaluating classification models such as efficientnasnet_m, efficientnasnet_s, efficientnet_lite0, efficientnet_lite1, efficientnet_lite2, efficientnet_lite3, efficientnet_lite4, googlenet, mobilenetv1, mobilenetv2, resnet18, and vargconvnet.
In the example, the annotation fileval.txtand source images in thevaldirectory are mainly used:
.
├── val.txt
└── val
Preprocess the dataset:
hb_eval_preprocess -m model_name -i imagenet/val -o ./pre_model_name
Cityscapes Dataset: This dataset is used for evaluating segmentation models such as deeplabv3plus_efficientnetb0, deeplabv3plus_efficientnetm1, deeplabv3plus_efficientnetm2, and fastscnn_efficientnetb0.
In the example, annotation files under./gtFine/valand source images under./leftImg8bit/valare mainly used.
.
├── gtFine
│ └── val
│ ├── frankfurt
│ ├── lindau
│ └── munster
└── leftImg8bit
└── val
├── frankfurt
├── lindau
└── munster
Preprocess the dataset:
hb_eval_preprocess -m model_name -i cityscapes/leftImg8bit/val -o ./pre_model_name
QAT Model Data Preprocessing:
QAT model data preprocessing requires executing the corresponding preprocessing script in ai_benchmark/x5/qat/tools/eval_preprocess under the x86 simulation environment.
Below we detail the datasets corresponding to models in the example package and their preprocessing operations.
Note: Before use, modify the dataset paths and save paths in the scripts to ensure proper execution.
ImageNet Dataset: This dataset is mainly used for evaluating QAT classification models such as mixvargenet_imagenet, mobilenetv1_imagenet, mobilenetv2_imagenet, resnet50_imagenet, horizon_swin_transformer_imagenet, and vargnetv2_imagenet.
Preprocess the dataset:
python3 imagenet.py --image-path=./standard_imagenet/val/ --save-path=./pre_model_name
VOC Dataset: This dataset is mainly used for evaluating the detection model yolo_mobilenetv1_voc.
Preprocess the dataset:
python3 voc.py --image-path=./VOCdevkit/VOC2012/JPEGImages/ --save-path=./pre_yolov3_mobilenetv1
COCO Dataset: This dataset is mainly used for evaluating QAT detection models fcos_efficientnetb0_mscoco and retinanet_vargnetv2_fpn_mscoco.
Preprocess the dataset:
# fcos_efficientnetb0_mscoco
python3 fcos_process.py --image-path=./mscoco/images/val2017/ --label-path=./mscoco/images/annotations/instances_val2017.json --save-path=./pre_fcos_efficientnetb0
# retinanet_vargnetv2_fpn_mscoco
python3 retinanet_process.py --image-path=./mscoco/images/val2017/ --label-path=./mscoco/images/annotations/instances_val2017.json --save-path=./pre_retinanet
Cityscapes: This dataset is mainly used for evaluating the QAT segmentation model unet_mobilenetv1_cityscapes. No preprocessing is required; use the validation set data directly.FlyingChairs: This dataset is mainly used for evaluating the QAT optical flow model pwcnet_pwcnetneck_flyingchairs. You can download this dataset from theFlyingChairs dataset official website <https://lmb.informatik.uni-freiburg.de/resources/datasets/FlyingChairs.en.html>_. In the example, data from FlyingChairs_release/data and the annotation file ./FlyingChairs_train_val.txt are mainly used.
.
├── FlyingChairs_release
│ └── data
│ ├── 00001_img1.ppm
│ ├── 00001_img2.ppm
│ └── 00001_flow.ppm
├── FlyingChairs_train_val.txt
Note:
{id}_img1.ppm and {id}_img2.ppm form an image pair, with image width 512 and height 384. The id ranges from 00001 to 22872. The label for each image pair is {id}_flow.flo.
FlyingChairs_train_val.txt is used to split the training and validation sets; a label value of 2 indicates the validation set.
Preprocess the dataset:
python3 pwcnet_process.py --input-path=./flyingchairs/FlyingChairs_release/data/ --val-file=./flyingchairs/FlyingChairs_train_val.txt --output-path=./pre_pwcnet_opticalflow
Kitti3D: This dataset is mainly used for evaluating the QAT detection model pointpillars_kitti_car. You can download this dataset from theKitti3D dataset official website <https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d>_.
.
├── kitti3d
├── data_object_calib.zip # Camera calibration matrices of object dataset
├── data_object_image_2.zip # Left color images of object dataset
├── data_object_label_2.zip # Training labels of object dataset
└── data_object_veloodyne.zip # Velodyne point cloud
.
├── kitti3d_origin
├── ImageSets
│ ├── test.txt
│ ├── train.txt
│ ├── trainval.txt
│ └── val.txt
├── testing
│ ├── calib
│ ├── image_2
│ └── velodyne
└── training
├── calib
├── image_2
├── label_2
└── velodyne
Preprocess the dataset:
python3 pointpillars_process.py --data-path=./kitti3d_origin --save-path=./pre_kitti3d --height=1 --width=150000
Culane: This dataset is mainly used for evaluating the QAT detection model ganet_mixvargenet_culane. You can download this dataset from theCulane dataset official website <https://xingangpan.github.io/projects/CULane.html>_.
.
├── culane
├── annotations_new.tar.gz
├── driver_23_30frame.tar.gz
├── driver_37_30frame.tar.gz
├── driver_100_30frame.tar.gz
├── driver_161_90frame.tar.gz
├── driver_182_30frame.tar.gz
├── driver_193_90frame.tar.gz
├── laneseg_label_w16.tar.gz
└── list.tar.gz
The annotations_new.tar.gz file should be extracted last to correct the original annotation files. It is recommended to decompress the downloaded dataset into the following structure:
.
├── culane # Root directory
├── driver_23_30frame # Dataset and annotations
│ ├── 05151640_0419.MP4 # A segment of the dataset containing individual frames
│ │ ├──00000.jpg # Source image
│ │ ├──00000.lines.txt # Annotation file, each line giving x, y coordinates of lane marking keypoints
│ ......
├── driver_37_30frame
├── driver_100_30frame
├── driver_161_90frame
├── driver_182_30frame
├── driver_193_90frame
├── laneseg_label_w16 # Lane segmentation labels
└── list # Training, validation, and test lists
Preprocess the dataset:
python3 ganet_process.py --image-path=./culane --save-path=./pre_culane
Nuscenes: This dataset is mainly used for evaluating QAT detection models fcos3d_efficientnetb0_nuscenes, centerpoint_pointpillar_nuscenes, lidar multitask model centerpoint_mixvargnet_multitask_nuscenes, and BEV models bev_gkt_mixvargenet_multitask_nuscenes, bev_lss_efficientnetb0_multitask_nuscenes, bev_ipm_efficientnetb0_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, and detr3d_efficientnetb3_nuscenes. You can download this dataset from theNuscenes dataset official website <https://www.nuscenes.org/nuscenes#download>_.
.
├── Nuscenes
├── nuScenes-map-expansion-v1.3.zip
├── nuScenes-map-expansion-v1.2.zip
├── nuScenes-map-expansion-v1.1.zip
├── nuScenes-map-expansion-v1.0.zip
├── v1.0-mini.tar
├── v1.0-test_blobs.tar
├── v1.0-test_meta.tar
├── v1.0-trainval01_blobs.tar
├── v1.0-trainval02_blobs.tar
├── v1.0-trainval03_blobs.tar
├── v1.0-trainval04_blobs.tar
├── v1.0-trainval05_blobs.tar
├── v1.0-trainval06_blobs.tar
├── v1.0-trainval07_blobs.tar
├── v1.0-trainval08_blobs.tar
├── v1.0-trainval09_blobs.tar
├── v1.0-trainval10_blobs.tar
└── v1.0-trainval_meta.tar
For lidar multitask models, additionally download the lidar segmentation labels (lidarseg) from the official website and update v1.0-trainval according to the Nuscenes official tutorial. It is recommended to decompress the downloaded dataset into the following structure:
.
├── Nuscenes
├── can_bus
├── lidarseg
├── maps
├── nuscenes
│ └── meta
│ ├── maps
│ ├── v1.0-mini
│ └── v1.0-trainval
├── samples
├── sweeps
├── v1.0-mini
└── v1.0-trainval
Note:
fcos3d_process.py not only generates preprocessed images but also processes the camera intrinsic parameters to generate the corresponding configuration file.
centerpoint_preprocess.py, bev_preprocess.py, and lidar_preprocess.py not only generate preprocessed data but also create a
val_gt_infos.pklfile under the preprocessed data path for accuracy calculation.bev_preprocess.py requires specifying the model name via the
--modelargument, with options including bev_gkt_mixvargenet_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, bev_ipm_efficientnetb0_multitask_nuscenes, bev_lss_efficientnetb0_multitask_nuscenes, and detr3d_efficientnetb3_nuscenes.
Preprocess the dataset:
# fcos3d_efficientnetb0_nuscenes
python3 fcos3d_process.py --src-data-dir=./Nuscenes --file-path=../../script/config/model/data_name_list/nuscenes_names.txt --save-path=./processed_fcos3d_images
# centerpoint_pointpillar_nuscenes
python3 centerpoint_preprocess.py --data-path=./Nuscenes --save-path=./nuscenes_lidar_val
# BEV models
python3 bev_preprocess.py --model=model_name --data-path=./Nuscenes --meta-path=./Nuscenes/meta --reference-path=../../script/config/reference_points --save-path=./nuscenes_bev_val
# centerpoint_mixvargnet_multitask_nuscenes
python3 lidar_preprocess.py --data-path=./Nuscenes --save-path=./nuscenes_lidar_val
Mot17: This dataset is used for evaluating the QAT tracking model motr_efficientnetb3_mot17. You can download this dataset from theMot17 dataset official website <https://opendatalab.com/MOT17>_.
.
├── valdata # Root directory
├── gt_val
│ ├── MOT17-02-SDP
│ ├── MOT17-04-SDP
│ ├── MOT17-05-SDP
│ ├── MOT17-09-SDP
│ ├── MOT17-10-SDP
│ ├── MOT17-11-SDP
│ ├── MOT17-13-SDP
├── images
│ └── train
│ ├── MOT17-04-SDP
│ ├── MOT17-05-SDP
│ ├── MOT17-09-SDP
│ ├── MOT17-10-SDP
│ ├── MOT17-11-SDP
│ ├── MOT17-13-SDP
└── mot17.val
Preprocess the dataset:
python3 motr_process.py --image-path=./valdata/images/train --save-path=./processed_motr
Carfusion: This dataset is used for evaluating the QAT detection model keypoints_efficientnetb0_carfusion. You can download this dataset from theCarfusion dataset official website <http://www.cs.cmu.edu/~ILIM/projects/IM/CarFusion/cvpr2018/index.html>_.
.
├── carfusion # Root directory
├── train
└── test
Preprocess the dataset:
# First, generate data required for evaluation (must run this script if using the dataset for the first time)
python3 gen_carfusion_data.py --src-data-path=carfusion --out-dir=cropped_data --num-workers 2
After running the first script, the directory structure becomes:
.
├── cropped_data # Root directory
├── test
└── simple_anno
Ensure the data-root path is at the same level as cropped_data, then run:
python3 keypoints_preprocess.py --data-root=./ --label-path=cropped_data/simple_anno/keypoints_test.json --save-path=./processed_carfusion
Argoverse1: This dataset is used for evaluating the QAT detection model densetnt_vectornet_argoverse1. You can download this dataset from theArgoverse1 dataset official website <https://www.argoverse.org/av1.html>_.
.
├── carfusion # Root directory
├── train
└── test
Preprocess the dataset:
# First, generate data required for evaluation (must run this script if using the dataset for the first time)
python3 gen_carfusion_data.py --src-data-path=carfusion --out-dir=cropped_data --num-workers 2
After running the first script, the directory structure becomes:
.
├── arogverse-1 # Root directory
├── map_files
└── val
In addition to generating preprocessed inputs, densetnt_process.py also generates corresponding evaluation meta files under src-path. Only the --src-path and --dst-path parameters are needed for evaluation; other parameters can be ignored.
python3 densetnt_process.py --src-path=arogverse-1 --dst-path=processed_arogverse1
SceneFlow: This dataset is used for evaluating the QAT depth estimation model stereonetplus_mixvargenet_sceneflow. You can download this dataset from theSceneFlow dataset official website <https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html>_.
.
├── SceneFlow # Root directory
├── FlyingThings3D
│ ├── disparity
│ ├── frames_finalpass
└── SceneFlow_finalpass_test.txt
Preprocess the dataset:
# stereonet_preprocess.py not only generates preprocessed data but also creates a ``val_gt_infos.pkl`` file under the preprocessed data path for accuracy calculation.
python3 stereonet_preprocess.py --data-path=SceneFlow/ --data-list=SceneFlow/SceneFlow_finalpass_test.txt --save-path=sceneflow_val
Generating lst Files
The workflow of the accuracy evaluation script in the example is as follows:
Based on the
image_list_fileparameter value inworkflow_accurary.json, locate the corresponding dataset’slstfile;Load each preprocessed file according to the file path information stored in the
lstfile, then perform inference.
Therefore, after generating the preprocessed files, the corresponding lst file must be generated, recording the path of each preprocessed file. This path depends on where the dataset is stored on the target device.
It is recommended to place it at the same level as the preprocessed data folder ./data/dataset_name/pre_model_name.
PTQ preprocessed dataset structure:
|── ptq
| |── data
| | |── cityscapes
| | | |── pre_deeplabv3plus_efficientnetb0
| | | | |── xxxx.bin # Preprocessed binary files
| | | | |── ....
| | | |── pre_deeplabv3plus_efficientnetb0.lst # lst file: records paths of each preprocessed file
| | | |── ....
| | |── ....
| |── model
| | |── ...
| |── script
| | |── ...
QAT preprocessed dataset structure:
|── qat
| |── data
| | |── carfusion
| | | |── pre_keypoints
| | | | |── xxxx # Preprocessed data
| | | | |── ....
| | | |── pre_carfusion.lst # lst file: records paths of each preprocessed file
| | |── ....
| |── model
| | |── ...
| |── script
| | |── ...
Corresponding lst file generation example:
Note: For all models except Densetnt_vectornet_argoverse1, BEV, Motr_efficientnetb3_mot17, and Stereonetplus_mixvargenet_sceneflow, use the following method:
find ../../../data/coco/pre_centernet_resnet101 -name "*bin*" > ../../../data/coco/pre_centernet_resnet101.lst
Note: The parameter after -name should be adjusted according to the format of the preprocessed dataset, e.g., bin, png.The paths stored in the generated lst file are relative paths: ../../../data/ or ../../../data/coco/pre_centernet_resnet101/, which aligns with the default configuration path in workflow_accuracy.json.
If you need to change the storage location of the preprocessed dataset, ensure that the corresponding lst file can be read by workflow_accuracy.json; additionally, ensure that the program can access the corresponding preprocessed files based on the path information in the lst file.
argoverse1:
sh generate_acc_lst.sh
The paths stored in the generated lst file are relative paths: ../../../data/argoverse1/densetnt/, which aligns with the default configuration path in workflow_accuracy.json.
For the models Densetnt_vectornet_argoverse1, Bev, Motr_efficientnetb3_mot17, and Stereonetplus_mixvargenet_sceneflow, refer to the following methods for generating lst files:
Densetnt:
sh generate_acc_lst.sh
The paths stored in the generated lst file are relative paths: ../../../data/argoverse1/densetnt/, which aligns with the default configuration path in workflow_accuracy.json.
Bev:
Taking the model bev_ipm_efficientnetb0_multitask as an example, this model has two inputs: images and reference points, with the image and reference point of the same frame sharing the same name. To ensure input correspondence, add sort to the find command to sort by filename. Reference generation method:
find ../../../data/nuscenes_bev/images -name "*bin*" | sort > ../../../data/nuscenes_bev/images.lst
find ../../../data/nuscenes_bev/reference_points0 -name "*bin*" | sort > ../../../data/nuscenes_bev/reference_points0.lst
For detr3d_efficientnetb3_nuscenes, which additionally requires inputs of coords, masks, and position embeddings, generate lst files as follows:
# detr3d_efficientnetb3_nuscenes:
find ../../../data/nuscenes_bev/coords0 -name "*bin*" | sort > ../../../data/nuscenes_bev/coords0.lst
find ../../../data/nuscenes_bev/coords1 -name "*bin*" | sort > ../../../data/nuscenes_bev/coords1.lst
find ../../../data/nuscenes_bev/coords2 -name "*bin*" | sort > ../../../data/nuscenes_bev/coords2.lst
find ../../../data/nuscenes_bev/coords3 -name "*bin*" | sort > ../../../data/nuscenes_bev/coords3.lst
find ../../../data/nuscenes_bev/masks -name "*bin*" | sort > ../../../data/nuscenes_bev/masks.lst
Note: In BEV models, reference_points in detr3d_efficientnetb3_nuscenes are used for post-processing and must have the correct path configured in workflow_accuracy.json so that the program can access the corresponding reference point files. Additionally, bev_ipm_4d_efficientnetb0_multitask is a temporal model that requires ordered input. Therefore, we provide a dedicated script gen_file_list.sh to generate lst files. Usage:
sh gen_file_list.sh
The paths stored in the generated lst file are relative paths: ../../../data/nuscenes_bev/, which aligns with the default configuration path in workflow_accuracy.json. If you need to change the storage location of the preprocessed dataset, ensure that the corresponding lst file can be read by workflow_accuracy.json, and that the program can access the corresponding preprocessed files based on the path information in the lst file.
Motr:
sh generate_acc_lst.sh
The paths stored in the generated lst file are relative paths: ../../../data/mot17/motr/, which aligns with the default configuration path in workflow_accuracy.json. If you need to change the storage location of the preprocessed dataset, ensure that the corresponding lst file can be read by workflow_accuracy.json, and that the program can access the corresponding preprocessed files based on the path information in the lst file.
Stereonetplus:
Taking the model stereonetplus_mixvargenet_sceneflow as an example, to ensure correspondence between left and right views of the same frame, add sort to the find command to sort by filename. Reference generation method:
find ../../../data/sceneflow/left -name "*png*" | sort > ../../../data/sceneflow/left.lst
find ../../../data/sceneflow/right -name "*png*" | sort > ../../../data/sceneflow/right.lst
The paths stored in the generated lst file are relative paths: ../../../data/mot17/motr/, which aligns with the default configuration path in workflow_accuracy.json. If you need to change the storage location of the preprocessed dataset, ensure that the corresponding lst file can be read by workflow_accuracy.json, and that the program can access the corresponding preprocessed files based on the path information in the lst file.
Data Mounting
Due to the relatively large dataset size, it is not suitable to store directly on the development board. Instead, use NFS mounting to allow the development board to access the data.
On the development PC (requires root privileges):
Edit
/etc/exports, add a line:/nfs *(insecure,rw,sync,all_squash,anonuid=1000,anongid=1000,no_subtree_check)./nfsindicates the local mount path and can be replaced with a user-specified directory.Run the command
exportfs -a -rto apply the changes in/etc/exports.
On the development board:
Create the mount directory:
mkdir -p /mnt.mount -t nfs {PC_IP}:/nfs /mnt -o nolock.
This completes mounting the PC’s /nfs folder to the board’s /mnt folder. Using this method, mount the folder containing preprocessed data to the board, and create a symbolic link of /data to the board’s /ptq directory, placing it at the same level as /script.
Model Inference
After mounting the data, log in to the development board. For instructions on logging in, refer to the Board Login section. After successful login, execute the accuracy.sh script under the centernet_resnet101/ directory as follows:
/userdata/ptq/script/detection/centernet_resnet101# sh accuracy.sh
../../aarch64/bin/example --config_file=workflow_accuracy.json --log_level=2
...
I0419 03:14:51.158655 39555 infer_method.cc:107] Predict DoProcess finished.
I0419 03:14:51.187361 39556 ptq_centernet_post_process_method.cc:558] PTQCenternetPostProcessMethod DoProcess finished, predict result: [{"bbox":[-1.518860,71.691170,574.934631,638.294922],"prob":0.750647,"label":21,"class_name":"
I0118 14:02:43.636204 24782 ptq_centernet_post_process_method.cc:558] PTQCenternetPostProcessMethod DoProcess finished, predict result: [{"bbox":[3.432283,164.936249,157.480042,264.276825],"prob":0.544454,"label":62,"class_name":"
...
The program on the development board will generate an eval.log file in the current directory, which contains the prediction results.
Accuracy Evaluation
Note:
Accuracy evaluation should be performed in the model conversion environment on the development PC.
PTQ Model Accuracy Evaluation:
The accuracy evaluation scripts for PTQ models are located in the ptq/tools/python_tools/accuracy_tools directory, including:
cls_eval.py: Evaluates classification models.coco_det_eval.py: Evaluates detection models using the COCO dataset.parsing_eval.py: Evaluates segmentation models using the Cityscapes dataset.voc_det_eval.py: Evaluates detection models using the VOC dataset.
Below are accuracy evaluation methods for different types of PTQ models:
Classification Models
For classification models using CIFAR-10 and ImageNet datasets:
#!/bin/sh
python3 cls_eval.py --log_file=eval.log --gt_file=val.txt
Remarks:
log_file: Prediction result file of the classification model.gt_file: Label file for the CIFAR-10 and ImageNet datasets.Detection Models
For detection models using the COCO dataset:
#!/bin/sh
python3 coco_det_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json
Remarks:
eval_result_path: Prediction result file of the detection model.annotation_path: Annotation file of the COCO dataset.
For detection models using the VOC dataset:
#!/bin/sh
python3 voc_det_eval.py --eval_result_path=eval.log --annotation_path=../Annotations --val_txt_path=../val.txt
Remarks:
eval_result_path: Prediction result file of the detection model.annotation_path: Annotation file of the VOC dataset.val_txt_path: Theval.txtfile located in the ImageSets/Main folder of the VOC dataset.Segmentation Models
For segmentation models using the Cityscapes dataset:
#!/bin/sh
python3 parsing_eval.py --log_file=eval.log --gt_path=cityscapes/gtFine/val
Remarks:
log_file: Prediction result file of the segmentation model.gt_path: Annotation file of the Cityscapes dataset.
QAT Model Accuracy Evaluation:
The accuracy evaluation scripts for QAT models are located in the qat/tools/python_tools/accuracy_tools directory, including:
bev_eval.py: Evaluates BEV models.centerpoint_eval.py: Evaluates the radar 3D modelcenterpoint_pointpillar_nuscenes.cls_eval.py: Evaluates classification models.densetnt_eval.py: Evaluates the trajectory prediction modeldensetnt_vectornet_argoverse1.detr_eval.py: Evaluates DETR detection models.fcos3d_eval.py: Evaluates the detection modelfcos3d_efficientnetb0_nuscenes.fcos_eval.py: Evaluates FCOS detection models.ganet_eval.py: Evaluates the detection modelganet_mixvargenet_culane.keypoints_eval.py: Evaluates the detection modelkeypoint_efficientnetb0_carfusion.lidar_multitask_eval.py: Evaluates the LiDAR multitask modelcenterpoint_mixvargnet_multitask_nuscenes.motr_eval.py: Evaluates the MOTR detection modelmotr_efficientnetb3_mot17.parsing_eval.py: Evaluates segmentation models using the Cityscapes dataset.pointpillars_eval.py: Evaluates the detection modelpointpillars_kitti_car.pwcnet_eval.py: Evaluates the optical flow modelpwcnet_pwcnetneck_flyingchairsusing the FlyingChairs dataset.retinanet_eval.py: Evaluates the detection modelretinanet_vargnetv2_fpn_mscoco.yolov3_eval.py: Evaluates YOLOv3 detection models.stereonet_eval.py: Evaluates the depth estimation modelstereonetplus_mixvargenet_sceneflow.
Below are accuracy evaluation methods for different types of QAT models:
BEV Models
For BEV models using the nuScenes dataset:
#!/bin/sh
python3 bev_eval.py --det_eval_path=bev_det_eval.log --seg_eval_path=bev_seg_eval.log --gt_files_path=./nuscenes_bev_val/val_gt_infos.pkl --meta_dir=./Nuscenes/meta/
# For detr3d_efficientnetb3_nuscenes, a BEV detection model, --seg_eval_path is not required
python3 bev_eval.py --det_eval_path=eval.log --gt_files_path=./nuscenes_bev_val/val_gt_infos.pkl --meta_dir=./Nuscenes/meta/
Remarks:
det_eval_path: Prediction result file for the BEV model’s detection task.seg_eval_path: Prediction result file for the BEV model’s segmentation task.gt_files_path: Ground truth file generated during preprocessing of the nuScenes dataset.meta_dir: Path to the meta information of the nuScenes dataset.Classification Models
For classification models using CIFAR-10 and ImageNet datasets:
#!/bin/sh
python3 cls_eval.py --log_file=eval.log --gt_file=val.txt
Remarks:
log_file: Prediction result file of the classification model.gt_file: Label file for the CIFAR-10 and ImageNet datasets.Detection Models
Example for detection models using the COCO dataset:
#!/bin/sh python3 fcos_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json --image_path=./mscoco/images/val2017/ # For QAT FCOS models, add --is_qat=True python3 fcos_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json --image_path=./mscoco/images/val2017/ --is_qat=True
Remarks:
eval_result_path: Prediction result file of the FCOS detection model.annotation_path: Annotation file of the COCO dataset.image_path: Original COCO dataset.is_qat: Whether the result is from a QAT FCOS model.
#!/bin/sh python3 retinanet_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json --image_path=./mscoco/images/val2017/
Remarks:
eval_result_path: Prediction result file of the RetinaNet detection model.annotation_path: Annotation file of the COCO dataset.image_path: Original COCO dataset.
#!/bin/sh python3 detr_eval.py --eval_result_path=eval.log --annotation_path=instances_val2017.json --image_path=./mscoco/images/val2017/
Remarks:
eval_result_path: Prediction result file of the DETR detection model.annotation_path: Annotation file of the COCO dataset.image_path: Original COCO dataset.
Example for detection models using the VOC dataset:
#!/bin/sh python3 yolov3_eval.py --eval_result_path=eval.log --annotation_path=../Annotations -- val_txt_path=../val.txt --image_height=416 --image_width=416
Remarks:
eval_result_path: Prediction result file of the detection model.annotation_path: Annotation file of the VOC dataset.val_txt_path: Theval.txtfile in the ImageSets/Main folder of the VOC dataset.image_height: Image height.image_width: Image width.
For detection models using the KITTI dataset:
#!/bin/sh python3 pointpillars_eval.py --eval_result_path=eval.log --annotation_path=./val_gt_infos.pkl
Remarks:
eval_result_path: Prediction result file of the detection model.annotation_path: Label fileval_gt_infos.pklgenerated during preprocessing of the KITTI 3D dataset.
For detection models using the CULane dataset:
#!/bin/sh python3 ganet_eval.py --eval_path=eval.log --image_path=./culane
Remarks:
eval_result_path: Prediction result file of the detection model.image_path: CULane dataset.
For detection models using the nuScenes dataset:
#!/bin/sh python3 fcos3d_eval.py --eval_result_path=eval.log --image_path=./Nuscenes
Remarks:
eval_result_path: Prediction result file of the detection model.image_path: nuScenes dataset.
#!/bin/sh python3 centerpoint_eval.py --predict_result_path=eval.log --gt_files_path=./nuscenes_lidar_val/val_gt_infos.pkl --meta_dir=./Nuscenes/meta/
Remarks:
predict_result_path: Prediction result file of the detection model.gt_files_path: Ground truth file generated during preprocessing of the nuScenes dataset.meta_dir: Path to the meta information of the nuScenes dataset.
For detection models using the CarFusion dataset:
#!/bin/sh python3 keypoints_eval.py --anno_path=./processed_carfusion/processed_anno.json --eval_result_path=eval.log
Remarks:
anno_path: Preprocessedprocessed_anno.jsonfile.eval_result_path: Prediction result file of the detection model.Segmentation Models
For segmentation models using the Cityscapes dataset:
#!/bin/sh python3 parsing_eval.py --log_file=eval.log --gt_path=cityscapes/gtFine/val
Remarks:
log_file: Prediction result file of the segmentation model.gt_path: Annotation file of the Cityscapes dataset.
Optical Flow Models
For optical flow models using the FlyingChairs dataset:
#!/bin/sh
python3 pwcnet_eval.py --log_file=eval.log --gt_path=./flyingchairs/FlyingChairs_release/data/ --val_file=./flyingchairs/FlyingChairs_train_val.txt
Remarks:
log_file: Prediction result file of the optical flow model.val_file: Label file of the FlyingChairs dataset.gt_path: Original files of the FlyingChairs dataset.Tracking Models
For tracking models using the MOT17 dataset:
#!/bin/sh
python3 motr_eval.py --eval_result_path=eval_log --gt_val_path=valdata/gt_val
Remarks:
eval_result_path: Directory containing prediction results of the tracking model.gt_val_path: Label files of the MOT17 dataset.Multitask Models
For LiDAR multitask models using the nuScenes dataset:
#!/bin/sh
python3 lidar_multitask_eval.py --det_eval_path=det_eval.log --seg_eval_path=seg_eval.log --gt_files_path=./nuscenes_lidar_val/val_gt_infos.pkl --data_dir=./Nuscenes
Remarks:
det_eval_path: Prediction result file for the detection task.seg_eval_path: Prediction result file for the segmentation task.gt_files_path: Ground truth file generated during preprocessing of the nuScenes dataset.data_dir: Path to the nuScenes dataset.Trajectory Prediction Models
For trajectory prediction models using the Argoverse1 dataset:
#!/bin/sh
python3 densetnt_eval.py --eval_result_path=eval.log --meta_path=argoverse1/meta
Remarks:
eval_result_path: Prediction result file for trajectory prediction.meta_path: Label file generated for the Argoverse1 dataset, located in themetadirectory of the original dataset after preprocessing.Depth Estimation Models
For depth estimation models using the Sceneflow dataset:
#!/bin/sh
python3 stereonet_eval.py --log_file=eval.log --gt_files=val_gt_infos.pkl
Remarks:
log_file: Prediction result file of the depth estimation model.gt_files: Ground truth file generated during preprocessing of the Sceneflow dataset.
Model Integration
Preprocessing
You can add custom model preprocessing as needed and deploy it on the CPU. Taking centerpoint_pointpillar_nuscenes as an example:
Add the preprocessing file
qat_centerpoint_preprocess_method.ccand its header fileqat_centerpoint_preprocess_method.h.2. Add model preprocessing configuration file.
Preprocessing File and Header File Addition
The preprocessing file qat_centerpoint_preprocess_method.cc should be placed under the ai_benchmark/code/src/method/ path,
and the header file qat_centerpoint_preprocess_method.h should be placed under the ai_benchmark/code/include/method/ path:
|── ai_benchmark
| |── code # Example source code
| | |── include
| | | |── method # Add header files in this folder
| | | | |── qat_centerpoint_preprocess_method.h
| | | | |── ...
| | |── src
| | | |── method # Add preprocessing .cc file in this folder
| | | | |── qat_centerpoint_preprocess_method.cc
| | | | |── ...
Model Preprocessing Configuration File Addition
|── ai_benchmark
| |── x5/qat/script # Example script directory
| | |── config
| | | |── preprocess
| | | | |── centerpoint_preprocess_5dim.json # Preprocessing configuration script
Single-Frame Latency Evaluation for Preprocessing
Run the sh latency.sh script to test the single-frame latency of preprocessing. Example output is shown below:
I0615 13:30:40.772293 3670 output_plugin.cc:91] Pre process latency: [avg: 20.295ms, max: 28.690ms, min: 18.512ms], Infer latency: [avg: 25.053ms, max: 31.943ms, min: 24.702ms], Post process latency: [avg: 52.760ms, max: 54.099ms, min: 51.992ms].
Where:
Pre processindicates preprocessing time.Inferindicates model inference time.Post processindicates postprocessing time.
Postprocessing
Postprocessing integration mainly consists of two steps, using CenterNet model integration as an example:
Add postprocessing file
ptq_centernet_post_process_method.ccand its header fileptq_centernet_post_process_method.h.Add model execution scripts and configuration files.
Postprocessing File and Header File Addition
The postprocessing code file can directly reuse any existing postprocessing file under the src/method directory. Only the InitFromJsonString function and the PostProcess function need to be modified.
The InitFromJsonString function mainly reads postprocessing-related parameter configurations from workflow.json; users can customize the corresponding input parameters.
The PostProcess function completes the postprocessing logic.
The postprocessing .cc file should be placed under ai_benchmark/code/src/method/,
and the .h header file should be placed under ai_benchmark/code/include/method/:
|── ai_benchmark
| |── code # Example source code
| | |── include
| | | |── method # Add header files in this folder
| | | | |── ptq_centernet_post_process_method.h
| | | | |── ...
| | |── src
| | | |── method # Add postprocessing .cc file in this folder
| | | | |── ptq_centernet_post_process_method.cc
| | | | |── ...
Add Model Execution Scripts and Configuration Files
After adding model execution scripts and configuration files, the directory structure reference is as follows:
For the centerpoint_pointpillar_nuscenes model:
|── ai_benchmark
| |── x5/qat/script # Example script directory
| | |── detection
| | | |── centerpoint_pointpillar_nuscenes
| | | | |── accuracy.sh # Accuracy test script
| | | | |── fps.sh # Performance test script
| | | | |── latency.sh # Single-frame latency example script
| | | | |── workflow_accuracy # Accuracy configuration folder
| | | | |── workflow_fps.json # Performance configuration file
| | | | |── workflow_latency.json # Single-frame latency configuration file
For the motr_efficientnetb3_mot17 model:
|── ai_benchmark
| |── x5/qat/script # Example script directory
| | |── tracking
| | | |── motr
| | | | |── accuracy.sh # Accuracy test script
| | | | |── fps.sh # Performance test script
| | | | |── generate_acc_lst.sh # Script to generate accuracy lst
| | | | |── latency.sh # Single-frame latency example script
| | | | |── workflow_accuracy # Accuracy configuration folder
| | | | |── workflow_fps.json # Performance configuration file
| | | | |── workflow_latency.json # Single-frame latency configuration file
For models other than centerpoint_pointpillar_nuscenes and motr_efficientnetb3_mot17:
|── ai_benchmark
| |── x5/ptq/script # Example script directory
| | |── detection
| | | |── centernet_resnet101
| | | | |── accuracy.sh # Accuracy test script
| | | | |── fps.sh # Performance test script
| | | | |── latency.sh # Single-frame latency example script
| | | | |── workflow_accuracy.json # Accuracy configuration file
| | | | |── workflow_fps.json # Performance configuration file
| | | | |── workflow_latency.json # Single-frame latency configuration file
Auxiliary Tools and Common Operations
Logging System Usage Instructions
The logging system mainly consists of two parts: example logs and model inference API DNN logs.
The example logs refer to application logs in the delivery package example code; DNN logs refer to logs within the lib dnn library.
Users can obtain different logs based on different requirements.
Example Logs
Log Levels. Example logs mainly use glog’s vlog, divided into four custom levels:
0(SYSTEM): This level is mainly used to output error messages;1(REPORT): This level is mainly used in example code to output performance data;2(DETAIL): This level is mainly used in example code to output current system status information;3(DEBUG): This level is mainly used in example code to output debugging information.
Log level setting rule: Suppose a levelPis set. If an event of levelQoccurs andQis less than or equal toP, the log will be enabled; otherwise, it will be suppressed. The default priority order is DEBUG > DETAIL > REPORT > SYSTEM.
Log Level Setting. Set the log level via the
log_levelparameter. When running an example, specify thelog_levelparameter to set the level.
For example, settinglog_level=0outputs only SYSTEM logs; settinglog_level=3outputs DEBUG, DETAIL, REPORT, and SYSTEM logs.
Model Inference API DNN Logs
For configuration details on model inference DNN API logs, please refer to the Configuration Information section in the Model Inference DNN API User Guide document.
Operator Latency Explanation
Model operator (OP) performance statistics are enabled by setting the HB_DNN_PROFILER_LOG_PATH environment variable. This section introduces model inference performance analysis, which helps developers understand the actual inference performance of models.
Details on the type and values of this variable are as follows:
Note:
export HB_DNN_PROFILER_LOG_PATH=${path}: Specifies the output path for OP node dumps. After the program exits normally, a profiler.log file will be generated.
Example Explanation
The following code block uses the mobilenetv1 model as an example. When enabling a single thread to RunModel and setting export HB_DNN_PROFILER_LOG_PATH=./, the output statistics are as follows:
{
"perf_result": {
"FPS": 677.6192525182025,
"average_latency": 11.506142616271973
},
"running_condition": {
"core_id": 0,
"frame_count": 200,
"model_name": "mobilenetv1_224x224_nv12",
"run_time": 295.151,
"thread_num": 1
}
}
***
{
"chip_latency": {
"BPU_inference_time_cost": {
"avg_time": 11.09122,
"max_time": 11.54,
"min_time": 3.439
},
"CPU_inference_time_cost": {
"avg_time": 0.18836999999999998,
"max_time": 0.4630000000000001,
"min_time": 0.127
}
},
"model_latency": {
"BPU_MOBILENET_subgraph_0": {
"avg_time": 11.09122,
"max_time": 11.54,
"min_time": 3.439
},
"Dequantize_fc7_1_HzDequantize": {
"avg_time": 0.07884999999999999,
"max_time": 0.158,
"min_time": 0.068
},
"MOBILENET_subgraph_0_output_layout_convert": {
"avg_time": 0.018765,
"max_time": 0.08,
"min_time": 0.01
},
"Preprocess": {
"avg_time": 0.0065,
"max_time": 0.058,
"min_time": 0.003
},
"Softmax_prob": {
"avg_time": 0.084255,
"max_time": 0.167,
"min_time": 0.046
}
},
"task_latency": {
"TaskPendingTime": {
"avg_time": 0.029375,
"max_time": 0.059,
"min_time": 0.009
},
"TaskRunningTime": {
"avg_time": 11.40324,
"max_time": 11.801,
"min_time": 4.008
}
}
}
The above output includes model_latency and task_latency. The model_latency section shows the execution time for each OP in the model, while the task_latency section shows the time consumption of various task modules during model execution.
Note:
The profiler.log file is only generated when the program exits normally.
Dump Tool
This section mainly introduces how to enable the dump tool. It generally does not need attention unless there are model accuracy anomalies.
By setting the environment variable export HB_DNN_DUMP_PATH=${path}, you can dump the input and output of each node during model inference. Based on the dump output, you can troubleshoot whether there are consistency issues between the development machine simulator and the development board: that is, whether the output results are identical for the same model and same input on both the development board and the development machine simulator.
6.3.4.3. On-Board Model Analysis Tool Description
Overview
This section introduces the quick validation tool for on-board model inference in the X5 algorithm toolchain, enabling developers to quickly obtain information about ***.bin models, model inference performance, model debugging, etc.
hrt_model_exec Tool Usage Instructions
The hrt_model_exec tool allows quick evaluation of model inference performance and retrieval of model information directly on the development board.
Currently, the tool provides three types of functions, as shown in the table below:
| No. | Subcommand | Description |
|---|---|---|
| 1 | model_info |
Retrieve model information, such as model input/output details. |
| 2 | infer |
Execute model inference and obtain inference results. |
| 3 | perf |
Perform model performance analysis and obtain analysis results. |
Tip:
The tool can also use the -v or --version command to check the dnn prediction library version.
For example: hrt_model_exec -v or hrt_model_exec –version
Input Parameter Description
Run hrt_model_exec, hrt_model_exec -h, or hrt_model_exec --help on the development board to get detailed usage parameters of the tool.
As shown in the figure below:
| No. | Parameter | Type | Description |
|---|---|---|---|
| 1 | model_file |
string | Path to model file(s), multiple paths can be separated by commas. |
| 2 | model_name |
string | Specify the name of a particular model within the model. |
| 3 | core_id |
int | Specify the execution core. 0: any core, 1: core0, 2: core1; default is 0. |
| 4 | input_file |
string | Model input information. Image input suffixes must be one of bin / JPG / JPEG / jpg / jpeg / png / PNG. Feature input suffixes must be one of bin / txt. Inputs must be separated by English commas ,. Example: xxx.jpg,input.txt. When the model contains operators requiring specific input data, it is recommended to use specified data for perf, such as the index input for the Gather operator needing to meet certain ranges. |
| 5 | roi_infer |
bool | Enable resizer model inference. If the model has resizer input sources, set to true, and configure corresponding input_file and roi parameters for each input source. |
| 6 | roi |
string | Specify the roi region required for resizer model inference. Multiple rois are separated by semicolons. Example: --roi="2,4,123,125;6,8,111,113" |
| 7 | frame_count |
int | Number of frames to execute model inference. |
| 8 | dump_intermediate |
string | Dump input and output of each layer in the model. - dump_intermediate=0: dump function is disabled by default. - dump_intermediate=1: input and output data of each layer node saved in bin format, with BPU node output as aligned data.- dump_intermediate=2: input and output data saved in both bin and txt formats, with BPU node output as aligned data. - dump_intermediate=3: input and output data saved in both bin and txt formats, with BPU node output as valid data. |
| 9 | enable_dump |
bool | Enable dumping model input and output, default is false. |
| 10 | dump_precision |
int | Control the number of decimal places for float-type data in txt format output, default is 9. |
| 11 | hybrid_dequantize_process |
bool | Save raw output after postprocessing. Effective when enable_dump=true, supports only 4D models. Postprocessing includes dequantization of fixed-point output and removal of padding. |
| 12 | dump_format |
string | Format for dumping model input and output. |
| 13 | dump_txt_axis |
int | Control line break rules for txt format input/output. |
| 14 | enable_cls_post_process |
bool | Enable classification postprocessing, default is false. Used with subcommand infer, currently supports only PTQ classification model postprocessing, prints classification results. |
| 15 | perf_time |
int | Duration for model execution. |
| 16 | thread_num |
int | Number of threads (parallelism), value indicates maximum concurrent tasks. For latency testing, set to 1 to avoid resource contention and achieve more accurate results. For throughput testing, set >2 (number of BPU cores), adjust thread count to maximize BPU utilization for more accurate throughput measurement. |
| 17 | profile_path |
string | Path for profiler log generation. Generates profiler.log and profiler.csv files to analyze OP latency and scheduling latency. Usually set as --profile_path="." to generate logs in the current directory. |
| 18 | dump_path |
string | Output path for dumped files. Files are generated when enable_dump or dump_intermediate is used. Files will be saved to the specified path; if the path does not exist, the tool will create it automatically. |
Usage Instructions
This section describes the specific usage methods of the three subfunctions of the hrt_model_exec tool.
model_info
Overview
This parameter is used to retrieve model information. Supported models include: QAT models, PTQ models.
Used together with model_file to obtain detailed model information;
Model information includes: model input/output properties hbDNNTensorProperties and model segmentation information stage.
The segmentation information refers to the ability to split one image into multiple inference stages. The stage information is [x1, y1, x2, y2], representing the top-left and bottom-right coordinates of the image region. Currently, X5’s bayes architecture supports such segmented model inference, while models on X3 have only one stage.
Tip:
If model_name is not specified, information for all models in the file will be output. If model_name is specified, only information for the corresponding model will be output.
Example Explanation
Single Model
hrt_model_exec model_info --model_file=xxx.bin
Multiple Models (output information for all models)
hrt_model_exec model_info --model_file=xxx.bin,xxx.bin
Multiple Models – Pack Model (output information for specified model)
hrt_model_exec model_info --model_file=xxx.bin --model_name=xx
Additional Input Parameter Explanation
Repeated Input
If parameters are specified repeatedly, the latter will overwrite the former. For example, if two model files are specified when retrieving model information, the latter one yyy.bin will be used:
hrt_model_exec model_info --model_file=xxx.bin --model_file=yyy.bin
If repeated input is specified without the command-line argument –model_file, the value after the command-line argument will be used, and untagged values will not be recognized.
For example, in the following case, yyy.bin will be ignored, and the parameter value will be xxx.bin:
hrt_model_exec model_info --model_file=xxx.bin yyy.bin
infer
Overview
This parameter is used to perform inference on one frame with a custom input image and output the inference result.
It must be used with input_file to specify the input image path. The tool will resize the image according to model information and prepare the model input.
Tip:
The program runs single-threaded on a single frame and outputs the model execution time.
Example Explanation
Single Model
hrt_model_exec infer --model_file=xxx.bin --input_file=xxx.jpg
Multiple Models
hrt_model_exec infer --model_file=xxx.bin,xxx.bin --model_name=xx --input_file=xxx.jpg
Resizer Model
The model has three inputs, with input source order [
ddr,resizer,resizer].To infer two frames, assuming the first frame inputs are [xx0.bin, xx1.jpg, xx2.jpg] with roi [2,4,123,125;6,8,111,113], and the second frame inputs are [xx3.bin, xx4.jpg, xx5.jpg] with roi [27,46,143,195;16,28,131,183], the inference command is as follows:
hrt_model_exec infer --roi_infer=true --model_file=xxx.bin --input_file="xx0.bin,xx1.jpg,xx2.jpg,xx3.bin,xx4.jpg,xx5.jpg" --roi="2,4,123,125;6,8,111,113;27,46,143,195;16,28,131,183"
Note: Multiple frame inputs are separated by commas, and rois are separated by semicolons.
Optional Parameters
| Parameter | Description |
|---|---|
core_id |
Specify the core ID for model inference: 0: any core, 1: core0, 2: core1; default is 0. On X5, only one BPU core exists, so configure 0 or 1. |
roi_infer |
Enable resizer model inference; set to true if model input includes resizer source, default is false. |
roi |
Effective when roi_infer is true; specify the roi regions for resizer model inference, separated by semicolons. |
frame_count |
Set the number of frames for infer execution. Repeated inference of a single frame, can be used with enable_dump to verify output consistency; default is 1. |
dump_intermediate |
Dump input and output data of each layer in the model; default value 0 means no dumping. 1: output files in bin format; 2: output in both bin and txt formats, with BPU node output as aligned data; 3: output in both bin and txt formats, with BPU node output as valid data. |
enable_dump |
Dump model output data; default is false. |
dump_precision |
Control number of decimal places for float-type data in txt format output; default is 9. |
hybrid_dequantize_process |
Control txt format output of float-type data. If output is fixed-point, apply dequantization. Currently supports only 4D models. |
dump_format |
Format for dumped model output files; options are bin or txt; default is bin. |
dump_txt_axis |
Line break rules for txt format model output; if output dimension is n, parameter range is [0, n]; default is -1, one data per line. |
enable_cls_post_process |
Enable classification postprocessing; currently supports only PTQ classification models; default is false. |
dump_path |
Specify dump output path; default is current directory. |
Multi-Input Model Explanation
The infer inference function supports multi-input models, including image input, binary file input, and text file input, with inputs separated by commas.
Model input information can be viewed via model_info.
Example Explanation
hrt_model_exec infer --model_file=xxx.bin --input_file=xxx.jpg,input.txt
Additional Input Parameter Explanation
input_file
Image input filenames must have one of the following suffixes: bin, JPG, JPEG, jpg, jpeg, png. Feature input suffixes must be either bin or txt.
Each input must be separated by an English comma ,. Example: xxx.jpg,input.txt.
enable_cls_post_process
Enable classification postprocessing. Used with subcommand infer; currently only supports postprocessing for PTQ classification models. When set to true, classification results are printed.
See the figure below:
roi_infer
If the model contains resizer input sources, both infer and perf functions require setting roi_infer to true, along with corresponding input_file and roi parameters for each input source.
For example, if the model has three inputs with source order [ddr, resizer, resizer], the command line for inferring two sets of input data is as follows:
// infer
hrt_model_exec infer --model_file=xxx.bin --input_file="xx0.bin,xx1.jpg,xx2.jpg,xx3.bin,xx4.jpg,xx5.jpg" --roi="2,4,123,125;6,8,111,113;27,46,143,195;16,28,131,183"
// perf
hrt_model_exec perf --model_file=xxx.bin --input_file="xx0.bin,xx1.jpg,xx2.jpg,xx3.bin,xx4.jpg,xx5.jpg" --roi="2,4,123,125;6,8,111,113;27,46,143,195;16,28,131,183"
Each roi input must be separated by a semicolon.
dump_intermediate
Dump input and output data of each layer node in the model. When dump_intermediate=0, dump function is disabled by default;
When dump_intermediate=1, input and output data of each layer node are saved in bin format, with BPU node output as aligned data;
When dump_intermediate=2, input and output data are saved in both bin and txt formats, with BPU node output as aligned data;
When dump_intermediate=3, input and output data are saved in both bin and txt formats, with BPU node output as valid data.
For example, if a model has two inputs with source order [pyramid, ddr], and you want to save each layer node’s input and output as bin files, with BPU node output saved as aligned type, the inference command line is as follows:
hrt_model_exec infer --model_file=xxx.bin --input_file="xx0.jpg,xx1.bin" --dump_intermediate=1
The dump_intermediate parameter supports both infer and perf modes.
hybrid_dequantize_process
Control txt format output of float-type data. The hybrid_dequantize_process parameter takes effect when enable_dump=true.
When enable_dump=true, if hybrid_dequantize_process=true, integer output data will be dequantized and all outputs saved as float-type in txt files, with model output as valid data, supporting configuration of dump_txt_axis and dump_precision;
If hybrid_dequantize_process=false, directly save model output aligned data without any processing.
For example, if a model has 3 outputs with tensor data types in order [float, int32, int16], and you want to output float-type valid data in txt format, the inference command line is as follows:
// Output float-type data
hrt_model_exec infer --model_file=xxx.bin --input_file="xx.bin" --enable_dump=true --hybrid_dequantize_process=true
The hybrid_dequantize_process parameter currently supports only 4D models.
perf
Overview
This parameter is used to test model inference performance.
Using this tool command, users do not need to provide input data; the program automatically constructs input tensors based on model information, with tensor data filled with random numbers.
The program defaults to running 200 frames in a single thread. When the perf_time parameter is specified, the frame_count parameter becomes invalid, and the program exits after the specified time.
After completion, the program outputs information such as number of threads, number of frames, total inference time, average latency, and frame rate.
Tip:
The program prints performance information every 200 frames: maximum, minimum, and average latency. If fewer than 200 frames are run, it prints once at the end.
Example Explanation
Single Model
hrt_model_exec perf --model_file=xxx.bin
Multiple Models
hrt_model_exec perf --model_file=xxx.bin,xxx.bin --model_name=xx
Resizer Model
The model has three inputs, with input source order [ddr, resizer, resizer].Infer two frames of data. Assume the first frame input is [xx0.bin,xx1.jpg,xx2.jpg] with ROI [2,4,123,125;6,8,111,113], and the second frame input is [xx3.bin,xx4.jpg,xx5.jpg] with ROI [27,46,143,195;16,28,131,183]. The inference command is as follows:
hrt_model_exec perf --roi_infer=true --model_file=xxx.bin --input_file="xx0.bin,xx1.jpg,xx2.jpg,xx3.bin,xx4.jpg,xx5.jpg" --roi="2,4,123,125;6,8,111,113;27,46,143,195;16,28,131,183"
Note: Multiple frame inputs are separated by commas, and ROIs are separated by semicolons.
Optional Parameters
| Parameter | Description |
|---|---|
core_id |
Specifies the core ID for model inference: 0: any core, 1: core0, 2: core1; default is 0. On X5, since there is only one BPU core, set to 0 or 1. |
input_file |
Model input information; multiple inputs can be separated by commas. |
roi_infer |
Enables resizer model inference; set to true if the model input includes a resizer source; default is false. |
roi |
Effective when roi_infer is true; specifies the roi regions required for resizer model inference, separated by semicolons. |
frame_count |
Sets the number of frames for perf to run; effective when perf_time is 0; default is 200. |
perf_time |
Sets the running time of perf in minutes; default is 0. |
thread_num |
Sets the number of threads for program execution, range [1, 8]; default is 1. Values greater than 8 are treated as 8. |
profile_path |
Path for profiler log output. Generates profiler.log and profiler.csv to analyze OP execution time and scheduling overhead. |
Multi-threaded Latency Data Explanation
The purpose of multi-threading is to fully utilize BPU resources. Multiple threads collectively process frame_count frames or run for perf_time minutes until data processing is complete or the execution time ends. During the multi-threaded perf process, the following command can be executed to obtain real-time BPU resource utilization:
hrut_somstatus -n 10000 –d 1
X3 output example:
=====================1=====================
temperature-->
CPU : 37.5 (C)
cpu frequency-->
min cur max
cpu0: 240000 1200000 1200000
cpu1: 240000 1200000 1200000
cpu2: 240000 1200000 1200000
cpu3: 240000 1200000 1200000
bpu status information---->
min cur max ratio
bpu0: 400000000 1000000000 1000000000 0
bpu1: 400000000 1000000000 1000000000 0
X5 output example:
=====================1=====================
temperature-->
DDR : 57.0 (C)
cat: /sys/class/hwmon/hwmon0/temp2_input: Connection timed out
BPU : 0.0 (C)
CPU : 56.6 (C)
cpu frequency-->
min(M) cur(M) max(M)
cpu0: 300 1500 1500
cpu1: 300 1500 1500
cpu2: 300 1500 1500
cpu3: 300 1500 1500
cpu4: 300 1500 1500
cpu5: 300 1500 1500
cpu6: 300 1500 1500
cpu7: 300 1500 1500
bpu status information---->
min(M) cur(M) max(M) ratio
bpu0: 500 1000 1000 0
ddr frequency information---->
min(M) cur(M) max(M)
ddr: 266 3200 3200
Remarks:
In perf mode, the single-threaded latency represents the actual measured on-board performance of the model.
For multi-threaded scenarios, the latency data reflects the per-thread single-frame processing time, which is longer than single-threaded latency due to concurrency overhead. However, the overall processing time decreases and the frame rate (FPS) increases.
Input Parameter Additional Notes
profile_path
Directory where profile log files are generated.
This parameter can be configured via environment variable: export HB_DNN_PROFILER_LOG_PATH=${path}, to view OP execution time and task scheduling overhead during model execution.
Typically, setting --profile_path="." suffices, meaning logs will be generated in the current directory, named profiler.log.
thread_num
Number of threads (parallelism), indicating the maximum number of concurrent tasks.
When measuring latency, set this value to 1 to avoid resource contention for more accurate latency measurement.
When measuring throughput, it is recommended to set it >2 (number of BPU cores), adjusting thread count to maximize BPU utilization for more accurate throughput evaluation.
Note: X5 has only one BPU core, so core_id can only be set to 0 or 1.
// Multi-threaded FPS
hrt_model_exec perf --model_file xxx.bin --thread_num 8 --core_id 0
// Latency
hrt_model_exec perf --model_file xxx.bin --thread_num 1 --core_id 1
Common Questions
How are Latency and FPS data calculated?
Latency refers to the average time consumed by a single process to perform inference, emphasizing the average per-frame inference time under sufficient resources. It is typically measured in single-core, single-thread mode on hardware. Pseudocode for measurement:
// Load model and prepare input and output tensor
...
// Loop run inference and get latency
{
int32_t const loop_num{1000};
start = std::chrono::steady_clock::now();
for(int32_t i = 0; i < loop_num; i++){
hbDNNInferCtrlParam infer_ctrl_param;
HB_DNN_INITIALIZE_INFER_CTRL_PARAM(&infer_ctrl_param);
hbDNNInfer(&task_handle,
&output,
input_tensors.data(),
dnn_handle,
&infer_ctrl_param);
// wait task done
hbDNNWaitTaskDone(task_handle, 0);
// release task handle
hbDNNReleaseTask(task_handle);
task_handle = nullptr;
}
end = std::chrono::steady_clock::now();
latency = (end - start) / loop_num;
}
// release tensor and model
FPS refers to the average number of frames inferred per second when multiple processes run inference concurrently, emphasizing model throughput under full resource utilization. It is typically measured in single-core, multi-thread mode on hardware. The method involves launching multiple threads for inference and calculating the total number of frames processed per second on average.
Why does FPS derived from Latency differ from the tool-measured FPS?
Latency and FPS are measured under different scenarios: Latency uses single-process (single-core, single-thread) inference, while FPS uses multi-process (single-core, multi-thread) inference. Therefore, derived values may not match. If the number of processes (threads) is set to 1 when measuring FPS, the calculated FPS from Latency will match the measured FPS.
How does the tool evaluate models with custom operators?
Refer to the example in horizon_runtime_sample/code/02_advanced_samples/custom_identity to develop custom operators. Compile the custom operator into a shared library and specify the library path before using the tool.
For example, if the shared library path is /userdata/plugins/libplugin.so, simply set the path when running the tool on a model containing the custom operator:
export HB_DNN_PLUGIN_PATH=/userdata/plugins/
hrt_bin_dump Tool Usage Guide
hrt_bin_dump is a layer dump tool for models, and its output files are binary files.
Input Parameter Description
| No. | Parameter | Type | Description | Notes |
|---|---|---|---|---|
| 1 | model_file |
string | Path to the model file. | Specifies the model file path; can dump input and output of all nodes in the model. |
| 2 | input_file |
string | Path to the input file. | Model input file, supports all types defined in hbDNNDataType. For IMG type, the file must be binary (suffix .bin), and size must match model input (e.g., YUV444 size: :math:height * width * 3). For TENSOR type, files can be binary (.bin) or text (.txt). Binary file size must match model input size; text file must contain at least required number of data entries (extra data will be discarded). Multiple inputs are separated by commas, e.g., for two inputs: --input_file=kite.bin,input.txt. |
| 3 | dump_path |
string | Output directory for the tool. | Must be a valid path. |
Usage Instructions
The tool provides functionality to dump model node inputs and outputs, including convolution layer outputs. Output files are in binary format.
Run hrt_bin_dump directly to view detailed usage.
See image below:
Tip:
The tool also supports -v or --version to check the version of the dnn inference library.
Example: hrt_bin_dump -v or hrt_bin_dump --version
Example Explanation
Using the MobileNetV1 model as an example, create an outputs folder and execute the following command:
./hrt_bin_dump --model_file=./mobilenetv1.bin --dump_path=./outputs --input_file=./input.bin
Refer to the screenshot below for the runtime log:
Output files can be found in the outputs/ folder. See screenshot below:
