4.1.2.1. Model Conversion Process¶
This section explains how to convert the floating-point models trained by using the opensource ML frameworks (such as Caffe, TensorFlow, PyTorch, etc.) to the D-Robotics hardware supported fixed-point models.
In most cases, the threshold values and weights of the floating-point models obtained from either the opensource ML frameworks or trained by yourself are floating-point numbers (float32) and each number occupies 4 bytes.
However, by converting the floating-point numbers to fixed-point numbers (int8), each number occupies only 1 byte, thus the computation operations in the embedded runtime can be dramatically reduced.
Therefore, it brings significant performance boost by converting the floating-point models to fixed-point models with no loss or very small loss.
Typically, model conversion can be divided into the following steps:
Check if there are unsupported OPs in the models to be converted.
Prepare 20~100 images for calibration use at the conversion stage.
Convert the floating-point models to fixed-point models using the floating-point conversion tools.
Evaluate the performance and accuracy of the converted models to ensure that there isn’t huge difference in model accuracy before and after the conversion.
Run models in simulator/dev board to validate model performance and accuracy.
Also please refer to below flowchart to learn more about the conversion process:
Functions of tools correspond with the conversion process:
4.1.2.1.1. Model Checking (hb_mapper checker)¶
Note
If you need to do this process in the sample folder, you need to execute the 00_init.sh script in the folder first to get the corresponding original model and dataset.
Before converting the floating-point models into the fixed-point models,
we should check if there are D-Robotics hardwares unsupported OPs in the floating-point models using the hb_mapper checker tool.
If yes, the tool will report the unsupported OP(s).
Usage of the hb_mapper checker tool please refer to the Model Checking Tool (hb_mapper checker) .
For instructions of using hb_mapper checker, refer to the Model Checking Tool (hb_mapper checker).
If unsupported ops are found during the process, it will prompt as follows:
ERROR D-RoboticsRT not support these cpu operators: {name of the unsupported OP}
Tip
More information about D-Robotics hardware supported OPs, refer to Model Conversion Toolchain Operator Support Constraint List.
The tool will print out an OP list which also shows the running location of these ops, on the CPU or BPU.
If the validation ends without error, you can proceed to the next step, as follows:
hb_mapper checker --model-type caffe --proto mobilenet_deploy.prototxt \
--model mobilenet.caffemodel --march bernoulli2
2021-01-08 17:33:53,117 INFO Start hb_mapper....
2021-01-08 17:33:53,118 INFO hb_mapper version 1.1.42
...
fc7 BPU id(0) HzSQuantizedConv
prob CPU -- Softmax
2021-01-08 17:33:53,329 INFO [Fri Jan 8 17:33:53 2021] End to D-Robotics NN Model Convert.
2021-01-08 17:33:53,332 INFO model deps info empty
2021-01-08 17:33:53,351 INFO End model checking....
Tip
If unsupported op exists, you can either contact D-Robotics technical staff for the op development plan or include the unsupported ops by using the Custom OP feature.
4.1.2.1.2. Calibration Image Preparation¶
Note
If you need to do this process in the sample folder, you need to execute the 00_init.sh script in the folder first to get the corresponding original model and dataset.
When converting the floating-point models, you need to prepare 20~100 images for calibration use at the calibration stage.
Input image formats may vary by input type and layout. In this stage, because both original (JPG, etc.) and the processed images are valid, you can either feed the calibration images used in the model training or feed your own processed images.
We recommend you preprocessing the calibration images on your own: Firstly, complete the operations such as image channel (BGR/RGB), data layout (NHWC/NCHW), and image resizing/padding (Resize&Padding). Secondly, set preprcess_on to False in the YAML file (i.e., mobilenet_config.yaml). The tool will then feed the images to the calibration stage after loading them as binary files.
Taking MobileNet as an example, the required transformer operations are as follows:
transformers = [
ShortSideResizeTransformer(short_size=256), # Pad the short side to 256 to maintain length-width ratio
CenterCropTransformer(crop_size=224), # Crop a 224\*224 image from the center of the image
HWC2CHWTransformer(), # Switch data layout from NHWC to NCHW
RGB2BGRTransformer(data_format="CHW"), # Switch color channel from RGB to BGR
ScaleTransformer(scale_value=255), # Switch number range from 0-1 to 0-255
]
Tip
If the color space in model training is BGR/RGB, then the color space of those images that will pass to the tool at the calibration stage must also be BGR/RGB. The tool can convert the color from BGR/RGB to YUV444/Gray automatically.
For example, in the above code block, the actual input of the MobileNet model is specified as NV12, but after 02_preprocess.sh conversion, the color space will be switched to BGR, and the remaining conversion from BGR to NV12 are automatically done by the tool.
4.1.2.1.3. Model Conversion (hb_mapper makertbin)¶
When you confirm that the floating-point model can be successfully converted by using the hb_mapper checker tool,
you can then convert the floating-point model to a D-Robotics hardware supported fixed-point model by using the hb_mapper makertbin tool.
This tool requires users to pass in the type (caffe / onnx ) of the model for conversion as well as a configuration file
(*.yaml) containing conversion requirements.
For specific configuration file settings and the insturctions of each parameter, refer to the descriptions in sections Specific Parameter Information and Configuration File Template .
When the model conversion process ends, it also prints the level of similarity between the floating-point model and fixed-point model to the log, you can therefore judge the similarity before and after conversion according to the Cosine Similarity field.
As shown in the example below, the value of Cosine Similarity is very close to 1, so the performance of the fixed-point model should be very close to that of the floating-point model before the conversion.
2023-03-01 20:26:50,149 INFO Start hb_mapper....
......
2023-03-01 20:27:21,427 INFO [Wed Mar 1 20:27:21 2023] End to compile the model with march bernoulli2.
2023-03-01 20:27:21,429 INFO The converted model node information:
======================================================================================================================
Node ON Subgraph Type Cosine Similarity Threshold In/Out DataType
----------------------------------------------------------------------------------------------------------------------
HZ_PREPROCESS_FOR_data BPU id(0) HzSQuantizedPreprocess 0.999988 127.000000 int8/int8
conv1 BPU id(0) HzSQuantizedConv 0.999922 2.937425 int8/int8
conv2_1/dw BPU id(0) HzSQuantizedConv 0.999378 2.040827 int8/int8
conv2_1/sep BPU id(0) HzSQuantizedConv 0.996680 4.486579 int8/int8
conv2_2/dw BPU id(0) HzSQuantizedConv 0.997340 3.545496 int8/int8
conv2_2/sep BPU id(0) HzSQuantizedConv 0.996384 2.791299 int8/int8
conv3_1/dw BPU id(0) HzSQuantizedConv 0.994165 1.417208 int8/int8
conv3_1/sep BPU id(0) HzSQuantizedConv 0.985451 2.188753 int8/int8
conv3_2/dw BPU id(0) HzSQuantizedConv 0.994921 1.822225 int8/int8
conv3_2/sep BPU id(0) HzSQuantizedConv 0.994251 1.841765 int8/int8
conv4_1/dw BPU id(0) HzSQuantizedConv 0.988263 1.043535 int8/int8
conv4_1/sep BPU id(0) HzSQuantizedConv 0.990294 1.736999 int8/int8
conv4_2/dw BPU id(0) HzSQuantizedConv 0.992460 0.990603 int8/int8
conv4_2/sep BPU id(0) HzSQuantizedConv 0.993468 1.574677 int8/int8
conv5_1/dw BPU id(0) HzSQuantizedConv 0.988949 0.823123 int8/int8
conv5_1/sep BPU id(0) HzSQuantizedConv 0.990803 1.265912 int8/int8
conv5_2/dw BPU id(0) HzSQuantizedConv 0.990202 0.772344 int8/int8
conv5_2/sep BPU id(0) HzSQuantizedConv 0.983443 1.530479 int8/int8
conv5_3/dw BPU id(0) HzSQuantizedConv 0.986502 0.783812 int8/int8
conv5_3/sep BPU id(0) HzSQuantizedConv 0.977642 1.927324 int8/int8
conv5_4/dw BPU id(0) HzSQuantizedConv 0.982337 0.996043 int8/int8
conv5_4/sep BPU id(0) HzSQuantizedConv 0.962062 2.167391 int8/int8
conv5_5/dw BPU id(0) HzSQuantizedConv 0.978872 1.923361 int8/int8
conv5_5/sep BPU id(0) HzSQuantizedConv 0.960184 3.578415 int8/int8
conv5_6/dw BPU id(0) HzSQuantizedConv 0.980317 2.463874 int8/int8
conv5_6/sep BPU id(0) HzSQuantizedConv 0.981055 4.124151 int8/int8
conv6/dw BPU id(0) HzSQuantizedConv 0.998241 0.667692 int8/int8
conv6/sep BPU id(0) HzSQuantizedConv 0.985220 0.983833 int8/int8
pool6 BPU id(0) HzSQuantizedGlobalAveragePool 0.993602 11.415899 int8/int8
fc7 BPU id(0) HzSQuantizedConv 0.995105 5.843800 int8/int32
prob CPU -- Softmax 0.985517 -- float/float
2023-03-01 20:27:21,430 INFO The quantify model output:
=======================================================================
Node Cosine Similarity L1 Distance L2 Distance Chebyshev Distance
-----------------------------------------------------------------------
prob 0.985517 0.000385 0.000203 0.185123
2023-03-01 20:27:21,432 INFO [Wed Mar 1 20:27:21 2023] End to D-Robotics NN Model Convert.
Note
As the above CosineSimilarity refers to the very first image in the calibration images, it cannot fully represent the model accuracy before and after the conversion.
After the model conversion, the following files will be generated under the folder model_output (default name):
\*_subgraph_0.html: Static performance evaluation file (with better readability)\*_subgraph_0.json: Static performance evaluation file\*_original_float_model.onnx: Original floating-point model (floating-point model)\*_optimized_float_model.onnx: Optimized floating-point model\*_calibrated_model.onnx: Calibration model\*_quantized_model.onnx: Fixed-point model\*.bin: Hybrid model to run in dev board
These model files are the output of the conversion at key stages and will be used in the subsequent steps.
Note
You can use the
03_classification/01_mobilenet/mapper/03_build.shscript to experience thehb_mapper makertbintool.If you want to learn more about the model conversion workflow, please read Model Quantification and Compilation.
If you want to learn more about calibration methods, please read Model Accuracy Analysis and Optimization.
4.1.2.1.4. Single Image Inference¶
The accuracy of the fixed-point model generated by the floating-point conversion must be evaluated.
You should have good understanding of the input/output structures of the model. You should also be able to accurately preprocess the input images of the model, postprocess the model outputs, and write the model execution scripts on your own.
You can refer to the sample code in 03_classification/01_mobilenet/mapper/04_inference.sh in D-Robotics model conversion sample package.
The code logic of the model accuracy validation are as follows:
import numpy as np
# Load D-Robotics dependency library
from horizon_tc_ui import HB_ONNXRuntime
from postprocess import postprocess
# Prepare the input for model running, here the `input.py` is the processed data
input_data = np.load("input.npy")
# Load model file
sess = HB_ONNXRuntime(model_file = "***_quantized_model.onnx")
# Obtain the model input & output node information
input_names = sess.input_names
output_names = sess.output_names
# Prepare the input data, here we assume the model has only one input
input_info = {input_names[0]: input_data}
# Model inference, the return value is a list that corresponds in order to the names specified by output_names
output = sess.run(output_names, input_info)
# Postprocessing
top_five_label_probs = postprocess(output)
As shown above, this single image inference script is used for validating model accuracy by inferencing a zebra image.
After preprocessing the image data from RGB to the intermediate type of input_type_rt (no -128 required). As input_type_rt of the example mobilenet is NV12, the corresponding intermediate type is YUV444_128, the infer_transformers` process needs to complete the conversion of RGB–>NV12–>YUV444. For informaiton about intermediate types, refer to the Model Conversion Interpretation.
Then, infer the model by passing the above image data by using the HB_ONNXRuntime command, post-processing after inference, and finally print out 5 of its most likely types.
The output of the script is shown as follows with the most possible class being label: 340:
I0108 18:11:47.398328 140427646048000 cls_inference.py:89] The input picture is classified to be:
label 340: prob 0.97
label 292: prob 0.02
label 282: prob 0.00
label 83: prob 0.00
label uses the ImageNet label classes, you can also find it in the 01_common/test_data/classes.txt file in the D-Robotics Model Conversion Sample Package, where the corresponding class of 340 is zebra, so the inference result is correct.
4.1.2.1.5. Model Accuracy Evaluations¶
It’s insufficient to determine the model accuracy by single image inference, so you still need to use scripts to evalute the model accuracy after the conversion.
To do so, you need some coding work to enable the model to loop the image inference and compare the inference results with standard results to get model accuracy results.
In model accuracy evaluations, images must be pre-processed and the model output must be post-processed, so here we provide a Python script as a sample.
The logic of this script is the same as that of single image inference, yet it must run on the entire dataset.
The script can evaluate the model output results and generate evaluation results.
Because it takes a long time to run the script, you can set the number of threads to run the evaluation by specifying the PARALLEL_PROCESS_NUM environment variable.
The output of the script is shown as follows:
===REPORT-START{MAPPER-EVAL}===
0.7011
===REPORT-END{MAPPER-EVAL}===
As you can see that the accuracy of the converted fixed-point model is 0.7011.
Note
Model accuracy may vary slightly due to the differences of operating systems and dependencies.
Model accuracy may vary slightly by iteration.
If you find that there is an accuracy loss after model conversion, please contact D-Robotics technical staff.
4.1.2.1.6. Model Performance Evaluations¶
One of the important performance indicators is the frame rate of a model running on the dev board.
To save developers from setting up a dev board environment, you can use the hb_perf tool to analyze the performance of the fixed-point model.
In the MobileNetv1 sample, after running the hb_perf mobilenetv1_224x224_nv12.bin command, you can find the model analysis file mobilenetv1_224x224_nv12.html under hb_perf_result/mobilenetv1_224x224_nv12/, as follows:
[horizon@gpu-dev model_output]$ hb_perf mobilenetv1_224x224_nv12.bin
2023-08-30 19:40:21,305 INFO log will be stored in horizon/.../mobilenetv1/hb_perf.log
2023-08-30 19:40:21,322 INFO Start hb_perf....
2023-08-30 19:40:21,323 INFO hb_perf version 1.21.0
2023-08-30 19:40:21,375 INFO ********* mobilenetv1_224x224_nv12 perf **********
......
2023-08-30 19:40:21,875 INFO file stored at : horizon/.../mobilenetv1_224x224_nv12/mobilenetv1_224x224_nv12.html
In mobilenetv1_224x224_nv12.html, we can see the overall model performance data.
When a model contains more than one segment, an individual performance analysis report on each segment running on the BPU will be generated.
Performance indicators in the above graph are explained as follows:
Model Name: Model name.BPU Model Latency(ms): overall time consumption of the model (unit: ms).Total DDR (loaded+stored) bytes per frame(MB per frame): Total amount of DDR space occupied by data loading and storage of the BPU part of the model (unit: MB/frame).Loaded Bytes per Frame: Loaded bytes per frame at the model runtime.Stored Bytes per Frame: Stored bytes per frame at the model runtime.
Note
The above BPU Model Latency(ms) refers to the time taken by the part of model to run on the BPU; However, the time taken by the part of the model that runs on the CPU, if there is any, is not counted.
4.1.2.1.7. [Reference] Supported Calibration Methods¶
We currently support the following calibration methods:
1.Default
Default is a strategy that automatically searches the calibrated quantization parameters to obtain a relatively good combination.
2.Mix
Mix is a search strategy that integrates multiple calibration methods, which automatically identifies quantization-sensitive nodes, selects the best from a group of calibration methods at node granularity, and finally build a hybrid calibration method absorbing the advantages of multiple calibration methods.
3.KL
KL learns from the Solution proposed by TensorRT, uses the KL entropy value to traverse the data distribution of each quantized layer, and determines threshold value by searching for the lowest KL entropy value.
As this method can cause more data saturation and smaller data quantization granularity, it more suitable than max for those neural network models with more concentrated data distribution.
4.Max
Max refers to a calibration method that automatically selects the max value in quantized layer as the threshold.
This method can cause oversized quantization granularity; however, it also causes less saturated points than the KL method, which makes it suitable for those neural network models with more discrete data distribution.
5.Load
This parameter is required when using the model exported by QAT.
6.Skip
If the model performance is the only concern without precision requirements, you can try the skip calibration method,
which uses max + internally generated random calibration data for calibration and does not require you to prepare calibration data,
so it is more suitable for the first attempts to validate the model structure.
Attention
As the skip method uses max + internally generated random calibration data for calibration, the model obtained cannot be used for accuracy verification.
4.1.2.1.8. [Reference] OP List¶
For more information about the operators and corresponding constraints currently supported by D-Robotics Algorithm Toolchain, please refer to Model Conversion Toolchain Operator Support Constraint List.