6.2.7. The hb_verifier Tool

The hb_verifier tool is used to validate the results of a specified fixed-point model and *.bin runtime model.

If you specify the image before using the tool, the hb_verifier will use the specified image, perform fixed-point model inference, runtime model board-side and runtime model on x86-side emulator, and compare the results of its three parties two by two and give a conclusion whether it passes or not(this process supports self-selection, so you can choose what to compare if you want).

If no image is specified before using the tool, the hb_verifier tool will use randomly generated tensor data for inference by default.

Attention

  • When performing inference on runtime models on the board side, you need to make sure that the given ip can be pinged and that hrt_tools is installed on the board side, if not, you can use the install.sh script under package/board in the OE package to install it.

    bash install.sh {board IP}

  • When performing inference on runtime models on the x86 side, you need to make sure that hrt_tools is installed on the host side, if not, you can use the install_host_package.sh script under package/host/host_package/ in the OE package to install it.

    bash install_host_package.sh -y

Before using this tool, the following should be brought to your attention.

Attention

  • The hb_verifier tool does not support comparing the node-changed bin models(other than Dequantize node change) with quanti.onnx.

    If, before using this tool, you used the hb_model_modifier tool to remove the last node before the output of the bin model and the node is a non-Dequantize node, or the yaml file is configured with the remove_node_type parameter, thus removing the last node before the output of the bin model and the node if it is a non-Dequantize node, then the hb_verifier tool will no longer support quanti.onnx for comparison with the bin model after the node is removed.

    If you want to solve the above problem, you need to avoid the above situation where the last node before the output of the bin model is deleted and the node is a non-Dequantize node.

  • Since the hb_verifier tool interacts with the board side via SSH, if you are using this tool inside an OE Docker container, do not use the docker attach command to connect to the container, using this command to connect to the container will cause SSH authentication to fail for interaction with the board side.

    It is recommended that you run run_docker.sh provided inside the OE package or use the docker exec -it {container ID} /bin/bash command to enter the container before using this tool.

6.2.7.1. How to Use

hb_verifier -m   ${quanti_model},${bin_model} \
            -b   ${board_ip} \
            -s   True / False \
            -i   ${input_img} \
            -c   ${digits}  \
            -r   True / False
            -u   Board username
            -p   Board password

6.2.7.2. Parameters

Parameters:
-h, --help

Displays help information and exits.

--version

Displays version information and exits.

-m, --model

The name of the fixed-point model and the bin model, with “,” to distinguish between multiple models.

-b, --board-ip

The ip address of the arm board used for on-board testing.

-s, --run-sim

Set whether to use libdnn for X86 environment to do bin model inference, default is False.

  • When this parameter is set to True, the tool will use the x86 environment’s libdnn to do bin model inference.

  • When this parameter is set to False, the tool will not use the x86 environment’s libdnn to do bin model inference.

-i, --input-img

Specify the image to be used during the inference test.

If not specified, randomly generated tensor data will be used.

If the specified image is a binary image file, the file should be in the form of a .bin extension.

The multi-input model adds images in the following two ways of passing parameters, with multiple images separated by “,”.

  • input_name1:image1,input_name2:image2, …

  • image1,image2…

Attention

In the muiti-batch model scenario, the hb_verifier does not support specify the input as binary data, and it is recommended to directly specify the input as a single-batch image, or you can use the random data instead of specify the input for your consistency verification.

-c, --compare_digits

Set the numerical precision of the comparison inference result (i.e. the number of decimal places to compare the value), if not specified, the tool will compare to five decimal places by default.

-r, --dump-all-nodes-results

Set whether to save the output results of each operator in the model and compare the results of operators with the same output name, default is False.

  • When this parameter is set to True, the tool will get the output of all the nodes in the model and do matching based on the names of the node outputs to make comparisons.

  • When this parameter is set to False, the tool will get only the final output of the model and compare it.

Attention

Please note that the dump feature is not currently supported in X86 environments for performance reasons.

-u, --username

Specify the development board user name, the default is root.

-p, --password

If you are using a development board with a password, you need to type in the board password.

Attention

If the board you are using does not have a password, do not type this voluntarily.

6.2.7.3. Examples of Reference Usage

  1. quanti.onnx model inference, bin model inference on the board side, bin model inference in the X86 environment, and a comparison of the three-way inference results.

hb_verifier -m quanti.onnx,model.bin -b *.*.*.* -s True (-i Optional)

  1. quanti.onnx model inference, bin model inference at the board side, and a comparison of the inference results between the two parties.

hb_verifier -m quanti.onnx,model.bin -b *.*.*.* (-i Optional)

  1. quanti.onnx model inference, bin model inference at the board side, the process will save the output of each operator in the two-sided model and compare the results with the same name of the operator output.

hb_verifier -m quanti.onnx,model.bin -b *.*.*.* -r True (-i Optional)

  1. quanti.onnx model inference, bin model inference in X86 environment, and a comparison of the inference results between the two parties.

hb_verifier -m quanti.onnx,model.bin -s True (-i Optional)

6.2.7.4. Description of the Output Content

Result comparison will be displayed at the client terminal, the tool will compare the results of multiple models running in different scenarios, and if there are no problems the following should be displayed:

Comparison results of original output is model_infer_output_0
raw output 0 and raw output 0 result Strict check PASSED
Quanti.onnx and Arm result Strict check PASSED

When the fixed-point model and the runtime model are not consistent in accuracy, the specific information of the inconsistent results will be output, as shown in the following log.

INFO ================== Sim infer log end ==========================
INFO ***************************************************************
INFO compare source: Quanti.onnx VS Arm
INFO compare model name: clr_320x800_bgr_quantized_model VS clr_320x800_bgr
Compare progress: 100%|###########################| 1/1 [00:00<00:00, 55.47it/s]
INFO =============== Original output comparison results =================
INFO Comparison results of original output is model_infer_output_0_output
INFO mismatch result num: 1000
INFO total result num: 1000
INFO mismatch rate: 1.0
INFO relative mismatch ratio: 0.9997149805034536
INFO max abs error: 8.36695
WARNING raw output output and raw output output result Strict check FAILED
WARNING Quanti.onnx and Arm result Strict check FAILED
INFO ***************************************************************
INFO ***************************************************************
INFO compare source: Quanti.onnx VS Sim
INFO compare model name: clr_320x800_bgr_quantized_model VS clr_320x800_bgr
Compare progress: 100%|##########################| 1/1 [00:00<00:00, 135.53it/s]
INFO =============== Original output comparison results =================
INFO Comparison results of original output is model_infer_output_0_output
INFO mismatch result num: 1000
INFO total result num: 1000
INFO mismatch rate: 1.0
INFO relative mismatch ratio: 0.9997149805034536
INFO max abs error: 8.36695
WARNING raw output output and raw output output result Strict check FAILED
WARNING Quanti.onnx and Sim result Strict check FAILED
INFO ***************************************************************
INFO ***************************************************************
INFO compare source: Arm VS Sim
INFO compare model name: clr_320x800_bgr VS clr_320x800_bgr
Compare progress: 100%|##########################| 1/1 [00:00<00:00, 150.69it/s]
INFO Arm and Sim result Strict check PASSED
INFO ***************************************************************

Among them.

  • mismatch result num is the number of inconsistent results of two models, including three inconsistencies:

    • mismatch.line_miss num is the number of inconsistent output results.

    • mismatch.line_diff num is the number of output result gaps that are too large.

    • mismatch.line_nan num is the number of output nan.

  • total result num is the total number of output data.

  • mismatch rate refers to the proportion of inconsistent data in the total number of output data.

  • relative mismatch ratio is the relative mismatch ratio, and the value with the largest mismatch ratio is displayed.

  • max abs error is the maximum absolute error.