4.1.2.11. The HB_ONNXRuntime Inference Library

The HB_ONNXRuntime is an ONNX model inference library for the x86 side packaged by D-Robotics based on the public version of ONNXRuntime. In addition to supporting the original ONNX models directly exported by Pytorch, TensorFlow, PaddlePaddle, MXNet and other training frameworks, it also supports inferring the various stages of ONNX models generated during the PTQ conversion process of the D-Robotics toolchain. The usage flow is shown as follows:

../../../../_images/hb_onnxruntime.png

Note

Note that the computing platform of the D-Robotics BPU architecture uses the int8 computational accuracy (the common accuracy for computing platforms in the industry). During PTQ conversion using the D-Robotics toolchain, while the conversion of the input of the bin model generated by the final conversion from input_type_rt to input_type_train color space will be done in conjunction with the processor hardware. However, the preprocessing node inserted at the front of the onnx model generated during the conversion process (non-featuremap input and norm_type is not specified as no_preprocess) does not contain the hardware conversion logic, so the actual input of the onnx model is just an intermediate type (except for featuremap input, all other types of input need to do -128 operation, i.e., from unit8 to int8), this data conversion will be handled internally by the HB_ONNXRuntime, but only in the following scenarios that don’t involve a loss of data:

  • The int8 model input/inputs: support the input/inputs of int8, uint8.

  • The uint8 model input/inputs: support the input/inputs of int8, uint8.

  • The float32 model input/inputs: support the input/inputs of int8, uint8 and float32.

If conversions that may result in data loss occur, such as those involving mixed-type inputs, the corresponding data conversion process must be completed on your own before proceeding with inference.

4.1.2.11.1. How to Use

The basic flow for loading ONNX model inference using HB_ONNXRuntime is shown below, and this sample code applies to inference for all ONNX models. Prepare the data according to the input type and layout requirements of different models:

import numpy as np
# Load D-Robotics dependency library
from horizon_tc_ui import HB_ONNXRuntime
# Prepare the input for model running
input_data = np.load("input.npy")
# Load model file
sess = HB_ONNXRuntime(model_file = "model.onnx")
# Obtain the model input & output node information
input_names = sess.input_names
output_names = sess.output_names
# Prepare the input data, here we assume the model has only one input
input_info = {input_names[0]: input_data}
# Model inference, the return value is a list that corresponds in order to the names specified by output_names
output = sess.run(output_names, input_info)

4.1.2.11.2. Parameters

  • output_names:

    Used to specify the output name, support to specify as None or custom configuration.

    • Specify as None, the tool will internally read the information of the output nodes in the model and give the inference result in the order of parsing.

    • If you customize the configuration, you can specify full or partial output_name and support modifying the order of outputs. Then, when the inference is complete, the output will be returned according to the output name and order that you specified.

  • input_info:

    Prepare the inputs for the model running according to the input type and layout, the format is required to be in the form of a dictionary, the input name and the input data should be the key value pairs, refer to the configuration example: {“input_name” : data}.