6.3.6. Model Accuracy Analysis

There are inevitable accuracy loss with the post-training model quantization that converting the floating-point models into the fixed-point models based on dozens or hundreds of calibration data. But it has been proofed by a large number of production experience that as long as the most optimized parameter combination can be found out, in most cases, D-Robotics’s conversion tools can keep the accuracy loss within 1%.

This section explains how to correctly analyze model accuracy. In case the evaluation results fail your expectations, please refer to the PTQ Model Accuracy Optimization section and try to optimize the accuracy. If you still can’t solve it, please don’t hesitate to visit the D-Robotics Developer Community and seek technical support.

You are expected to understand how to evaluate model accuracy when reading this section. This section explains how to run the model inference using the outputs of model conversion. As previously described, successful model conversion consists of the following outputs:

  • *_original_float_model.onnx

  • *_optimized_float_model.onnx

  • *_calibrated_model.onnx

  • *_quantized_model.onnx

  • *.bin

Although the final bin model is the one that will be deployed to the computing platform, in order to facilitate the accuracy evaluation on Ubuntu development machines. We provide *_quantized_model.onnx to complete this accuracy evaluation process. The model has already been quantized and has the same accuracy results as the final bin model. The basic process for loading the ONNX model inference model using the D-Robotics development library is shown below, and this illustrative code is not only applicable to the quantized model, but also to the original and optimized models. You only need to prepare corresponding data in line with different input types and layouts.

import numpy as np
# Load D-Robotics dependency library
from horizon_tc_ui import HB_ONNXRuntime

# Prepare the input for model running
input_data = np.load("input.npy")
# Load model file
sess = HB_ONNXRuntime(model_file = "***_quantized_model.onnx")
# Obtain the model input & output node information
input_names = sess.input_names
output_names = sess.output_names
# Prepare the input data, here we assume the model has only one input
input_info = {input_names[0]: input_data}
# Model inference, the return value is a list that corresponds in order to the names specified by output_names
output = sess.run(output_names, input_info)

In addition, the input data preparation process is the most error-prone part. Compared with the accuracy validation process during the original floating-point model design and training, you are expected to further adjust the inference input data after data pre-processing, especially data format (RGB, NV12 etc.), accuracy (INT8, FLOAT32 etc.) and layout (NCHW or NHWC). How to specifically adjust the input data depends jointly on your specified input_type_train, input_layout_train, input_type_rt and input_layout_rt when converting the model. For the parameter configuration, please refer to the Model Conversion Interpretation.

For example, there is an original floating-point model for classification trained using ImageNet, which has only one input node. The input node can accept three-channel images with BGR sequence and input data layout is NCHW. At the original floating-point model design and training stage, the data pre-processing prior to validation dataset inference is shown as below:

  1. The image length and width are scaled equally, and the short side is scaled to 256.

  2. Obtain 224x224 image using the center_crop method.

  3. Subtract mean value by the channel.

  4. Multiply scale ratio.

When converting this original floating-point model using D-Robotics’s conversion tools, specify the input_type_train as bgr, input_layout_train as NCHW, input_type_rt as bgr and input_layout_rt as NHWC.

According to the rules described in the Model Conversion Interpretation, the *_quantized_model.onnx accepts bgr_128 with NCHW layout. In correspondence with the above-mentioned sample, the your_custom_data_prepare part of pre-processing should be the following:

# This sample uses the skimage library and there are differences when using the opencv library
# Note that the mean subtraction and scale multiplication operations is not shown in the transformers
# The mean and scale operations have been fused into the model,
# Refer to the previous norm_type/mean_values/scale_values configurations
def your_custom_data_prepare_sample(image_file):
  # When reading images using the skimage library, the layout is NHWC
  image = skimage.img_as_float(skimage.io.imread(image_file))
  # Uniformly scale the images and resize the short side to 256
  image = ShortSideResize(image, short_size=256)
  # Obtain 224x224 images using the CenterCrop
  image = CenterCrop(image, crop_size=224)
  # The channel sequence is RGB when reading the results using the skimage,
  # converting to the BGR sequence needed for bgr_128
  image = RGB2BGR(image)
  # If the original model is NCHW input (except input_type_rt is nv12)
  if layout == "NCHW":
    image = HWC2CHW(image)
  # skimage reads values in the range [0.0,1.0] and adjusts them to the range needed by bgr
  image = image * 255
  # the bgr_128 subtracts 128 from bgr
  image = image - 128
  # bgr_128 uses int8
  image = image.astype(np.int8)

  return image