4.1.2.2. The hb_mapper Tools

The hb_mapper is a tool that maps floating-point models to quantized models and provides some additional validation features.

It contains three subcommands/tools:

  • hb_mapper checker: Model checking tool

  • hb_mapper makertbin: Model compilation/conversion tool

  • hb_mapper infer: Model inference tool, which also dumps vector outputs of conv layers at various stages.

4.1.2.2.1. Model Checking Tool (hb_mapper checker)

In In engineering practice, since not all floating-point models can be converted to quantized models, checking is required before conversion. This checking process will go through the process of a model conversion.

However, for those more time-consuming procedures, we simplified the checking process. This command will output the check results and the OP deployment on the device when it finishes the checking of the model.

How To Use:

hb_mapper checker --model-type ${model_type} \
                  --march ${march} \
                  --proto ${caffe_proto} \
                  --model ${caffe_model/onnx_model} \
                  --input-shape ${input_node_name} ${input_shape} \
                  --output ${output}
Parameters:
--help

Shows help information and exit.

--model-type

Type of the model to be converted, can be caffe or onnx.

--march

BPU’s micro architectures, can be bernoulli2 (X3 series) or bayes (J5), the default is bayes.

--proto

Specify the Prototxt file of the Caffe model.

--model

Floating-point model file of Caffe/ONNX.

--input-shape

Input node and input shape of the node, optional, where the shape should be separated by x, e.g., data 1x3x224x224.

--output

Deprecated, log information is stored in hb_mapper_checker.log.

Note

If you find that the The converted model node information results during check does not match the The converted model node information results during makertbin, it may be because there is actually a default yaml configuration during the check process, and if you configured the yaml before the conversion, some parameter differences may cause this If you configure yaml before conversion, some parameters may cause this situation. The yaml configuration parameters that may cause this to happen include: norm_type , mean_value , scale_value , calibration_type , run_on_bpu, run_on_cpu, optimization , set_node_data_type and node_info .

4.1.2.2.2. Model Compiling Tool (hb_mapper makertbin)

This tool generates a quantized ONNX model and a runtime model simulating the on-board running situation.

For specific configuration file settings and the insturctions of each parameter, refer to the descriptions in sections Specific Parameter Information and Configuration File Template .

How To Use(fast-perf mode is not turned on):

hb_mapper makertbin --config ${config_file}  \
                    --model-type  ${model_type}

How To Use(fast-perf mode is turned on):

hb_mapper makertbin --fast-perf --model ${caffe_model/onnx_model} --model-type ${model_type} \
                    --proto ${caffe_proto} \
                    --march ${march} \
                    --input-shape ${input_node_name} ${input_shape}
Parameters:
--help

Shows help information and exit.

-c, --config

Configuration file for the model compilation, in YAML format.

--model-type

caffe or onnx.

--fast-perf

Turn on fast-perf mode, this mode will generate the bin model that can have the highest performance when running on the board side during the conversion process, so that you can easily use it for the model performance evaluation later.

If you turned on fast-perf mode, you also need to configure the following:

--model Floating-point model file of Caffe/ONNX.

--proto Specify the Prototxt file of the Caffe model.

--march BPU’s micro architectures, can be bernoulli2 (X3 series) or bayes (J5), the default is bayes.

-i, --input-shape Optional parameter that specifies the shape of the input node of the model. Currently, this configuration only takes effect when fast-perf is turned on. It is used in the following way:

  • Specify the shape information of a single input node, example of how to use: --input-shape input_1 1x3x224x224.

  • Specify the shapes of multiple input nodes, example of how to use: --input-shape input_1 1x3x224x224 --input-shape input_2 1x3x224x224.

Attention

If you do not specify the --input-shape , the tool will only support the model with the first dimension of the dynamic input node [-1, 0, ?] and the first dimension of the dynamic input node will be set to 1 by default.

The log file generated by the compilation will be stored in the directory where the command is executed under the default name hb_mapper_makertbin.log.

4.1.2.2.3. Inference Tool (hb_mapper infer)

Attention

  • The hb_mapper infer is limited by onnxruntime and does not support dynamic shape infer, so explicit input model shape is required.

  • This tool doesn’t support inferring the model that contains shape information with ?.

  • This tool only supports the non-featuremap model which input is four-dimensional and output is less than or equal to four-dimensional.

This tool runs inference using the FPM and HGM and saves inference results to the specified subfolder under --output-dir.

To validate and analyze if the FPM compilation is correct, you can specify layer_out_dump in the configuration file to True. This tool will dump the inference results of conv and output node, and you can analyze whether the model is correctly compiled by using the vec_diff tool.

How To Use:

hb_mapper infer --config ${config_file} \
                --model-file ${quantized_model_file}  \
                --model-type ${caffe/onnx} \
                --image-file ${input_node} ${image_file} \
                --input-layout ${input_layout} \
                --output-dir ${quantized_output_dir}

When using hb_mapper infer, please use the same configuration file as the one used for hb_mapper makertbin to ensure that the input data processing in the configuration file is the same.

In other words, when running hb_mapper infer, use the same images and data as when you run hb_mapper makertbin.

Attention

Your choices of input data when running hb_mapper infer are related to below parts in the configuration file:

  • preprocess_on: True: The tool can receive the JPEG images, automatically perform preprocessing operations such as resizing, and convert them into the input_type_rt format.

  • preprocess_on: False: The tool can only receive the pre-processed binary image files and you need to preprocess the images by yourself and convert images into corresponding binary files (refer to the script 02_preprocess.sh ).

Parameters:
--help

Shows help information and exit.

-c, --config

Configuration file for model compilation.

--model-file

Model file for performing the inference, can be either the floating-point model or quantized ONNX model.

--model-type

Type of the original floating-point model specified for performing the inference, can be caffe or onnx.

--image-file

Input node name and the corresponding image file for performing the inference, e.g., input_name1 kite.jpg.

--input-layout

Layout of the input model (optional).

--output-dir

Directory to save the inference results. For the quantized model, the inference results will be dequantized floating-point data.

The output of this tool are under the output_dir directory, following the naming rule: ${layername}_float.bin.

4.1.2.2.4. Key Configuration Parameter (calibration_parameters.preprocess_on)

calibration_parameters:
    # The directory where reference images for the model quantization are saved (supported image formats:
    # JPEG, BMP, etc.). The input images should cover typical scenarios. Usually 20~100 images are selected from
    # the test dataset. Do not use images for uncommon scenarios, such as overexposed, oversaturated, vague,
    # pure black, or pure white.
    # If there are multiple input parameters, use ';' to separate them.
    cal_data_dir: './calibration_data_bgr_f32'
    # When the size of the image file is different from that from the model training and the preprocess_on parameter
    # is set to True, it will use the default pre-processing method (i.e., skimage resize) to resize or crop the
    # image to the specified size. Otherwise (preprocess_on: False) you will have to change the image to the
    # size in the model training in advance.

When you set preprocess_on=True:

The tool can automatically pre-process the calibration images when the preprocess_on is set to True. In this mode, you need to specify the directory to save those calibration used JPEG images under cal_data_dir.

When calibrating the model, the tool will load JPEG images using the skimage library, resize images to the input_shape, which is specified in the configuration file, using the resize method of skimage and change them to the format specified by input_type_rt.

For example, if the input is a 608x608 JPEG image, the default pre-processing will then resize it into a 224x224 BGR (NCHW) image and adjust the range of pixel value to 0~255.

The default pre-processing logic is as follows:

def data_transformer(norm_type, input_dim, input_type_train):
    image_width = input_dim[2]
    image_height = input_dim[1]
    transformers = [
        ResizeTransformer((image_height, image_width)),
        HWC2CHWTransformer(),  # to CXHXW
        RGB2BGRTransformer(),
    ]

    transformers.append(ScaleTransformer(255))

When you set preprocess_on=False:

You will need to process the images by yourself. Modify the images to the format specified by input_type_train and save them as binary files. The tool will automatically convert the image format from input_type_train to input_type_rt.

Note

File format: uint8/float32 as described in Row-major order.

4.1.2.2.5. Configuration File Reference Example

4.1.2.2.5.1. Reference and Descriptions

The configuration file is in the YAML format, whose parameters are as follows:

# Model conversion related parameters
model_parameters:
    # Model file of floating-point Caffe neural network data
    caffe_model: '../../../01_common/model_zoo/mapper/classification/mobilenet/mobilenet.caffemodel'
    # File describes the structure of Caffe neural network
    prototxt: '../../../01_common/model_zoo/mapper/classification/mobilenet/mobilenet_deploy.prototxt'
    # Applicable BPU architecture
    march: "bernoulli2"
    # If set to True, dump the intermediate results of each layer in model conversion
    layer_out_dump: False
    # Output control parameter of log file(s)
    # If set to 'debug', output the details of model conversion
    # If set to 'info', output only the important information
    # If set to 'warn', output the information ranked higher than 'warn' and 'error'
    working_dir: 'model_output'
    # Name prefix of the model files generated by the model conversion and used for on-board execution
    output_model_file_prefix: 'mobilenetv1_224x224_nv12'


# Model input related parameters
# Use ";" to separate multiple nodes and use None for the default settings
input_parameters:
    # (Optional) input node name of the model
    # It shall be the same as the one in the model file, otherwise an error will be triggered
    # If left blank, the node name in the model file will be used
    input_name: "data"
    # Data format to be passed to the neural network when actually running on the network, including: NV12/RGB/BGR/YUV444/Gray/FeatureMap
    # If the input data is YUV444 and BGR(NCHW) is used in model training,
    # then hb_mapper will automatically insert the YUV to BGR(NCHW) conversion
    input_type_rt: 'nv12'
    # Input data layout to which the HGM needs to adapt, can be NHWC/NCHW
    # If input_type_rt is set to NV12, then this parameter left unconfigured
    # input_layout_rt: ''
    # Input data format during network training, can be RGB/BGR/Gray/FeatureMap/YUV444
    input_type_train: 'bgr'
    # Input data layout during network training, can be NHWC/NCHW
    input_layout_train: 'NCHW'
    # Input size of model network, separated by 'x', optional
    # If left blank, the network input size in the model file will be used,
    # which, if otherwise, will be overridden
    # input_shape: ''
    # Preprocessing methods of the network input, as follows:
    # 'no_preprocess': Do nothing
    # 'data_mean': Subtracts by the channel mean_value
    # 'data_scale': Multiplies number of image pixels by data_scale ratio
    # 'data_mean_and_scale': Subtracts by the channel mean_value and multiplies by scale ratio
    norm_type: 'data_mean_and_scale'
    # Mean value subtracted from the image
    # If the channel mean_value is used, the values must be separated by spaces
    mean_value: 103.94 116.78 123.68
    # Scale factor for the image preprocessing
    # If the channel scale is used, the values must be separated by spaces
    scale_value: 0.017

calibration_parameters:
    # The directory where reference images for the model quantization are saved (supported image formats:
    # JPEG, BMP, etc.). The input images should cover typical scenarios. Usually 20~100 images are selected from
    # the test dataset. Do not use images for uncommon scenarios, such as overexposed, oversaturated, vague,
    # pure black, or pure white.
    # If there are multiple input parameters, use ';' to separate them.
    cal_data_dir: './calibration_data_bgr_f32'
    # Data type of the binary file of the calibration data, can be float32 or uint8.
    # Use ";" to separate multiple nodes
    cal_data_type: 'float32'
    # When the size of the image file is different from that from the model training and the preprocess_on parameter
    # is set to True, it will use the default pre-processing method (i.e., skimage resize) to resize or crop the
    # image to the specified size. Otherwise (preprocess_on: False), you will have to change the image to the
    # size in the model training in advance.

    # preprocess_on: False
    # Type of the algorithm for model quantization, can be mix, kl, max, or load, usually the default is enough,
    # which means an automatic searching strategy that tries to get the best parameter combination from
    # a series of calibration quantization parameters
    #
    # We recommend you trying the above strategy first
    #
    # If the accuracy cannot meet your expectations, you can then try mix
    # mix is a search strategy integrating multiple calibration methods, which automatically identifies
    # quantization-sensitive nodes and pick the best method from different calibration methods at node granularity
    #
    # If the above two still cannot meet your expectations, try to use kl or max
    # and carefully adjust the quantization parameters for optimal accuracy
    #
    # Both kl and max are algorithm types for model quantization, usually using kl is enough to meet the requirements
    # In addition, when exporting a model using QAT, set this parameter to load.
    calibration_type: 'default'

# Compiler related parameters
compiler_parameters:
    # Compilation strategy, can be 'bandwidth' or 'latency':
    # 'bandwidth': Optimizes the DDR access bandwidth
    # while the 'latency': Optimizes the inference duration
    compile_mode: 'latency'
    # The default value of debug is True, that means turnning on the debug mode of the compiler,
    # which can output information related to performance simulation,
    # such as frame rate, DDR bandwidth usage, etc.
    debug: True
    # Specifies the number of cores used in model compilation
    # If left blank, the default single core is used
    # To compile dual-core models, delete the "# " below to enable dual-core mode:
    # core_num: 2
    # Optimization level range: O0~O3
    # O0: Not optimized, with fastest compilation speed and the lowest optimization level
    # O1-O3: as the level increases, the running speed of the compiled model get faster while the compilation becomes longer
    # O2 is recommended for fastest verification
    optimize_level: 'O3'

4.1.2.2.5.2. About input_type_rt/ input_type_train

To boost the ASIC performance, 2 assumptions have been made in the design of ASIC micro architecture:

  1. All inputs are quantized int8 data.

  2. All camera captured data are NV12.

Therefore, if you use the RGB (NCHW) format in the model training and expect the model to process NV12 data efficiently, then you will need to configure as follow:

input_parameters:
    input_type_rt: 'nv12'
    input_type_train: 'rgb'
    input_layout_train: 'NCHW'

Tip

When Gray format is used in the model training, while in practice, the model input is NV12, you can set both input_type_rt and input_type_train to gray during model conversion, and use only the y channel address of the NV12 data in embedded application development.

In addition to converting the input data to NV12, you can also use different RGB-orders in the training and runtime infer. More information about the image types supported by input_type_rt / input_type_train, refer to the table below (Y for supported image types, while N for unsupported image types):

input_type_train \ input_type_rt

nv12

yuv444

rgb

bgr

gray

featuremap

yuv444

Y

Y

N

N

N

N

rgb

Y

Y

Y

Y

N

N

bgr

Y

Y

Y

Y

N

N

gray

N

N

N

N

Y

N

featuremap

N

N

N

N

N

Y

Note

To meet the requirements of D-Robotics ASICs on input data type (int8) and reduce the inference costs, when input_type_rt is of the RGB(NHWC/NCHW)/BGR(NHWC/NCHW) type, the input data type of the models converted by using the conversion tool will all be int8. That is, for regular image formats, pixel values should be subtracted by 128, which has already been done by the API and you do not need to do it again.