中文 EN

6.1. Introduction

The X5 algorithm toolchain is an algorithmic solution developed based on the X5 processor, which helps you convert floating-point models into fixed-point models and rapidly deploy self-developed algorithm models on the X5.

Most models trained on GPUs are floating-point models, meaning their parameters are stored in float type. The BPU (Binary Processing Unit) in the X5 processor uses INT8 computation precision (a common standard across industry processors) and can only run fixed-point quantized models. The process of converting a trained floating-point model into a fixed-point model is called quantization. Depending on whether adjustments are made to the parameters after quantization, we classify quantization methods into QAT (Quantization-Aware Training) and PTQ (Post-Training Quantization). The operational differences between these two methods are illustrated in the figure below (left: PTQ, right: QAT):

qat_ptq_contrast

Post-Training Quantization (PTQ) requires only a small set of calibration data to calibrate the pre-trained floating-point model, directly converting the trained FP32 network into a fixed-point network. This process does not require any retraining of the original floating-point model—only a few hyperparameters are adjusted to complete quantization. It is simple and fast, and has been widely adopted in both edge and cloud scenarios. For more details about the PTQ approach, please refer to the Post-Training Quantization (PTQ) section.

Quantization-Aware Training (QAT) involves retraining the model after quantization. Since fixed-point values cannot be used in backward gradient computation, in practice, fake quantization nodes are inserted before certain operations (OPs) during training to capture the range of data flowing through these OPs. This information is then used during deployment to properly quantize the nodes. We need to iteratively optimize accuracy during training to obtain the best quantization parameters. As this method requires model retraining, it demands higher technical expertise from the operator. For more details about the QAT approach, please refer to the Quantization-Aware Training (QAT) section.

The toolchain consists of PTQ, QAT, and embedded compilation components. The schematic diagram of the toolchain structure is shown below:

toolchain_framework

6.1.1. Usage Notes

This section is intended for developers using the X5 processor and provides important usage notes regarding the X5 algorithm toolchain.

6.1.1.1. Floating-Point Model (FP32) Notes

  • Supports quantizing Caffe 1.0 version Caffe floating-point models and ONNX floating-point models with ir_version≤7, opset10, or opset11 into fixed-point models supported by the X5 processor;

  • Floating-point models trained with other frameworks must first be exported into ONNX floating-point models that meet the version requirements specified above before quantization can be performed;

  • Model input dimensions only support fixed 4D inputs in NCHW or NHWC format, e.g., 1x3x224x224 or 1x224x224x3; dynamic dimensions and non-4D inputs are not supported;

  • Floating-point models should not contain post-processing operators, such as NMS computation.

6.1.1.2. Model Operator List Description

  • Currently, all Caffe and ONNX operators supported by the X5 processor are listed. Other operators not listed are not supported due to hardware limitations of the X5 processor’s BPU. For the detailed list of supported operators, please refer to the Model Operator Support List section.