4.2.3.6. Building Quantization-friendly Floating-point Models¶

4.2.3.6.1. Background¶

Not all models are suitable for quantification. In the actual production environment, we often encounter some models with poor accuracy after quantization. The fundamental reason is that the model in the floating point stage is not suitable for quantization. Here are some common floating-point situations that are not suitable for quantization. In practice, you can use the debug tool to find the part of the model that is not suitable for quantization.

4.2.3.6.2. Phenomena Not Suitable for Quantization¶

After the QAT accuracy problem occurs, we can use the debug tool provided in the Model Accuracy DEBUG Tool to properly analyze each part of the model. The results of the analysis reveal the parts of the model that are not suitable for quantification

4.2.3.6.2.1. Extensive Use of OPs Not Suitable for Quantization¶

Due to the limitations of the quantization method and the compiler, the current quantized op implementation will be subject to some limitations or errors. The error of this operator generally has two manifestations. The first one is that the accuracy will be affected when QAT is used (in rare cases), and the second one is that the accuracy will be affected when QAT is transferred to Quantized. The following table is used here to briefly list some common situations. For detailed OP impact, please refer to the output of the debug tool.

Representative OPs	Reasons not suitable for quantization	Influence sphere
Softmax, LayerNorm	QAT has multiple quantization nodes, and Quantized is the combination of multiple lookup tables	QAT, Quantized
LUT OPs (Cos, Exp, Pow, Sin, Sqrt, Sigmoid, Tanh)	Implemented by LUT, and there is a risk of accuracy when converting to quantized	Quantized
Conv/AvgPool with large kernel size	Large kernel size leads to inaccurate statistics. It is recommended to use conventional kernels, such as 2x2, 3x3	QAT
Concat with a wide range of inputs	Similar to the large kernel size of conv, the statistics are inaccurate	QAT
Improper use of QuantStub	Refer to the check of the input situation in the next section	QAT

It should be noted here that using these OPs do not necessarily lead to low accuracy. It also needs to take the specific model, the specific algorithm, and the frequency of use into consideration. If used heavily, you need to consider the impact of these OPs on quantization.

4.2.3.6.2.2. Asymmetric Input or High Resolution Requirements¶

There are generally two types of model inputs. The first is common raw data (image, radar, etc.), and the other is an auxiliary input of the model (such as the position code of transformer). All these inputs need to be fed into the quantized model via the quantization settings.

At present, the input quantization method is relatively fixed, using Plugin or community QuantStub, doing symmetric quantization with scale=1/128.0 or any value. In this case, there are certain requirements for the input floating-point data. Taking image input as an example, the input range of the original image (whether it is RGB or YUV) is [0, 255].

Raw input range is [0, 255], not suitable for symmetric quantization.
Complete the conversion about 0 symmetry, the input range will be [-128, 128], suitable for symmetric quantization, but not suitable for fixed scale=1/128.0.
When the normalization has been done, the input range [-1, 1] is suitable for symmetric quantization with fixed scale=1/128.0. This distribution analysis of the input is very common in radar scenarios. It is suggested that a proper analysis of the input data is performed first before radar processing.

4.2.3.6.2.2.1. Solution¶

Adjust the scale value of QuantStub under the condition of int8.
Use QuantStub of int16 qconfig .
Normalize the input in 0 symmetry according to the input range.

4.2.3.6.2.3. Large Feature Map in Model¶

Checking the intermediate output of the model is generally to analyze the data distribution (min, max, etc.) of the floating-point model to see if there are obvious outliers (such as large values in the thousands or tens of thousands). In this case, the process of the QAT training differs significantly from that of the floating-point training, resulting in poor QAT accuracy.

4.2.3.6.2.3.1. Solution¶

It is recommended to check the model structure. In the floating-point training phase, the OP with larger value is followed by BN, ReLU and other normalization operations.
Use int16 quantization for output layers with a large range in the model.

4.2.3.6.2.4. Large Range of Model Weight¶

The large weight range of the model is similar to that of the intermediate output of the model. In this case, the difference between the QAT and the floating-point model training process will also be too large, resulting in poor QAT accuracy.

4.2.3.6.2.4.1. Solution¶

Use int16 quantization for a wide range of weights in the model.
Adjust weight decay appropriately. (may have an impact on floating point precision)

4.2.3.6.3. Tips¶

4.2.3.6.3.1. Not Suitable for Quantification Does Not Mean That It Cannot Be Quantified¶

QAT still has a certain ability of model training, so just the fact that it does not lend itself to quantification does not mean that it cannot be quantified. In some cases, even if the above unsuitability for quantification occurs, it can still be quantified very well. Therefore, the quantization-friendly floating-point model is built to assist in the analysis of problems with the floating-point model when there is an obvious problem with the quantization accuracy.

4.2.3.6.3.2. int16 Quantization¶

int16, which is currently supported, is conditionally supported for input_channel * kernel_size * kernel_size <= 256, and the speed of a single OP will be twice as slow. Therefore, in most cases, int16 quantization can only help to figure out accuracy problems, which is not suitable for deployment on the board.