4.2.3.7. QAT Experience Summary¶

4.2.3.7.1. Hyperparameter¶

Except the hyper-parameters in the following table, other parameters are recommended to be consistent in the QAT stage with the floating point stage.

Hyperparameter	Recommended configuration	Advanced configuration (try if the recommended configuration doesn’t work)	Remark
LR	Starting from 0.001, do 2 lr decay with scale=0.1 with StepLR	1. Adjust lr between 0.0001->0.001, with lr decay of 1-2; 2. Try to replace StepLR with CosLR; 3. QAT uses AMP, and appropriately adjusts lr, too large will lead to nan.
Epoch	10% of floating point epochs	According to the convergence of loss and metric, consider whether it is necessary to extend the epoch appropriately.
Weight decay	consistent with floating point	It is recommended to make appropriate adjustments around 4e-5. If the weight decay is too small, the weight variance will be too large, and if the weight decay is too large, the weight variance of the output layer of the task with large output will be too large.
optimizer	consistent with floating point	When qat accuracy is low, it is recommended to try SGD
transforms (data augmentation)	consistent with floating point	The QAT stage can be appropriately weakened. For example, the color conversion of classification can be removed, and the scale range of RandomResizeCrop can be appropriately reduced	Reduced data augmentation may also benefit floating point models

The adjustable parameters of the QAT stage are shown below:

Quantization related parameters	Recommended configuration	Advanced configuration (try if the recommended configuration doesn’t work)
network output layer	use high precision output
averaging_constant(qconfig_params)	1. When calibration is not used, use the default value; 2. When calibration is used, weight averaging_constant=1.0, activation averaging_constant=0.0	1. When there is a large gap between calibration accuracy and floating point, do not set activation averaging_constant to 0.0; 2. Generally, weight averaging_constant does not need to be set to 0.0, the actual situation can be adjusted between (0, 1.0]
ReLU	consistent with floating point	When the featuremap value is huge, try to use ReLU with an upper bound, such as ReLU6, ReLU8, etc.