6.4. Delve deeper

Quantization refers to the technique of performing calculations and storing tensors at bit widths below floating point precision. Quantitative models use integers instead of floating-point values to perform some or all operations on tensors. Compared to the typical FP32 model, horizon_plugin_pytorch supports INT8 quantization, which reduces model size by 4 times and reduces memory bandwidth requirements by 4 times. Hardware support for INT8 computing is usually 2 to 4 times faster than FP32 computing. Quantization is mainly a technology that accelerates reasoning, and quantitative operations only support forward calculations.

horizon_plugin_pytorch provides quantization operations that adapt to BPUs and supports quantization awareness training, which uses a pseudo-quantization module to model quantization errors in forward calculations and backpropagation. Note that the entire computational process of quantitative training is performed using floating point operations. At the end of quantization-aware training, horizon_plugin_pytorch provides a transformation function that converts the trained model into a fixed-point model, using more compact model representations and high-performance vectorization operations on the BPU.

This chapter introduces you in detail the quantitative training tool of horizon_plugin_pytorch developed by D-Robotics based on PyTorch.