4.1. Post-training Quantization (PTQ)

D-Robotics provides you with the PTQ scheme. The following sections will guide you on how to use Post Training Quantization (PTQ) .

PTQ Principle and Steps: This section guides you through the use of PTQ from PTQ principles, model preparation, model verification, model quantization & compilation, performance analysis and optimization, and accuracy analysis and optimization.

PTQ Tools Guide: This section provides you with a detailed introduction to the PTQ toolkit provided by algorithm toolchain.

PTQ Model Conversion Samples Guide: This section introduces conversion sample package of the horizon_model_convert_sample model and its usage instructions. Provide a quick sample of converting a floating-point model to a fixed-point model using the floating-point model conversion toolchain, including a single inference and accuracy verification sample.

Appendix: This section introduces the descriptions and analysis of norm_type related parameters and related calculation formulas, as well as the concept of each transformer used in image scaling and cropping, parameter descriptions and examples, and general suggestions for solving common abnormalities and failures.

• 4.1.1. PTQ Principle and Steps
  • 4.1.1.1. PTQ Conversion Process and Steps
  • 4.1.1.2. Operator Constraints
  • 4.1.1.3. Floating-point Model Preparation
  • 4.1.1.4. Check the Model
    • 4.1.1.4.1. Use the hb_mapper checker Command to Check Your Model
    • 4.1.1.4.2. Exception Handling
    • 4.1.1.4.3. Interpret Model Check Results
    • 4.1.1.4.4. A Guide to Optimize the Check Results
  • 4.1.1.5. Prepare Calibration Data
  • 4.1.1.6. Model Quantization and Compilation
    • 4.1.1.6.1. Convert the Model Using the hb_mapper makertbin Tool
      • 4.1.1.6.1.1. Configuration File Template
      • 4.1.1.6.1.2. Specific Parameter Information
      • 4.1.1.6.1.3. param_value Configuration
    • 4.1.1.6.2. Model Conversion Interpretation
    • 4.1.1.6.3. Interpret Conversion Results
    • 4.1.1.6.4. Interpret Conversion Output
  • 4.1.1.7. Model Performance Analysis and Optimization
    • 4.1.1.7.1. Use the hb_perf Tool to Evaluate Model Performance
    • 4.1.1.7.2. Evaluate Actual Model Performance on Dev Board
    • 4.1.1.7.3. Model Performance Optimization
  • 4.1.1.8. Model Accuracy Analysis and Optimization
    • 4.1.1.8.1. Model Accuracy Analysis
    • 4.1.1.8.2. Accuracy Optimization
    • 4.1.1.8.3. Further Improve Model Accuracy Using the QAT Solution
      • 4.1.1.8.3.1. About Quantization
      • 4.1.1.8.3.2. About Model Conversion
      • 4.1.1.8.3.3. About Model Quantization & Compilation Workflow
      • 4.1.1.8.3.4. QAT Model Quantization & Compilation Introduction
  • 4.1.1.9. Other Dev Tools (Optional)
    • 4.1.1.9.1. Pack Multiple Bin Models into One File
    • 4.1.1.9.2. View Model Information
    • 4.1.1.9.3. Modify the Nodes of Bin Models
  • 4.1.1.10. Post-training Quantization (PTQ) FAQ
    • 4.1.1.10.1. How to understand the two forms of BPU acceleration and CPU computation mentioned in the operator constraint?
    • 4.1.1.10.2. How does model segmentation affect the performance?
    • 4.1.1.10.3. Why some OPs supported by BPU at the tail part of the model running on CPU?
    • 4.1.1.10.4. Does it support asymmetric quantization?
    • 4.1.1.10.5. How to understand D-Robotics’ default calibration method?
    • 4.1.1.10.6. How to understand D-Robotics’ mix calibration method?
    • 4.1.1.10.7. How to understand the compiler optimization level parameters in yaml files?
    • 4.1.1.10.8. Why does the input size derived from the nv12 model hb_perf not match the prediction library?
    • 4.1.1.10.9. Is the data layout of the inputs to the quantized model and the on-board bin model necessarily the same?
    • 4.1.1.10.10. How to compile to get a multi-batch model?
    • 4.1.1.10.11. Is it normal for the order of model inputs to change during the conversion of a multi-input model?
  • 4.1.1.11. Custom OP Development
    • 4.1.1.11.1. General Descriptions
    • 4.1.1.11.2. Custom OP Included Model Conversion
      • 4.1.1.11.2.1. Modify Model File
        • 4.1.1.11.2.1.1. Caffe Model
        • 4.1.1.11.2.1.2. ONNX Model
      • 4.1.1.11.2.2. OP Implementation
    • 4.1.1.11.3. Run Custom OP on Dev Board
      • 4.1.1.11.3.1. A C++ Template of Custom OP
      • 4.1.1.11.3.2. Custom OP Registration
• 4.1.2. PTQ Tools Guide
  • 4.1.2.1. Model Conversion Process
    • 4.1.2.1.1. Model Checking (hb_mapper checker)
    • 4.1.2.1.2. Calibration Image Preparation
    • 4.1.2.1.3. Model Conversion (hb_mapper makertbin)
    • 4.1.2.1.4. Single Image Inference
    • 4.1.2.1.5. Model Accuracy Evaluations
    • 4.1.2.1.6. Model Performance Evaluations
    • 4.1.2.1.7. [Reference] Supported Calibration Methods
    • 4.1.2.1.8. [Reference] OP List
  • 4.1.2.2. The hb_mapper Tools
    • 4.1.2.2.1. Model Checking Tool (hb_mapper checker)
    • 4.1.2.2.2. Model Compiling Tool (hb_mapper makertbin)
    • 4.1.2.2.3. Inference Tool (hb_mapper infer)
    • 4.1.2.2.4. Key Configuration Parameter (calibration_parameters.preprocess_on)
    • 4.1.2.2.5. Configuration File Reference Example
      • 4.1.2.2.5.1. Reference and Descriptions
      • 4.1.2.2.5.2. About input_type_rt/ input_type_train
  • 4.1.2.3. The hb_perf Tool
    • 4.1.2.3.1. How to Use
    • 4.1.2.3.2. Parameters
    • 4.1.2.3.3. Output Contents
  • 4.1.2.4. The vec_diff Tool
    • 4.1.2.4.1. How to Use
    • 4.1.2.4.2. Parameters
    • 4.1.2.4.3. Output Contents
  • 4.1.2.5. The hb_model_info Tool
    • 4.1.2.5.1. How to Use
    • 4.1.2.5.2. Parameters
    • 4.1.2.5.3. Output Contents
  • 4.1.2.6. The hb_pack Tool
    • 4.1.2.6.1. How to Use
    • 4.1.2.6.2. Parameters
    • 4.1.2.6.3. Output Contents
  • 4.1.2.7. The hb_verifier Tool
    • 4.1.2.7.1. How to Use
    • 4.1.2.7.2. Parameters
    • 4.1.2.7.3. Examples of Reference Usage
    • 4.1.2.7.4. Description of the Output Content
  • 4.1.2.8. The hb_model_modifier Tool
    • 4.1.2.8.1. How to Use
    • 4.1.2.8.2. Parameters
    • 4.1.2.8.3. Output Contents
  • 4.1.2.9. The hb_custom_op Tool
    • 4.1.2.9.1. How to Use
    • 4.1.2.9.2. Parameters
    • 4.1.2.9.3. Output Contents
  • 4.1.2.10. The hb_eval_preprocess Tool
    • 4.1.2.10.1. How to Use
    • 4.1.2.10.2. Parameters
    • 4.1.2.10.3. Output Contents
  • 4.1.2.11. The HB_ONNXRuntime Inference Library
    • 4.1.2.11.1. How to Use
    • 4.1.2.11.2. Parameters
  • 4.1.2.12. Accuracy Debug Tool
    • 4.1.2.12.1. Quickstart
      • 4.1.2.12.1.1. Calibration Models and Data Storage
      • 4.1.2.12.1.2. Accuracy Debug Module Import and Usage
      • 4.1.2.12.1.3. Analysis Result Display
    • 4.1.2.12.2. Function Description
      • 4.1.2.12.2.1. get_sensitivity_of_nodes
      • 4.1.2.12.2.2. plot_acc_error
      • 4.1.2.12.2.3. plot_distribution
      • 4.1.2.12.2.4. get_channelwise_data_distribution
      • 4.1.2.12.2.5. sensitivity_analysis
      • 4.1.2.12.2.6. runall
• 4.1.3. PTQ Model Conversion Samples Guide
  • 4.1.3.1. Release Introduction
    • 4.1.3.1.1. Model Quantification Example Package: horizon_model_convert_sample
    • 4.1.3.1.2. Model Release: model_zoo
  • 4.1.3.2. Common Algorithm Model Samples
    • 4.1.3.2.1. Where to Find
    • 4.1.3.2.2. Prepare Dataset
      • 4.1.3.2.2.1. Dataset Download Address
      • 4.1.3.2.2.2. Dataset Reference Structure
        • 4.1.3.2.2.2.1. ImageNet
        • 4.1.3.2.2.2.2. COCO
        • 4.1.3.2.2.2.3. VOC
        • 4.1.3.2.2.2.4. Cityscapes
        • 4.1.3.2.2.2.5. CIFAR-10
    • 4.1.3.2.3. Prepare Models
      • 4.1.3.2.3.1. Fcos_efficientnetb0
      • 4.1.3.2.3.2. Fcos_efficientnetb1
      • 4.1.3.2.3.3. Fcos_efficientnetb2
      • 4.1.3.2.3.4. MobileNetv1/v2
      • 4.1.3.2.3.5. GoogleNet
      • 4.1.3.2.3.6. ResNet18
      • 4.1.3.2.3.7. Mobilenet_onnx
      • 4.1.3.2.3.8. EfficientNet_Lite0/1/2/3/4
      • 4.1.3.2.3.9. YOLOv2_Darknet19
      • 4.1.3.2.3.10. YOLOv3_Darknet53
      • 4.1.3.2.3.11. YOLOv5s
      • 4.1.3.2.3.12. SSD_MobileNetv1
      • 4.1.3.2.3.13. EfficientDetd0
      • 4.1.3.2.3.14. CenterNet_Resnet50
      • 4.1.3.2.3.15. Fcos_efficientnetb0
      • 4.1.3.2.3.16. UNet_mobilenet
      • 4.1.3.2.3.17. DeeplabV3plus_efficientnetb0
      • 4.1.3.2.3.18. Fastscnn_efficientnetb0
    • 4.1.3.2.4. A Demonstration of the Algorithm Model Samples
      • 4.1.3.2.4.1. Docker Container Prep
      • 4.1.3.2.4.2. Obtain the Original Model and Calibration Dataset
      • 4.1.3.2.4.3. Check If the Model is Executable
      • 4.1.3.2.4.4. Prepare Calibration Dataset
      • 4.1.3.2.4.5. Building Heterogeneous Models
      • 4.1.3.2.4.6. Single Image Inference
      • 4.1.3.2.4.7. Accuracy Evaluation
    • 4.1.3.2.5. FAQ
      • 4.1.3.2.5.1. How to match the inference results of the ONNX original floating-point model and the hb_mapper makertbin tool generated *_original_float_model.onnx model?
      • 4.1.3.2.5.2. Why is the reproduced accuracy slightly different from the indicators in the documentation?
      • 4.1.3.2.5.3. Why is the fixed-point model accuracy not aligned with the on-board accuracy of the bin file in the ai_benchmark example?
  • 4.1.3.3. Other Algorithm Model Samples
    • 4.1.3.3.1. How to Use
    • 4.1.3.3.2. Miscellaneous Samples
      • 4.1.3.3.2.1. lenet_gray
        • 4.1.3.3.2.1.1. Directory
        • 4.1.3.3.2.1.2. Description
      • 4.1.3.3.2.2. resnet50_feature
        • 4.1.3.3.2.2.1. Directory
        • 4.1.3.3.2.2.2. Description
      • 4.1.3.3.2.3. vector-diff
        • 4.1.3.3.2.3.1. Directory
        • 4.1.3.3.2.3.2. Output List
      • 4.1.3.3.2.4. multi_input_example
        • 4.1.3.3.2.4.1. Directory
        • 4.1.3.3.2.4.2. Description
      • 4.1.3.3.2.5. model_verifier
        • 4.1.3.3.2.5.1. Directory
        • 4.1.3.3.2.5.2. Description
      • 4.1.3.3.2.6. model_info
        • 4.1.3.3.2.6.1. Directory
        • 4.1.3.3.2.6.2. Description
      • 4.1.3.3.2.7. mobilenet_bgr
        • 4.1.3.3.2.7.1. Directory
        • 4.1.3.3.2.7.2. Description
      • 4.1.3.3.2.8. mobilenet_yuv444
        • 4.1.3.3.2.8.1. Directory
        • 4.1.3.3.2.8.2. Description
    • 4.1.3.3.3. User Custom OP Sample
      • 4.1.3.3.3.1. Directory
      • 4.1.3.3.3.2. Description
• 4.1.4. Appendix
  • 4.1.4.1. norm_type Configuration Description
    • 4.1.4.1.1. Explanation of Parameters
    • 4.1.4.1.2. Calculation Formula and Example Description
      • 4.1.4.1.2.1. Calculation Formula for Normalization of Data During Model Training
      • 4.1.4.1.2.2. Calculation Formula for Model Inference
  • 4.1.4.2. Image Processing Transformer Description
    • 4.1.4.2.1. AddTransformer
    • 4.1.4.2.2. MeanTransformer
    • 4.1.4.2.3. ScaleTransformer
    • 4.1.4.2.4. NormalizeTransformer
    • 4.1.4.2.5. TransposeTransformer
    • 4.1.4.2.6. HWC2CHWTransformer
    • 4.1.4.2.7. CHW2HWCTransformer
    • 4.1.4.2.8. CenterCropTransformer
    • 4.1.4.2.9. PILCenterCropTransformer
    • 4.1.4.2.10. LongSideCropTransformer
    • 4.1.4.2.11. PadResizeTransformer
    • 4.1.4.2.12. ResizeTransformer
    • 4.1.4.2.13. PILResizeTransformer
    • 4.1.4.2.14. ShortLongResizeTransformer
    • 4.1.4.2.15. PadTransformer
    • 4.1.4.2.16. ShortSideResizeTransformer
    • 4.1.4.2.17. PaddedCenterCropTransformer
    • 4.1.4.2.18. BGR2RGBTransformer
    • 4.1.4.2.19. RGB2BGRTransformer
    • 4.1.4.2.20. RGB2GRAYTransformer
    • 4.1.4.2.21. BGR2GRAYTransformer
    • 4.1.4.2.22. RGB2GRAY_128Transformer
    • 4.1.4.2.23. RGB2YUV444Transformer
    • 4.1.4.2.24. BGR2YUV444Transformer
    • 4.1.4.2.25. BGR2YUV444_128Transformer
    • 4.1.4.2.26. RGB2YUV444_128Transformer
    • 4.1.4.2.27. BGR2YUVBT601VIDEOTransformer
    • 4.1.4.2.28. RGB2YUVBT601VIDEOTransformer
    • 4.1.4.2.29. YUVTransformer
    • 4.1.4.2.30. ReduceChannelTransformer
    • 4.1.4.2.31. BGR2NV12Transformer
    • 4.1.4.2.32. RGB2NV12Transformer
    • 4.1.4.2.33. NV12ToYUV444Transformer
    • 4.1.4.2.34. WarpAffineTransformer
    • 4.1.4.2.35. F32ToS8Transformer
    • 4.1.4.2.36. F32ToU8Transformer
  • 4.1.4.3. Common Failure Resolutions
    • 4.1.4.3.1. Common Failure - hb_mapper checker
    • 4.1.4.3.2. Common Failure - hb_mapper makertbin
    • 4.1.4.3.3. Common Failure - hb_model_modifier
    • 4.1.4.3.4. Common Failure - hb_model_verifier
    • 4.1.4.3.5. Common Failure - hb_onnxruntime
    • 4.1.4.3.6. Common Failure - libDNN