4.2.3. TUTORIAL

This section describes Floating Point Model Preparation, Operator Fusion, Set up Different BPU Architectures, Heterogeneous model QAT, FX based quantization, Building Quantization-friendly Floating-point Models, QAT Experience Summary, Model Precision Debug Tool, and Ideas of Quantization Precision Debug.

• 4.2.3.1. Floating Point Model Preparation
  • 4.2.3.1.1. Operator Substitution
  • 4.2.3.1.2. Insert Quantization and Dequantization Nodes
  • 4.2.3.1.3. Set up Quantization Parameters
  • 4.2.3.1.4. Example of Custom Floating Point Model
• 4.2.3.2. Operator Fusion
  • 4.2.3.2.1. Why Do Operator Fusion
  • 4.2.3.2.2. Speed up Computation
  • 4.2.3.2.3. Improve Quantization Accuracy
  • 4.2.3.2.4. Fusion Operator
  • 4.2.3.2.5. Purpose of BN (Batch Normalization) Absorption
  • 4.2.3.2.6. BN Absorption Method
  • 4.2.3.2.7. Operator Fusion Example
• 4.2.3.3. Set Up Different BPU Architectures
• 4.2.3.4. Heterogeneous Model QAT
  • 4.2.3.4.1. Differences Between Heterogeneous Models and Non-heterogeneous Models
  • 4.2.3.4.2. Description of Main Interface Parameters
    • 4.2.3.4.2.1. horizon_plugin_pytorch.quantization.prepare_qat_fx
    • 4.2.3.4.2.2. horizon_plugin_pytorch.quantization.prepare_calibraiton_fx
    • 4.2.3.4.2.3. horizon_plugin_pytorch.quantization.convert_fx
    • 4.2.3.4.2.4. horizon_plugin_pytorch.utils.onnx_helper.export_to_onnx
  • 4.2.3.4.3. Example
• 4.2.3.5. FX Based Quantization
  • 4.2.3.5.1. Restriction
• 4.2.3.6. Building Quantization-friendly Floating-point Models
  • 4.2.3.6.1. Background
  • 4.2.3.6.2. Phenomena Not Suitable for Quantization
    • 4.2.3.6.2.1. Extensive Use of OPs Not Suitable for Quantization
    • 4.2.3.6.2.2. Asymmetric Input or High Resolution Requirements
      • 4.2.3.6.2.2.1. Solution
    • 4.2.3.6.2.3. Large Feature Map in Model
      • 4.2.3.6.2.3.1. Solution
    • 4.2.3.6.2.4. Large Range of Model Weight
      • 4.2.3.6.2.4.1. Solution
  • 4.2.3.6.3. Tips
    • 4.2.3.6.3.1. Not Suitable for Quantification Does Not Mean That It Cannot Be Quantified
    • 4.2.3.6.3.2. int16 Quantization
• 4.2.3.7. QAT Experience Summary
  • 4.2.3.7.1. Hyperparameter
  • 4.2.3.7.2. Quantization Parameters
• 4.2.3.8. Model Precision Debug Tool
  • 4.2.3.8.1. Similarity
    • 4.2.3.8.1.1. Example
  • 4.2.3.8.2. Visualization
    • 4.2.3.8.2.1. Model Visualization
    • 4.2.3.8.2.2. PT File Visualization
  • 4.2.3.8.3. Statistics
    • 4.2.3.8.3.1. Example
  • 4.2.3.8.4. Step Quantization
    • 4.2.3.8.4.1. Example
  • 4.2.3.8.5. Shared OP Check
    • 4.2.3.8.5.1. Example
  • 4.2.3.8.6. Fuse Check
    • 4.2.3.8.6.1. Example
  • 4.2.3.8.7. Single-operator Conversion Precision Debugging
    • 4.2.3.8.7.1. Example
  • 4.2.3.8.8. Quantitative Configuration Check
    • 4.2.3.8.8.1. Example
  • 4.2.3.8.9. Model Weight Comparison
    • 4.2.3.8.9.1. Example
  • 4.2.3.8.10. Deploy Device Check of Hybrid Models
    • 4.2.3.8.10.1. Example
  • 4.2.3.8.11. Integrated Interface
    • 4.2.3.8.11.1. Example
  • 4.2.3.8.12. GPU Memory Profiler
    • 4.2.3.8.12.1. Example
• 4.2.3.9. Ideas of Quantization Precision Debug
  • 4.2.3.9.1. BaseLine
    • 4.2.3.9.1.1. Quantization Precision Standard
  • 4.2.3.9.2. Flow Chart
  • 4.2.3.9.3. NAN
  • 4.2.3.9.4. Low Calibration Precision
  • 4.2.3.9.5. Low QAT Precision
  • 4.2.3.9.6. Low Quantized Precision