中文 EN

6. Quantitative toolchain development guide

• 6.1. Introduction
  • 6.1.1. Usage Notes
    • 6.1.1.1. Floating-Point Model (FP32) Notes
    • 6.1.1.2. Model Operator List Description
• 6.2. Getting Started Guide
  • 6.2.1. Overview
  • 6.2.2. Environment Setup
    • 6.2.2.1. Hardware Requirements
    • 6.2.2.2. Development Machine Deployment
  • 6.2.3. Quick Start
    • 6.2.3.1. Development Environment Preparation
    • 6.2.3.2. Model Preparation
    • 6.2.3.3. Model Verification
    • 6.2.3.4. Model Conversion
    • 6.2.3.5. On-Board Model Execution
• 6.3. Advanced Guide
  • 6.3.1. Environment Installation
    • 6.3.1.1. Instructions for Using Delivered Packages
      • Source Code Directory Structure of the Sample Package
    • 6.3.1.2. Development Machine Deployment
      • Development Machine Preparation
      • Using Docker Environment
    • 6.3.1.3. Board Deployment
    • 6.3.1.4. Version Management Tool Usage Instructions
      • ddk_vcs list
      • ddk_vcs install
      • ddk_vcs uninstall
      • ddk_vcs patch
      • ddk_vcs show
  • 6.3.2. PTQ Principle and Step-by-Step Guide
    • 6.3.2.1. Introduction
    • 6.3.2.2. Model Preparation
    • 6.3.2.3. Model Verification
      • Using the hb_mapper checker Tool to Verify Models
      • Handling Verification Exceptions
      • Interpreting Verification Results
      • Optimization Guidance from Verification Results
    • 6.3.2.4. Model Conversion
      • Preparing Calibration Data
      • Using the hb_mapper makertbin Tool for Model Conversion
      • Model Conversion YAML Configuration Parameters
        • Model parameter group
        • Enter information parameter group
        • Calibration parameter group
        • Compile parameter group
        • Custom operator parameter group
        • X5 int16 configuration instructions
        • Instructions for preprocessing HzPreprocess operator
        • [Reference] Supported calibration methods
      • Interpretation internal process
      • Interpretation of conversion results
      • Interpretation of Conversion Outputs
    • 6.3.2.5. Model Performance Analysis
      • Performance Evaluation on Development Machine
      • Performance Measurement on Development Board
      • Model Performance Optimization
        • Check YAML Parameters Affecting Model Performance
        • Handling CPU Operators
        • High-Performance Model Design Recommendations
    • 6.3.2.6. Model Accuracy Analysis
      • Accuracy Analysis
      • Accuracy Optimization
        • Significant Accuracy Loss (Over 4%)
        • Minor Accuracy Loss Optimization (1.5%~3%)
        • Accuracy Debug Tool
          • Usage Instructions
          • Function Description
          • get_sensitivity_of_nodes
          • plot_acc_error
          • plot_distribution
          • get_channelwise_data_distribution
          • sensitivity_analysis
    • 6.3.2.7. Improving Model Accuracy Using QAT (Quantization-Aware Training)
    • 6.3.2.8. Other Tool Usage Instructions
      • hb_mapper infer Tool
      • hb_perf Tool
      • hb_pack Tool
      • hb_model_info Tool
      • hb_model_modifier Tool
      • hb_verifier Tool
      • hb_eval_preprocess Tool
      • HB_ONNXRuntime Inference Library
  • 6.3.3. Model Operator Support List
    • 6.3.3.1. Usage Limitations
    • 6.3.3.2. X5 Supported Caffe Operators
    • 6.3.3.3. X5 Supported ONNX Operators
  • 6.3.4. Model board running application development instructions
    • 6.3.4.1. Model Inference DNN API usage example description
      • Overview
        • Introduction to the sample code package structure
      • Environment Construction
        • Development Board Preparation
        • Compilation
      • Example usage
        • basic_samples example
        • quick_start
        • api_tutorial
        • advanced_samples
        • misc
      • Auxiliary Tools and Common Operations
        • Logs
        • Example Logs
        • Model Inference DNN API Logs
    • 6.3.4.2. Public Model Performance and Accuracy Evaluation Guide
      • Introduction
      • Release Package Description
        • Example Code Package Structure
        • Example Models
        • Public Datasets
      • Environment Setup
        • Development Board Preparation
        • Compilation Environment Preparation
      • Evaluation Example Usage Guide
        • JSON Configuration File Parameter Information
          • Input Configuration (input_config)
          • Output Configuration (output_config)
        • Performance Evaluation
          • Evaluation Script Usage
            • Command-line Parameter Explanation
            • Result Visualization
        • Accuracy Evaluation
          • Data Preprocessing
          • Generating lst Files
          • Data Mounting
          • Model Inference
          • Accuracy Evaluation
      • Model Integration
        • Preprocessing
          • Preprocessing File and Header File Addition
          • Model Preprocessing Configuration File Addition
          • Single-Frame Latency Evaluation for Preprocessing
        • Postprocessing
          • Postprocessing File and Header File Addition
          • Add Model Execution Scripts and Configuration Files
      • Auxiliary Tools and Common Operations
        • Logging System Usage Instructions
          • Example Logs
          • Model Inference API DNN Logs
        • Operator Latency Explanation
        • Dump Tool
    • 6.3.4.3. On-Board Model Analysis Tool Description
      • Overview
      • hrt_model_exec Tool Usage Instructions
        • Input Parameter Description
        • Usage Instructions
        • model_info
        • Additional Input Parameter Explanation
        • infer
        • Multi-Input Model Explanation
        • Additional Input Parameter Explanation
        • perf
        • Multi-threaded Latency Data Explanation
        • Input Parameter Additional Notes
        • Common Questions
          • How are Latency and FPS data calculated?
          • Why does FPS derived from Latency differ from the tool-measured FPS?
          • How does the tool evaluate models with custom operators?
      • hrt_bin_dump Tool Usage Guide
        • Input Parameter Description
        • Usage Instructions
        • Example Explanation
  • 6.3.5. Model Inference Interface Description
    • 6.3.5.1. Model Inference Library Version Information Retrieval API
      • hbDNNGetVersion()
    • 6.3.5.2. Model Loading/Release API
      • hbDNNInitializeFromFiles()
      • hbDNNInitializeFromDDR()
      • hbDNNRelease()
    • 6.3.5.3. Model Information Retrieval API
      • hbDNNGetModelNameList()
      • hbDNNGetModelHandle()
      • hbDNNGetInputCount()
      • hbDNNGetInputName()
      • hbDNNGetInputTensorProperties()
      • hbDNNGetOutputCount()
      • hbDNNGetOutputName()
      • hbDNNGetOutputTensorProperties()
    • 6.3.5.4. Model Inference API
      • hbDNNInfer()
      • hbDNNRoiInfer()
      • hbDNNWaitTaskDone()
      • hbDNNReleaseTask()
    • 6.3.5.5. Model Memory Operation API
      • hbSysAllocMem()
      • hbSysAllocCachedMem()
      • hbSysFlushMem()
      • hbSysFreeMem()
      • hbSysWriteMem()
      • hbSysReadMem()
      • hbSysRegisterMem()
      • hbSysUnregisterMem()
    • 6.3.5.6. Data Types and Data Structures
      • Version Information
      • Model-related Types
      • System-related Types
      • Preprocessing Classes
    • 6.3.5.7. Data Layout and Alignment Rules
      • Data Layout
      • BPU Alignment Restrictions
        • Model Input Requirements
        • Alignment and Valid Data
      • Introduction to NV12
        • YUV Format
        • NV12 Layout
    • 6.3.5.8. Model Inference DEBUG Methods
      • Error Codes
      • Configuration Information
      • Notes on Using Simulator on Development Machine
• 6.4. Delve deeper
  • 6.4.1. Environment Dependencies
  • 6.4.2. Quick Start
    • 6.4.2.1. Basic Workflow
    • 6.4.2.2. Obtain the Floating-Point Model
    • 6.4.2.3. Calibration
    • 6.4.2.4. Quantization Aware Training
    • 6.4.2.5. Convert to Fixed-Point Model
    • 6.4.2.6. Model Deployment
  • 6.4.3. Development Guide
    • 6.4.3.1. Requirements for Floating-Point Models
      • symbolic_trace
      • Partial Operator Support
      • Building Quantization-Friendly Models
    • 6.4.3.2. Detailed Explanation of qconfig
      • What is qconfig?
      • How to Obtain qconfig
      • How to Set qconfig
      • qconfig Templates
    • 6.4.3.3. Calibration Guide
      • Procedure and Example
      • Introduction to Common Algorithms
      • Parameter Tuning Tips
      • Observer Parameter Documentation
    • 6.4.3.4. Quantization-Aware Training Guide
      • Workflow and Example
        • prepare_qat_fx
        • Loading Calibration Model Parameters
        • Training Iteration
      • Pseudo-Quantization Operators
        • Pseudo-Quantization Process
        • Statistical Methods
        • Learning-Based Methods
    • 6.4.3.5. Heterogeneous Model Guide
      • Introduction to Heterogeneous Models
      • Workflow
      • Operator Limitations
      • Main Interface Parameters
      • Workflow and Example
    • 6.4.3.6. Guide to Precision Tuning Tools
      • Quick Start
      • API Reference
        • Model Structure Checker
        • QuantAnalysis Class
          • auto_find_bad_case
          • set_bad_case
          • load_bad_case
          • save_bad_case
          • set_model_profiler_dir
          • run
          • compare_per_layer
          • sensitivity
          • clean
        • ModelProfiler Class
          • get_info_manager
          • table
          • tensorboard
    • 6.4.3.7. Cross-Device Inference Instructions for Quantized Deployment of PT Models
      • PT model execution device differs from tracing device
      • Multi-GPU Parallel Inference
    • 6.4.3.8. Common Issues
      • Import Errors
      • Unable to properly prepare_calibration/qat
      • Forward fails after prepare_qat
      • Compilation Error
      • Quantization Accuracy Anomalies
      • Error when loading pt file using torch.jit.load
    • 6.4.3.9. Common Usage Misconceptions
      • Configuration Errors
      • Methodological Errors
      • Network Structure Errors
      • Operator-Level Errors
      • Model-Level Errors
  • 6.4.4. In-Depth Exploration
    • 6.4.4.1. FX Quantization Principle Introduction
      • Quantization Workflow
        • Fuse (Optional)
        • Prepare
        • Convert
        • Eager Mode Compatibility
    • 6.4.4.2. RGB888 Data Deployment
      • Scenario
      • Introduction to YUV Format
      • Preprocessing RGB Input During Training
      • Real-Time Conversion of YUV Input During Inference
        • Operator Definition
        • Usage Method
    • 6.4.4.3. Model Segmented Deployment
      • Scenario
      • Method
    • 6.4.4.4. Operator Fusion
      • BN Absorption
      • Fusion of Add and ReLU(6)
      • Implementation Principle
      • Supported Fusion Operators
    • 6.4.4.5. Adaround (Experimental Feature)
      • Basic Principle
      • Interface Definition
      • Usage
        • list/tuple (Recommended)
        • torch.utils.data.DataLoader
    • 6.4.4.6. Automatic Calibration (Experimental Feature)
      • Basic Principle
      • Interface Definition
      • Usage
        • list/tuple (Recommended)
        • torch.utils.data.DataLoader
  • 6.4.5. API Manual
    • 6.4.5.1. March
    • 6.4.5.2. qconfig
      • Example of qconfig Definition
    • 6.4.5.3. Fake Quantization Operators
    • 6.4.5.4. QAT
    • 6.4.5.5. ONNX
    • 6.4.5.6. TorchScript Model Saving and Loading
    • 6.4.5.7. Horizon Operators
    • 6.4.5.8. Model Compilation
  • 6.4.6. Appendix
    • 6.4.6.1. Eager Mode
      • Differences from FX Mode
      • Workflow
        • Construct Floating-Point Model
        • Pre-train Floating-Point Model
        • Set BPU Architecture
        • Operator Fusion
        • Convert Floating-Point Model to Quantized Model
        • Quantization Training
        • Convert Quantized Model to Fixed-Point Model
        • Check and Compile Fixed-Point Inference Model
    • 6.4.6.2. Supported Public Operators
      • General Notes
      • torch function class
      • torch.nn.functional function class
      • torch.nn Module class
      • torch.quantization Module Classes
      • torch.Tensor Methods
      • torchvision Classes