6.1. Efficient Model Design Guide¶
6.1.1. Overview¶
The D-Robotics X3 processor is a second-generation low-power, high-performance intelligent processor for smart robots and other edge AI scenarios, released in 2020. with its BPU® (Bernoulli 2.0 architecture engine) designed in 2017, and therefore also targeted and substantially optimized for new forms of convolution such as Group Conv and Depthwise Conv, which were emerging at that time. The following sections provides some general recommendations for designing efficient models on X3 processors.
6.1.2. General Recommendations¶
6.1.2.1. Recommendation 1: Using D-Robotics’ Efficient Implementation Backbone¶
On the X3 processor, the BPU of Bernoulli architecture is optimized for GroupConv, especially DepthwiseConv, therefore, we recommend the MobileNetv2, EfficientNet_lite with Depthwise+Pointwise structure, and D-Robotics’ VarGNet based on GroupConv’s hand-designed self-developed model as Backbone of the model, which is the most significant improvement in the overall performance of the model.
Note that D-Robotics’ EfficientNet_lite example model is derived from the EfficientNet_lite . Compared to the official EfficientNet implementation, the SE structure containing the Sigmoid lookup operator has been removed and the Swish activation has been replaced with Relu (see Recommendation 3: Avoid Using The Lookup Operator for the impact of the lookup operator).
6.1.2.2. Recommendation 2: Select BPU Operator to Build the Model¶
The execution efficiency of BPU hardware is much higher than that of CPU, and the scheduling switch between CPU and BPU operators in the model also introduces quantization and anti-quantization nodes (the running time on CPU is proportional to the shape size, when it is located at the end of the model, it is recommended to remove it by hb_model_modifier tool and merge it into the pre and post processing code, for details, please refer to Modifier Tool Output Contents ), so you should try to choose the full BPU operator to obtain efficient performance when designing the model. The operators supported by the X3 processor can be found in Model Conversion Toolchain Operator Support Constraint List section.
6.1.2.3. Recommendation 3: Avoid Using the Lookup Operator¶
For some operators that cannot be supported natively by BPU (including Cos, Sin, Sigmoid, Softplus, Sqrt, Tanh, and other activation functions), D-Robotics has developed look up table operators that corresponds to the HzLUT (D-Robotics Look up Table) operator type in the model conversion log. The look-up table operator itself is not as computationally efficient as the operators natively supported by BPU such as ReLU and ReLU6, but the small number of look-up table operators does not affect the overall performance of the model too much. However, it should be noted that when the size of the align_shape (NxCxHxW) of the execution of the table lookup operator exceeds 8192, it will fall back to CPU computation. Therefore, it is more recommended to replace it with ReLU/ReLU6.
6.1.2.4. Recommendation 4: Avoid Using BN, Add, Relu Operators Alone¶
The floating-point conversion toolchain performs some operator fusion, operator splitting, redundant computational merging, constant folding and other graph optimization operations in the Optimizer phase of model conversion. Usually, structures such as Conv+BN, Conv+BN+ReLU, Conv+Add(Elementwise)+ReLU, etc. are available for mathematically equivalent fusion. In contrast, BN, Add, and ReLU used separately from Conv insert the Unit_Conv node before the operator (internal quantization logic, without manual insertion by the user), and thus generating additional computational effort.
6.1.2.5. Recommendation 5: Follow BPU Data Alignment Rules to Fully Utilize MAC Computing Resources¶
The recommendation is based on the limitations of the BPU hardware itself, i.e., if the model design does not follow the BPU data alignment rules, the BPU will automatically padding during inference, resulting in a waste of MAC computational resources. The data alignment rules of the Bernoulli architecture BPU include:
Dimension |
X3 Alignment Rules |
|---|---|
Width |
8 alignment |
Channel |
8 alignment |
Group Conv |
Channel and Kernel num 8 alignment within each group |