4.1.4.3. Common Failure Resolutions

This section introduces you to some of the abnormal failures you may encounter when using D-Robotics X3 toolchain products. For these failures, we provide you with the possible causes of the corresponding failures and general suggestions for solving them, so that you can quickly locate the problem and solve it.

4.1.4.3.1. Common Failure - hb_mapper checker

Background information: model check command (hb_mapper checker)

In practical engineering, since not all floating-point models can be converted to quantized models, a check is required before conversion, and this check process will complete the process of a model conversion. But for the more time-consuming steps, the processing is simplified. This command outputs the check result and the deployment of OP on the device after completing the check of the model.

Failure scenarios: The following are common failure scenarios when using hb_mapper checker

1.Scenario 1

ERROR The shape of model input:input is [xxx] which has dimensions of 0.
Please specify input-shape parameter.

Possible Cause: The model input is a dynamic shape.

Suggested Solution: You can use the parameter -input-shape "input_name input_shape" to specify the input node’s shape information.

2.Scenario 2

ERROR D-RoboticsRT not support these cpu operators: {op_type}

Possible Cause: The CPU operator used is a CPU operator that is not supported by D-Robotics.

Suggested Solution: You can replace operators according to the operator support list we provide; If the unsupported CPU operator is the core operator of the model, please contact D-Robotics for development evaluation.

3.Scenario 3

Unsupported op {op_type}

Possible Cause: The BPU operator used is a BPU operator not supported by D-Robotics.

Suggested Solution: If the overall performance of the model can meet your needs, you can ignore this log; If the overall performance of the model cannot meet your expectations, you can replace the operator according to the list of operator support we provide.

4.Scenario 4

ERROR nodes:['{op_type}'] are specified as domain:xxx, which are not supported by official onnx.
Please check whether these ops are official onnx ops or defined by yourself

Possible Cause: The custom operator used is a custom operator that is not supported by D-Robotics.

Suggested Solution: You can replace the operator according to the list of operator support provided by us.

4.1.4.3.2. Common Failure - hb_mapper makertbin

Background information: model compilation commands (hb_mapper makertbin)

According to the configuration file and the type of model, this command generates ONNX quantization models as well as runtime models for simulating the on-board situation.

Failure scenarios: The following are common failure scenarios when using hb_mapper makertbin

1.Scenario 1

Layer {op_name}
  xxx expect data shape range:[[xxx][xxx]], but the data shape is [xxx]
Layer {op_name}
  Tensor xxx expects be n dimensions, but m provided

Possible Cause: The {op_name} operator exceeded the support limit and is backed off to CPU computation.

Suggested Solution: If the performance loss caused by the CPU operator is acceptable to you, you do not need to pay attention to this information; if the performance does not meet your requirements, you can modify the op to the range supported by BPU according to the content of the operator support list we provide.

2.Scenario 2

ERROR There is an error in pass: {op_name}. Error message:xxx

Possible Cause: The {op_name} operator optimization failed.

Suggested Solution: Please collect the models and .log files and provide them to D-Robotics technicians for analysis and processing.

3.Scenario 3

Error There is an error in pass:constant_folding.
Error message: Could not find an implementation for the node {op_name}

Possible Cause: The operator is not supported by onnxruntime at this time.

Suggested Solution: You can replace the operator according to the operator support list we provide. If the unsupported operator is a core operator, please contact D-Robotics for development evaluation of this.

4.Scenario 4

Start to parse the onnx model
core dump

Possible Cause: The model parsing failed (probably only a name was specified for one output/input node when exporting the model).

Suggested Solution: It is recommended that you re-export onnx and confirm its validity (export the onnx model without specifying the output/input name, or specify a name for each output/input node in turn).

5.Scenario 5

Start to calibrate/quantize the model
core dump

Start to compile the model
core dump

Possible Cause: The model quantization/compilation failed.

Suggested Solution: Please collect the models and .log files and provide them to D-Robotics technicians for analysis and processing.

6.Scenario 6

ERROR model conversion faild: Inferred shape and existing shape differ in dimension x: (n) vs (m)

Possible Cause: The input shape of the onnx model is illegal, or the tool optimization pass is wrong.

Suggested Solution: Please ensure the validity of the onnx model. If the onnx model can be reasoned normally, please provide the model to D-Robotics technical staff for analysis and processing.

7.Scenario 7

WARNING got unexpected input/output/sumin threshold on conv {op_name}! value: xxx

Possible Cause: The data preprocessing is wrong, or the value of this node weight is too small/too large.

Suggested Solution: Please check your data preprocessing for errors; we recommend that you use the BN operator to optimize the data distribution.

8.Scenario 8

ERROR hbdk-cc compile hbir model failed with returncode -n

Possible Cause: The model compilation failed.

Suggested Solution: Please collect the models and .log files and provide them to D-Robotics technicians for analysis and processing.

9.Scenario 9

ERROR {op_type}  only support 4 dim input

Possible Cause: The toolchain does not support this op input dimension to be non-four-dimensional at this time.

Suggested Solution: We recommend that you adjust this op input dimension to a four-dimensional input.

10.Scenario 10

ERROR {op_type} Not support this attribute/mode=xxx

Possible Cause: The toolchain does not support this property for op at this time.

Suggested Solution: You can replace it according to the operator support list we provide or contact D-Robotics for development evaluation of this.

11.Scenario 11

ERROR There is no node can execute on BPU in this model,
please make sure the model has at least one conv node which is supported by BPU.

Possible Cause: There are no quantifiable BPU nodes in the model.

Suggested Solution: Please ensure the validity of the onnx model, and at least one conv is used in the model; If the above conditions are satisfied, please collect the models and .log files and provide them to D-Robotics technicians for analysis and processing.

12.Scenario 12

ERROR The opset version of the onnx model is n, only model with opset_version 10/11 is supported

Possible Cause: The model opset version exceeds toolchain support limit.

Suggested Solution: Please re-export the model and ensure opset_version=10 or 11.

13.Scenario 13

Conversion error after using run_on_bpu.

Possible Cause: This operator is not supported for run_on_bpu at this time

Suggested Solution: The run_on_bpu temporarily supports only Relu/Softmax/Reshape/pooling (maxpool, avgpool, etc.) operators of the specified model and CPU*+Transpose combinations (you can run both CPU*+Transpose on the BPU by declaring the Transpose node name, CPU* specifically refers to BPU support op). If the aforementioned conditions are met but still run_on_bpu fails, please contact D-Robotics technical staff to analyze and deal with it; if the aforementioned conditions are not met, you can contact D-Robotics technical staff to evaluate the development.

14.Scenario 14:

ERROR : There is an ERROR during shape inference,···,The error model has been saved as shape_inference_fail.onnx

Possible Cause: The model is illegal or the tool parsing failed.

Suggested Solution: Please provide the .log file and the generated shape_inference_fail.onnx to D-Robotics technical staff for cause analysis.

4.1.4.3.3. Common Failure - hb_model_modifier

Background information: The hb_model_modifier tool is used to delete Transpose and Quantize nodes on the input side and Transpose, Dequantize, DequantizeFilter, Cast, Reshape and Softmax nodes on the output side of the specified runtime model. The information of the deleted nodes is stored in the BIN model and can be viewed via hb_model_info.

Failure scenarios: The following are common failure scenarios when using hb_model_modifier

Scenario:

ERROR Can not find value info {op_name}

Possible Cause: This failure is a D-Robotics known failure and has been fixed in OE1.1.14.

Suggested Solution: Please completely update the OE development package or upgrade horizon-tc-ui to version 1.7.8.

4.1.4.3.4. Common Failure - hb_model_verifier

Background information: The hb_model_verifier is used to verify the results of the specified fixed-point model and runtime model. The tool use the specified image to perform fixed-point model inference, inference on the runtime model board and the x86-side simulator, and compare the results of the three parties to give a conclusion whether it passes or not.

Failure scenarios: The following are common failure scenarios when using hb_model_verifier

1.Scenario 1

ERROR Arm result does not exist, program halted

Possible Cause: After this failure occurs, you can check the terminal execution log for relevant prompts, usually due to board connection failure or board-side reasoning failure.

Suggested Solution: If you see a connection failure message, please check again if the board can be pinged in the current environment. If the relevant prompt is a board-side inference failure, you can check the specific reason for the inference failure by executing infer.sh under the /userdata/model_verifier_test path on the board side.

2.Scenario 2

ERROR Quanti onnx and Arm result Strict check FAILED

Possible Cause: The model consistency comparison failed.

Suggested Solution: Please provide your model to D-Robotics’ technical staff for analysis and processing.

4.1.4.3.5. Common Failure - hb_onnxruntime

Background information: hb_onnxruntime is mainly the class for onnx model inference.

Failure scenarios: The following are common failure scenarios when using hb_onnxruntime

1.Scenario 1

ERROR [ONNXRuntimeError] : 2 INVALID_ARGUMENT : Unexpected input data type.
Actual: (N11onnxruntime17PrimitiveDataTypexxx), expected: (N11onnxruntime17PrimitiveDataTypexxx)

Possible Cause: The input data format does not match the model.

Suggested Solution: In general, the input format of the floating-point onnx model is float32, and the input format of the quantized model is int8. The properties of the input nodes of the onnx model can be viewed using the visualization tool.

2.Scenario 2

[libprotobuf FATAL google/protobuf/stubs/common.cc:83] This program was compiled against version 3.6.1 of the Protocol Buffer runtime library,
which is not compatible with the installed version on (3.19.4).

Possible Cause: The version of protobuf used by torch conflicts with the version of protobuf used by horizon, and needs to be imported before torch.

Suggested Solution: You can import from horizon_tc_ui import HB_ONNXRuntime on the first line. importing other APIs with the same error also applies.

4.1.4.3.6. Common Failure - libDNN

Background information: libDNN is mainly the inference library for horizon models.

Failure scenarios: The following are common failure scenarios when using libDNN

1.Scenario 1

(common.h:79): HR:ERROR: op_name:xxx invalid attr key xxx

Possible Cause: The libDNN does not support a certain property of this op for the time being ( subsequently we will gradually move the operator constraint forward to the model transformation stage to remind).

Suggested Solution: You can replace it according to the operator support list we provide or contact D-Robotics for a development evaluation of this.

2.Scenario 2

(hb_dnn_ndarray.cpp:xxx): data type of ndarray do not match specified type. NDArray dtype_: n, given m

Possible Cause: The libDNN does not support this input type for now ( subsequently we will gradually move the operator constraints forward to the model transformation stage to remind).

Suggested Solution: You can replace it according to the operator support list we provide or contact D-Robotics for a development evaluation of this.

3.Scenario 3

(validate_util.cpp:xxx) tensor aligned shape size is xxx , but tensor hbSysMem memSize is xxx,
tensor hbSysMem memSize should >= tensor aligned shape size!

Possible Cause: The insufficient memory for input data request.

Suggested Solution: Since when using hrt_model_exec model_info to view the aligned shape of the model input node, memory space is requested by aligned shape*size_of(tensor type). Here we recommend: if your libDNN version is higher than 1.5.4b, we recommend to use hbDNNTensorProperties.alignedByteSize to request memory space directly, if your libDNN version is lower than 1.5.4b then use aligned*size to request memory space directly.

4.Scenario 4

(bpu_model_info.cpp:xxx): HR:ERROR: hbm model input feature names must be equal to graph node input names

Possible Cause: This failure is a D-Robotics known failure, it is a known problem of the hb_model_modifer. This failure has been fixed in OE1.1.14.

Suggested Solution: Please completely update the OE development package or upgrade horizon-tc-ui to version 1.7.8.