6.6.2. Common Failure Resolutions¶
This section introduces you to some of the abnormal failures you may encounter when using D-Robotics X5 toolchain products.
For these failures, we provide you with the possible causes of the corresponding failures and general suggestions for solving them, so that you can quickly locate the problems and solve them.
6.6.2.1. Common Failure - hb_mapper checker¶
Background: model checking command (hb_mapper checker)
In In engineering practice, since not all floating-point models can be converted to quantized models, checking is required before conversion. This checking process will go through the process of a model conversion.
However, for those more time-consuming procedures, we simplified the checking process. This command will output the check results and the OP deployment on the device when it finishes the checking of the model.
Failure Scenarios: Common failure scenarios when using hb_mapper checker are as follows:
1.Scenario 1
ERROR The shape of model input:input is [xxx] which has dimensions of 0.
Please specify input-shape parameter.
Possible Cause: Model input is a dynamic shape.
Suggested Solution: You can use the parameter -input-shape "input_name input_shape" to specify the input node’s shape information.
2.Scenario 2
ERROR HorizonRT not support these cpu operators: {op_type}
Possible Cause: CPU operator used is a CPU operator that is not supported by Horizon.
Suggested Solution: You can replace operators according to the operator support list we provide. If the unsupported CPU operator is the core operator of the model, contact D-Robotics for development evaluation.
3.Scenario 3
Unsupported op {op_type}
Possible Cause: BPU operator used is a BPU operator not supported by Horizon.
Suggested Solution: If the overall performance of the model can meet your needs, you can ignore this log; If otherwise, you can replace the operator according to the operator support list we provide.
4.Scenario 4:
ERROR nodes:['{op_type}'] are specified as domain:xxx, which are not supported by official onnx.
Please check whether these ops are official onnx ops or defined by yourself
Possible Cause: Custom operator used is a custom operator that is not supported by Horizon.
Suggested Solution: You can replace the operator according to the operator support list we provide, or refer to custom operator development materials on the D-Robotics Developer Community.
6.6.2.2. Common Failure - hb_mapper makertbin¶
Background: model compilation commands (hb_mapper makertbin)
According to the configuration file and the model type, this command generates ONNX quantization models as well as the runtime models for simulating the on-board situations.
Failure Scenarios: Common failure scenarios when using hb_mapper makertbin are as follows:
1.Scenario 1:
Layer {op_name}
xxx expect data shape range:[[xxx][xxx]], but the data shape is [xxx]
Layer {op_name}
Tensor xxx expects be n dimensions, but m provided
Possible Cause: {op_name} operator exceeds the support limit and is pushed back to the CPU computation.
Suggested Solution: If the performance loss caused by the CPU operator is acceptable to you, then you can skip this information; if otherwise, you can modify the op to the range supported by BPU according to the operator support list we provide.
2.Scenario 2:
ERROR There is an error in pass: {op_name}. Error message:xxx
Possible Cause: {op_name} operator optimization fails.
Suggested Solution: You can collect the models and .log files and provide them to D-Robotics technical staff for analysis and processing.
3.Scenario 3:
Error There is an error in pass:constant_folding.
Error message: Could not find an implementation for the node {op_name}
Possible Cause: Operator is not supported by onnxruntime for now.
Suggested Solution: You can replace the operator according to the operator support list we provide. If the unsupported operator is a core operator, contact D-Robotics for development evaluation.
4.Scenario 4:
Start to parse the onnx model
core dump
Possible Cause: Model parsing fails (probably because only the name for one output/input node is specified when exporting the model).
Suggested Solution: You can re-export ONNX and make sure it is valid (export the ONNX model without specifying the output/input name, or specify a name for each output/input node in turn).
5.Scenario 5:
Start to calibrate/quantize the model
core dump
Start to compile the model
core dump
Possible Cause: Model quantization/compilation fails.
Suggested Solution: Collect the models and .log files and provide them to D-Robotics technical staff for analysis and processing.
6.Scenario 6:
ERROR model conversion failed: Inferred shape and existing shape differ in dimension x: (n) vs (m)
Possible Cause: Input shape of the ONNX model is illegal, or the tool optimization pass is wrong.
Suggested Solution: Make sure the ONNX model is valid. If the ONNX model can be inferred normally, please provide the model to D-Robotics technical staff for analysis and processing.
7.Scenario 7:
WARNING got unexpected input/output/sumin threshold on conv {op_name}! value: xxx
Possible Cause: Data preprocessing is wrong, or the value of this node weight is too small/too large.
Suggested Solution: Check your data preprocessing for errors. We recommend BN operator for the optimization of the data distribution.
8.Scenario 8:
ERROR hbdk-cc compile hbir model failed with returncode -n
Possible Cause: Model compilation fails.
Suggested Solution: Collect the models and .log files and provide them to D-Robotics technical staff for analysis and processing.
9.Scenario 9:
ERROR {op_type} only support 4 dim input
Possible Cause: Toolchain does not support non-4D input of this op for now.
Suggested Solution: We recommend adjusting the input dimension of this op to a 4D input.
10.Scenario 10:
ERROR {op_type} Not support this attribute/mode=xxx
Possible Cause: Toolchain does not support this property for op for now.
Suggested Solution: You can replace it according to the operator support list we provide or contact D-Robotics for development evaluation.
11.Scenario 11:
ERROR There is no node can execute on BPU in this model,
please make sure the model has at least one conv node which is supported by BPU.
Possible Cause: There are no BPU nodes can be quantized in the model.
Suggested Solution: Make sure the ONNX model is valid and at least one conv is used in the model; If these conditions are satisfied, collect the models and .log files and provide them to D-Robotics technical staff for analysis and processing.
12.Scenario 12:
ERROR The opset version of the onnx model is n, only model with opset_version 10/11 is supported
Possible Cause: Model opset version exceeds toolchain support limit.
Suggested Solution: Re-export the model and make sure opset_version is 10 or 11.
13.Scenario 13:
Conversion error after using run_on_bpu.
Possible Cause: Run_on_bpu operation of this operator is not supported for now.
Suggested Solution: Support of run_on_bpu now includes only certain operators of the specified models such as Relu, Softmax, Reshape, pooling (maxpool, avgpool, etc.), as well as CPU*+Transpose combinations (you can run both CPU*+Transpose on the BPU by declaring the Transpose node name, CPU* specifically refers to the operators supported by the BPU).
If these conditions are met but you still encounters run_on_bpu failure, contact D-Robotics technical staff for analysis and processing; if otherwise, you still can contact D-Robotics technical staff for development evaluation.
14.Scenario 14:
ERROR unsupported model: BAYES-E not support execute one model on 2core simultaneously now
Possible Cause: X5 currently does not support compiling dual-core models.
Suggested Solution: Set core_num to 1 in the YAML configuration file.
15.Scenario 15:
ERROR : There is an ERROR during shape inference,···,The error model has been saved as shape_inference_fail.onnx
Possible Cause: The model is illegal or the tool parsing failed.
Suggested Solution: Please provide the .log file and the generated shape_inference_fail.onnx to D-Robotics technical staff for cause analysis.
6.6.2.3. Common Failure - hb_model_modifier¶
Background: hb_model_modifier tool is used to delete certain input/output nodes (Input: Transpose and Quantize; Output: Transpose, Dequantize, DequantizeFilterCast, Reshape, and Softmax) of the specified runtime model.
The information of the deleted nodes is stored in the BIN model and can be viewed by using hb_model_info.
Failure Scenarios: Common failure scenarios when using hb_model_modifier are as follows:
Scenario:
ERROR Can not find value info {op_name}
Possible Cause: It is a known failure and has been fixed in OE1.1.14.
Suggested Solution: Fully update the OE development package or upgrade horizon-tc-ui to version 1.7.8.
6.6.2.4. Common Failure - hb_verifier¶
Background: hb_verifier is used to validate the results of a specified fixed-point model and *.bin runtime model.
If you specify the image before using the tool, the hb_verifier will use the specified image, perform fixed-point model inference,
runtime model board-side and runtime model on x86-side emulator, and compare the results of its three parties two by two and give a conclusion
whether it passes or not(this process supports self-selection, so you can choose what to compare if you want).
If no image is specified before using the tool, the hb_verifier tool will use randomly generated tensor data for inference by default.
Failure Scenarios: Common failure scenarios when using hb_verifier are as follows:
Scenario:
ERROR Quanti onnx and Arm result Strict check FAILED
Possible Cause: Model consistency comparison fails.
Suggested Solution: Provide your model to D-Robotics’s technical staff for analysis and processing.
6.6.2.5. Common Failure - hb_onnxruntime¶
Background: hb_onnxruntime is the class for ONNX model inference.
Failure Scenarios: Common failure scenarios when using hb_onnxruntime are as follows:
1.Scenario 1:
ERROR [ONNXRuntimeError] : 2 INVALID_ARGUMENT : Unexpected input data type.
Actual: (N11onnxruntime17PrimitiveDataTypexxx), expected: (N11onnxruntime17PrimitiveDataTypexxx)
Possible Cause: Input data format and the model does not match.
Suggested Solution: In general, the input format of the floating-point ONNX model is float32, while the input format of the quantized model is int8. You can use the visualization tool to view the properties of the input nodes in the ONNX model.
2.Scenario 2:
[libprotobuf FATAL google/protobuf/stubs/common.cc:83] This program was compiled against version 3.6.1 of the Protocol Buffer runtime library,
which is not compatible with the installed version on (3.19.4).
Possible Cause: Version of protobuf used by torch conflicts with the one used by D-Robotics, which should be imported before torch.
Suggested Solution: You can import it by adding from horizon_tc_ui import HB_ONNXRuntime to the first line, which also applies to other APIs with the same error.
6.6.2.6. Common Failure - libDNN¶
Background: libDNN is the inference library for D-Robotics models.
Failure Scenarios: Common failure scenarios when using libDNN are as follows:
1.Scenario 1:
(common.h:79): HR:ERROR: op_name:xxx invalid attr key xxx
Possible Cause: Certain properties of this op may not be supported by LibDNN for now (subsequently we will gradually the reminder of the operator constraint to the model conversion stage).
Suggested Solution: You can replace it according to the operator support list we provide or contact D-Robotics for development evaluation.
2.Scenario 2:
(hb_dnn_ndarray.cpp:xxx): data type of ndarray do not match specified type. NDArray dtype_: n, given m
Possible Cause: LibDNN does not support this input type for now (subsequently we will gradually move the reminder of operator constraints forward to the model conversion stage).
Suggested Solution: You can replace it according to the operator support list we provide or contact D-Robotics for development evaluation.
3.Scenario 3:
(validate_util.cpp:xxx) tensor aligned shape size is xxx , but tensor hbSysMem memSize is xxx,
tensor hbSysMem memSize should >= tensor aligned shape size!
Possible Cause: Insufficient memory for input data request.
Suggested Solution: When using hrt_model_exec model_info to view the aligned shape of the input node of the model,
the memory space is requested by aligned shape*size_of(tensor type) , so we recommend using hbDNNTensorProperties.alignedByteSize directly to
request the memory space if the libDNN version is higher than 1.5.4b;
if the libDNN version is lower than that, you can use aligned shape*size_of(tensor type) to request the memory space.
4.Scenario 4:
(bpu_model_info.cpp:xxx): HR:ERROR: hbm model input feature names must be equal to graph node input names
Possible Cause: It is a known failure of hb_model_modifer, which has been fixed in OE1.1.14.
Suggested Solution: Fully update the OE development package or upgrade horizon-tc-ui to version 1.7.8.