中文 EN

6.3.5. Model Inference Interface Description

The header files and library files used by the model inference API can be obtained from the package/host/host_package/x5_aarch64/dnn folder in 《Delivery Artifacts Description》.

6.3.5.1. Model Inference Library Version Information Retrieval API

hbDNNGetVersion()

【Function Prototype】

const char *hbDNNGetVersion()

【Function Description】

Retrieve version information of the DNN inference library.

【Parameters】

None

【Return Type】

Returns version information.

6.3.5.2. Model Loading/Release API

hbDNNInitializeFromFiles()

【Function Prototype】

int32_t hbDNNInitializeFromFiles(hbPackedDNNHandle_t *packedDNNHandle, const char **modelFileNames, int32_t modelFileCount)

【Function Description】

Create and initialize packedDNNHandle from files. The caller can use the returned packedDNNHandle across functions and threads.

【Parameters】

• [out] packedDNNHandle Horizon DNN handle pointing to multiple models.

• [in] modelFileNames Paths to model files.

• [in] modelFileCount Number of model files.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNInitializeFromDDR()

【Function Prototype】

int32_t hbDNNInitializeFromDDR(hbPackedDNNHandle_t *packedDNNHandle, const void **modelData, int32_t *modelDataLengths, int32_t modelDataCount)

【Function Description】

Create and initialize packedDNNHandle from memory. The caller can use the returned packedDNNHandle across functions and threads.

【Parameters】

• [out] packedDNNHandle Horizon DNN handle pointing to multiple models.

• [in] modelData Pointers to model data in memory.

• [in] modelDataLengths Lengths of model data.

• [in] modelDataCount Number of model data entries.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNRelease()

【Function Prototype】

int32_t hbDNNRelease(hbPackedDNNHandle_t packedDNNHandle)

【Function Description】

Release models pointed to by packedDNNHandle.

【Parameters】

• [in] packedDNNHandle Horizon DNN handle pointing to multiple models.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

6.3.5.3. Model Information Retrieval API

hbDNNGetModelNameList()

【Function Prototype】

int32_t hbDNNGetModelNameList(const char ***modelNameList, int32_t *modelNameCount, hbPackedDNNHandle_t packedDNNHandle)

【Function Description】

Retrieve the list and count of model names pointed to by packedDNNHandle.

【Parameters】

• [out] modelNameList List of model names.

• [out] modelNameCount Number of model names.

• [in] packedDNNHandle Horizon DNN handle pointing to multiple models.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNGetModelHandle()

【Function Prototype】

int32_t hbDNNGetModelHandle(hbDNNHandle_t *dnnHandle, hbPackedDNNHandle_t packedDNNHandle, const char *modelName)

【Function Description】

Obtain a handle to one model from the model list pointed to by packedDNNHandle. The caller can use the returned dnnHandle across functions and threads.

【Parameters】

• [out] dnnHandle DNN handle pointing to one model.

• [in] packedDNNHandle DNN handle pointing to multiple models.

• [in] modelName Name of the model.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNGetInputCount()

【Function Prototype】

int32_t hbDNNGetInputCount(int32_t *inputCount, hbDNNHandle_t dnnHandle)

【Function Description】

Retrieve the number of input tensors for the model pointed to by dnnHandle.

【Parameters】

• [out] inputCount Number of model input tensors.

• [in] dnnHandle DNN handle pointing to one model.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNGetInputName()

【Function Prototype】

int32_t hbDNNGetInputName(const char **name, hbDNNHandle_t dnnHandle, int32_t inputIndex)

【Function Description】

Retrieve the name of an input tensor for the model pointed to by dnnHandle.

【Parameters】

• [out] name Name of the model input tensor.

• [in] dnnHandle DNN handle pointing to one model.

• [in] inputIndex Index of the model input tensor.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNGetInputTensorProperties()

【Function Prototype】

int32_t hbDNNGetInputTensorProperties(hbDNNTensorProperties *properties, hbDNNHandle_t dnnHandle, int32_t inputIndex)

【Function Description】

Retrieve properties of a specific input tensor for the model pointed to by dnnHandle.

【Parameters】

• [out] properties Information about the input tensor.

• [in] dnnHandle DNN handle pointing to one model.

• [in] inputIndex Index of the model input tensor.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNGetOutputCount()

【Function Prototype】

int32_t hbDNNGetOutputCount(int32_t *outputCount, hbDNNHandle_t dnnHandle)

【Function Description】

Retrieve the number of output tensors for the model pointed to by dnnHandle.

【Parameters】

• [out] outputCount Number of model output tensors.

• [in] dnnHandle DNN handle pointing to one model.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNGetOutputName()

【Function Prototype】

int32_t hbDNNGetOutputName(const char **name, hbDNNHandle_t dnnHandle, int32_t outputIndex)

【Function Description】

Retrieve the name of an output tensor for the model pointed to by dnnHandle.

【Parameters】

• [out] name Name of the model output tensor.

• [in] dnnHandle DNN handle pointing to one model.

• [in] outputIndex Index of the model output tensor.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbDNNGetOutputTensorProperties()

【Function Prototype】

int32_t hbDNNGetOutputTensorProperties(hbDNNTensorProperties *properties, hbDNNHandle_t dnnHandle, int32_t outputIndex)

【Function Description】

Retrieve properties of a specific output tensor for the model pointed to by dnnHandle.

【Parameters】

• [out] properties Information about the output tensor.

• [in] dnnHandle DNN handle pointing to one model.

• [in] outputIndex Index of the model output tensor.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

6.3.5.4. Model Inference API

hbDNNInfer()

【Function Prototype】

int32_t hbDNNInfer(hbDNNTaskHandle_t *taskHandle, hbDNNTensor **output, const hbDNNTensor *input, hbDNNHandle_t dnnHandle, hbDNNInferCtrlParam *inferCtrlParam)

【Function Description】

Execute inference tasks based on input parameters. The caller can use the returned taskHandle across functions and threads.

【Parameters】

• [out] taskHandle Pointer to task handle.

• [in/out] output Output of the inference task.

• [in] input Input of the inference task.

• [in] dnnHandle Pointer to DNN handle.

• [in] inferCtrlParam Parameters controlling the inference task.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

Note:

When submitting tasks using this interface, taskHandle should be set to nullptr in advance, unless appending tasks to a specified taskHandle (i.e., using the inferCtrlParam::more feature).

Up to 32 model tasks can exist simultaneously.

For batch models, it is allowed to set memory addresses of input tensors separately. For example: if the model’s input validShape/alignedShape is [4, 3, 224, 224], four hbDNNTensor instances can be allocated, each with validShape/alignedShape set to [1, 3, 224, 224], storing data for each batch. When the model has multiple inputs, the order of input should be input0[batch0], input0[batch1], …, inputn[batch0], inputn[batch1], ….

hbDNNRoiInfer()

【Function Prototype】

int32_t hbDNNRoiInfer(hbDNNTaskHandle_t *taskHandle, hbDNNTensor **output, const hbDNNTensor *input, hbDNNRoi *rois, int32_t roiCount, hbDNNHandle_t dnnHandle, hbDNNInferCtrlParam *inferCtrlParam)

【Function Description】

Execute ROI inference tasks based on input parameters. The caller can use the returned taskHandle across functions and threads.

【Parameters】

• [out] taskHandle Pointer to task handle.

• [in/out] output Output of the inference task.

• [in] input Input of the inference task.

• [in] rois ROI box information.

• [in] roiCount Number of ROI boxes.

• [in] dnnHandle Pointer to DNN handle.

• [in] inferCtrlParam Parameters controlling the inference task.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

Note:

If using X5, please follow the rules below:

• input_count : Number of model input branches

• output_count : Number of model output branches

• resizer_count : Number of input branches with resizer as source (≤input_count). When processing a batch of data, one resizer input branch handles one ROI.

• roiCount : Total number of ROIs, numerically equal to batch * resizer_count

• data_batch : Number of data batches the model needs to infer, numerically equal to roiCount / resizer_count

• model_batch : Internal batch size of the model, i.e., the batch_size actually fed into the model during inference. The X5 algorithm toolchain supports compiling models as batch models.

Input/Output Example Explanation:

Taking a complex multi-input model as an example, assume the model has 3 input branches (2 resizer sources, 1 DDR source) and 1 output branch, compiled with batch=2. The model needs to process 3 batches of data, totaling 6 ROIs (i.e., 2 ROIs per batch), then we have the following information:

• input_count = 3

• output_count = 1

• resizer_count = 2

• roiCount = 6

• data_batch = 3

• model_batch = 2

Therefore, the number of input_tensors with independent addresses required for the model to infer these 3 batches of data is input_count * data_batch = 9.

Suppose the static input/output information of the model is as follows:

• Model input (model_info):

  • tensor_0_resizer: [2, 3, 128, 128]

  • tensor_1_resizer: [2, 3, 256, 256]

  • tensor_2_ddr: [2, 80, 1, 100]

• Model output (model_info):

  • tensor_out: [2, 100, 1, 56]

Then the dynamic information during model inference would be:

• Model input (input_tensors):

  • [1x3x128x128, 1x3x256x256, 1x80x1x100, 1x3x128x128, 1x3x256x256, 1x80x1x100, 1x3x128x128, 1x3x256x256, 1x80x1x100]

• Model output (output_tensors):

  • [4x100x1x56]

Among them, since model_batch = 2, the underlying BPU can process 2 batches of data in a single execution; and because data_batch = 3, the formula for calculating the highest dimension of output_tensor is ceil[(data_batch) / model_batch] * model_batch, which must be an integer multiple of model_batch, as required by BPU hardware instructions. Missing inputs will be automatically ignored in computation.

Interface Limitations:

• Regarding batch size limit: the range should be [1, 255].

• When submitting tasks via this interface, taskHandle should be set to nullptr in advance, unless appending tasks to a specified taskHandle (i.e., using inferCtrlParam::more feature).

• roi size requirements: :math:2 <= width <= 4096, :math:2 <= height <= 4096.

• Original image size requirements: :math:1 <= W <= 4096, :math:16 <= stride <= 131072, where stride must be a multiple of 16.

• Output size requirements: :math:2 <= Wout, :math:2 <= Hout.

• ROI scaling factor limit: :math:0 <= step <= 262143, step calculation formula: :math:step = ((src\_len - 1)*65536 + (dst\_len - 1)/2)/(dst\_len - 1), where src_len is the W or H of the ROI, and dst_len is the required W or H of the model.

• Up to 32 model tasks can exist simultaneously.

hbDNNWaitTaskDone()

【Function Prototype】

int32_t hbDNNWaitTaskDone(hbDNNTaskHandle_t taskHandle, int32_t timeout)

【Function Description】

Wait for task completion or timeout.

【Parameters】

• [in] taskHandle Pointer to task handle.

• [in] timeout Timeout setting (unit: milliseconds).

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

Note:

1. timeout > 0 means the waiting duration;

2. timeout <= 0 means wait indefinitely until the task completes.

hbDNNReleaseTask()

【Function Prototype】

int32_t hbDNNReleaseTask(hbDNNTaskHandle_t taskHandle)

【Function Description】

Release the task. If the task has not started execution, it will be canceled and released immediately. If already executing, it will be canceled and released after reaching certain nodes.

【Parameters】

• [in] taskHandle Pointer to task handle.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

6.3.5.5. Model Memory Operation API

hbSysAllocMem()

【Function Prototype】

int32_t hbSysAllocMem(hbSysMem *mem, uint32_t size)

【Function Description】

Allocate BPU memory.

【Parameters】

• [in] size Size of memory to allocate.

• [out] mem Memory pointer.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbSysAllocCachedMem()

【Function Prototype】

int32_t hbSysAllocCachedMem(hbSysMem *mem, uint32_t size)

【Function Description】

Allocate cached BPU memory.

【Parameters】

• [in] size Size of memory to allocate.

• [out] mem Memory pointer.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbSysFlushMem()

【Function Prototype】

int32_t hbSysFlushMem(hbSysMem *mem, int32_t flag)

【Function Description】

Flush cached BPU memory.

【Parameters】

• [in] mem Memory pointer.

• [in] flag Flush flag.

【Return Type】

• Returns 0 if the API executed successfully, otherwise execution failed.

hbSysFreeMem()

【Function Prototype】

int32_t hbSysFreeMem(hbSysMem *mem)

【Function Description】

Free BPU memory.

【Parameters】

• [in] mem Memory pointer.

【Return Type】

• Returns 0 indicates the API executed successfully; otherwise, the execution failed.

hbSysWriteMem()

【Function Prototype】

int32_t hbSysWriteMem(hbSysMem *dest, char *src, uint32_t size)

【Function Description】

Write to BPU memory.

【Parameters】

• [out] dest Memory pointer.

• [in] src Data pointer.

• [in] size Data size.

【Return Type】

• Returns 0 indicates the API executed successfully; otherwise, the execution failed.

hbSysReadMem()

【Function Prototype】

int32_t hbSysReadMem(char *dest, hbSysMem *src, uint32_t size)

【Function Description】

Read from BPU memory.

【Parameters】

• [out] dest Data pointer.

• [in] src Memory pointer.

• [in] size Data size.

【Return Type】

• Returns 0 indicates the API executed successfully; otherwise, the execution failed.

hbSysRegisterMem()

【Function Prototype】

int32_t hbSysRegisterMem(hbSysMem *mem)

【Function Description】

Register a memory region with known physical address as a memory identifier usable by the BPU. The resulting memory is cacheable.

【Parameters】

• [in/out] mem Memory pointer.

【Return Type】

• Returns 0 indicates the API executed successfully; otherwise, the execution failed.

hbSysUnregisterMem()

【Function Prototype】

int32_t hbSysUnregisterMem(hbSysMem *mem)

【Function Description】

Unregister a memory identifier previously registered by hbSysRegisterMem.

【Parameters】

• [in] mem Memory pointer.

【Return Type】

• Returns 0 indicates the API executed successfully; otherwise, the execution failed.

6.3.5.6. Data Types and Data Structures

Version Information

HB_DNN_VERSION_MAJOR

##define HB_DNN_VERSION_MAJOR 1U

Major version number of DNN.

HB_DNN_VERSION_MINOR

##define HB_DNN_VERSION_MINOR 1U

Minor version number of DNN.

HB_DNN_VERSION_PATCH

##define HB_DNN_VERSION_PATCH 0U

Patch version number of DNN.

Note:

Note that the version numbers in this section may vary with versions. The version numbers shown here are for reference only. Please refer to the actual release you obtained for the correct version.

Model-related Types

HB_DNN_TENSOR_MAX_DIMENSIONS

##define HB_DNN_TENSOR_MAX_DIMENSIONS 8

Maximum number of tensor dimensions is set to 8.

HB_DNN_INITIALIZE_INFER_CTRL_PARAM

##define HB_DNN_INITIALIZE_INFER_CTRL_PARAM(param) \
{                                                 \
    (param)->bpuCoreId = HB_BPU_CORE_ANY;         \
    (param)->dspCoreId = HB_DSP_CORE_ANY;         \
    (param)->priority = HB_DNN_PRIORITY_LOWEST;   \
    (param)->more = false;                        \
    (param)->customId = 0;                        \
    (param)->reserved1 = 0;                       \
    (param)->reserved2 = 0;                       \
}

Initialize inference control parameters.

hbPackedDNNHandle_t

typedef void *hbPackedDNNHandle_t;

DNN handle pointing to multiple packed models.

hbDNNHandle_t

typedef void *hbDNNHandle_t;

DNN handle pointing to a single model.

hbDNNTaskHandle_t

typedef void *hbDNNTaskHandle_t;

Task handle pointing to a single task.

hbDNNTensorLayout

typedef enum {
  HB_DNN_LAYOUT_NHWC = 0,
  HB_DNN_LAYOUT_NCHW = 2,
  HB_DNN_LAYOUT_NONE = 255,
} hbDNNTensorLayout;

Tensor layout format. NHWC stands for Number, Height, Width, and Channel.

• Members

Member Name	Description
HB_DNN_LAYOUT_NONE	Layout format is undefined.
HB_DNN_LAYOUT_NHWC	Layout format is NHWC.
HB_DNN_LAYOUT_NCHW	Layout format is NCHW.

hbDNNDataType

typedef enum {
  HB_DNN_IMG_TYPE_Y,
  HB_DNN_IMG_TYPE_NV12,
  HB_DNN_IMG_TYPE_NV12_SEPARATE,
  HB_DNN_IMG_TYPE_YUV444,
  HB_DNN_IMG_TYPE_RGB,
  HB_DNN_IMG_TYPE_BGR,
  HB_DNN_TENSOR_TYPE_S4,
  HB_DNN_TENSOR_TYPE_U4,
  HB_DNN_TENSOR_TYPE_S8,
  HB_DNN_TENSOR_TYPE_U8,
  HB_DNN_TENSOR_TYPE_F16,
  HB_DNN_TENSOR_TYPE_S16,
  HB_DNN_TENSOR_TYPE_U16,
  HB_DNN_TENSOR_TYPE_F32,
  HB_DNN_TENSOR_TYPE_S32,
  HB_DNN_TENSOR_TYPE_U32,
  HB_DNN_TENSOR_TYPE_F64,
  HB_DNN_TENSOR_TYPE_S64,
  HB_DNN_TENSOR_TYPE_U64,
  HB_DNN_TENSOR_TYPE_MAX
} hbDNNDataType;

Tensor data type. S means signed, U means unsigned, F means floating-point, and the number indicates bit width.

HB_DNN_IMG_TYPE_NV12 and HB_DNN_IMG_TYPE_NV12_SEPARATE both represent NV12 data, differing only in storage format.

When inferring models with NV12 input, users can change the tensor’s tensorType property to either HB_DNN_IMG_TYPE_NV12 or HB_DNN_IMG_TYPE_NV12_SEPARATE based on actual needs.

• Members

Member Name	Description
HB_DNN_IMG_TYPE_Y	Tensor type is image with only Y channel.
HB_DNN_IMG_TYPE_NV12	Tensor type is an NV12 image.
HB_DNN_IMG_TYPE_NV12_SEPARATE	Tensor type is image with separate Y and UV channels.
HB_DNN_IMG_TYPE_YUV444	Tensor type is image in YUV444 format.
HB_DNN_IMG_TYPE_RGB	Tensor type is image in RGB format.
HB_DNN_IMG_TYPE_BGR	Tensor type is image in BGR format.
HB_DNN_TENSOR_TYPE_S4	Tensor type is signed 4-bit.
HB_DNN_TENSOR_TYPE_U4	Tensor type is unsigned 4-bit.
HB_DNN_TENSOR_TYPE_S8	Tensor type is signed 8-bit.
HB_DNN_TENSOR_TYPE_U8	Tensor type is unsigned 8-bit.
HB_DNN_TENSOR_TYPE_F16	Tensor type is floating-point 16-bit.
HB_DNN_TENSOR_TYPE_S16	Tensor type is signed 16-bit.
HB_DNN_TENSOR_TYPE_U16	Tensor type is unsigned 16-bit.
HB_DNN_TENSOR_TYPE_F32	Tensor type is floating-point 32-bit.
HB_DNN_TENSOR_TYPE_S32	Tensor type is signed 32-bit.
HB_DNN_TENSOR_TYPE_U32	Tensor type is unsigned 32-bit.
HB_DNN_TENSOR_TYPE_F64	Tensor type is floating-point 64-bit.
HB_DNN_TENSOR_TYPE_S64	Tensor type is signed 64-bit.
HB_DNN_TENSOR_TYPE_U64	Tensor type is unsigned 64-bit.
HB_DNN_TENSOR_TYPE_MAX	Maximum tensor type index.

hbDNNTensorShape

typedef struct {
  int32_t dimensionSize[HB_DNN_TENSOR_MAX_DIMENSIONS];
  int32_t numDimensions;
} hbDNNTensorShape;

Tensor shape. For example, a 224x224 BGR color image has numDimensions=4. If the layout is NHWC, then the dimensionSize array stores the values Number=1, Height=224, Width=224, and Channel=3 in order.

• Members

Member Name	Description
dimensionSize	Size of each tensor dimension.
numDimensions	Number of tensor dimensions.

hbDNNQuantiShift

typedef struct {
  int32_t shiftLen;
  uint8_t *shiftData;
} hbDNNQuantiShift;

Shift data for quantization/dequantization.

For input: If the collected floating-point data is data[i] and the corresponding shift data is shift[i], then the data fed into the model is: integer part of :math:data[i] * (1 << shift[i]);

For output: If the inference result is data[i] and the corresponding shift data is shift[i], then the final inference result is: :math:data[i] / (1 << shift[i]).

Note:

Here, shiftLen is determined by the data data according to the per-axis or per-tensor quantization/dequantization method. When data data is quantized/dequantized in per-tensor mode, shiftLen equals 1, and quantizeAxis value can be ignored; otherwise, it equals the size of the quantizeAxis dimension of data data.

• Members

Member Name	Description
shiftLen	Length of shift data.
shiftData	Starting address of shift data.

hbDNNQuantiScale

typedef struct {
  int32_t scaleLen;
  float *scaleData;
  int32_t zeroPointLen;
  int8_t *zeroPointData;
} hbDNNQuantiScale;

Scaling data for quantization/dequantization.

For input: If the collected floating-point data is data[i], the corresponding scale data is scale[i], and zero point offset data is zeroPoint[i], then the data fed into the model is: :math:g((data[i] / scale[i]) + zeroPoint[i]), where :math:g(x) = clip(nearbyint(x)), using fesetround(FE_TONEAREST) rounding method, clipped to: U8: :math:g(x)∈[0, 255], S8: :math:g(x)∈[-128, 127];

For output: If the inference result is data[i], the corresponding scale data is scale[i], and zero point offset data is zeroPoint[i], then the final inference result is: :math:(data[i] - zeroPoint[i]) * scale[i].

Note:

Here, scaleLen is determined by the data data according to the per-axis or per-tensor quantization/dequantization method. When data data is quantized/dequantized in per-tensor mode, scaleLen equals 1, and quantizeAxis value can be ignored; otherwise, it equals the size of the quantizeAxis dimension of data data. zeroPointLen should be consistent with scaleLen.

• Members

Member Name	Description
scaleLen	Length of scale data.
scaleData	Starting address of scale data.
zeroPointLen	Length of zero point offset data.
zeroPointData	Starting address of zero point offset data.

hbDNNQuantiType

typedef enum {
  NONE, 
  SHIFT,
  SCALE,
} hbDNNQuantiType;

Quantization/dequantization type for fixed-point and floating-point conversion. NONE means no processing is required; SHIFT type parameters are stored in the hbDNNQuantiShift structure, and SCALE type parameters are stored in the hbDNNQuantiScale structure.

• Members

Member Name	Description
NONE	No quantization.
SHIFT	Quantization type is SHIFT.
SCALE	Quantization type is SCALE.

hbDNNTensorProperties

typedef struct {
  hbDNNTensorShape validShape;
  hbDNNTensorShape alignedShape;
  int32_t tensorLayout;
  int32_t tensorType;
  hbDNNQuantiShift shift;
  hbDNNQuantiScale scale;
  hbDNNQuantiType quantiType;
  int32_t quantizeAxis;
  int32_t alignedByteSize;
  int32_t stride[HB_DNN_TENSOR_MAX_DIMENSIONS];
} hbDNNTensorProperties;

Tensor properties.

When alignedShape is obtained from model information, it represents the shape after alignment. After input data preparation, alignedShape should match the actual input shape of the tensor.

• Members

Member Name	Description
validShape	Shape of valid tensor content.
alignedShape	Shape of aligned tensor content.
tensorLayout	Tensor layout format.
tensorType	Tensor data type.
shift	Quantization shift.
scale	Quantization scale.
quantiType	Quantization type.
quantizeAxis	Quantization axis, effective only for per-axis quantization.
alignedByteSize	Memory size of aligned tensor content.
stride	Stride of each dimension in validShape.

Note:

The tensor properties obtained via API reflect model requirements. You may modify certain properties based on actual input, but currently only alignedShape and tensorType are allowed to be modified, and must comply with requirements.

alignedShape:

1. If you prepare input according to alignedShape, no need to change alignedShape.

2. If you prepare input according to validShape, you must set alignedShape to validShape; the inference library will internally perform padding.

tensorType:

When inferring models with NV12 input, you may change the tensor’s tensorType to HB_DNN_IMG_TYPE_NV12 or HB_DNN_IMG_TYPE_NV12_SEPARATE based on actual needs.

hbDNNTaskPriority

typedef enum {
  HB_DNN_PRIORITY_LOWEST = 0,
  HB_DNN_PRIORITY_HIGHEST = 255,
  HB_DNN_PRIORITY_PREEMP = HB_DNN_PRIORITY_HIGHEST,
} hbDNNTaskPriority;

Task priority configuration, with default parameters provided.

hbDNNTensor

typedef struct {
  hbSysMem sysMem[4];
  hbDNNTensorProperties properties;
} hbDNNTensor;

Tensor. Used to store input and output information. Tensors of type NV12_SEPARATE require 2 hbSysMem entries; others require only 1.

• Members

| Member Name | Description | |——————-|————————————–| | sysMem | Memory storing the tensor. | | properties | Tensor properties. |

hbDNNRoi

typedef struct {
  int32_t left;
  int32_t top;
  int32_t right;
  int32_t bottom;
} hbDNNRoi;

Rectangular region of interest. :math:W∈[left, right], H∈[top, bottom].

• Members

Member Name	Description
left	Left-top width pixel of ROI.
top	Left-top height pixel of ROI.
right	Right-bottom width pixel of ROI.
bottom	Right-bottom height pixel of ROI.

hbDNNInferCtrlParam

typedef struct {
  int32_t bpuCoreId;
  int32_t dspCoreId;
  int32_t priority;
  int32_t more;
  int64_t customId;
  int32_t reserved1;
  int32_t reserved2;
} hbDNNInferCtrlParam;

Control parameters for model inference. bpuCoreId and dspCoreId control which BPU and DSP cores are used for inference. X3 does not support DSP inference, so dspCoreId is used as a placeholder only. The more parameter is used in scenarios involving batch processing of small models. When you want to obtain outputs only after all tasks are completed, set more to 1 for all tasks except the last one, which should have more set to 0. Up to 255 small model inferences are supported. When the small model is a resizer model, each ROI may be considered a small model. The customId parameter allows user-defined priority for task ordering (e.g., timestamp, frame ID), where smaller values indicate higher priority. Priority order: priority > customId.

• Members

Member Name	Description
bpuCoreId	BPU core ID.
dspCoreId	DSP core ID.
priority	Task priority.
more	Whether more tasks follow this one.
customId	Custom priority identifier.
reserved1	Reserved field 1.
reserved2	Reserved field 2.

System-related Types

hbBPUCore

typedef enum {
  HB_BPU_CORE_ANY = 0,
  HB_BPU_CORE_0 = (1 << 0),
  HB_BPU_CORE_1 = (1 << 1)
} hbBPUCore;

BPU core enumeration. The X5 system has only one BPU core and only supports specifying HB_BPU_CORE_ANY or HB_BPU_CORE_0.

• Members

Member Name	Description
HB_BPU_CORE_ANY	Any BPU core.
HB_BPU_CORE_0	BPU core 0.
HB_BPU_CORE_1	BPU core 1.

hbDSPCore

typedef enum {
  HB_DSP_CORE_ANY = 0,
  HB_DSP_CORE_0 = (1 << 0),
  HB_DSP_CORE_1 = (1 << 1)
} hbDSPCore;

DSP core enumeration.

Member Name	Description
HB_DSP_CORE_ANY	Any DSP core.
HB_DSP_CORE_0	DSP core 0.
HB_DSP_CORE_1	DSP core 1.

hbSysMem

typedef struct {
  uint64_t phyAddr;
  void *virAddr;
  uint32_t memSize;
} hbSysMem;

System memory structure, used for allocating system memory.

• Members

Member Name	Description
phyAddr	Physical address.
virAddr	Virtual address.
memSize	Memory size.

hbSysMemFlushFlag

typedef enum {
  HB_SYS_MEM_CACHE_INVALIDATE = 1,
  HB_SYS_MEM_CACHE_CLEAN = 2
} hbSysMemFlushFlag;

Parameters for synchronizing system memory with cache. There is a cache layer between CPU and memory, which may cause inconsistency between cache and memory data. To ensure the latest data is always available, data must be updated before CPU reads and after CPU writes. Before CPU reads, data from memory should be loaded into cache. After CPU writes, data from cache should be flushed back to memory.

• Members

Member Name	Description
HB_SYS_MEM_CACHE_INVALIDATE	Synchronize memory to cache, used before CPU read.
HB_SYS_MEM_CACHE_CLEAN	Synchronize cache to memory, used after CPU write.

Preprocessing Classes

HB_DNN_INITIALIZE_RESIZE_CTRL_PARAM

#define HB_DNN_INITIALIZE_RESIZE_CTRL_PARAM(param)     \
  {                                                     \
    (param)->bpuCoreId = HB_BPU_CORE_ANY;              \
    (param)->resizeType = HB_DNN_RESIZE_TYPE_BILINEAR; \
    (param)->priority = HB_DNN_PRIORITY_LOWEST;        \
    (param)->reserved1 = 0;                             \
    (param)->reserved2 = 0;                             \
    (param)->reserved3 = 0;                             \
    (param)->reserved4 = 0;                             \
  }

Initializes control parameters.

hbDNNResizeType

typedef enum {
  HB_DNN_RESIZE_TYPE_BILINEAR = 0,
} hbDNNResizeType;

Resize type.

• Members

Member Name	Description
HB_DNN_RESIZE_TYPE_BILINEAR	Resize type is bilinear interpolation.

hbDNNResizeCtrlParam

typedef struct {
  int32_t bpuCoreId;
  int32_t priority;
  hbDNNResizeType resizeType;
  int32_t reserved1;
  int32_t reserved2;
  int32_t reserved3;
  int32_t reserved4;
} hbDNNResizeCtrlParam;

Control parameters for Resize.

• Members

Member Name	Description
bpuCoreId	BPU core ID.
priority	Task priority.
resizeType	Resize type.
reserved1	Reserved field 1.
reserved2	Reserved field 2.
reserved3	Reserved field 3.
reserved4	Reserved field 4.

6.3.5.7. Data Layout and Alignment Rules

Data Layout

Internally, hardware uses a special data layout to improve computational efficiency, arranging feature maps and kernels used in the same batch of convolution operations adjacently in memory. The following briefly introduces the concept of data layout in the X5 processor.

Variables in neural network models can be represented as a 4-dimensional tensor, with each number being an element of the tensor—this is referred to as natural layout. Elements from different dimensions are tightly packed together according to certain rules into small independent blocks. These blocks are then treated as new elements to form a new 4-dimensional tensor, which is called a tensor with data layout.

Input and output data use different layout formats. Users can obtain layout description information via APIs, and data with different layouts cannot be directly compared.

Note:

When performing data layout transformation, if padding is required, it is recommended to set the padding value to zero.

Two data layouts are introduced here: NHWC_NATIVE and NCHW_NATIVE. Taking NHWC_NATIVE as an example, its data layout is as follows:

N0H0W0C0	N0H0W0C1	……
N0H0W1C0	N0H0W1C1	……
……	……	……
N0H1W0C0	N0H1W0C1	……
……	……	……
N1H0W0C0	N1H0W0C1	……
……	……	……

A tensor of size N×H×W×C can be expressed using the following 4-level loop:

for (int32_t n = 0; n < N; n++) {
    for (int32_t h = 0; h < H; h++) {
        for (int32_t w = 0; w < W; w++) {
            for (int32_t c = 0; c < C; c++) {
                int32_t native_offset = n*H*W*C + h*W*C + w*C + c;
            }
        }
    }
}

The only difference between NCHW_NATIVE and NHWC_NATIVE is the loop order, so it is not listed separately.

caution In the following text, “native” specifically refers to this layout.

BPU Alignment Restrictions

This section describes alignment restrictions when using the BPU.

Model Input Requirements

The BPU does not restrict model input size or parity. Inputs like YOLO’s 416×416 are supported, as well as SqueezeNet’s 227×227. NV12 input is special—it requires both height (H) and width (W) to be even numbers, to meet the requirement that UV is half the size of Y.

Alignment and Valid Data

The BPU has alignment restrictions on data. Alignment requirements and actual data layout are represented by validShape, alignedShape, and stride in hbDNNTensorProperties.

• validShape is the actual valid shape;

• alignedShape is the shape satisfying alignment requirements. Due to hardware characteristics, alignedShape is always represented as a 4-dimensional shape;

• stride indicates the stride of each dimension in validShape. For models with NV12 input, stride is always 0 because such models only require W to be 16-byte aligned.

Currently, 4D model tensors can obtain correct data layout via validShape and alignedShape. For tensors in X5 models with more than 4 dimensions, correct layout can be obtained using validShape and stride.

Introduction to NV12

YUV Format

YUV format is primarily used to optimize the transmission of color video signals. YUV consists of three components: Y represents luminance (i.e., grayscale), while U and V represent chrominance, describing color and saturation, and are used to specify pixel colors.

NV12 Layout

NV12 image format belongs to the YUV420SP format in the YUV color space. Every four Y components share one set of U and V components. Y components are stored sequentially, while U and V are interleaved.

The layout is as follows:

6.3.5.8. Model Inference DEBUG Methods

Error Codes

HB_DNN_SUCCESS = 0                              // Operation succeeded
HB_DNN_INVALID_ARGUMENT = -6000001              // Invalid argument
HB_DNN_INVALID_MODEL = -6000002                 // Invalid model
HB_DNN_MODEL_NUMBER_EXCEED_LIMIT = -6000003     // Number of models exceeds limit
HB_DNN_INVALID_PACKED_DNN_HANDLE = -6000004     // Invalid packed handle
HB_DNN_INVALID_DNN_HANDLE = -6000005            // Invalid handle
HB_DNN_CAN_NOT_OPEN_FILE = -6000006             // File does not exist
HB_DNN_OUT_OF_MEMORY = -6000007                 // Insufficient memory
HB_DNN_TIMEOUT = -6000008                       // Timeout
HB_DNN_TASK_NUM_EXCEED_LIMIT = -6000009         // Number of tasks exceeds limit
HB_DNN_TASK_BATCH_SIZE_EXCEED_LIMIT = -6000010  // Batch size exceeds limit
HB_DNN_INVALID_TASK_HANDLE = -6000011           // Invalid task handle
HB_DNN_RUN_TASK_FAILED = -6000012               // Task execution failed
HB_DNN_MODEL_IS_RUNNING = -6000013              // Task is running
HB_DNN_INCOMPATIBLE_MODEL = -6000014            // Incompatible model
HB_DNN_API_USE_ERROR = -6000015                 // API usage error
HB_DNN_MULTI_PROGRESS_USE_ERROR = -6000016      // Multi-process usage error

HB_SYS_SUCCESS = 0                              // Operation succeeded
HB_SYS_INVALID_ARGUMENT = -6000129              // Invalid argument
HB_SYS_OUT_OF_MEMORY = -6000130                 // Insufficient memory
HB_SYS_REGISTER_MEM_FAILED = -6000131           // Failed to register memory

Configuration Information

1. Log levels. Logs in dnn are mainly divided into 4 levels:

   • HB_DNN_LOG_NONE = 0: No log output;

   • HB_DNN_LOG_WARNING = 3: Used to output warning messages in code;

   • HB_DNN_LOG_ERROR = 4: Used to output error messages in code;

   • HB_DNN_LOG_FATAL = 5: Used to output fatal errors that cause program termination.

2. Log level setting rules:

   A log will be printed if its level is greater than or equal to the set level; otherwise, it will be suppressed. The lower the set log level (except level 0, which disables all logs), the more messages will be printed.
   For example: Setting log level to 3 (i.e., WARNING) means logs of levels 3, 4, and 5 will be printed.
   The default log level in the inference library is HB_DNN_LOG_WARNING, so the following log levels will be printed:
   WARNING, ERROR, FATAL.

3. How to set log level:
   Use the environment variable HB_DNN_LOG_LEVEL to set the log level.
   Example: export HB_DNN_LOG_LEVEL=3 will output logs at WARNING level and above.

4. Common Environment Variables

    HB_DNN_LOG_LEVEL                // Set log level.
    HB_DNN_PLUGIN_PATH              // Custom directory for CPU operator dynamic libraries.
    HB_DNN_PROFILER_LOG_PATH        // Path to dump profiling information on model execution stages.
    HB_DNN_SIM_PLATFORM             // x86 simulator platform setting; can be set to BERNOULLI, BERNOULLI2, BAYES, BAYESE.
    HB_DNN_SIM_BPU_MEM_SIZE         // Set BPU memory size for x86 simulator, in MB.
   

Notes on Using Simulator on Development Machine

1. When using the simulator on a development machine, the target processor architecture can be specified by setting the environment variable HB_DNN_SIM_PLATFORM. For example:

   • export HB_DNN_SIM_PLATFORM=BAYESE means simulating the x5 platform;

2. If the HB_DNN_SIM_PLATFORM environment variable is not set, the simulator platform will be determined by the architecture of the first loaded model. For example, if the first loaded model is of BAYESE architecture, the platform will be set to x5 by default.

3. Before performing resize operations in the development machine simulator, the platform must be specified using the HB_DNN_SIM_PLATFORM environment variable.