9.2. BPU SDK API DOC

This document mainly introduces the API, data, structures, layout, and alignment rules of D-Robotics OpenExplorer Toolchain Runtime.

By reading this document, you can use APIs to complete operations such as model loading and releasing, model information retrieval, and model inference on dev boards provided by D-Robotics.

9.2.1. Data Types and Data Structures

9.2.1.1. Version Category

Note

Note that the version numbers of the version information type in this section may vary by version. Version numbers here are for reference only and the one you get may be different.

9.2.1.1.1. HB_DNN_VERSION_MAJOR

#define HB_DNN_VERSION_MAJOR 1U

Major version number of DNN.

9.2.1.1.2. HB_DNN_VERSION_MINOR

#define HB_DNN_VERSION_MINOR 1U

Minor version number of DNN.

9.2.1.1.3. HB_DNN_VERSION_PATCH

#define HB_DNN_VERSION_PATCH 0U

Version number of DNN patch.

9.2.1.2. Model Category

9.2.1.2.1. HB_DNN_TENSOR_MAX_DIMENSIONS

#define HB_DNN_TENSOR_MAX_DIMENSIONS 8

The maximum dimension of the tensor is set to 8.

9.2.1.2.2. HB_DNN_INITIALIZE_INFER_CTRL_PARAM

#define HB_DNN_INITIALIZE_INFER_CTRL_PARAM(param) \
{                                                  \
    (param)->bpuCoreId = HB_BPU_CORE_ANY;         \
    (param)->priority = HB_DNN_PRIORITY_LOWEST;   \
    (param)->reserved1 = 0;                        \
    (param)->reserved2 = 0;                        \
    (param)->reserved3 = 0;                        \
    (param)->reserved4 = 0;                        \
    (param)->more = false;                         \
}

Initializes the control parameters.

9.2.1.2.3. hbPackedDNNHandle_t

typedef void *hbPackedDNNHandle_t;

DNN handle, pointing to multiple packaged models.

9.2.1.2.4. hbDNNHandle_t

typedef void *hbDNNHandle_t;

DNN handle, pointing to a single model.

9.2.1.2.5. hbDNNTaskHandle_t

typedef void *hbDNNTaskHandle_t;

Task handle, pointing to a task.

9.2.1.2.6. hbDNNTaskDoneCb

typedef void (*hbDNNTaskDoneCb)(hbDNNTaskHandle_t taskHandle, int32_t status,
                                void *userdata);

User-defined method that needs to be executed after task done.

  • Parameter

    • [in] taskHandle Task handle pointer.

    • [in] status task status code.

    • [in] userdata user-defined data.

9.2.1.2.7. hbDNNTensorLayout

typedef enum {
  HB_DNN_LAYOUT_NHWC = 0,
  HB_DNN_LAYOUT_NCHW = 2,
  HB_DNN_LAYOUT_NONE = 255,
} hbDNNTensorLayout;

Tensor layout form.

NHWC stands for Number, Height, Width, and Channel, respectively.

  • Member

Member Name

Description

HB_DNN_LAYOUT_NONE

No layout is defined.

HB_DNN_LAYOUT_NHWC

Layout is NHWC.

HB_DNN_LAYOUT_NCHW

Layout is NCHW.

9.2.1.2.8. hbDNNDataType

typedef enum {
  HB_DNN_IMG_TYPE_Y,
  HB_DNN_IMG_TYPE_NV12,
  HB_DNN_IMG_TYPE_NV12_SEPARATE,
  HB_DNN_IMG_TYPE_YUV444,
  HB_DNN_IMG_TYPE_RGB,
  HB_DNN_IMG_TYPE_BGR,
  HB_DNN_TENSOR_TYPE_S4,
  HB_DNN_TENSOR_TYPE_U4,
  HB_DNN_TENSOR_TYPE_S8,
  HB_DNN_TENSOR_TYPE_U8,
  HB_DNN_TENSOR_TYPE_F16,
  HB_DNN_TENSOR_TYPE_S16,
  HB_DNN_TENSOR_TYPE_U16,
  HB_DNN_TENSOR_TYPE_F32,
  HB_DNN_TENSOR_TYPE_S32,
  HB_DNN_TENSOR_TYPE_U32,
  HB_DNN_TENSOR_TYPE_F64,
  HB_DNN_TENSOR_TYPE_S64,
  HB_DNN_TENSOR_TYPE_U64,
  HB_DNN_TENSOR_TYPE_MAX
} hbDNNDataType;

The type of the tensor.

S stands for Signed, U for Unsigned, and F for Floating-point, followed by a number that stands for the number of bits.

HB_DNN_IMG_TYPE_NV12 and HB_DNN_IMG_TYPE_NV12_SEPARATE both represent NV12 data, but there are differences in storage.

  • Member

Member Name

Description

HB_DNN_IMG_TYPE_Y

Tensor type: Image with Y channel only.

HB_DNN_IMG_TYPE_NV12

Tensor type: An NV12 Image.

HB_DNN_IMG_TYPE_NV12_SEPARATE

Tensor type: Image with Y & UV channels as input.

HB_DNN_IMG_TYPE_YUV444

Tensor type: Image with YUV444 as input.

HB_DNN_IMG_TYPE_RGB

Tensor type: Image with RGB as input.

HB_DNN_IMG_TYPE_BGR

Tensor type: Image with BGR as input.

HB_DNN_TENSOR_TYPE_S4

Tensor type: Signed 4 bit.

HB_DNN_TENSOR_TYPE_U4

Tensor type: Unsigned 4 bit.

HB_DNN_TENSOR_TYPE_S8

Tensor type: Signed 8 bit.

HB_DNN_TENSOR_TYPE_U8

Tensor type: Unsigned 8 bit.

HB_DNN_TENSOR_TYPE_F16

Tensor type: 16 bit floating-point.

HB_DNN_TENSOR_TYPE_S16

Tensor type: Signed 16 bit.

HB_DNN_TENSOR_TYPE_U16

Tensor type: Unsigned 16 bit.

HB_DNN_TENSOR_TYPE_F32

Tensor type: 32 bit floating-point.

HB_DNN_TENSOR_TYPE_S32

Tensor type: Signed 32 bit.

HB_DNN_TENSOR_TYPE_U32

Tensor type: Unsigned 32 bit.

HB_DNN_TENSOR_TYPE_F64

Tensor type: 64 bit floating-point.

HB_DNN_TENSOR_TYPE_S64

Tensor type: Signed 64 bit.

HB_DNN_TENSOR_TYPE_U64

Tensor type: Unsigned 64 bit.

HB_DNN_TENSOR_TYPE_MAX

Maximum tensor type number.

9.2.1.2.9. hbDNNTensorShape

typedef struct {
  int32_t dimensionSize[HB_DNN_TENSOR_MAX_DIMENSIONS];
  int32_t numDimensions;
} hbDNNTensorShape;

The shape of the tensor.

For example, for a 224x224 BGR color space image numDimensions=4, if the layout form is NHWC, then Number=1, Height=224, Width=224, and Channel=3 of the image are stored in the dimensionSize array in order.

  • Member

Member Name

Description

dimensionSize

Size of each dimension of the tensor.

numDimensions

Dimension of the tensor.

9.2.1.2.10. hbDNNQuantiShift

typedef struct {
  int32_t shiftLen;
  uint8_t *shiftData;
} hbDNNQuantiShift;

Quantized/anti-quantized shifted data.

Input: If the floating-point data data[i] is collected, and the corresponding shift data is shift[i], then the inference data sent to the model is \(data[i] * (1 << shift[i])\) rounded.

Output:: If the corresponding shift data of the inference result data[i] is shift[i], then the final inference result is \(data[i] / (1 << shift[i])\).

The shiftLen is determined by data according to per-axis or per-tensor quantization/dequantion. When the data is quantized/dequantized by per-tensor, the shiftLen is equal to 1, ignoring quantizeAxis; otherwise it is equal to the quantizeAxis dimension value of data.

  • Member

Member Name

Description

shiftLen

Length of shift data.

shiftData

First address of shift data.

9.2.1.2.11. hbDNNQuantiScale

typedef struct {
  int32_t scaleLen;
  float *scaleData;
  int32_t zeroPointLen;
  int8_t *zeroPointData;
} hbDNNQuantiScale;

Quantitative/anti-quantitative scale data.

Input: If the floating point data data[i] is collected, the corresponding scale data is scale[i], and the corresponding zero-point offset data is zeroPoint[i], then the inference data sent to the model is \(g((data[i] / scale[i]) + zeroPoint[i])\), in which, \(g(x) = clip(round(x))\), it is truncated to U8: \(g(x)∈[0, 255]\), S8: \(g(x)∈[-128, 127]\).

Output: If the corresponding scale data of the inference result data[i] is scale[i], and the corresponding zero-point offset data is zeroPoint[i], then the final inference result is \((data[i] - zeroPoint[i])* scale[i]\).

The scaleLen is determined by data according to per-axis or per-tensor quantization/dequantion. When the data is quantized/dequantized by per-tensor, the scaleLen is equal to 1, ignoring quantizeAxis; otherwise it is equal to the quantizeAxis dimension value of data. The value of zeroPointLen is equal to scaleLen.

  • Member

Member Name

Description

scaleLen

Length of scale data.

scaleData

First address of scale data.

zeroPointLen

Length of zero-point offset data.

zeroPointLen

First address of zero-point offset data.

9.2.1.2.12. hbDNNQuantiType

typedef enum {
  NONE,
  SHIFT,
  SCALE,
} hbDNNQuantiType;

Quantization/antiquantization type for fixed-point floating-point conversions.

NONE: No data processing is needed.

SHIFT: The quantization/anti-quantization parameters corresponding to the SHIFT type are stored in the hbDNNQuantiShift structure.

SCALE: The quantization/anti-quantization parameters corresponding to SCALE are stored in the hbDNNQuantiScale structure.

  • Member

Member Name

Description

NONE

No quantization.

SHIFT

Quantization type: SHIFT.

SCALE

Quantization type: SCALE.

9.2.1.2.13. hbDNNTensorProperties

typedef struct {
  hbDNNTensorShape validShape;
  hbDNNTensorShape alignedShape;
  int32_t tensorLayout;
  int32_t tensorType;
  hbDNNQuantiShift shift;
  hbDNNQuantiScale scale;
  hbDNNQuantiType quantiType;
  int32_t quantizeAxis;
  int32_t alignedByteSize;
  int32_t stride[HB_DNN_TENSOR_MAX_DIMENSIONS];
} hbDNNTensorProperties;

Tensor information.

  • Member

Member Name

Description

validShape

Shape of the valid content of the tensor.

alignedShape

Shape of the aligned content of the tensor.

tensorLayout

Tensor layout form.

tensorType

Tensor type.

shift

Shift of quantization.

scale

Scale of quantization.

quantiType

Quantization type.

quantizeAxis

Axis of quantization, effective only when quantified by per-axis.

alignedByteSize

Memory size of the aligned content of the tensor.

stride

Step size of each dimension of validShape.

Note

The tensor information obtained through the interface is required by the model. You can modify the corresponding tensor information according to the actual input. Currently, only alignedShape and tensorType are allowed to be modified, and the requirements must be met.

alignedShape:

  1. If you prepare your input according to alignedShape, you don’t need to change alignedShape.

  2. If you prepare input based on validShape, you need to change alignedShape to validShape, and the inference library will perform padding operations on the data internally.

tensorType:

When inferring a model with NV12 input, the user can change the tensorType attribute of the tensor to HB_DNN_IMG_TYPE_NV12 or HB_DNN_IMG_TYPE_NV12_SEPARATE according to the actual situation.

9.2.1.2.14. hbDNNTaskPriority

typedef enum {
  HB_DNN_PRIORITY_LOWEST = 0,
  HB_DNN_PRIORITY_HIGHEST = 255,
  HB_DNN_PRIORITY_PREEMP = HB_DNN_PRIORITY_HIGHEST,
} hbDNNTaskPriority;

Task priority configuration, default parameters provided.

9.2.1.2.15. hbDNNTensor

typedef struct {
  hbSysMem sysMem[4];
  hbDNNTensorProperties properties;
} hbDNNTensor;

Tensor.

Used to store the input/output information. The tensor of type NV12_SEPARATE requires two hbSysMem, while the others require one.

  • Member

Member Name

Description

sysMem

Memory to store the tensor.

properties

Tensor information.

9.2.1.2.16. hbDNNRoi

typedef struct {
  int32_t left;
  int32_t top;
  int32_t right;
  int32_t bottom;
} hbDNNRoi;

Rectangular region of interest. \(W∈[left, right], H∈[top, bottom]\).

  • Member

Member Name

Description

left

X coordinate of the ROI’s top-left point.

top

Y coordinate of the ROI’s top-left point.

right

X coordinate of the ROI’s bottom-right point.

bottom

Y coordinate of the ROI’s bottom-right point.

9.2.1.2.17. hbDNNInferCtrlParam

typedef struct {
  int32_t bpuCoreId;
  int32_t priority;
  int32_t more;
  int32_t reserved1;
  int32_t reserved2;
  int32_t reserved3;
  int32_t reserved4;
} hbDNNInferCtrlParam;

Control parameters for model inference.

bpuCoreId is used to control the cores used by the inference model BPU node;

where the more parameter is used for batch processing small models. When you prefer getting the output after all tasks are executed, set the `more of all the tasks to 1 except for the last task, whose more should be set to 0. Batch inference of up to 255 different small models are supported. When the small models are resizer models, each ROI may be considered as a small model.

customId is used for user customized priorities, which defines the task priority, can be timestamp, frame ID, etc. Lower values mean higher priorities. Priority level: priority > customId.

  • Member

Member Name

Description

bpuCoreId

BPU core ID.

priority

Task priority.

more

Whether the task is followed by subsequent tasks.

customId

Custom priority.

reserved1

Reserved field 1.

Reserved2

Reserved field 2.

9.2.1.3. System Category

9.2.1.3.1. hbBPUCore

typedef enum {
  HB_BPU_CORE_ANY = 0,
  HB_BPU_CORE_0 = (1 << 0),
  HB_BPU_CORE_1 = (1 << 1)
} hbBPUCore;

BPU core enumeration, X5 system only have one bpu core, just support HB_BPU_CORE_ANY and HB_BPU_CORE_0.

  • Member

Member Name

Description

HB_BPU_CORE_ANY

Any BPU core.

HB_BPU_CORE_0

BPU core 0.

HB_BPU_CORE_1

BPU core 0.

9.2.1.3.2. hbSysMem

typedef struct {
  uint64_t phyAddr;
  void *virAddr;
  uint32_t memSize;
} hbSysMem;

Structure of system memory for requesting the system memory.

  • Member

Member Name

Description

phyAddr

Physical address.

virAddr

Virtual address.

memSize

Memory size.

9.2.1.3.3. hbSysMemFlushFlag

typedef enum {
  HB_SYS_MEM_CACHE_INVALIDATE = 1,
  HB_SYS_MEM_CACHE_CLEAN = 2
} hbSysMemFlushFlag;

Parameter for sync between system memory and cache.

There is a cache area between the CPU and memory, where the contents in the cache can be out of sync with those in memory. To get the latest data every time, we need to update the data before CPU reading data and after CPU writing data. Before CPU reads data, update data from memory to cache. After CPU writes the data, flush data from cache to memory.

../../../_images/hbSysMemFlushFlag.png

  • Member

Member Name

Description

HB_SYS_MEM_CACHE_INVALIDATE

Sync memory data to cache prior to CPU reading operations.

HB_SYS_MEM_CACHE_CLEAN

Sync cache data to memory after CPU writing operations.

9.2.1.4. Plug-in Category

9.2.1.4.1. hbDNNLayerCreator

typedef hobot::dnn::Layer *(*hbDNNLayerCreator)();

User-defined layer creation method.

9.2.2. API

9.2.2.1. Version Info

9.2.2.1.1. hbDNNGetVersion()

char const *hbDNNGetVersion();

Gets version info of DNN prediction library.

  • Return value

    • Returns version info.

9.2.2.2. Model Loading & Releasing

9.2.2.2.1. hbDNNInitializeFromFiles()

int32_t hbDNNInitializeFromFiles(hbPackedDNNHandle_t *packedDNNHandle,
                                 char const **modelFileNames,
                                 int32_t modelFileCount);

Complete the creation and initialization of packedDNNHandle from the file. The caller can use the returned packedDNNHandle across functions and threads.

  • Parameter

    • [out] packedDNNHandle D-Robotics DNN handle, pointing to multiple models.

    • [in] modelFileNames Path of the model files.

    • [in] modelFileCount Number of the model files.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.2.2. hbDNNInitializeFromDDR()

int32_t hbDNNInitializeFromDDR(hbPackedDNNHandle_t *packedDNNHandle,
                               const void **modelData,
                               int32_t *modelDataLengths,
                               int32_t modelDataCount);

Completes the creation and initialization of packedDNNHandle from the memory. The caller can use the returned packedDNNHandle across functions and threads.

  • Parameter

    • [out] packedDNNHandle D-Robotics DNN handle, pointing to multiple models.

    • [in] modelData Pointer to the model file.

    • [in] modelDataLengths Length of the model data.

    • [in] modelDataCount Number of the model data.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.2.3. hbDNNRelease()

int32_t hbDNNRelease(hbPackedDNNHandle_t packedDNNHandle);

Releases the model that packedDNNHandle points to.

  • Parameter

    • [in] packedDNNHandle D-Robotics DNN handle, pointing to multiple models.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3. Model Info

9.2.2.3.1. hbDNNGetModelNameList()

int32_t hbDNNGetModelNameList(char const ***modelNameList,
                              int32_t *modelNameCount,
                              hbPackedDNNHandle_t packedDNNHandle);

Gets the name list and number of the models that packedDNNHandle points to.

  • Parameter

    • [out] modelNameList List of model names.

    • [out] modelNameCount Number of model names.

    • [in] packedDNNHandle D-Robotics DNN handle, pointing to multiple models.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3.2. hbDNNGetModelHandle()

int32_t hbDNNGetModelHandle(hbDNNHandle_t *dnnHandle,
                            hbPackedDNNHandle_t packedDNNHandle,
                            char const *modelName);

Get the handle of a model from the model list that packedDNNHandle points to. The caller can use the returned dnnHandle across functions and threads.

  • Parameter

    • [out] dnnHandle DNN handle, pointing to one model.

    • [in] packedDNNHandle DNN handle, pointing to multiple models.

    • [in] modelName Model name.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3.3. hbDNNGetInputCount()

int32_t hbDNNGetInputCount(int32_t *inputCount,
                           hbDNNHandle_t dnnHandle);

Gets the number of the input tensors of the model that dnnHandle points to.

  • Parameter

    • [out] inputCount Number of input tensors of the model.

    • [in] dnnHandle DNN handle, pointing to one model.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3.4. hbDNNGetInputName()

int32_t hbDNNGetInputName(char const **name,
                          hbDNNHandle_t dnnHandle,
                          int32_t inputIndex);

Gets the name of the input tensor of the model that dnnHandle points to.

  • Parameter

    • [out] name Name of the input tensor of the model.

    • [in] dnnHandle DNN handle, pointing to one model.

    • [in] inputIndex Index of the input tensor of the model.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3.5. hbDNNGetInputTensorProperties()

int32_t hbDNNGetInputTensorProperties(hbDNNTensorProperties *properties,
                                      hbDNNHandle_t dnnHandle,
                                      int32_t inputIndex);

Gets the property of the specific input tensor of the model that dnnHandle points to.

  • Parameter

    • [out] properties Info of the input tensor.

    • [in] dnnHandle DNN handle, pointing to one model.

    • [in] inputIndex Index of the input tensor of the model.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3.6. hbDNNGetOutputCount()

int32_t hbDNNGetOutputCount(int32_t *outputCount,
                            hbDNNHandle_t dnnHandle);

Gets the number of the output tensors of the model that dnnHandle points to.

  • Parameter

    • [out] outputCount Number of the output tensors of the model.

    • [in] dnnHandle DNN handle, pointing to one model.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3.7. hbDNNGetOutputName()

int32_t hbDNNGetOutputName(char const **name,
                           hbDNNHandle_t dnnHandle,
                           int32_t outputIndex);

Gets the names of the output tensors of the model that dnnHandle points to.

  • Parameter

    • [out] name Name of the output tensor of the model.

    • [in] dnnHandle DNN handle, pointing to one model.

    • [in] outputIndex Index of the output tensor of the model.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.3.8. hbDNNGetOutputTensorProperties()

int32_t hbDNNGetOutputTensorProperties(hbDNNTensorProperties *properties,
                                       hbDNNHandle_t dnnHandle,
                                       int32_t outputIndex);

Gets the property of the specific output tensor of the model that dnnHandle points to.

  • Parameter

    • [out] properties Info of the output tensor.

    • [in] dnnHandle DNN handle, pointing to one model.

    • [in] outputIndex Index of the output tensor of the model.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.4. Model Inference

9.2.2.4.1. hbDNNInfer()

int32_t hbDNNInfer(hbDNNTaskHandle_t *taskHandle,
                   hbDNNTensor **output,
                   hbDNNTensor const *input,
                   hbDNNHandle_t dnnHandle,
                   hbDNNInferCtrlParam *inferCtrlParam);

Performs the inference task based on the input parameters. The caller can use the returned taskHandle across functions and threads.

  • Parameter

    • [out] taskHandle Task handle pointer.

    • [in/out] output Output of the inference task.

    • [in] input Input of the inference task.

    • [in] dnnHandle DNN handle pointer.

    • [in] inferCtrlParam Parameter of the control inference task.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

Note

Before submitting tasks using this interface, the taskHandle should be set to nullptr, unless it is a task to be appended to the specified taskHandle (i.e., using the inferCtrlParam::more function).

Up to 32 coexisting model tasks are supported.

For batch models, it is allowed to set the memory addresses of the input tensors separately. For example: the input validShape/alignedShape of the model is [4, 3, 224, 224], you can apply for four hbDNNTensors, and the validShape/alignedShape of each hbDNNTensor is set to [1, 3, 224, 224], storing the data of each batch. When the model has multiple inputs, the order of input should be input0[batch0], input0[batch1], …, inputn[batch0], inputn[batch1], …

9.2.2.4.2. hbDNNRoiInfer()

int32_t hbDNNRoiInfer(hbDNNTaskHandle_t *taskHandle,
                      hbDNNTensor **output,
                      hbDNNTensor const *input,
                      hbDNNRoi *rois,
                      int32_t roiCount,
                      hbDNNHandle_t dnnHandle,
                      hbDNNInferCtrlParam *inferCtrlParam);

Perform the ROI inference task based on the input parameter. The caller can use the returned taskHandle across functions and threads.

  • Parameter

    • [out] taskHandle Task handle pointer.

    • [in/out] output Output of the inference task.

    • [in] input Input of the inference task.

    • [in] rois ROI box info.

    • [in] roiCount Number of ROI boxes.

    • [in] dnnHandle DNN handle pointer.

    • [in] inferCtrlParam Parameter of the control inference task.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

Note

Concept Description:

  • input_count : model input count.

  • output_count : model output count.

  • resizer_count : input count of resizer input source in the model input count, one batch data for a resizer input should have one roi.

  • roi_count : the number of all rois, its value should be greater than or equal to resizer_count.

  • data_batch : the number of data batches that need to be inferred, with each resizer input source corresponding to an roi, which equals to roi_num / resizer_count.

  • model_batch : represents the number of batches processed by BPU in a single process, user can compile the model into a batch model.

  • tensor_alignedByteSize : retrieve the properties information of the input and output tensor from the model information, representing the required memory size of the tensor.

Input Description:

  • input tensor nums: input_count * data_batch.

  • input tensor order: roi_0_tensor_0, roi_0_tensor_1, …, roi_0_tensor_n, roi_1_tensor_0, roi_1_tensor_1, …, roi_1_tensor_n, … .

  • when model_batch > 1 , prepare a single batch tensor for inference, and each batch needs to be given an independent tensor address. Such as: 4x3x224x224(resizer)、4x80x1x100(ddr), prepare a batch with independent address tensor input (not required to be a multiple of 4) according to the number of batches to be processed, when batch==5, the input tensors is as follows: [1x3x224x224, 1x80x1x100, 1x3x224x224, 1x80x1x100, 1x3x224x224, 1x80x1x100, 1x3x224x224, 1x80x1x100, 1x3x224x224, 1x80x1x100], given 10 tensors in order, and BPU will process 4 batches 2 times model_batch , and missing inputs will be automatically skipped.

Output Description:

  • output tensor nums: output_count

  • output tensor memory layout: tensor_0[roi_0, roi_1, …], …, tensor_n[roi_0, roi_1, …].

  • when model_batch = 1 , output tensor memory size should be data_batch * tensor_alignedByteSize .

  • when model_batch > 1 , output tensor memory size should be ceil[(float)(data_batch) / (float)(model_batch)] * tensor_alignedByteSize, such as: data_batch = 5 , model_batch = 4 , and the output memory size should be 2 * tensor_alignedByteSize.

Index Description:

  • Prepare the input parameter input - the subscript range of the input array corresponding to the i-th data_batch is math:[i * input_count, \((i + 1 ) * input\_count), i=[0,data\_batch)\).

  • Prepare the input parameter rois: the input of each resizer input source should match a ROI, and the rois array subscript range corresponding to the i-th data_batch is math:[i * resizer_count, \((i + 1) * resizer\_count), i=[0,data\_batch)\). The ROI order of each batch should be consistent with the input order.

Restriction Description:

  • About the data_batch limit: it should be in the range [1, 255].

  • Before submitting tasks using this interface, the taskHandle should be set to nullptr, unless it is a task to be appended to the specified taskHandle (i.e., using the inferCtrlParam::more function).

  • roi size requirements are \(2 <= width <= 4096\), \(2 <= height <= 4096\).

  • The size requirements of the original image are \(1 <= W <= 4096\), \(16 <= stride <= 131072\), stride must be a multiple of 16.

  • The output size requirements are \(2 <= Wout\), \(2 <= Hout\).

  • The roi scale limits \(0 <= step <= 262143\), step calculate formula \(step = ((src\_len - 1)*65536 + (dst\_len - 1)/2)/(dst\_len - 1)\), src_len is roi’s W or H, dst_len is model’s W or H.

  • Up to 32 coexisting model tasks are supported.

9.2.2.4.3. hbDNNWaitTaskDone()

int32_t hbDNNWaitTaskDone(hbDNNTaskHandle_t taskHandle,
                          int32_t timeout);

Waits for the task to complete or timeout.

  • Parameter

    • [in] taskHandle Task handle pointer.

    • [in] timeout Time-out setting (unit: ms).

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

Note

  1. timeout > 0: Waiting time.

  2. timeout <= 0: Waits until the task is completed.

9.2.2.4.4. hbDNNReleaseTask()

int32_t hbDNNReleaseTask(hbDNNTaskHandle_t taskHandle);

Releases the task. If the task is not executed, then it will be cancelled and released directly; or if it is already in execution, it will be cancelled and released after reaching certain nodes.

  • Parameter

    • [in] taskHandle Task handle pointer.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.4.5. hbDNNSetTaskDoneCb()

int32_t hbDNNSetTaskDoneCb(hbDNNTaskHandle_t taskHandle, hbDNNTaskDoneCb cb,
                           void *userdata);

Registrs a callback function that needs to be executed after task is completed.

  • Parameter

    • [in] taskHandle Task handle pointer.

    • [in] cb Callback function pointer.

    • [in] userData user-defined data.

  • Return value

    • If 0 is returned, the callback function is registered successfully, otherwise it fails.

Note

This interface can register a callback function, which will be called to execute user-defined functions after the task execution is completed. If no custom input is required, userdata can be set to nullptr.

9.2.2.5. Memory Operations

9.2.2.5.1. hbSysAllocMem()

int32_t hbSysAllocMem(hbSysMem *mem, uint32_t size);

Requests BPU memory.

  • Parameter

    • [in] size Size of allocated memory.

    • [out] mem Memory pointer.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.5.2. hbSysAllocCachedMem()

int32_t hbSysAllocCachedMem(hbSysMem *mem, uint32_t size);

Requests cached BPU memory.

  • Parameter

    • [in] size Size of allocated memory.

    • [out] mem Memory pointer.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.5.3. hbSysFlushMem()

int32_t hbSysFlushMem(hbSysMem *mem, int32_t flag);

Refreshes the cached BPU memory.

  • Parameter

    • [in] mem Memory pointer.

    • [in] flag Refreshes the flag.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.5.4. hbSysFreeMem()

int32_t hbSysFreeMem(hbSysMem *mem);

Releases BPU memory.

  • Parameter

    • [in] mem Memory pointer.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.5.5. hbSysWriteMem()

int32_t hbSysWriteMem(hbSysMem *dest, char *src, uint32_t size);

Writes to BPU memory.

  • Parameter

    • [out] dest Memory pointer.

    • [in] src Data pointer.

    • [in] size Data size.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.5.6. hbSysReadMem()

int32_t hbSysReadMem(char *dest, hbSysMem *src, uint32_t size);

Reads BPU memory.

  • Parameter

    • [out] dest Data pointer.

    • [in] src Memory pointer.

    • [in] size Data size.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.5.7. hbSysRegisterMem()

int32_t hbSysRegisterMem(hbSysMem *mem);

Register known memory location of physical address as a BPU accessible memory flag, the obtained memory is cacheable.

  • Parameter

    • [in/out] mem Memory pointer.

  • Return

    • If returns 0, then API execution is successful. Otherwise API execution has failed.

9.2.2.5.8. hbSysUnregisterMem()

int32_t hbSysUnregisterMem(hbSysMem *mem);

Cancel the hbSysRegisterMem registered memory flag.

  • Parameter

    • [in] mem Memory pointer.

  • Return

    • If returns 0, then API execution is successful. Otherwise API execution has failed.

9.2.2.6. Plug-ins

9.2.2.6.1. hbDNNRegisterLayerCreator()

int32_t hbDNNRegisterLayerCreator(char const *layerType,
                                  hbDNNLayerCreator layerCreator);

Creation method of the registered layer.

  • Parameter

    • [in] layerType Layer type.

    • [in] layerCreator Creation method of the layer.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.6.2. hbDNNUnregisterLayerCreator()

int32_t hbDNNUnregisterLayerCreator(char const *layerType);

Unregisters the layer.

  • Parameter

    • [in] layerType Layer type.

  • Return value

    • If 0 is returned, the API is executed successfully, otherwise it fails.

9.2.2.7. Status code

9.2.2.7.1. hbDNNGetErrorDesc()

char const *hbDNNGetErrorDesc(int32_t errorCode);

Translate error codes into natural language.

  • Parameter

    • [in] errorCode dnn error status code.

  • Return value

    • return char * , translate internal error codes into natural language.

9.2.3. Data Layout and Alignment Rules

9.2.3.1. Data Layout

To improve the efficiency of hardware computation, special data layout is adopted, making the feature map and kernel used in the same multiply-add batch in the convolution calculation juxtaposed next to each other in the memory. Here we briefly explain the data layout concept in X5.

The variables in a neural network model can be represented by a 4-dimensional tensor, where each number is an element of this tensor, the natural layout as we call it. We arrange different elements of different dimensions closely together as per certain rules to form a separate block. If we consider these blocks as new elements, then we get a new 4-dimensional tensor. We call it a tensor with data layout.

The input and output data use different data layouts, you can get the layout information by using APIGetlayout. Different layout data cannot be directly compared with each other.

Note

When transforming the data layout, the padding value is recommended to be set to 0.

Here we introduce two types of data layouts: NHWC_NATIVE and NCHW_NATIVE. Take NHWC_NATIVE as an example, its data layout is listed below:

N0H0W0C0

N0H0W0C1

……

N0H0W1C0

N0H0W1C1

……

……

……

……

N0H1W0C0

N0H1W0C1

……

……

……

……

N1H0W0C0

N1H0W0C1

……

……

……

……

A tensor of the size N*H*W*C can be represented by the following 4-fold cycles:

for (int32_t n = 0; n < N; n++) {
    for (int32_t h = 0; h < H; h++) {
        for (int32_t w = 0; w < W; w++) {
            for (int32_t c = 0; c < C; c++) {
                int32_t native_offset = n*H*W*C + h*W*C + w*C + c;
            }
        }
    }
}

Where, NCHW_NATIVE and NHWC_NATIVE are the same except the layout loop order, which we do not list separately here.

Attention

All native occurrences in the next sections refer specifically to this layout.

9.2.3.2. BPU Alignment Restriction Rules

This section describes the alignment restriction rules when using BPU.

9.2.3.2.1. Model Input Requirements

BPU does not restrict the model input size or parity. Both 416x416 inputs (e.g., YOLO) and 227x227 inputs (e.g., SqueezeNet) can be supported.

For NV12, Both H & W of the input are required to be even to meet the requirement that the UV is half of the Y.

9.2.3.2.2. Alignment and Efficient Data

The BPU has alignment requirements for data. Alignment requirements and actual data arrangement are represented by validShape, alignedShape, and stride in hbDNNTensorProperties.

  • validShape: Valid shape.

  • alignedShape: Shape after alignment. Because of hardware characteristics, alignedShape is represented by four-dimensional data.

  • stride: Step size of each dimension of validshape. The model imported by NV12 is special, and its stride is all 0, because the model imported by NV12 only requires W 16 alignment.

Tensors equal to four dimensions can obtain correct arrangement of data through validShape and alignedShape, while tensors larger than four dimensions can obtain the correct data arrangement through validShape and stride.

In subsequent scenarios, considering the alignment requirements, it is recommended to apply for memory space according to the size of alignedbytesize.

9.2.3.3. NV12 Introduction

9.2.3.3.1. YUV format

The YUV format is mainly used to optimize the transmission of color video signals.

YUV has three components: Y, U, and V, where Y for brightness, i.e., grayscale value, and U and V for chroma, used to describe the image color and saturation and specify the pixel color.

9.2.3.3.2. NV12 layout

NV12 belongs to the YUV420SP image format in the YUV color space, where every four Y components share the same set of U and V components, with Y in sequential order and U and V in cross order.

The arrangement is as follows:

../../../_images/nv12_layout.png

9.2.4. Error Code

HB_DNN_SUCCESS = 0                              // Execution successful
HB_DNN_INVALID_ARGUMENT = -6000001              // Invalid parameter
HB_DNN_INVALID_MODEL = -6000002                 // Invalid model
HB_DNN_MODEL_NUMBER_EXCEED_LIMIT = -6000003     // Number of models exceeds the limit
HB_DNN_INVALID_PACKED_DNN_HANDLE = -6000004     // Invalid packed handle
HB_DNN_INVALID_DNN_HANDLE = -6000005            // Invalid handle
HB_DNN_CAN_NOT_OPEN_FILE = -6000006             // File not exist
HB_DNN_OUT_OF_MEMORY = -6000007                 // Memory not enough
HB_DNN_TIMEOUT = -6000008                       // Time out
HB_DNN_TASK_NUM_EXCEED_LIMIT = -6000009         // Number of tasks exceeds the limit
HB_DNN_TASK_BATCH_SIZE_EXCEED_LIMIT = -6000010  // Number of batch task exceeds the limit
HB_DNN_INVALID_TASK_HANDLE = -6000011           // Invalid task handle
HB_DNN_RUN_TASK_FAILED = -6000012               // Task execution failed
HB_DNN_MODEL_IS_RUNNING = -6000013              // Task in progress
HB_DNN_INCOMPATIBLE_MODEL = -6000014            // Incompatible models
HB_DNN_API_USE_ERROR = -6000015                 // API usage error
HB_DNN_MULTI_PROGRESS_USE_ERROR = -6000016      // Multi progress usr error

HB_SYS_SUCCESS = 0                              // Execution successful
HB_SYS_INVALID_ARGUMENT = -6000129              // Invalid parameter
HB_SYS_OUT_OF_MEMORY = -6000130                 // Memory not enough
HB_SYS_REGISTER_MEM_FAILED = -6000131           // Failed to register memory

9.2.5. Configuration Info

9.2.5.1. Common Environment Variables

HB_DNN_LOG_LEVEL                // Sets the log level.
HB_DNN_PLUGIN_PATH              // Directory to the dynamic link library of the custom CPU operator.
HB_DNN_PROFILER_LOG_PATH        // Dump path of time-consuming statistics at each stage of model running.
HB_DNN_SIM_PLATFORM             // X86 emulator simulation platform settings.
HB_DNN_SIM_BPU_MEM_SIZE         // X86 emulator sets BPU memory size, in MB.

9.2.5.2. Log Level Setting Instructions

  1. Log level. Logs in dnn fall into four levels:

    • HB_DNN_LOG_NONE = 0: No log is generated;

    • HB_DNN_LOG_WARNING = 3: Outputs alarm messages in the code;

    • HB_DNN_LOG_ERROR = 4: Outputs error messages in the code;

    • HB_DNN_LOG_FATAL = 5: Outputs the error messages in the code that cause exit.

  2. Log level setting rules.

    • If the occurring log level >= the set level, the log can be printed; otherwise, it is blocked.

    • Smaller log levels print more information (except level 0, which means generating no logs). For example: If the log level is 3, i.e., the WARNING level, then logs of level 3, 4, and 5 can all be printed. By default, the log level of the prediction library is HB_DNN_LOG_WARNING, meaning that the following levels of logs can be printed: WARNING, ERROR, and FATAL.

9.2.5.3. Platform Configuration Instructions with x86 Emulator Simulation

  1. The x86 emulator simulates the different processor architectures of D-Robotics by different options, as follows:

    • export HB_DNN_SIM_PLATFORM=BERNOULLI: Simulates D-Robotics xj2 platform.

    • export HB_DNN_SIM_PLATFORM=BERNOULLI2: Simulates D-Robotics xj3 platform.

    • export HB_DNN_SIM_PLATFORM=BAYES: Simulates D-Robotics j5 platform.

    • export HB_DNN_SIM_PLATFORM=BAYESE: Simulates D-Robotics x5 platform.

  2. If you have not set the environment variable HB_DNN_SIM_PLATFORM, the simulator platform will be set according to the model architecture loaded for the first time. For example, if the first loaded model is BAYESE, the program will set the platform to x5 by default.