5.1. Embedded Application Development¶

5.1.1. General Descriptions¶

This chapter describes how to develop applications, how to deploy and run converted models in D-Robotics’ platform and some matters need attention.

Attention

Prior to application development, please be sure that you’ve completed the development environment preparations as described in the Environment Deployment.

The simplest application development can be divided into 3 stages: project creation, project implementation and operation. However, given the fact that the development of actual business scenarios are more complicated, here we’d like to offer more instructions about the concept of multi-model control and suggestions on application tuning.

5.1.2. Create a New Project¶

It is recommended by D-Robotics to manage your application development engineering using CMake. As described in the Prerequisites chapter, by now, you should have installed CMake. Before reading this section, you’re expected to understand how to use CMake.

D-Robotics’ development library provides arm architecture based dependency environment and deb board application programs. Engineering dependency information for arm programs are listed as follows:

D-Robotics evaluation library libdnn.so in the: ~/.horizon/ddk/xj3_aarch64/dnn/lib/ directory.
D-Robotics compiler dependency libhbrt_bernoulli_aarch64.so in the: ~/.horizon/ddk/xj3_aarch64/dnn/lib/ directory.
The aarch64-linux-gnu-gcc C compiler.
The aarch64-linux-gnu-g++ C++ compiler.

To create a new project, users need to compile the CMakeLists.txt file. The script defines the path to compiler tool, the CMakeLists.txt file defines the paths to some compilation options, dependency libs and header files. Refer to below code block:

cmake_minimum_required(VERSION 2.8)

project(your_project_name)

# libdnn.so depends on system software dynamic link library, use -Wl,-unresolved-symbols=ignore-in-shared-libs to shield during compilation
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -Wl,-unresolved-symbols=ignore-in-shared-libs")

set(CMAKE_CXX_FLAGS_DEBUG " -Wall -Werror -g -O0 ")
set(CMAKE_C_FLAGS_DEBUG " -Wall -Werror -g -O0 ")
set(CMAKE_CXX_FLAGS_RELEASE " -Wall -Werror -O3 ")
set(CMAKE_C_FLAGS_RELEASE " -Wall -Werror -O3 ")

if (NOT CMAKE_BUILD_TYPE)
    set(CMAKE_BUILD_TYPE Release)
endif ()

message(STATUS "Build type: ${CMAKE_BUILD_TYPE}")

# define dnn lib path
set(DNN_PATH "~/.horizon/ddk/xj3_aarch64/dnn/")

set(DNN_LIB_PATH ${DNN_PATH}/lib)

include_directories(${DNN_PATH}/include)
link_directories(${DNN_LIB_PATH})

add_executable(user_app main.cc)
target_link_libraries(user_app
                      dnn
                      pthread
                      rt
                      dl)

Note

In the above sample, the compiler’s postion was not specified and is to be specified at the project compilation stage. Please refer to the descriptions in the: Compile And Run The Project section.

5.1.3. Implement the Project¶

This section explains how to run the aforementioned converted bin models in D-Robotics’ platform. The simplest procedure should cover: model loading, input data preparations, output memory preparations, inference and result parsing. Please refer to below model loading and deploying sample code:

#include <iostream>

#include "dnn/hb_dnn.h"
#include "dnn/hb_sys.h"

int main(int argc, char **argv) {
  // Step 1: load the model
  hbPackedDNNHandle_t packed_dnn_handle;
  const char* model_file_name= "./mobilenetv1.bin";
  hbDNNInitializeFromFiles(&packed_dnn_handle, &model_file_name, 1);

  // Step 2: obtain model names
  const char **model_name_list;
  int model_count = 0;
  hbDNNGetModelNameList(&model_name_list, &model_count, packed_dnn_handle);

  // Step 3: obtain dnn_handle
  hbDNNHandle_t dnn_handle;
  hbDNNGetModelHandle(&dnn_handle, packed_dnn_handle, model_name_list[0]);

  // Step 4: prepare input data
  hbDNNTensor input;
  hbDNNTensorProperties input_properties;
  hbDNNGetInputTensorProperties(&input_properties, dnn_handle, 0);
  input.properties = input_properties;
  auto &mem = input.sysMem[0];

  int yuv_length = 224 * 224 * 3;
  hbSysAllocCachedMem(&mem, yuv_length);
  //memcpy(mem.virAddr, yuv_data, yuv_length);
  //hbSysFlushMem(&mem, HB_SYS_MEM_CACHE_CLEAN);

  // Step 5: prepare space for model output data
  int output_count;
  hbDNNGetOutputCount(&output_count, dnn_handle);
  hbDNNTensor *output = new hbDNNTensor[output_count];
  for (int i = 0; i < output_count; i++) {
  hbDNNTensorProperties &output_properties = output[i].properties;
  hbDNNGetOutputTensorProperties(&output_properties, dnn_handle, i);

  // Obtain model output size
  int out_aligned_size = 4;
  for (int j = 0; j < output_properties.alignedShape.numDimensions; j++) {
    out_aligned_size =
        out_aligned_size * output_properties.alignedShape.dimensionSize[j];
  }

  hbSysMem &mem = output[i].sysMem[0];
  hbSysAllocCachedMem(&mem, out_aligned_size);
}

  // Step 6: inference
  hbDNNTaskHandle_t task_handle = nullptr;
  hbDNNInferCtrlParam infer_ctrl_param;
  HB_DNN_INITIALIZE_INFER_CTRL_PARAM(&infer_ctrl_param);
  hbDNNInfer(&task_handle,
              &output,
              &input,
              dnn_handle,
              &infer_ctrl_param);

  // Step 7: wait until the end of the task
  hbDNNWaitTaskDone(task_handle, 0);
  // Step 8: parse model output, the sample is to obtain TOP1 class of MobileNetv1
  float max_prob = -1.0;
  int max_prob_type_id = 0;
  hbSysFlushMem(&(output->sysMem[0]), HB_SYS_MEM_CACHE_INVALIDATE);
  float *scores = reinterpret_cast<float *>(output->sysMem[0].virAddr);
  int *shape = output->properties.validShape.dimensionSize;
  for (auto i = 0; i < shape[1] * shape[2] * shape[3]; i++) {
    if(scores[i] < max_prob)
      continue;
    max_prob = scores[i];
    max_prob_type_id = i;
  }

  std::cout << "max id: " << max_prob_type_id << std::endl;
  // Free data
  hbSysFreeMem(&(input.sysMem[0]));
  hbSysFreeMem(&(output->sysMem[0]));

  // Free the model
  hbDNNRelease(packed_dnn_handle);

  return 0;
}

To keep it simple, some data in above sample use known constants. But in development, you should obtain size and data type using the hbDNNGetInputTensorProperties/hbDNNGetOutputTensorProperties etc. interfaces.

Note

At the input data preparations stage, a snippet of the memcpy code is commented out at input data preparation stage. This snippet refers to the step to prepare input sample based on model’s input format and copy it into the input.sysMem[0]. The aforementioned input_type_rt and input_layout_rt parameters jointly determine the types of model input. Please refer to the descriptions in the: Model Conversion Interpretation section for more information.

More comprehensive engineering guidance please refer to the BPU SDK API DOC section.

5.1.4. Compile and Run the Project¶

You need to specify the environment variable ‘LINARO-GCC-ROOT’ based on the GCC environment of the target platform to obtain the correct cross compilation tool to perform compilation, for example:

gcc9.3

export LINARO_GCC_ROOT=/opt/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu

gcc6.5

export LINARO_GCC_ROOT=/opt/gcc-linaro-6.5.0-2018.12-x86_64_aarch64-linux-gnu

Note

The setting path of the environment variable ‘LINARO-GCP_ROOT’ needs to be the same as the directory decompressed by the cross compilation tool, otherwise it will result in failure.

Along with CMake engineering configurations as described in the Create A New Project section, please see below compilation script:

DIR=$(cd "$(dirname "$0")";pwd)

export CC=${LINARO_GCC_ROOT}/bin/aarch64-linux-gnu-gcc
export CXX=${LINARO_GCC_ROOT}/bin/aarch64-linux-gnu-g++

rm -rf build_arm
mkdir build_arm
cd build_arm

cmake ${DIR}

make -j8

Based on the descriptions in the Environment Deployment, you should’ve installed the required compiler, so here you only need to configure your compiler for your project in the above script.

Copy and run your arm programs in D-Robotics’ dev board and do not forget to copy the dependency files of the programs to the dev board, too. Then configure dependencies in the startup script. For example, dependency of our sample program is: libhbrt_bernoulli_aarch64.libdnn.so and libdnn.so. They are in the ~/.horizon/ddk/xj3_aarch64/dnn/lib/ directory and must be copied in the the operating environment in the dev board. You’re recommended to copy the libraries into the /userdata/lib directory, so that the dependency path information you should specify is as below:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/userdata/lib

5.1.5. Multi-model Control Strategy¶

5.1.5.1. Group Control Strategy¶

Resource contention is inevitable in those scenarios where multiple models exist as each model has to execute inference using limited resources. To facilitate your multi-model execution, D-Robotics presents the model preemption control strategy.

5.1.5.2. Model Preemption Control¶

There isn’t task preemption feature in the BPU computing unit hardware of the X3 ASICs. Each inference task, once entered the BPU and begins model computing, it always occupies the BPU till the end of the task. Hence other tasks have to wait in line.

This can cause the problem that the BPU computing resources are constantly occupied solely by an enormous inference task and affect the inference executions of other higher priority models. To tackle such problem, the Runtime SDK can, based on model priority, implement the BPU resource preemption feature through software.

Please pay attention to the following factors:

When executing inference in BPU, the compiled data command model are denoted by 1 or more function-call(s). The function-call refers to BPU’s atomic execution unit and multiple function-call tasks line up in BPU hardware line and are distributed sequentially. A model inference task will be considered accomplished when all of it function-calls are executed.
Based on the above descriptions, we can see that it is easier that the basic unit of the model task preemption is designed as function-call, so that when BPU finishes executing a function-call, it can suspend the existing model for now, switch to execute another model, then resume executing the previous model when the latter was executed. However there are 2 problems, the first problem is that the compiler compiled model function-calls are merged together, in other words, these is only one function-call which cannot be preempted; the second problem is that the execution time of each function-call varies, it can be either very prolonged or uncertain, and therefore makes the timing to preempt uncertain, or affect the results of preemption.

To solve the above-mentioned 2 problems, D-Robotics provides support in model conversion and system software layers. Here below describes the implement principles and way to proceed:

Firstly, at model conversion stage, specify the max_time_per_fc parameter in the compiler_parameters of YAML configuration file. This parameter is used for specifying the execution time (in microsecond) of each function-call, whose default value is 0 (i.e. no limits). Let’s assume that the execution time of a certain function-call is 10ms, so when compiling the model, by specifying the max_time_per_fc as 500, this function-call will be split into 20 function-calls.
Secondly, there is a BPLAT_CORELIMIT environment variable used for specifying the granularity of model preemption. If specified as 0, the model preemption will be disabled. Therefore, to execute higher-priority tasks ASAP, when you develop with the devboard, first run export BPLAT_CORELIMIT=1 to specify this environment variable as 1, so that when the underlying layer of the system receive function-calls, it will determine the priority and put those high priority tasks into an independent queue, thus when one function-call is executed, the higher-priority task will be able to preempt BPU.
Nextly, as the model preemption mechanism is implemented in the libdnn, continue to specify the hbDNNInferCtrlParam.priority parameter provided by the infer API of dnn as HB_DNN_PRIORITY_PREEMP(255), so that your task will become a high-priority task.

5.1.6. Suggestions on Application Optimization¶

D-Robotics suggested application optimization strategy includes 2 perspectives: engineering task scheduling and algorithm task integration.

In terms of engineering task scheduling, we recommended you to utilize some workflow scheduling management tools, in order to make full use of the parallel-processing ability at different task stages. Typical an application can be divided into 3 stages: pre-processing, model inference and output post-processing. The simplified workflow is shown as below:

To implement parallel-processing at different stages by taking full advantages of the workflow management, the ideal task processing workflow can be as shown below:

In terms of algorithm task integration, D-Robotics recommends you to utilize multi-task models. Because it can on the one hand to a certain extend avoid the difficuties brough by the management of multi-model scheduling; On the other hand, multi-task model can also share the computing volume of the backbone to the full, and compared with using single models, it can apparently decrease computing volume at the entire application level and therefore reach higher overall performance. Based on D-Robotics’ past cooperation with a large number of customers, multi-task is a frequently-used application optimization strategy.

5.1.7. Other Dev Tools¶

hrt_bin_dump is the layer dump tool for model, the output file of the tool is a binary file, how to use the tool please refer to hrt_bin_dump Tool Introduction section.
hrt_model_exec is a model execution tool that can evaluate the inference performance of the model and get the model information directly on the development board. On the one hand, it allows the user to get a realistic understanding of the model’s real performance; On the other hand, it also helps the user to understand the speed limit that the model can achieve, which is a guideline for the target limit of application tuning. hrt_model_exec provides three types of functions including model inference infer, model performance analysis perf and viewing model information model_info respectively, how to use the tool please refer to hrt_model_exec Tool Introduction section.

5.1.8. FAQ¶

5.1.8.1. What is BPU memory Cache?¶

As described in BPU SDK API DOC, BPU memory functions hbSysAllocCachedMem and hbSysAllocMem are used for allocating BPU read/write memory.

In which, a parameter called hbSysAllocCachedMem is used for allocating the cacheable memory space, and a supporting function, hbSysFlushMem, is used for refreshing the cache.

The cache mechanism is determined by the memory architecture of the BPU, as shown in the following figure. The cache between CPU and memory is used as a data cache; however, there is no cache between BPU and memory. Therefore, the misuse of the cache can cause problems in data reading/writing accuracy and efficiency.

When the CPU has finished writing data, it needs to actively flush the data in the cache to the memory, otherwise the BPU will read the old data.
When the BPU has finished writing data, it also needs to actively invalidate the data in the cache, otherwise the CPU will preferentially read the old data previously cached.
In the process of continuous model inference, we recommend applying for memory with cache for the input and output to improve the CPU efficiency in repeated reading and writing.

5.1.8.2. What is the physical/virtual address in BPU memory?¶

In the X3 processor architecture, BPU and CPU share the memory, and a physically contiguous section of memory can be requested through the hbSysAllocCachedMem and hbSysAllocMem interfaces.

The return values of these functions are wrapped in the hbSysMem data structure, and the phyAddr and virAddr fields correspond to the physical and virtual addresses of its memory space, respectively.

As this memory space is contiguous, both physical and virtual addresses can be represented, read, and written by the first address. However, in practice, it is recommended to use virtual addresses in preference in non-essential scenarios.

5.1.8.3. How to convert camera dumped NV12 images into other formats e.g. BGR etc.?¶

D-Robotics’ X3 ASICs don’t come with hardware accelerator to convert pixel space, so some customers hope to accelerate pixel space conversion using BPU via API interfaces. But to avoid BPU’s inference efficiency to be affected by such feature, after rigorous evaluations, we’ve decided not to open up the interfaces for now.

However, users can still accelerate this operation in ARM CPU using the open source libYUV lib. Throughout test, when converting 720P NV12 images into BGR, conversion latency was shortened by 7ms and can satisfy business requirements in most scenarios.