4.2.4.2. D-Robotics Operator API

horizon_plugin_pytorch.nn.functional.filter(*inputs, threshold, idx_range=None)

Filter.

The output order is different with bpu, because that the compiler do some optimization and slice input following complex rules, which is hard to be done by plugin.

All inputs are filtered along HW by the max value within a range in channel dim of the first input. Each NCHW input is splited, transposed and flattened to List[Tensor[H * W, C]] first. If input is QTensor, the output will be dequantized.

Parameters

  • inputs – Data in NCHW format. Each input shold have the same size in N, H, W. The output will be selected according to the first input.

  • threshold (float) – Threshold, the lower bound of output.

  • idx_range (tuple) – The index range of values counted in compare of the first input. Defaults to None which means use all the values.

Returns

A list with same length of batch size, and each element contains:

  • max_value: Flattened max value within idx_range in channel dim.

  • max_idx: Flattened max value index in channel dim.

  • coord: The original coordinates of the output data in the input data in the shape of [M, (h, w)].

  • (multi) data: Filtered data in the shape of [M, C].

Return type

list

horizon_plugin_pytorch.nn.functional.point_pillars_preprocess(points_list, pc_range, voxel_size, max_voxels, max_points_per_voxel, use_max, norm_range, norm_dims)

Preprocess PointPillars.

Parameters

  • points_list (list) – [(M1, ndim), (M2, ndim), ...], List of PointCloud data.

  • pc_range – (6,), indicate voxel range, format: [x_min, y_min, z_min, x_max, y_max, z_max].

  • voxel_size – (3,), xyz, indicate voxel size.

  • max_voxels (int) – Indicate maximum voxels.

  • max_points_per_voxel (int) – Indicate maximum points contained in a voxel.

  • use_max (bool) – Whether to use max_voxels, for deploy should be True.

  • norm_range – Feature range, like [x_min, y_min, z_min, ..., x_max, y_max, z_max, ...].

  • norm_dims – Dims to do normalize.

Returns

(features, coords), encoded feature and coordinates in (idx, z, y, x) format.

Return type

tuple

class horizon_plugin_pytorch.nn.detection_post_process.DetectionPostProcess(score_threshold: int = 0, regression_scale=None, background_class_idx=None, size_threshold=None, image_size=None, pre_decode_top_n=None, post_decode_top_n=None, iou_threshold=None, pre_nms_top_n=None, post_nms_top_n=None, nms_on_each_level: bool = False, mode: str = 'normal')

General post process for object detection models.

Compatible with YOLO, SSD, RetinaNet, Faster-RCNN (RPN & RCNN), etc. Note that this is a float OP, please use after DequantStubs.

Parameters

  • score_threshold (int) – Filter boxes whose score is lower than this. Defaults to 0.

  • regression_scale (tuple) – Scale to be multiplyed to box regressions. Defaults to None.

  • background_class_idx (int) – Specify the class index to be ignored. Defaults to None.

  • size_threshold (float) – Filter bixes whose height or width smaller than this. Defaults to None.

  • image_size (tuple) – Clip boxes to image sizes. Defaults to None.

  • pre_decode_top_n (int) – Get top n boxes by objectness (first element in the score vector) before decode. Defaults to None.

  • post_decode_top_n (int) – Get top n boxes by score after decode. Defaults to None.

  • iou_threshold (float) – IoU threshold for nms. Defaults to None.

  • pre_nms_top_n (int) – Get top n boxes by score before nms. Defaults to None.

  • post_nms_top_n (int) – Get top n boxes by score after nms. Defaults to None.

  • nms_on_each_level (bool) – Whether do nms on each level seperately. Defaults to False.

  • mode (str) –

    Only support ‘normal’ and ‘yolo’. If set to ‘yolo’:

    1. Box will be filtered by objectness rathen than classification scores.

    2. dx, dy in regressions will be treated as absolute offset.

    3. Objectness will be multiplyed to classification scores.

    Defaults to ‘normal’.

forward(boxes, scores, regressions, image_shapes=None)

Forward pass of DetectionPostProcess.

class horizon_plugin_pytorch.nn.bgr_to_yuv444.BgrToYuv444(channel_reversal: bool = False)

Convert image color format from bgr to yuv444.

Parameters

channel_reversal (bool) – Color channel order, set to True when used on RGB input. Defaults to False.

forward(input)

Forward pass of BgrToYuv444.

class horizon_plugin_pytorch.nn.detection_post_process_v1.DetectionPostProcessV1(num_classes, box_filter_threshold, class_offsets, use_clippings, image_size, nms_threshold, pre_nms_top_k, post_nms_top_k, nms_padding_mode=None, nms_margin=0.0, use_stable_sort=None, bbox_min_hw=(0, 0))

Post process for object detection models. Only supported on bernoulli2.

This operation is implemented on BPU, thus is expected to be faster than cpu implementation. This operation requires input_scale = 1 / 2 ** 4, or a rescale will be applied to the input data. So you can manually set the output scale of previous op (Conv2d for example) to 1 / 2 ** 4 to avoid the rescale and get best performance and accuracy.

Major differences with DetectionPostProcess:

  1. Each anchor will generate only one pred bbox totally, but in DetectionPostProcess each anchor will generate one bbox for each class (num_classes bboxes totally).

  2. NMS has a margin param, box2 will only be supressed by box1 when box1.score - box2.score > margin (box1.score > box2.score in DetectionPostProcess).

  3. A offset can be added to the output class indices (using class_offsets).

Parameters

  • num_classes (int) – Class number.

  • box_filter_threshold (float) – Default threshold to filter box by max score.

  • class_offsets (list) – Offset to be added to output class index for each branch.

  • use_clippings (bool) – Whether clip box to image size. If input is padded, you can clip box to real content by providing image size.

  • image_size (tuple) – Fixed image size in (h, w), set to None if input have different sizes.

  • nms_threshold (float) – IoU threshold for nms.

  • pre_nms_top_k (int) – Maximum number of bounding boxes in each image before nms.

  • post_nms_top_k (int) – Maximum number of output bounding boxes in each image.

  • nms_padding_mode (str) – The way to pad bbox to match the number of output bounding bouxes to post_nms_top_k, can be None, pad_zero or rollover.

  • nms_margin (float) – Only supress box2 when box1.score - box2.score > nms_margin.

  • use_stable_sort (bool) – Whether use stable sort.

  • bbox_min_hw (tuple) – Minimum height and width of selected bounding boxes.

forward(data, anchors, image_sizes=None)

Forward pass of DetectionPostProcessV1.

Parameters

  • data – (N, (4 + num_classes) * anchor_num, H, W)

  • anchors – (N, anchor_num * 4, H, W)

  • image_sizes – Defaults to None.

Returns

List of (bbox (x1, y1, x2, y2), score, class_idx).

Return type

list

horizon_plugin_pytorch.functional.centered_yuv2bgr(input, swing='studio', mean=(128.0,), std=(128.0,), q_scale=0.0078125)

Convert color space.

Convert images from centered YUV444 BT.601 format to transformed and quantized BGR. Only use this operator in the quantized model. Insert it after QuantStub. Pass the scale of QuantStub to the q_scale argument and set scale of QuantStub to 1 afterwards.

Parameters

  • input – Input images in centered YUV444 BT.601 format, centered by the pyramid with -128.

  • swing (str) – studio for YUV studio swing (Y: -112~107, U, V: -112~112). full for YUV full swing (Y, U, V: -128~127). Default is studio.

  • mean – BGR mean, a list of float, or torch.Tensor, can be a scalar [float], or [float, float, float] for per-channel mean.

  • std – BGR standard deviation, a list of float, or torch.Tensor, can be a scalar [float], or [float, float, float] for per-channel std.

  • q_scale – BGR quantization scale.

Returns

Transformed and quantized image in BGR color, dtype is qint8.

Return type

QTensor

horizon_plugin_pytorch.functional.centered_yuv2rgb(input, swing='studio', mean=(128.0,), std=(128.0,), q_scale=0.0078125)

Convert color space.

Convert images from centered YUV444 BT.601 format to transformed and quantized RGB. Only use this operator in the quantized model. Insert it after QuantStub. Pass the scale of QuantStub to the q_scale argument and set scale of QuantStub to 1 afterwards.

Parameters

  • input – Input images in centered YUV444 BT.601 format, centered by the pyramid with -128.

  • swing (str) – studio for YUV studio swing (Y: -112~107, U, V: -112~112). full for YUV full swing (Y, U, V: -128~127). Default is studio.

  • mean – RGB mean, a list of float, or torch.Tensor, can be a scalar [float], or [float, float, float] for per-channel mean.

  • std – RGB standard deviation, a list of float, or torch.Tensor, can be a scalar [float], or [float, float, float] for per-channel std.

  • q_scale – RGB quantization scale.

Returns

Transformed and quantized image in RGB color, dtype is qint8.

Return type

QTensor

horizon_plugin_pytorch.functional.quantized_conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1, padding_mode='zeros')

Quantized 2D convolution.

Functional quantized conv2d aligned with torch.nn.functional.conv2d.

Parameters

  • input – Input tensor or QTensor of shape (N, C, H, W).

  • weight – Convolution weight.

  • bias – Optional bias.

  • stride – Same meaning as torch.nn.functional.conv2d.

  • padding – Same meaning as torch.nn.functional.conv2d.

  • dilation – Same meaning as torch.nn.functional.conv2d.

  • groups – Same meaning as torch.nn.functional.conv2d.

  • padding_mode (str) – Padding mode.

Returns

Convolution output.

Return type

Tensor or QTensor