4.1.4.2. Image Processing Transformer Description

This section will explain the concept and parameters of each transformer you use when scaling and cropping images, and provide you with reference to the use of examples to make it easier for you to transformer operations.

Before you start reading the contents of the document, please note the following.

Attention

The image data is three-dimensional data, but the default first dimension of the transformer provided by D-Robotics is N-dimensional, and it will process the data with the first dimension split cycle, so if you need to do processing on the image, please provide four-dimensional data.

4.1.4.2.1. AddTransformer

Description:

This transforms add the value to all the pixel values in the input image. The transformer converts the data format to float32 on output.

Parameters:

  • value: the value to be added to each pixel, note that value can be negative, e.g. -128.

Examples of use:

# Subtract 128 from image data
``AddTransformer(-128)``

# Add 127 to the image data
``AddTransformer(127)``

4.1.4.2.2. MeanTransformer

Description:

This transforms subtract the mean_value from all pixel values in the input image.

Parameters:

  • means: The added value of each pixel, note that the value can be a negative number, e.g. -128.

  • data_format: The layout type of the input, the range of values is [“CHW”, “HWC”], the default “CHW”.

Examples of use:

# Each pixel minus 128.0 ,the input type is CHW
MeanTransformer(np.array([128.0, 128.0, 128.0]))

# Each pixel subtracts a different value: 103.94, 116.78, 123.68, the type of input is HWC
MeanTransformer(np.array([103.94, 116.78, 123.68]), data_format="HWC")

4.1.4.2.3. ScaleTransformer

Description:

This transforms do multiplication of all pixel values in the input image by the data_scale factor.

Parameters:

  • scale_value: The factor to be multiplied, such as 0.0078125 or 1/128.

Examples of use:

# Adjust all pixels in the range of -128 to 127 to between -1 and 1.
ScaleTransformer(0.0078125)
# or
ScaleTransformer(1/128)

4.1.4.2.4. NormalizeTransformer

Description:

This transformer is used to normalize the input image. The transformer converts the data format to float32 on output.

Parameters:

  • std: The value that needs to be divided for the first input image.

Examples of use:

# Adjusts all pixels in the range [-128, 127] to between -1 and 1
NormalizeTransformer(128)

4.1.4.2.5. TransposeTransformer

Description:

This transformer is used to do layout conversion operations.

Parameters:

  • order: The order of the input image after layout conversion (the order is related to the original layout order). For example: the order of HWC is 0,1,2, when you need to convert to CHW, the order is (2,0,1).

Examples of use:

# HWC to CHW
TransposeTransformer((2, 0, 1))
# CHW to HWC
TransposeTransformer((1, 2, 0))

4.1.4.2.6. HWC2CHWTransformer

Description:

This transformer is used to convert NHWC to NCHW.

Parameters: Not involving.

Examples of use:

# NHWC to NCHW
HWC2CHWTransformer()

4.1.4.2.7. CHW2HWCTransformer

Description:

This transformer is used to convert NCHW to NHWC.

Parameters: Not involving.

Examples of use:

# NCHW to NHWC
CHW2HWCTransformer()

4.1.4.2.8. CenterCropTransformer

Description:

This transformer will cut out a square image from the center of the image by directly truncating the value. This transformer will convert the data format to float32 when outputting. When the value of data_type is uint8, the output is uint8.

Parameters:

  • crop_size: The size of the sides of the center cropped square.

  • data_type: The type of the output result, the value range is [“float”, “uint8”].

Examples of use:

# In the form of 224*224, do center cropping, the default output type is float32
CenterCropTransformer(crop_size=224)

# In the form of 224*224, do center cropping, and the output type is uint8
CenterCropTransformer(crop_size=224, data_type="uint8")

4.1.4.2.9. PILCenterCropTransformer

Description:

This transformer will use the PIL method to crop a square image from the center of the image. This transformer will convert the data format to float32 when outputting.

Parameters:

  • size: The size of the sides of the center cropped square.

Examples of use:

# In a 224*224 way, use PIL to do center cropping
PILCenterCropTransformer(size=224)

4.1.4.2.10. LongSideCropTransformer

Description:

This transformer is used for long-edge cropping. This transformer will convert the data format to float32 when outputting.

When the width is larger than the height value, a square whose center is based on the height will be cropped, such as width 100, height 70, and the size after cropping is 70*70.

When the height is larger than the width value, a rectangle with the same center as the width and the height is half of the difference + width will be cropped, such as width 70, height 100, and the size after cropping is 70*(100-70) /2+70 , which is a rectangle of size 70*85.

Parameters: Not involving.

Examples of use:

LongSideCropTransformer()

4.1.4.2.11. PadResizeTransformer

Description:

This transformer uses the padding method to enlarge the image. The transformer converts the data format to float32 when outputting.

Parameters:

  • target_size: Target size, value is a tuple, e.g. (240,240).

  • pad_value: The value to be padded into the array, the default value is 127.

  • pad_position: The position of the padding. The range of values is [“boundary”, “bottom_right”], and the default value is “boundary”.

Examples of use:

# Crop a size of 512*512, padding to the bottom right corner, with a padding value of 0
PadResizeTransformer((512, 512), pad_position='bottom_right', pad_value=0)

# Crop a size of 608*608, padding to the border, with a padding value of 127
PadResizeTransformer(target_size=(608, 608))

4.1.4.2.12. ResizeTransformer

Description:

This transformer is used to resize the image.

Parameters:

  • target_size: Target size, value is a tuple, e.g. (240,240). (240,240): The first 240 represents a height of 240 and the second 240 represents a width of 240.

  • mode: Image processing mode, takes a range of values (“skimage”, “opencv”), default value is “skimage”.

  • method: Interpolation method, this Parameters only works when mode is skimage. The range of values is 0-5, and the default value is 1, of which :

    • 0 for Nearest-neighbor;

    • 1 for Bi-linear(default);

    • 2 for Bi-quadratic;

    • 3 for Bi-cubic;

    • 4 for Bi-quartic;

    • 5 for Bi-quintic.

  • data_type: The type of the output, in the range (uint8, float), defaults to the float type. When it is set to uint8, the output type is uint8 , otherwise it is float32.

  • interpolation: The interpolation method. This parameter only takes effect when mode is opencv. The default is null and takes values in the range (opencv’s interpolation method). Currently, interpolation only supports empty, or INTER_CUBIC in opencv two interpolation methods, when interpolation is empty, the default use INTER_LINEAR method.

    The following are the interpolation methods supported in opencv and their descriptions (interpolation methods not currently supported will be gradually supported in subsequent iterations).

    • INTER_NEAREST: Nearest Neighbor Interpolation;

    • INTER_LINEAR: Bi-linear interpolation, which is used by default when the interpolation is empty.

    • INTER_CUBIC: Bi-cubic interpolation within a 4x4 pixel neighborhood.

    • INTER_AREA: Resampling using pixel area relation.It may be the preferred method for image decimation as it can provide moiré-free results.But when the image is scaled, it is similar to INTER_NEAREST method.

    • INTER_LANCZOS4: Lanczos interpolation of 8x8 neighborhood.

    • INTER_LINEAR_EXACT: Bit-accurate bilinear interpolation.

    • INTER_NEAREST_EXACT: Bit-exact nearest neighbor interpolation. This will produce the same results as the nearest neighbor method in PIL, scikit-image or Matlab.

    • INTER_MAX: Mask for interpolation code.

    • WARP_FILL_OUTLIERS: flag, padding all target image pixels. If some of them correspond to outliers in the source image, set them to zero.

    • WARP_INVERSE_MAP； Flag, inverter.

Examples of use:

# Resize the input image to 224*224, use opencv to process the image, interpolation is bilinear, output is float32
ResizeTransformer(target_size=(224, 224), mode='opencv', method=1)

# Resize the input image to 256*256, use skimage to process the image, interpolate it bilinearly, and output it as float32
ResizeTransformer(target_size=(256, 256))

# Resize input image to 256*256, use skimage to process the image, interpolation is bilinear, output is uint8
ResizeTransformer(target_size=(256, 256), data_type="uint8")

4.1.4.2.13. PILResizeTransformer

Description:

This transformer uses the PIL library to do image resizing.

Parameters:

  • size: Target size, value is a tuple, e.g. (240,240).

  • interpolation: Specify the interpolation method, the value range: (Image.NEAREST, Image.BILINEAR, Image.BICUBIC, Image.LANCZOS), the default value is Image.BILINEAR.

    • Image.NEAREST: Nearest neighbor sampling;

    • Image.BILINEAR: Linear interpolation;

    • Image.BICUBIC: Cubic spline interpolation;

    • Image.LANCZOS: High quality downsampling filter.

Examples of use:

# Adjust the input image size to 256*256 and the interpolation method is linear interpolation
PILResizeTransformer(size=256)

# Adjust the input image size to to 256*256 and the interpolation method is a high-quality downsampling filter
PILResizeTransformer(size=256, interpolation=Image.LANCZOS)

4.1.4.2.14. ShortLongResizeTransformer

Description:

This transformer is used to scale the input image according to the original scale, and the size of the new image is related to the parameters set. The operation is performed as follows.

1.First, divide the size of short_size by the minimum value of the width and height of the original image, and use this value as the scaling factor.

2.When the scaling factor is multiplied by the maximum value of the width and height of the original image and the result is greater than the value of long_size, the scaling factor will be changed to long_size divided by the maximum value of the width and height of the original image.

3.Use the resize method in opencv to re-crop the image according to the scaling factor obtained above.

Parameters:

  • short_size: The expected length of the short edge after cutting.

  • long_size: The expected length of the short edge after cutting.

  • include_im: The default value is True. When set to True, it will return the original image in addition to the processed image.

Examples of use:

# Short side length is 20, long side length is 100, return the processed image and the original image
ShortLongResizeTransformer(short_size=20, long_size=100)

4.1.4.2.15. PadTransformer

Description:

This transformer resizes the image by dividing the size value of the target size by the maximum value of the width or height of the input image, and then multiplying this factor by the original width and height. Then according to the size of the new image, divide it by size_divisor and round it up, then multiply it by size_divisor to generate a new image for the new width and height.

Parameters:

  • size_divisor: Size divisor , default is 128.

  • target_size: Target size, default is 512.

Examples of use:

# The pad size is 1024*1024
PadTransformer(size_divisor=1024, target_size=1024)

4.1.4.2.16. ShortSideResizeTransformer

Description:

According to the desired length of the short side, this transformer use the current ratio of the long and short sides, and the center crop out the operation of the new image size.

Parameters:

  • short_size: Expected length of short side.

  • data_type: The type of the output result, the value range is (“float”, “uint8”), the default value is “float32”, output in float32 type, when set to uint8, the output type will be uint8.

  • interpolation: Specifies the interpolation method, the value range is the interpolation method used in opencv, the default is empty.

    Currently, interpolation only supports empty, or INTER_CUBIC in opencv, two interpolation methods, when interpolation is empty, the default use INTER_LINEAR method.

    The following are the interpolation methods supported in opencv and their descriptions (interpolation methods not currently supported will be gradually supported in subsequent iterations).

    • INTER_NEAREST: Nearest Neighbor Interpolation;

    • INTER_LINEAR: Bi-linear interpolation, which is used by default when the interpolation is empty.

    • INTER_CUBIC: Bi-cubic interpolation within a 4x4 pixel neighborhood.

    • INTER_AREA: Resampling using pixel area relation.It may be the preferred method for image decimation as it can provide moiré-free results.But when the image is scaled, it is similar to INTER_NEAREST method.

    • INTER_LANCZOS4: Lanczos interpolation of 8x8 neighborhood.

    • INTER_LINEAR_EXACT: Bit-accurate bilinear interpolation.

    • INTER_NEAREST_EXACT: Bit-exact nearest neighbor interpolation. This will produce the same results as the nearest neighbor method in PIL, scikit-image or Matlab.

    • INTER_MAX: Mask for interpolation code.

    • WARP_FILL_OUTLIERS: flag, padding all target image pixels. If some of them correspond to outliers in the source image, set them to zero.

    • WARP_INVERSE_MAP； Flag, inverter.

Examples of use:

# Adjust the short side size to 256 and the interpolation method is bilinear interpolation
ShortSideResizeTransformer(short_size=256)

# Resize the short side to 256 and the interpolation method is Lanczos interpolation within the 8x8 pixel neighborhood
ShortSideResizeTransformer(short_size=256, interpolation=Image.LANCZOS4)

4.1.4.2.17. PaddedCenterCropTransformer

Description:

This transformer uses padding to crop the center of the image.

Attention

Only for EfficientNet-lite related instance models.

The calculation method is:

  1. Calculate the factor, int((float( image_size ) / ( image_size + crop_pad )).

  2. Calculate the size of the center size, coefficient * np.minimum( the height of the original image, the width of the original image )).

  3. According to the calculated size, make the center crop.

Parameters:

  • image_size: The size of the image, the default value is 224.

  • crop_pad: The size of the center pad, the default value is 32.

Examples of use:

# Crop size is 240*240, padding value is 32
PaddedCenterCropTransformer(image_size=240, crop_pad=32)

# Crop size is 224*224, padding value is 32
PaddedCenterCropTransformer()

4.1.4.2.18. BGR2RGBTransformer

Description:

This transformer converts the input format from BGR to RGB.

Parameters:

  • data_format: Data format, the range of values is (CHW,HWC), the default value is CHW.

Examples of use:

# When layout is NCHW, convert BGR to RGB
BGR2RGBTransformer()

# When layout is NHWC,convert BGR to RGB
BGR2RGBTransformer(data_format="HWC")

4.1.4.2.19. RGB2BGRTransformer

Description:

This transformer converts the input format from RGB to BGR.

Parameters:

  • data_format: Data format, the range of values is (CHW,HWC), the default value is CHW.

Examples of use:

# When layout is NCHW, convert RGB to BGR
RGB2BGRTransformer()

# When layout is NHWC, convert RGB to BGR
RGB2BGRTransformer(data_format="HWC")

4.1.4.2.20. RGB2GRAYTransformer

Description:

This transformer converts the input format from RGB to GRAY.

Parameters:

  • data_format: The input layout type, in the range (“CHW”, “HWC”), the default is “CHW”..

Examples of use:

# When layout is NCHW, convert RGB to GRAY
RGB2GRAYTransformer(data_format='CHW')

# When layout is NHWC, convert RGB to GRAY
RGB2GRAYTransformer(data_format='HWC')

4.1.4.2.21. BGR2GRAYTransformer

Description:

This transformer converts the input format from BGR to GRAY.

Parameters:

  • data_format: The input layout type, in the range [“CHW”, “HWC”], the default value is “CHW”..

Examples of use:

# When layout is NCHW, convert BGR to GRAY
BGR2GRAYTransformer(data_format='CHW')

# When layout is NHWC, convert BGR to GRAY
BGR2GRAYTransformer(data_format='HWC')

4.1.4.2.22. RGB2GRAY_128Transformer

Description:

This transformer converts the input format from RGB to GRAY_128. GRAY_128 takes values in the range (-128,127).

Parameters:

  • data_format: The input layout type, the range of values is [“CHW”, “HWC”], the default value is “CHW”, this field is required.

Examples of use:

# When layout is NCHW, convert RGB to GRAY_128
RGB2GRAY_128Transformer(data_format='CHW')

# When layout is NHWC, convert RGB to GRAY_128
RGB2GRAY_128Transformer(data_format='HWC')

4.1.4.2.23. RGB2YUV444Transformer

Description:

This transformer converts the input format from RGB to YUV444.

Parameters:

  • data_format: The input layout type, the range is [“CHW”, “HWC”], the default value is “CHW”, this field is required.

Examples of use:

# layout 为 NCHW 时，做 RGB 转成 YUV444
RGB2YUV444Transformer(data_format='CHW')

# layout 为 NHWC 时，做 RGB 转成 YUV444
RGB2YUV444Transformer(data_format='HWC')

4.1.4.2.24. BGR2YUV444Transformer

Description:

This transform the input format from BGR to YUV444.

Parameters:

  • data_format: The input layout type, the range of values is [“CHW”, “HWC”], the default value is “CHW”, this item is required.

Examples of use:

# When layout is NCHW, convert BGR to YUV444
BGR2YUV444Transformer(data_format='CHW')

# When layout is NHWC, convert BGR to YUV444
BGR2YUV444Transformer(data_format='HWC')

4.1.4.2.25. BGR2YUV444_128Transformer

Description:

This transformer converts the input format from BGR to YUV444_128. YUV444_128 takes values in the range (-128,127).

Parameters:

  • data_format: The input layout type, the range of values is [“CHW”, “HWC”], the default value is “CHW”, this item is required.

Examples of use:

# When layout is NCHW, convert BGR to YUV444_128
BGR2YUV444_128Transformer(data_format='CHW')

# When layout is NHWC, convert BGR to YUV444_128
BGR2YUV444_128Transformer(data_format='HWC')

4.1.4.2.26. RGB2YUV444_128Transformer

Description:

This transformer converts the input format from RGB to YUV444_128. YUV444_128 takes values in the range of (-128,127).

Parameters:

  • data_format: The input layout type, the range of values is [“CHW”, “HWC”], the default value is “CHW”, this field is required.

Examples of use:

# When layout is NCHW , convert RGB to YUV444_128
RGB2YUV444_128Transformer(data_format='CHW')

# When layout is NHWC, convert RGB to YUV444_128
RGB2YUV444_128Transformer(data_format='HWC')

4.1.4.2.27. BGR2YUVBT601VIDEOTransformer

Description:

This transformer converts the input format from BGR to YUV_BT601_Video_Range.

YUV_BT601_Video_Range, some camera input data are YUV BT601 (Video Range) format, the range of values is 16~235, this transformer is adapted to this format of data generated.

Parameters:

  • data_format: The input layout type, the range of values is [“CHW”,”HWC”], the default value is “CHW”, this field is required.

Examples of use:

# When layout is NCHW, convert BGR to YUV_BT601_Video_Range
BGR2YUVBT601VIDEOTransformer(data_format='CHW')

# When layout is NHWC, convert BGR to YUV_BT601_Video_Range
BGR2YUVBT601VIDEOTransformer(data_format='HWC')

4.1.4.2.28. RGB2YUVBT601VIDEOTransformer

Description:

This transformer converts the input format from RGB to YUV_BT601_Video_Range.

YUV_BT601_Video_Range, some camera input data are YUV BT601 (Video Range) format, the range of values is 16~235, this transformer is adapted to this format of data generated.

Parameters:

  • data_format: The input layout type, the range of values is [“CHW”,”HWC”], the default value is “CHW”, this field is required.

Examples of use:

# When layout is NCHW, convert RGB to YUV_BT601_Video_Range
RGB2YUVBT601VIDEOTransformer(data_format='CHW')

# When layout is NHWC, convert RGB to YUV_BT601_Video_Range
RGB2YUVBT601VIDEOTransformer(data_format='HWC')

4.1.4.2.29. YUVTransformer

Description:

The transformer converts the input format to YUV444.

Parameters:

  • color_sequence: Color sequence, this field is required.

Examples of use:

# Converting BGR read-in images to YUV444
YUVTransformer(color_sequence="BGR")

# Converting RGB read-in images to YUV444
YUVTransformer(color_sequence="RGB")

4.1.4.2.30. ReduceChannelTransformer

Description:

This transformer will reduce the C channel to a single channel. The transformer is mainly for C channel, such as shape 1*3*224*224 to 1*1*224*224. When using layout must be aligned with data_format value, to avoid causing the wrong channel deletion.

Parameters:

  • data_format: The input layout type, the range is [“CHW”, “HWC”], the default value is “CHW”..

Examples of use:

# Delete the C channel with layout as NCHW
ReduceChannelTransformer()
# Or
ReduceChannelTransformer(data_format="CHW")

# Delete the C channel with layout as NHWC
ReduceChannelTransformer(data_format="HWC")

4.1.4.2.31. BGR2NV12Transformer

Description:

This transformer converts the input format from BGR to NV12.

Parameters:

  • data_format: The input layout type, the range is [“CHW”,”HWC”], the default value is “CHW”.

  • cvt_mode: cvt mode, the value range is (rgb_calc, opencv), the default value is rgb_calc.

    • rgb_calc: Image processing using mergeUV;

    • opencv: Image processing using opencv.

Examples of use:

# When the layout is NCHW, convert from BGR to NV12, and the image is processed by the rgb_calc
BGR2NV12Transformer()
# Or
BGR2NV12Transformer(data_format="CHW")

# When the layout is NHWC, convert from BGR to NV12, and the image is processed by the opencv
BGR2NV12Transformer(data_format="HWC", cvt_mode="opencv")

4.1.4.2.32. RGB2NV12Transformer

Description:

This transformer converts the input format from RGB to NV12.

Parameters:

  • data_format: The input layout type, in the range [“CHW”, “HWC”], the default value is “CHW”.

  • cvt_mode: cvt mode, the range of values is (rgb_calc,opencv), the default value is rgb_calc.

    • rgb_calc: Image processing using mergeUV;

    • opencv: Image processing using opencv.

Examples of use:

# When the layout is NCHW, convert RGB to NV12, and the image is processed by the rgb_calc
RGB2NV12Transformer()
# Or
RGB2NV12Transformer(data_format="CHW")

# When the layout is NHWC, convert RGB to NV12, and the image is processed by the opencv
RGB2NV12Transformer(data_format="HWC", cvt_mode="opencv")

4.1.4.2.33. NV12ToYUV444Transformer

Description:

This transformer converts the input format from NV12 to YUV444.

Parameters:

  • target_size: Target size, value is a tuple, e.g. (240,240).

  • yuv444_output_layout: yuv444 output layout, the range is (HWC,CHW), the default value is “HWC”.

Examples of use:

# layout is NCHW , size is 768*768, convert nv12 to yuv444
NV12ToYUV444Transformer(target_size=(768, 768))

# layout is NHWC , size is 224*224, convert nv12 to yuv444
NV12ToYUV444Transformer((224, 224), yuv444_output_layout="HWC")

4.1.4.2.34. WarpAffineTransformer

Description:

This transformer is used to do image affine transformations.

Parameters:

  • input_shape: The input shape value.

  • scale: The factor to be multiplied.

Examples of use:

# The size is 512*512, the length of the long side is 1.0
WarpAffineTransformer((512, 512), 1.0)

4.1.4.2.35. F32ToS8Transformer

Description:

This transformer is used to do the input format conversion from float32 to int8.

Parameters: Not involving.

Examples of use:

# Conversion of input format from float32 to int8
F32ToS8Transformer()

4.1.4.2.36. F32ToU8Transformer

Description:

This transformer is used to convert the input format from float32 to uint8.

Parameters: Not involving.

Examples of use:

# Conversion of input format from float32 to uint8
F32ToU8Transformer()