5.16. ISP - AF Statistics Module

5.16.1. Function Description

Auto-focus achieves its goal by analyzing image contrast and automatically adjusting the camera lens. The statistical information for the auto-focus algorithm is provided by the AFM module within the ISP. The AFM module calculates the image sharpness value FV (Focus Value) through built-in filters that perform convolution on RAW data. The AFM module’s statistics position is at the end of the RAW domain.

The AFM module contains FIR and IIR high-pass and low-pass filters. The FIR high-pass filter result and the IIR high-pass filter result are fused, and the low-pass filter results are fused together, ultimately outputting two groups of statistics: high-pass and low-pass. The statistics area consists of a fixed number of 15x15 zones, and the overall statistics area size is configurable.

5.16.2. Pipeline Introduction

Block

5.16.3. AFM Statistics Module Interface

AFM statistics data retrieval interface

int32_t hbn_isp_get_af_statistics(hbn_vnode_handle_t vnode_fd, hbn_isp_af_statistics_t *p_data);

typedef struct hbn_isp_af_statistics_s {
        uint32_t sharpnessLowPass[HBN_ISP_AFM_BLOCK_NUM];
        uint32_t sharpnessHighPass[HBN_ISP_AFM_BLOCK_NUM];
        uint32_t histLowData[HBN_ISP_AFM_BLOCK_NUM];
        uint32_t histHighData[HBN_ISP_AFM_BLOCK_NUM];
        uint32_t frame_id;      // Corresponding frame id (backup)
} hbn_isp_af_statistics_t;

AFM statistics module parameter interface

int32_t hbn_isp_get_afm_attr(hbn_vnode_handle_t vnode_fd, hbn_isp_afm_attr_t *p_attr);
int32_t hbn_isp_set_afm_attr(hbn_vnode_handle_t vnode_fd, hbn_isp_afm_attr_t *p_attr);

typedef struct hbn_isp_afm_ldg_s {
	float iir_highpass_y[HBN_ISP_AFM_LDG_Y_NUM];	/**< IIR high pass on Y axis */
	uint8_t iir_highpass_x[HBN_ISP_AFM_LDG_X_NUM];	/**< IIR high pass on X axis */
	float iir_lowpass_y[HBN_ISP_AFM_LDG_Y_NUM];	/**< IIR low pass on Y axis */
	uint8_t iir_lowpass_x[HBN_ISP_AFM_LDG_X_NUM];	/**< IIR low pass on X axis */
	float fir_highpass_y[HBN_ISP_AFM_LDG_Y_NUM];	/**< FIR high pass on Y axis */
	uint8_t fir_highpass_x[HBN_ISP_AFM_LDG_X_NUM];	/**< FIR high pass on X axis */
	float fir_lowpass_y[HBN_ISP_AFM_LDG_Y_NUM];	/**< FIR low pass on Y axis */
	uint8_t fir_lowpass_x[HBN_ISP_AFM_LDG_X_NUM];	/**< FIR low pass on X axis */
} hbn_isp_afm_ldg_t;

typedef struct hbn_isp_afm_coring_s {
	uint8_t iir_highpass[HBN_ISP_AFM_CORING_COEFF_NUM];	/**< IIR high pass */
	uint8_t iir_lowpass[HBN_ISP_AFM_CORING_COEFF_NUM];	/**< IIR low pass */
	uint8_t fir_highpass[HBN_ISP_AFM_CORING_COEFF_NUM];	/**< FIR high pass */
	uint8_t fir_lowpass[HBN_ISP_AFM_CORING_COEFF_NUM];	/**< FIR low pass */
} hbn_isp_afm_coring_t;

typedef struct hbn_isp_afm_iir_s {
	uint8_t data_shift;	/**< Data shift */
	uint8_t data_shift_v;	/**< Data shift Vertical*/
	float highpass_weight;	/**< High pass weight */
	float lowpass_weight;	/**< Low pass weight */
	float highpass_coeff[HBN_ISP_AFM_IIR_FILTER_NUM][HBN_ISP_AFM_IIR_FILTER_COEFF_NUM];  /**< High pass coefficient */
	float lowpass_coeff[HBN_ISP_AFM_IIR_FILTER_NUM][HBN_ISP_AFM_IIR_FILTER_COEFF_NUM];   /**< Low pass coefficient */
} hbn_isp_afm_iir_t;

typedef struct hbn_isp_afm_fir_s {
	uint8_t data_shift;		/**< Data shift */
	uint8_t data_shift_v;		/**< Data shift Vertical*/
	float highpass_weight;		/**< High pass weight */
	float lowpass_weight;		/**< Low pass weight */
	uint8_t highdata_shift;
	uint8_t highdata_shift_v;
	uint8_t lowdata_shift;
	uint8_t lowdata_shift_v;
	uint8_t highpass_coeff[HBN_ISP_AFM_FIR_HIGHPASS_COEFF_NUM];	/**< High pass coefficient */
	uint8_t lowpass_coeff[HBN_ISP_AFM_FIR_LOWPASS_COEFF_NUM];	/**< Low pass coefficient */
} hbn_isp_afm_fir_t;

typedef struct hbn_isp_afm_attr_s {
	uint16_t bls;					/**< BLS components */
	uint8_t hist_threshold;				/**< Histogram threshold */
	float gamma;					/**< Gamma value */
	uint16_t hw_roi[HBN_ISP_AFM_BLOCK_POINT_NUM];	/**< Hardware ROI */
	hbn_isp_afm_ldg_t ldg;				/**< Level Dependent Gain configuration */
	hbn_isp_afm_coring_t coring;			/**< Coring configuration */
	hbn_isp_afm_iir_t iir;				/**< IIR configuration */
	hbn_isp_afm_fir_t fir;				/**< FIR configuration */
} hbn_isp_afm_attr_t;

5.16.4. Input Image Cropping

The AFM module supports configuration of the input image statistics position through the hw_roi parameter in the hbn_isp_afm_attr_t structure. The four structure members correspond to the starting and ending positions of x and y respectively.

5.16.5. Input Image Preprocessing

The AFM module includes two preprocessing steps before filter calculation: BLC and Gamma. Parameters can be configured to remove a certain black level. The Gamma parameter only requires configuring a single Gamma coefficient, and the software will automatically calculate the corresponding Gamma curve and configure it to the hardware.

These two parameters correspond to the bls and gamma parameters in the hbn_isp_afm_attr_t structure respectively.

5.16.6. Output Data Post-processing

The AFM module includes two post-processing steps after filter calculation: Coring and LDG. These two parameters correspond to the coring and ldg parameters in the hbn_isp_afm_attr_t structure respectively.

The coring function eliminates noise in low-light areas to increase sharpness. Pixel values below a certain threshold are clipped according to the coring curve. The first parameter x1: if input x < x1, then y = 0. The second parameter x2: if x1 < x < x2, then y = (x - x1) * x2 / 16. The third parameter x3: the slope of the linear mapping stage. The default parameters are as follows, representing a mapping slope of 1 in the range 32 to 230.

Block

Level Dependent Gain (LDG) adjusts sharpness according to pixel values. It increases low pixel values and reduces high pixel values according to the LDG curve. LDG is applied after the IIR filter and FIR filter.

Taking the high-pass IIR filter as an example, iirHighPassY[0~2]: the Y-axis of the LDG curve for high-pass IIR filtering, representing the amplification factor for input x, i.e., 1.0 means y = x * 1, 2.0 means y = 2 * x. iirHighPassX[0~3]: the X-axis of the LDG curve for high-pass IIR filtering, corresponding to x[0~3] in the figure, representing the numerical range for determining input x. Different ranges determine the mapped amplification slope.

Block

5.16.7. Data Fusion

AF statistics data includes highdata and lowdata, which represent the weighted results of FIR and IIR high-pass filter results, and the weighted results of FIR and IIR low-pass filter results, respectively.

highdata = (fir.highdata * fir.highpassweight + iir.highdata * iir.highpassweight) / 2;
lowdata = (fir.lowdata * fir.lowpassweight + iir.lowdata * iir.lowpassweight) / 2;

It is recommended that the sum of iir and fir highpassweight equals 2, and the sum of iir and fir lowpassweight equals 2, so that no additional multiplier is introduced to the data.

5.16.8. Point Light Source Processing

When a point light source appears in the scene, the statistics data may exhibit an abnormal phenomenon where the image appears sharper as it becomes more blurred due to the influence of halos. For this situation, AFM has a brightness statistics module that can help users develop debugging strategies. By configuring a brightness threshold, you can obtain the number of pixels that exceed or fall below this threshold. Users can use this to determine whether a region is a light source, and then choose to exclude that region’s data, or determine whether the region is in focus or out of focus based on the change in the number of bright spots.

The brightness threshold is the hist_threshold parameter in the hbn_isp_afm_attr_t structure.

5.16.9. Data Shift

Parameter	Meaning
fir.highDataShift/fir.lowDataShift	For each pixel, the number of shifts before squaring the horizontal gradient of the FIR filter
fir.lowDataShiftV/fir.highDataShiftV	For each pixel, the number of shifts before squaring the vertical gradient of the FIR filter
fir.data_shift/iir.data_shift	During block statistics, the number of shifts after summing the sharpness of each row in the block. This parameter applies to both high-pass and low-pass
fir.data_shift_v/iir.data_shift_v	During block statistics, the number of shifts after summing all points in the block. This parameter applies to both high-pass and low-pass

5.16.10. IIR Parameter Generation Steps Reference

The interface uses a Chebyshev Type II filter, with the first parameter being 3rd order and the last parameter being “bandpass”. These two parameters cannot be changed. Then use tf2sos to decompose it into 2nd order. The correspondence between the resulting parameters and the JSON parameters is as follows:

The three parameters of a2 correspond to the first three parameters of the first column in JSON, and the second parameter of b3 corresponds to the fourth parameter of the first column in JSON.
The three parameters of a3 correspond to the first three parameters of the second column in JSON, and the second parameter of b2 corresponds to the fourth parameter of the second column in JSON.
The three parameters of a1 correspond to the first three parameters of the third column in JSON, and the second parameter of b1 corresponds to the fourth parameter of the third column in JSON.

matlab_iir

5.16.11. Recommended Parameter Configuration

"AFM_3": {
	"bls": 0,
	"coring": {
		"firHighPass": [32,230,16],
		"firLowPass": [32,230,16],
		"iirHighPass": [32,230,16],
		"iirLowPass": [32,230,16]
	},
	"fir": {
		"dataShift": 12,
		"dataShiftV": 3,
		"highPassWeight": 1.0,
		"lowPassWeight": 1.0,
		"highDataShift": 0,
		"highDataShiftV": 0,
		"lowDataShift": 3,
		"lowDataShiftV": 3,
		"highPassCoeff": [2, 1],
		"lowPassCoeff": [12, 6, 4, 3]
	},
	"gamma": 2.2,
	"histThreshold": 128,
	"hwRoi": [0,0,1920,1080],
	"iir": {
		"dataShift": 3,
		"dataShiftV": 3,
		"highPassWeight": 1.0,
		"highPassCoeff": [[0.573, -1.4474, 0.774, -1.98],[0.573, -1.875, 0.915, -1.31],[0.217, -1.467, 0.566, 0.0]],
		"lowPassWeight": 1.0,
		"lowPassCoeff": [[0.6, -0.445, 0.624, -1.92],[0.6, -1.67, 0.823, 0.215],[0.383, -1.465, 0.566, 0.0]]
	},
	"ldg": {
		"firHighPassX": [16, 96, 160, 208],
		"firHighPassY": [4.0, 8.0, 4.0],
		"firLowPassX": [16, 96, 160, 208],
		"firLowPassY": [4.0, 8.0, 4.0],
		"iirHighPassX": [16, 96, 160, 208],
		"iirHighPassY": [1.0, 2.0, 1.0],
		"iirLowPassX": [16, 96, 160, 208],
		"iirLowPassY": [1.0, 2.0, 1.0]
	},
}

5.16.12. Debugging Suggestions

The coring parameter is used to remove the influence of noise. In scenarios with high exposure gain, appropriately increasing the first parameter of coring can reduce numerical fluctuations caused by noise, but the overall FV value will decrease.
If a double-peaked curve appears in high-contrast scenes, this may be caused by excessive vertical filter data. You can adjust the FIR filter kernel parameters and appropriately reduce the vertical direction parameters to minimize the influence of vertical contrast.
In scenes with insufficient information, the statistics output by AFM are typically small. You can appropriately reduce datashift at each level and increase the LDG values in the middle region to amplify the final statistics. However, it is not recommended to adjust the parameters too far from the defaults, as this will affect the statistics results in normal scenes.