7.3.8. Model Precision Debug Tool¶
When a precision problem occurred in QAT or quantized model, users can use the various tools described below to analyze the model and figure out the low precision reason.
7.3.8.1. Similarity¶
If the QAT/quantized model has lower precision than the floating point model, users can use the similarity comparison tool to compare the similarity of the output of each layer in the model, and figure out which layer or which operator leading to serious precision drop.
# from horizon_plugin_pytorch.utils.quant_profiler import featuremap_similarity
def featuremap_similarity(
model1: torch.nn.Module,
model2: torch.nn.Module,
inputs: Any,
similarity_func: Union[str, Callable] = "Cosine",
threshold: Optional[Real] = None,
devices: Union[torch.device, tuple, None] = None,
out_dir: Optional[str] = None,
):
"""
Compute the similarity of feature maps. The input models can be floating/
fused/calibration/qat/quantized model.
Arguments:
model1: can be float/fused/calibration/qat/quantized model
model2: can be float/fused/calibration/qat/quantized model
inputs: the input data feed to model
similarity_func: similarity computation function. Support "Cosine",
"MSE", "L1", "KL", "SQNR", or any user-defined Callable object. If
it is a user-defined object, it should return a scalar or tensor
with only one number. Otherwise the result shown may be unexpected.
Default: "Cosine"
threshold: if similarity value exceeds or less than this threshold,
the featuremap info will be shown on the screen. If threshold is
none, it will be set to different values according to different
similarity functions. Default: None
devices: run model on which devices (cpu, gpu). If can be:
1) None. Run model with given inputs;
2) torch.device. Both models and given inputs will be moved on this
specified device;
3) tuple. A tuple of 2 torch.devices. The two models will be moved
on specified devices seperatedly. It may be used to compare the
CPU and GPU results difference.
out_dir: path to save the result txt and picture. If None, will save in
the current directory. Default: None
Returns:
A List of list. Each list is each layer similarity info in format
[index, module name, module type, similarity, scale, atol,
atol(N scale), single op error(N scale)]
"""
Users must pay attention to the following points when using the function:
By default, the function will print:
the similarity of the corresponding layer.
scaleof the result.the maximum error of result (
atol).single operator error (single operator error = the maximum error of the result
atol/ thescaleof the result).single operator error of the result with the same inputs. The single operator error refers to how many
scalesthat each layer result differs, including the influence of cumulative error in that the inputs of this layer may be different under the influence of results deviation of previous layers. The single operator error with the same inputs, is calculated by manually setting the input of this layer to be exactly the same and then comparing the outputs in two models. It should be within a fewscalesin theory. If it differs much, there may be problems in the operator conversion.
The function supports any two-stage models compared in any order of input or on any two
devices. It is recommended that the two stage models input in the order offloat/qat/quantized, such as (float, qat) and (qat, quantized). If the order is (qat, float), it has no effect on the similarity and single op error, but the single op error under the same input may be confused, because it is impossible to generate an input corresponding to the floating point model to the qat model. In addition, because the model parameters will change after qat training, it is not meaningful to directly compare the similarity between floating point and the qat model after training. It is recommended to compare the similarity between floating point and the qat model that has been calibrated and not trained.The function saves the result to a .txt file. It also draws the curve of the similarity and saves it as an image. The following files will be generated:
similarity.txt: Print the results of the similarity and single operator error of each layer in the order of model forward.ordered_op_error_similarity.txt: The result of sorted the operator error with the same input from high to low, which is convenient for users to quickly figure out which operator has the largest convert error.similarity.html: An interactive image shows the curve of similarity of each layer as the model forward. You can zoom in and out, and move the cursor to the corresponding point to see the specific similarity value.
If the model holds multiple inputs, multiple inputs should be combined into a
tupleand passed to theinputsparameter.If all the outputs of a certain layer are 0, the similarity result is also 0 when calculating the similarity using cosine. In this case, users can check whether all the outputs of this layer are 0, or confirm whether the outputs are the same according to the
atoland other indicators. If the outputs of a certain layer are exactly the same, the result is inf when calculating the similarity using the signal-to-noise ratio .If
device=None, the function will not move the model and inputs between devices. Users must ensure that the model and the model input are on the same device .The function prints the similarity of each layer of the output layer by layer in the format
layer name - similarity result.If the module name has the suffix ‘(I)’, it means that the operator is
Identityin a certain modelIf the module name has the suffix ‘(I vs I)’, it means that the operator is an
Identityin the two models to be comparedIf the module name has the suffix ‘(i)’ (i >= 1), it means that this layer is a shared operator and has been shared i times, and it is currently the i+1st call. Shared operators called for the first time have no suffix like other operators.
If two different floating-point models are compared, the similarity output is empty.
7.3.8.1.1. Example¶
import torch
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
import horizon_plugin_pytorch as D-Robotics
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.utils.quant_profiler import featuremap_similarity
class Net(nn.Module):
def __init__(self, quant=False, share_op=True):
super(Net, self).__init__()
self.quant_stubx = QuantStub()
self.quant_stuby = QuantStub()
self.mul_op = FloatFunctional()
self.cat_op = FloatFunctional()
self.quantized_ops = nn.Sequential(
nn.ReLU(),
nn.Sigmoid(),
nn.Softmax(),
nn.SiLU(),
horizon_nn.Interpolate(
scale_factor=2, recompute_scale_factor=True
),
horizon_nn.Interpolate(
scale_factor=2.3, recompute_scale_factor=True
),
nn.AvgPool2d(kernel_size=4),
nn.Upsample(scale_factor=1.3, mode="bilinear"),
nn.UpsamplingBilinear2d(scale_factor=0.7),
)
self.dequant_stub = DeQuantStub()
self.float_ops = nn.Sequential(
nn.Tanh(),
nn.LeakyReLU(),
nn.PReLU(),
nn.UpsamplingNearest2d(scale_factor=0.7),
)
self.quant = quant
self.share_op = share_op
def forward(self, x, y):
x = self.quant_stubx(x)
y = self.quant_stuby(y)
z = self.mul_op.mul(x, y)
x = self.cat_op.cat((x, y), dim=1)
if self.share_op:
x = self.cat_op.cat((x, y), dim=1)
x = self.quantized_ops(x)
x = self.dequant_stub(x)
if not self.quant:
x = self.float_ops(x)
return x
set_march(March.XXX)
device = torch.device("cuda")
float_net = Net(quant=True, share_op=True).to(device)
float_net.qconfig = horizon.quantization.get_default_qat_qconfig()
qat_net = horizon.quantization.prepare_qat(float_net, inplace=False)
qat_net = qat_net.to(device)
data = torch.arange(1 * 3 * 4 * 4) / 100 + 1
data = data.reshape((1, 3, 4, 4))
data = data.to(torch.float32).to(device)
featuremap_similarity(float_net, qat_net, (data, data))
The following files will be generated in the current directory or the directory specified by the out_dir parameter:
similarity.txt
---------------------------------------------------------------
Note:
* Suffix '(I)' means this layer is Identity in one model
* Suffix '(I vs I)' means this layer is Identity in both models
* Suffix '(i)'(i >= 1) means this op is shared i times
---------------------------------------------------------------
+---------+----------------------------+----------------------------------------------------------------------------------+--------------+-----------+----------------+------------------+------------------------+
| Index | Module Name | Module Type | Similarity | qscale | Acc Error | Acc Error | Op Error with Same |
| | | | | | (float atol) | (N out_qscale) | Input (N out_qscale) |
|---------+----------------------------+----------------------------------------------------------------------------------+--------------+-----------+----------------+------------------+------------------------|
| 0 | quant_stubx | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | 1.0000000 | 0.0115294 | 0.0000000 | 0 | 0 |
| 1 | quant_stuby | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | 1.0000000 | 0.0115294 | 0.0000000 | 0 | 0 |
| 2 | mul_op | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999989 | 0.0168156 | 0.0168156 | 1 | 1 |
| 3 | cat_op | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999971 | 0.0167490 | 0.0334979 | 2 | 0 |
| 4 | cat_op(1) | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999980 | 0.0167490 | 0.0334979 | 2 | 0 |
| 5 | quantized_ops.0 | <class 'horizon_plugin_pytorch.nn.qat.relu.ReLU'> | 0.9999980 | 0.0167490 | 0.0334979 | 2 | 0 |
| 6 | quantized_ops.1 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000000 | 0.0070079 | 0.0000000 | 0 | 0 |
| 7 | quantized_ops.2.sub | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999999 | 0.0000041 | 0.0000041 | 1 | 1 |
| 8 | quantized_ops.2.exp | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000000 | 0.0000305 | 0.0000305 | 1 | 1 |
| 9 | quantized_ops.2.sum | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 1.0000000 | 0.0002541 | 0.0005081 | 2 | 2 |
| 10 | quantized_ops.2.reciprocal | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000001 | 0.0000037 | 0.0000186 | 5 | 5 |
| 11 | quantized_ops.2.mul | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 1.0000000 | 0.0009545 | 0.0000000 | 0 | 0 |
| 12 | quantized_ops.3 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000000 | 0.0005042 | 0.0000000 | 0 | 0 |
| 13 | quantized_ops.4 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | 1.0000000 | 0.0005042 | 0.0005042 | 1 | 1 |
| 14 | quantized_ops.5 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | 0.9999999 | 0.0005042 | 0.0005042 | 1 | 0 |
| 15 | quantized_ops.6 | <class 'horizon_plugin_pytorch.nn.qat.avg_pool2d.AvgPool2d'> | 0.9999995 | 0.0005022 | 0.0005022 | 1 | 1 |
| 16 | quantized_ops.7 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.Upsample'> | 0.9999998 | 0.0005022 | 0.0005022 | 1 | 0 |
| 17 | quantized_ops.8 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.UpsamplingBilinear2d'> | 1.0000000 | 0.0005022 | 0.0000000 | 0 | 0 |
| 18 | dequant_stub | <class 'horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub'> | 1.0000000 | | 0.0000000 | 0 | 0 |
+---------+----------------------------+----------------------------------------------------------------------------------+--------------+-----------+----------------+------------------+------------------------+
ordered_op_error_similarity.txt
---------------------------------------------------------------
Note:
* Suffix '(I)' means this layer is Identity in one model
* Suffix '(I vs I)' means this layer is Identity in both models
* Suffix '(i)'(i >= 1) means this op is shared i times
---------------------------------------------------------------
+---------+----------------------------+----------------------------------------------------------------------------------+--------------+-----------+----------------+------------------+------------------------+
| Index | Module Name | Module Type | Similarity | qscale | Acc Error | Acc Error | Op Error with Same |
| | | | | | (float atol) | (N out_qscale) | Input (N out_qscale) |
|---------+----------------------------+----------------------------------------------------------------------------------+--------------+-----------+----------------+------------------+------------------------|
| 10 | quantized_ops.2.reciprocal | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000001 | 0.0000037 | 0.0000186 | 5 | 5 |
| 9 | quantized_ops.2.sum | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 1.0000000 | 0.0002541 | 0.0005081 | 2 | 2 |
| 2 | mul_op | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999989 | 0.0168156 | 0.0168156 | 1 | 1 |
| 7 | quantized_ops.2.sub | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999999 | 0.0000041 | 0.0000041 | 1 | 1 |
| 8 | quantized_ops.2.exp | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000000 | 0.0000305 | 0.0000305 | 1 | 1 |
| 13 | quantized_ops.4 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | 1.0000000 | 0.0005042 | 0.0005042 | 1 | 1 |
| 15 | quantized_ops.6 | <class 'horizon_plugin_pytorch.nn.qat.avg_pool2d.AvgPool2d'> | 0.9999995 | 0.0005022 | 0.0005022 | 1 | 1 |
| 0 | quant_stubx | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | 1.0000000 | 0.0115294 | 0.0000000 | 0 | 0 |
| 1 | quant_stuby | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | 1.0000000 | 0.0115294 | 0.0000000 | 0 | 0 |
| 3 | cat_op | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999971 | 0.0167490 | 0.0334979 | 2 | 0 |
| 4 | cat_op(1) | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 0.9999980 | 0.0167490 | 0.0334979 | 2 | 0 |
| 5 | quantized_ops.0 | <class 'horizon_plugin_pytorch.nn.qat.relu.ReLU'> | 0.9999980 | 0.0167490 | 0.0334979 | 2 | 0 |
| 6 | quantized_ops.1 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000000 | 0.0070079 | 0.0000000 | 0 | 0 |
| 11 | quantized_ops.2.mul | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | 1.0000000 | 0.0009545 | 0.0000000 | 0 | 0 |
| 12 | quantized_ops.3 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | 1.0000000 | 0.0005042 | 0.0000000 | 0 | 0 |
| 14 | quantized_ops.5 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | 0.9999999 | 0.0005042 | 0.0005042 | 1 | 0 |
| 16 | quantized_ops.7 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.Upsample'> | 0.9999998 | 0.0005022 | 0.0005022 | 1 | 0 |
| 17 | quantized_ops.8 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.UpsamplingBilinear2d'> | 1.0000000 | 0.0005022 | 0.0000000 | 0 | 0 |
| 18 | dequant_stub | <class 'horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub'> | 1.0000000 | | 0.0000000 | 0 | 0 |
+---------+----------------------------+----------------------------------------------------------------------------------+--------------+-----------+----------------+------------------+------------------------+
similarity.html
7.3.8.2. Visualization¶
Horizon_plugin_pytorch supports model visualization in any stage. The visualization here refers to the visualization of the model structure. Onnx is exported by default and can be viewed using netron .
7.3.8.2.1. Model Visualization¶
# from horizon_plugin_pytorch.utils.onnx_helper import export_to_onnx, export_quantized_onnx
export_to_onnx(
model,
args,
f,
export_params=True,
verbose=False,
training=TrainingMode.EVAL,
input_names=None,
output_names=None,
operator_export_type=OperatorExportTypes.ONNX_FALLTHROUGH,
do_constant_folding=True,
example_outputs=None,
dynamic_axes=None,
enable_onnx_checker=False,
)
export_quantized_onnx(
model,
args,
f,
export_params=True,
verbose=False,
training=TrainingMode.EVAL,
input_names=None,
output_names=None,
operator_export_type=OperatorExportTypes.ONNX_FALLTHROUGH,
opset_version=None,
do_constant_folding=True,
example_outputs=None,
dynamic_axes=None,
keep_initializers_as_inputs=None,
custom_opsets=None,
)
The meaning of the parameter remains the same as torch.onnx.export, the only difference is operator_export_type=OperatorExportTypes.ONNX_FALLTHROUGH
Notes:
Use
export_to_onnxto export the float/qat model.Use
export_quantized_onnxto export the quantized model.The granularity of visualization is:
Customized operators in the plugin include floating point and quantized operators, and the internal implementation of operators will not be visualized.
The visualization granularity of the community operators used in floating point model is determined by pytorch community.
Examples:
import torch
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
import horizon_plugin_pytorch as D-Robotics
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.utils.onnx_helper import (
export_to_onnx,
export_quantized_onnx,
)
class Net(nn.Module):
def __init__(self, quant=False, share_op=True):
super(Net, self).__init__()
self.quant_stubx = QuantStub()
self.quant_stuby = QuantStub()
self.mul_op = FloatFunctional()
self.cat_op = FloatFunctional()
self.quantized_ops = nn.Sequential(
nn.ReLU(),
nn.Sigmoid(),
nn.Softmax(),
nn.SiLU(),
horizon_nn.Interpolate(
scale_factor=2, recompute_scale_factor=True
),
horizon_nn.Interpolate(
scale_factor=2.3, recompute_scale_factor=True
),
nn.AvgPool2d(kernel_size=4),
nn.Upsample(scale_factor=1.3, mode="bilinear"),
nn.UpsamplingBilinear2d(scale_factor=0.7),
)
self.dequant_stub = DeQuantStub()
self.float_ops = nn.Sequential(
nn.Tanh(),
nn.LeakyReLU(),
nn.PReLU(),
nn.UpsamplingNearest2d(scale_factor=0.7),
)
self.quant = quant
self.share_op = share_op
def forward(self, x, y):
x = self.quant_stubx(x)
y = self.quant_stuby(y)
z = self.mul_op.mul(x, y)
x = self.cat_op.cat((x, y), dim=1)
if self.share_op:
x = self.cat_op.cat((x, y), dim=1)
x = self.quantized_ops(x)
x = self.dequant_stub(x)
if not self.quant:
x = self.float_ops(x)
return x
set_march(March.XXX)
device = torch.device("cuda")
float_net = Net(quant=True, share_op=True).to(device)
float_net.qconfig = horizon.quantization.get_default_qat_qconfig()
qat_net = horizon.quantization.prepare_qat(float_net, inplace=False)
qat_net = qat_net.to(device)
quantized_net = horizon.quantization.convert(qat_net, inplace=False)
data = torch.arange(1 * 3 * 4 * 4) / 100 + 1
data = data.reshape((1, 3, 4, 4))
data = data.to(torch.float32).to(device)
export_to_onnx(float_net, (data, data), "float_test.onnx")
export_to_onnx(qat_net, (data, data), "qat_test.onnx")
export_quantized_onnx(quantized_net, (data, data), "quantized_test.onnx")
7.3.8.2.2. PT File Visualization¶
To support the visualization of torchscript models, users need to install the patched netron , and directly use netron to open the PT file. The installation method is shown below.
# install netron
pip install netron>=6.0.2
# use the script in horizon_plugin_pytorch to patch netron
python -m horizon_plugin_pytorch.utils.patch_netron
7.3.8.3. Statistics¶
The functions directly calculate the statistics of the input and output of each layer in the model, and save the results. Min/max/mean/var/scale info is saved by default. Statistics can help users judge whether the data distribution is suitable for quantification, and evaluate which quantification precision to use.
# from horizon_plugin_pytorch.utils.quant_profiler import get_raw_features, profile_featuremap
def get_raw_features(
model: torch.nn.Module,
example_inputs: Any,
prefixes: Tuple = (),
types: Tuple = (),
device: torch.device = None,
preserve_int: bool = False,
use_class_name: bool = False,
skip_identity: bool = False,
):
"""
Use hooks to get raw features to be profiled. Default insert hooks in all leaf modules. If the origin model is too large to show info in tensorboard, use prefixes or types to insert hooks in specific modules.
Arguments:
model: can be float/fused/calibration/qat/quantized model
example_inputs: the input data feed to model
prefixes: get features info by the prefix of qualified name. Default: tuple().
types: get features info by module type. Default: tuple().
device: run the model on which device. Default: None
preserve_int: if True, record each operator result in int type. Default: False
use_class_name: if True, record class name not class type. Default: False
skip_identity: if True, the result of the Identity module will not recorded. Default: False
Returns:
output(List(dict)): A list of dict. Each dict contains:
"module_name": (str) the module name in the model
"module_type": (str) the module type
"attr": (str) the attr of module. Maybe input/output/weight/bias. Multi-inputs will be suffixed by input-i(i>=0)
"data": (Tensor) the featuremap
"scale": (Tensor, None) the scale of the feature if it has.
"ch_axis": (int) the channel axis of the quantized data for each channel.
"ff_method": (str) actual function name if module_type is FloatFunctional or QFunctional, otherwise, None
"""
def profile_featuremap(
featuremap: List[Dict],
with_tensorboard: bool = False,
tensorboard_dir: Optional[str] = None,
print_per_channel_scale: bool = False,
show_per_channel: bool = False,
out_dir: Optional[str] = None,
file_name: Optional[str] = None,
):
"""Profile featuremap value with log or tensorboard.
Print min/max/mean/var/scale of each feature profiled by `get_raw_features`by default. If `with_tensorboard` is set to True, the histogram of each feature will be shown in tensorboard, which is useful to view the data distribution.
If you want to get more info about features, define your customized profile functions to process the results of `get_raw_features`.
Arguments:
featuremap: raw featuremaps returned by `get_raw_features`
with_tensorboard: whether to use the tensorboard. Default: False
tensorboard_dir: path to the tensorboard log file. Default: None
print_per_channel_scale: whether to print the quantization scale per channel. Default: False
show_per_channel: show each featuremap in per channel ways in tensorboard. Default: False
out_dir: path to save the result .txt and image files. If None, save them in the current directory. Default: None
file_name: result file name. If None, save the result .txt and image files with the name 'statistic'.(statistic.txt and statistic.html). Default: None
"""
Notes:
By default, the two interfaces are used in combination.
profile_featuremap(get_raw_features(model, example_inputs), with_tensorboard=True).By default, the statistics results will be saved to
statistic.txt. The results will also be plotted and saved tostatistic.html, which can be opened and viewed in a browser.To collect other information, you can customize the feature map statistical processing function to process the data returned by
get_raw_features.The function
get_raw_featuresinsertshooksto record the input and output of each layer in the model. However, the pytorch communityhooksdo not supportkwargsnow (refer to here), which will cause two problems:cat((x,y), 1): The parameterdim=1will be filtered out and only two tensors x and y will be recorded, which is as expected.cat(x=(x,y), dim=1): Two keyword arguments will not work at the runtime of the hook. Make sure that thetorch.tensorparameters in model forward are not passed in the form of keyword parameters.
7.3.8.3.1. Example¶
import torch
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
import horizon_plugin_pytorch as D-Robotics
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.utils.quant_profiler import (
get_raw_features,
profile_featuremap,
)
class Net(nn.Module):
def __init__(self, quant=False, share_op=True):
super(Net, self).__init__()
self.quant_stubx = QuantStub()
self.quant_stuby = QuantStub()
self.mul_op = FloatFunctional()
self.cat_op = FloatFunctional()
self.quantized_ops = nn.Sequential(
nn.ReLU(),
nn.Sigmoid(),
nn.Softmax(),
nn.SiLU(),
horizon_nn.Interpolate(
scale_factor=2, recompute_scale_factor=True
),
horizon_nn.Interpolate(
scale_factor=2.3, recompute_scale_factor=True
),
nn.AvgPool2d(kernel_size=4),
nn.Upsample(scale_factor=1.3, mode="bilinear"),
nn.UpsamplingBilinear2d(scale_factor=0.7),
)
self.dequant_stub = DeQuantStub()
self.float_ops = nn.Sequential(
nn.Tanh(),
nn.LeakyReLU(),
nn.PReLU(),
nn.UpsamplingNearest2d(scale_factor=0.7),
)
self.quant = quant
self.share_op = share_op
def forward(self, x, y):
x = self.quant_stubx(x)
y = self.quant_stuby(y)
z = self.mul_op.mul(x, y)
x = self.cat_op.cat((x, y), dim=1)
if self.share_op:
x = self.cat_op.cat((x, y), dim=1)
x = self.quantized_ops(x)
x = self.dequant_stub(x)
if not self.quant:
x = self.float_ops(x)
return x
set_march(March.XXX)
device = torch.device("cuda")
float_net = Net(quant=True, share_op=True).to(device)
float_net.qconfig = horizon.quantization.get_default_qat_qconfig()
qat_net = horizon.quantization.prepare_qat(float_net, inplace=False)
qat_net = qat_net.to(device)
data = torch.arange(1 * 3 * 4 * 4) / 100 + 1
data = data.reshape((1, 3, 4, 4))
data = data.to(torch.float32).to(device)
profile_featuremap(get_raw_features(qat_net, (data, data)), True)
The following files will be generated in the current directory or the directory specified by the out_dir parameter:
statistic.txt
+----------------+----------------------------+----------------------------------------------------------------------------------+---------------------+------------+------------+------------+-----------+-----------+
| Module Index | Module Name | Module Type | Input/Output/Attr | Min | Max | Mean | Var | Scale |
|----------------+----------------------------+----------------------------------------------------------------------------------+---------------------+------------+------------+------------+-----------+-----------|
| 0 | quant_stubx | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | input | -2.9995410 | 2.9934216 | -0.0161597 | 3.0371325 | |
| 0 | quant_stubx | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | output | -3.0000000 | 3.0000000 | -0.0133929 | 3.2358592 | 1.0000000 |
| 1 | quant_stuby | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | input | 0.5000594 | 0.9993884 | 0.7544266 | 0.0207558 | |
| 1 | quant_stuby | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | output | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 2 | mul_op[mul] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-0 | -3.0000000 | 3.0000000 | -0.0133929 | 3.2358592 | 1.0000000 |
| 2 | mul_op[mul] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-1 | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 2 | mul_op[mul] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | output | -3.0000000 | 3.0000000 | -0.0133929 | 3.2358592 | 1.0000000 |
| 3 | cat_op[cat] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-0-0 | -3.0000000 | 3.0000000 | -0.0133929 | 3.2358592 | 1.0000000 |
| 3 | cat_op[cat] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-0-1 | -3.0000000 | 3.0000000 | -0.0133929 | 3.2358592 | 1.0000000 |
| 3 | cat_op[cat] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | output | -3.0000000 | 3.0000000 | -0.0133929 | 3.2346549 | 1.0000000 |
| 4 | cat_op(1)[cat] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-0 | -3.0000000 | 3.0000000 | -0.0133929 | 3.2346549 | 1.0000000 |
| 4 | cat_op(1)[cat] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-1 | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 4 | cat_op(1)[cat] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | output | -3.0000000 | 3.0000000 | 0.3244048 | 2.3844402 | 1.0000000 |
| 5 | quantized_ops.0 | <class 'horizon_plugin_pytorch.nn.qat.relu.ReLU'> | input | -3.0000000 | 3.0000000 | 0.3244048 | 2.3844402 | 1.0000000 |
| 5 | quantized_ops.0 | <class 'horizon_plugin_pytorch.nn.qat.relu.ReLU'> | output | 0.0000000 | 3.0000000 | 0.8363096 | 0.7005617 | 1.0000000 |
| 6 | quantized_ops.1 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | input | 0.0000000 | 3.0000000 | 0.8363096 | 0.7005617 | 1.0000000 |
| 6 | quantized_ops.1 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | output | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 7 | quantized_ops.2.sub[sub] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-0 | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 7 | quantized_ops.2.sub[sub] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-1 | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 7 | quantized_ops.2.sub[sub] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 8 | quantized_ops.2.exp | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 8 | quantized_ops.2.exp | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | output | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 9 | quantized_ops.2.sum[sum] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 9 | quantized_ops.2.sum[sum] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | output | 18.0000000 | 18.0000000 | 18.0000000 | 0.0000000 | 1.0000000 |
| 10 | quantized_ops.2.reciprocal | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | input | 18.0000000 | 18.0000000 | 18.0000000 | 0.0000000 | 1.0000000 |
| 10 | quantized_ops.2.reciprocal | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 11 | quantized_ops.2.mul[mul] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-0 | 1.0000000 | 1.0000000 | 1.0000000 | 0.0000000 | 1.0000000 |
| 11 | quantized_ops.2.mul[mul] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input-1 | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 11 | quantized_ops.2.mul[mul] | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 12 | quantized_ops.3 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 12 | quantized_ops.3 | <class 'horizon_plugin_pytorch.nn.qat.segment_lut.SegmentLUT'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 13 | quantized_ops.4 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 13 | quantized_ops.4 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 14 | quantized_ops.5 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 14 | quantized_ops.5 | <class 'horizon_plugin_pytorch.nn.qat.interpolate.Interpolate'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 15 | quantized_ops.6 | <class 'horizon_plugin_pytorch.nn.qat.avg_pool2d.AvgPool2d'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 15 | quantized_ops.6 | <class 'horizon_plugin_pytorch.nn.qat.avg_pool2d.AvgPool2d'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 16 | quantized_ops.7 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.Upsample'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 16 | quantized_ops.7 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.Upsample'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 17 | quantized_ops.8 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.UpsamplingBilinear2d'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 17 | quantized_ops.8 | <class 'horizon_plugin_pytorch.nn.qat.upsampling.UpsamplingBilinear2d'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 18 | dequant_stub | <class 'horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub'> | input | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 |
| 18 | dequant_stub | <class 'horizon_plugin_pytorch.nn.qat.stubs.DeQuantStub'> | output | 0.0000000 | 0.0000000 | 0.0000000 | 0.0000000 | |
+----------------+----------------------------+----------------------------------------------------------------------------------+---------------------+------------+------------+------------+-----------+-----------+
statistic.html
If with_tensorboard=True is set, the tensorboard log file will be generated in the specified directory, which can be opened and viewed with tensorboard.
7.3.8.4. Step Quantization¶
In cases where the metrics do not go up due to difficulties in training the QAT model, you may need to use step-by-step quantization to find the accuracy bottleneck, by setting qconfig=None to keep some operators in QAT models still floating point.
# from horizon_plugin_pytorch.quantization import prepare_qat
def prepare_qat(
model: torch.nn.Module,
mapping: Optional[Dict[torch.nn.Module, torch.nn.Module]] = None,
inplace: bool = False,
optimize_graph: bool = False,
hybrid: bool = False,
):
"""
Arguments:
hybrid: whether to generate a hybrid model that some intermediate operations are computed in float. There are some constraints for this functionality now:
1. The hybrid model cannot pass check_model and cannot be compiled.
2. Some quantized operations cannot directly accept input from float operation. You need to manually insert QuantStub.
"""
Attention
Quantized operator → floating point operator: The output type of the quantized operator is
QTensor.QTensoris not allowed to be used as the input of floating point operator by default, which causes aNotImplementedErrorerror when forwarding. To solve this problem, users can use the above interface to abolish this restriction.Floating point operator → quantized operator: The implementation of quantization operators in QAT model is generally in the form of
float operator + FakeQuant. So in most cases, the quantization operators can directly useTensoras input. Due to the requirement of alignment with quantized operators, a few operators need the input scale information during QAT, which restricts the input type to beQTensor. Some checks are added for this case. Users must insertQuantStubmanually between floating point and quantized operators when encountering related errors.
7.3.8.4.1. Example¶
import numpy as np
import pytest
import torch
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn import qat
from horizon_plugin_pytorch.quantization import (
get_default_qat_qconfig,
prepare_qat,
)
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
class HyperQuantModel(nn.Module):
def __init__(self, channels=3) -> None:
super().__init__()
self.quant = QuantStub()
self.conv0 = nn.Conv2d(channels, channels, 1)
self.conv1 = nn.Conv2d(channels, channels, 1)
self.conv2 = nn.Conv2d(channels, channels, 1)
self.dequant = DeQuantStub()
def forward(self, input):
x = self.quant(input)
x = self.conv0(x)
x = self.conv1(x)
x = self.conv2(x)
return self.dequant(x)
def set_qconfig(self):
self.qconfig = get_default_qat_qconfig()
self.conv1.qconfig = None
shape = np.random.randint(10, 20, size=4).tolist()
data = torch.rand(size=shape)
model = HyperQuantModel(shape[1])
model.set_qconfig()
set_march(March.XXX)
qat_model = prepare_qat(model, hybrid=True)
assert isinstance(qat_model.conv0, qat.Conv2d)
# conv1 module in QAT model is still float conv
assert isinstance(qat_model.conv1, nn.Conv2d)
assert isinstance(qat_model.conv2, qat.Conv2d)
qat_model(data)
7.3.8.6. Fuse Check¶
The correctness of the model fusion includes two aspects:
Whether the operators that can be fused are all fused.
Is the fused operator correct?
This interface can only check the first case. For the second case, please use the similarity comparison tool to compare the feature similarity of the model before and after fusion. If it is found that the similarity of all features after a specific fused operator is very slow, there may be a problem in this operator fusion process. The fusion process combines several operators into one and replaces original operators in the other positions with Identity, so it may be normal that the feature similarity is low in these Identity positions.
This interface only allows floating point model as input.
# from horizon_plugin_pytorch.utils.quant_profiler import check_unfused_operations
def check_unfused_operations(
model: torch.nn.Module, example_inputs, print_tabulate=True
):
"""
Check unfused modules in a model.
Note: This function is only capable to check unfused modules. For the correctness of fusion, please use `featuremap_similarity` to compare the feature between fused and unfused model.
Arguments:
model (torch.nn.Module): The input model.
example_inputs (Any[Tensor]): The input data feed to model.
print_tabulate (bool, optional): Whether to print the result as tabulate. By default, True.
Returns:
List[List[str]]:
The qualified name of modules that can be fused.
"""
7.3.8.6.1. Example¶
import horizon_plugin_pytorch as D-Robotics
import numpy as np
import torch
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.utils.quant_profiler import check_unfused_operations
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
class Conv2dModule(nn.Module):
def __init__(
self,
in_channels,
out_channels,
kernel_size=1,
stride=1,
padding=0,
dilation=1,
groups=1,
bias=True,
padding_mode="zeros",
):
super().__init__()
self.conv2d = nn.Conv2d(
in_channels,
out_channels,
kernel_size,
stride,
padding,
dilation,
groups,
bias,
padding_mode,
)
self.add = FloatFunctional()
self.bn_mod = nn.BatchNorm2d(out_channels)
self.relu_mod = nn.ReLU()
def forward(self, x, y):
x = self.conv2d(x)
x = self.bn_mod(x)
x = self.add.add(x, y)
x = self.relu_mod(x)
return x
def fuse_model(self):
from horizon_plugin_pytorch.quantization import fuse_modules
fuse_list = ["conv2d", "bn_mod", "add", "relu_mod"]
fuse_modules(
self,
fuse_list,
inplace=True,
)
class TestFuseNet(nn.Module):
def __init__(self, channels) -> None:
super().__init__()
self.convmod1 = Conv2dModule(channels, channels)
self.convmod2 = Conv2dModule(channels, channels)
self.convmod3 = Conv2dModule(channels, channels)
self.shared_conv = nn.Conv2d(channels, channels, 1)
self.bn1 = nn.BatchNorm2d(channels)
self.bn2 = nn.BatchNorm2d(channels)
self.sub = FloatFunctional()
self.relu = nn.ReLU()
def forward(self, x, y):
x = self.convmod1(x, y)
x = self.convmod2(y, x)
x = self.convmod3(x, y)
x = self.shared_conv(x)
x = self.bn1(x)
y = self.shared_conv(y)
y = self.bn2(y)
x = self.sub.sub(x, y)
x = self.relu(x)
return x
def fuse_model(self):
self.convmod1.fuse_model()
self.convmod3.fuse_model()
shape = np.random.randint(10, 20, size=4).tolist()
data0 = torch.rand(size=shape)
data1 = torch.rand(size=shape)
float_net = TestFuseNet(shape[1])
float_net.fuse_model()
check_unfused_operations(float_net, (data0, data1))
Output:
name type
------------------- ------------------------------------------------
shared_conv(shared) <class 'torch.nn.modules.conv.Conv2d'>
bn1 <class 'torch.nn.modules.batchnorm.BatchNorm2d'>
name type
------------------- ------------------------------------------------
shared_conv(shared) <class 'torch.nn.modules.conv.Conv2d'>
bn2 <class 'torch.nn.modules.batchnorm.BatchNorm2d'>
name type
----------------- --------------------------------------------------------------------------------
convmod2.conv2d <class 'torch.nn.modules.conv.Conv2d'>
convmod2.bn_mod <class 'torch.nn.modules.batchnorm.BatchNorm2d'>
convmod2.add <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'>
convmod2.relu_mod <class 'torch.nn.modules.activation.ReLU'>
7.3.8.7. Single-operator Conversion Precision Debugging¶
If the accuracy degrades after QAT to fixed point conversion, users may need to replace some operators in the quantized model with QAT to verify if these operators cause the accuracy problem.
# from horizon_plugin_pytorch.utils.quant_profiler import set_preserve_qat_mode
def set_preserve_qat_mode(model: nn.Module, prefixes=(), types=(), value=True):
"""
Make modules in the model to preserve qat mode in convert by setting
mod.preserve_qat_mode attribute. It can be used on float model or qat
model.
Note:
1) For fused module, only conv.preserve_qat_mode = True,
fused.preserve_qat_mode = True. So setting the fused.preserve_qat_mode
= True is same as setting conv.preserve_qat_mode = True. For example,
class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__()
self.conv = torch.nn.Conv2d()
self.bn = torch.nn.BatchNorm2d()
self.add = FloatFunctional()
self.relu = torch.nn.Relu()
float_model = Model()
# set float conv is OK
set_preserve_qat_mode(float_model, types=(torch.nn.Conv2d,))
# set float bn does not work
set_preserve_qat_mode(float_model, types=(torch.nn.BatchNorm2d,))
float_model.fuse_modules()
float_model.qconfig = get_default_qat_qconfig()
qat_model = prepare_qat(float_model)
# After fuse and convert, set conv via float type is also OK.
# All conv modules and fused modules(convbn, convbnadd, ...)
# will set preserve_qat_mode = True
set_preserve_qat_mode(qat_model, types=(torch.nn.Conv2d,))
# To set exactly one fused module, use 'prefixes' arg.
# convbnaddrelu is fused on "add" position
set_preserve_qat_mode(qat_model, prefixes=("add",))
2) If float model uses torch functions(torch.add, torch.pow, ...) and
is converted by fx, this functions will be converted to D-Robotics ops
automatically. To set these functions preserve_qat_mode = True, please
set corresponding D-Robotics ops preserve_qat_mode = True in qat model.
For example,
class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__()
self.add = torch.add
float_model = Model()
# convert by fx
qat_model = prepare_qat_fx(float_model)
# set by types is OK. All FloatFunctional in qat model will
# be set preserve_qat_mode = True
set_preserve_qat_mode(qat_model, types=(FloatFunctional,))
# To set exactly this add, use 'prefixes' arg
# "add_generated_add_0" is the generated add module name
set_preserve_qat_mode(qat_model, prefixes=("add_generated_add_0",))
Arguments:
model (nn.Module): The model to be modified.
prefixes (tuple, optional):
Set preserve_qat_mode by the prefix of qualified name.
By default, tuple().
types (tuple, optional):
Set preserve_qat_mode by module type. Defaults to tuple().
If float model, types must be float module types
If QAT model, types can be float or qat module types
value (bool, optional):
Set preserve_qat_mode to this value.
By default, True.
"""
7.3.8.7.1. Example¶
import horizon_plugin_pytorch as D-Robotics
import numpy as np
import torch
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.utils.quant_profiler import set_preserve_qat_mode
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
class Conv2dModule(nn.Module):
def __init__(
self,
in_channels,
out_channels,
kernel_size=1,
stride=1,
padding=0,
dilation=1,
groups=1,
bias=True,
padding_mode="zeros",
):
super().__init__()
self.conv2d = nn.Conv2d(
in_channels,
out_channels,
kernel_size,
stride,
padding,
dilation,
groups,
bias,
padding_mode,
)
self.add = FloatFunctional()
self.bn_mod = nn.BatchNorm2d(out_channels)
self.relu_mod = nn.ReLU()
def forward(self, x, y):
x = self.conv2d(x)
x = self.bn_mod(x)
x = self.add.add(x, y)
x = self.relu_mod(x)
return x
def fuse_model(self):
from horizon_plugin_pytorch.quantization import fuse_modules
fuse_list = ["conv2d", "bn_mod", "add", "relu_mod"]
fuse_modules(
self,
fuse_list,
inplace=True,
)
class TestFuseNet(nn.Module):
def __init__(self, channels) -> None:
super().__init__()
self.convmod1 = Conv2dModule(channels, channels)
self.convmod2 = Conv2dModule(channels, channels)
self.convmod3 = Conv2dModule(channels, channels)
self.shared_conv = nn.Conv2d(channels, channels, 1)
self.bn1 = nn.BatchNorm2d(channels)
self.bn2 = nn.BatchNorm2d(channels)
self.sub = FloatFunctional()
self.relu = nn.ReLU()
def forward(self, x, y):
x = self.convmod1(x, y)
x = self.convmod2(y, x)
x = self.convmod3(x, y)
x = self.shared_conv(x)
x = self.bn1(x)
y = self.shared_conv(y)
y = self.bn2(y)
x = self.sub.sub(x, y)
x = self.relu(x)
return x
def fuse_model(self):
self.convmod1.fuse_model()
self.convmod3.fuse_model()
model = TestFuseNet(3)
model.fuse_model()
model.qconfig = horizon.quantization.get_default_qat_qconfig()
# use the interface to set, or manually set preserve_qat_mode=True
set_preserve_qat_mode(float_net, ("convmod1"), ())
model.convmod1.preserve_qat_mode = True
set_march(March.XXX)
horizon.quantization.prepare_qat(model, inplace=True)
quant_model = horizon.quantization.convert(model.eval(), inplace=False)
# convmod1.add in the fixed-point model is still qat.ConvAddReLU2d
assert isinstance(quant_model.convmod1.add, qat.ConvAddReLU2d)
7.3.8.8. Quantitative Configuration Check¶
This function checks the quantization configuration of each layer in the QAT model. Input must be a QAT model. The results will be saved into qconfig_info.txt .
# from horizon_plugin_pytorch.utils.quant_profiler import check_qconfig
def check_qconfig(
model: torch.nn.Module,
example_inputs: Any,
prefixes: Tuple = (),
types: Tuple = (),
custom_check_func: Optional[Callable] = None,
out_dir: Optional[str] = None,
):
"""Check the quantization configuration of the QAT model.
This function
1) checks activation and weight quantization configurations of each layer in the model. These infos will be saved in "qconfig_info.txt".
2) checks input and output types of each layer in the model.
By default, this function prints warnings when checking:
1) activation = None
2) fixed scale observer
3) not qint8 weight
4) model input and output types are different
If you want to check more info, define a customized check function and use `custom_check_func` parameter.
Arguments:
model: MUST be qat model
example_inputs (Any[Tensor]): The input data feed to model.
prefixes: get features info by the prefix of qualified name. Default: tuple().
types: get features info by module type. Default: tuple().
custom_check_func: a user-defined function to check other info. This function is invoked in module hooks, so it has the same signature with torch.nn.Module hooks:
func(module, input, output) -> None
out_dir: path to save the result .txt file 'qconfig_info.txt'. If None, will save in the current directory. Default: None
"""
7.3.8.8.1. Example¶
import horizon_plugin_pytorch as D-Robotics
import numpy as np
import torch
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.quantization import (
convert,
get_default_qat_qconfig,
prepare_qat,
fuse_modules,
)
from horizon_plugin_pytorch.quantization.observer import FixedScaleObserver
from horizon_plugin_pytorch.utils.quant_profiler import check_qconfig
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
class Conv2dModule(nn.Module):
def __init__(
self,
in_channels,
out_channels,
kernel_size=1,
stride=1,
padding=0,
dilation=1,
groups=1,
bias=True,
padding_mode="zeros",
):
super().__init__()
self.conv2d = nn.Conv2d(
in_channels,
out_channels,
kernel_size,
stride,
padding,
dilation,
groups,
bias,
padding_mode,
)
self.add = FloatFunctional()
self.bn_mod = nn.BatchNorm2d(out_channels)
self.relu_mod = nn.ReLU()
def forward(self, x, y):
x = self.conv2d(x)
x = self.bn_mod(x)
x = self.add.add(x, y)
x = self.relu_mod(x)
return x
def fuse_model(self):
from horizon_plugin_pytorch.quantization import fuse_modules
fuse_list = ["conv2d", "bn_mod", "add", "relu_mod"]
fuse_modules(
self,
fuse_list,
inplace=True,
)
class TestFuseNet(nn.Module):
def __init__(self, channels) -> None:
super().__init__()
self.convmod1 = Conv2dModule(channels, channels)
self.convmod2 = Conv2dModule(channels, channels)
self.convmod3 = Conv2dModule(channels, channels)
self.shared_conv = nn.Conv2d(channels, channels, 1)
self.bn1 = nn.BatchNorm2d(channels)
self.bn2 = nn.BatchNorm2d(channels)
self.sub = FloatFunctional()
self.relu = nn.ReLU()
def forward(self, x, y):
x = self.convmod1(x, y)
x = self.convmod2(y, x)
x = self.convmod3(x, y)
x = self.shared_conv(x)
x = self.bn1(x)
y = self.shared_conv(y)
y = self.bn2(y)
x = self.sub.sub(x, y)
x = self.relu(x)
return x
def fuse_model(self):
self.convmod1.fuse_model()
self.convmod3.fuse_model()
float_net = TestFuseNet(3)
float_net.fuse_model()
set_march(March.XXX)
# some unsupported or special cases
float_net.qconfig = get_default_qat_qconfig(weight_dtype="qint16")
float_net.sub.qconfig = get_default_qat_qconfig(
activation_qkwargs={
"observer": FixedScaleObserver,
"scale": 1 / 2 ** 15,
"dtype": "qint16",
}
)
qat_net = prepare_qat(float_net)
shape = np.random.randint(10, 20, size=4).tolist()
shape[1] = 3
data0 = torch.rand(size=shape)
data1 = torch.rand(size=shape)
check_qconfig(qat_net, (data0, data1))
Output:
qconfig_info.txt
Each layer out qconfig:
+-----------------+----------------------------------------------------------------------------------+--------------------+-------------+---------------+-----------+
| Module Name | Module Type | Input dtype | out dtype | per-channel | ch_axis |
|-----------------+----------------------------------------------------------------------------------+--------------------+-------------+---------------+-----------|
| quantx | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | [torch.float32] | qint8 | False | -1 |
| quanty | <class 'horizon_plugin_pytorch.nn.qat.stubs.QuantStub'> | [torch.float32] | qint8 | False | -1 |
| convmod1.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | ['qint8', 'qint8'] | qint8 | False | -1 |
| convmod2.conv2d | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | ['qint8'] | qint8 | False | -1 |
| convmod2.bn_mod | <class 'horizon_plugin_pytorch.nn.qat.batchnorm.BatchNorm2d'> | ['qint8'] | qint8 | False | -1 |
| convmod2.add | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | ['qint8', 'qint8'] | qint8 | False | -1 |
| convmod3.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | ['qint8', 'qint8'] | qint8 | False | -1 |
| shared_conv | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | ['qint8'] | qint8 | False | -1 |
| bn1 | <class 'horizon_plugin_pytorch.nn.qat.batchnorm.BatchNorm2d'> | ['qint8'] | qint8 | False | -1 |
| shared_conv(1) | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | ['qint8'] | qint8 | False | -1 |
| bn2 | <class 'horizon_plugin_pytorch.nn.qat.batchnorm.BatchNorm2d'> | ['qint8'] | qint8 | False | -1 |
| sub | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | ['qint8', 'qint8'] | qint16 | False | -1 |
+-----------------+----------------------------------------------------------------------------------+--------------------+-------------+---------------+-----------+
Weight qconfig:
+-----------------+--------------------------------------------------------------+----------------+---------------+-----------+
| Module Name | Module Type | weight dtype | per-channel | ch_axis |
|-----------------+--------------------------------------------------------------+----------------+---------------+-----------|
| convmod1.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | qint16 | True | 0 |
| convmod2.conv2d | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 | True | 0 |
| convmod3.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | qint16 | True | 0 |
| shared_conv | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 | True | 0 |
| shared_conv(1) | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 | True | 0 |
+-----------------+--------------------------------------------------------------+----------------+---------------+-----------+
Please check if these operators qconfigs are expected.
+-----------------+----------------------------------------------------------------------------------+------------------------------------------------------------------+
| Module Name | Module Type | Msg |
|-----------------+----------------------------------------------------------------------------------+------------------------------------------------------------------|
| convmod1.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | qint16 weight!!! |
| convmod2.conv2d | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 weight!!! |
| convmod3.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | qint16 weight!!! |
| shared_conv | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 weight!!! |
| shared_conv(1) | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 weight!!! |
| sub | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | Fixed scale 3.0517578125e-05 |
| sub | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input dtype ['qint8', 'qint8'] is not same with out dtype qint16 |
+-----------------+----------------------------------------------------------------------------------+------------------------------------------------------------------+
screen output
Please check if these operators qconfigs are expected.
+-----------------+----------------------------------------------------------------------------------+------------------------------------------------------------------+
| Module Name | Module Type | Msg |
|-----------------+----------------------------------------------------------------------------------+------------------------------------------------------------------|
| convmod1.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | qint16 weight!!! |
| convmod2.conv2d | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 weight!!! |
| convmod3.add | <class 'horizon_plugin_pytorch.nn.qat.conv2d.ConvAddReLU2d'> | qint16 weight!!! |
| shared_conv | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 weight!!! |
| shared_conv(1) | <class 'horizon_plugin_pytorch.nn.qat.conv2d.Conv2d'> | qint16 weight!!! |
| sub | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | Fixed scale 3.0517578125e-05 |
| sub | <class 'horizon_plugin_pytorch.nn.quantized.functional_modules.FloatFunctional'> | input dtype ['qint8', 'qint8'] is not same with out dtype qint16 |
+-----------------+----------------------------------------------------------------------------------+------------------------------------------------------------------+
7.3.8.9. Model Weight Comparison¶
This function calculates the similarity of the weight of each layer in the model. The results will be shown on the screen and saved into the file at the same time by default. Users can also set with_tensorboard=True to draw the histograms of weight in tensorboard, which is convenient for more intuitive comparison.
# from horizon_plugin_pytorch.utils.quant_profiler import compare_weights
def compare_weights(
float_model: torch.nn.Module,
qat_quantized_model: torch.nn.Module,
similarity_func="Cosine",
with_tensorboard: bool = False,
tensorboard_dir: Optional[str] = None,
out_dir: Optional[str] = None,
) -> Dict[str, Dict[str, torch.Tensor]]:
"""Compare weights of float/qat/quantized models.
This function compares weights of each layer based on torch.quantization._numeric_suite.compare_weights. The weight similarity and atol will be printed on the screen and saved in "weight_comparison.txt".
If you want to see histogram of weights, set with_tensorboard=True.
Arguments:
float_model: floating-point model
qat_quantized_model: qat or quantized model
similarity_func: similarity computation function. Support "Cosine", "MSE", "L1", "KL", "SQNR" or any user-defined Callable object. If it is a user-defined object, it should return a scalar or tensor with only one number. Otherwise the result shown may be unexpected. Default: "Cosine"
with_tensorboard: whether to use tensorboard. Default: False
tensorboard_dir: path to the tensorboard log file. Default: None
out_dir: path to save the result .txt and image files. If None, save them in the current directory. Default: None
Returns:
A weight comparison dict with schema:
* KEY (str): module name (Eg. layer1.0.conv.weight)
* VALUE (dict): a dict of the corresponding weights in two models:
"float": weight value in floating-point model
"quantized": weight value in qat/quantized model
"""
7.3.8.9.1. Example¶
import horizon_plugin_pytorch as D-Robotics
import numpy as np
import torch
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.quantization import (
convert,
get_default_qat_qconfig,
prepare_qat,
fuse_modules,
)
from horizon_plugin_pytorch.utils.quant_profiler import compare_weights
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
# skip Resnet18 definition here
float_net = Resnet18().to(device)
float_net.fuse_model()
set_march(March.XXX)
float_net.qconfig = get_default_qat_qconfig()
qat_net = prepare_qat(float_net)
qat_net(data)
quantized_net = convert(qat_net)
quantized_net(data)
compare_weights(float_net, qat_net)
The results will be shown on the screen and saved in weight_comparsion.txt.
+-------------------------------------+--------------+-----------+
| Weight Name | Similarity | Atol |
|-------------------------------------+--------------+-----------|
| conv1.conv.weight | 1.0000000 | 0.0000000 |
| layer1.0.conv_cell1.conv.weight | 1.0000000 | 0.0000000 |
| layer1.0.shortcut.conv.weight | 1.0000000 | 0.0000000 |
| layer1.0.conv_cell2.skip_add.weight | 1.0000000 | 0.0000000 |
| layer1.1.conv_cell1.conv.weight | 1.0000000 | 0.0000000 |
| layer1.1.conv_cell2.conv.weight | 1.0000000 | 0.0000000 |
| layer2.0.conv_cell1.conv.weight | 1.0000000 | 0.0000000 |
| layer2.0.shortcut.conv.weight | 1.0000000 | 0.0000000 |
| layer2.0.conv_cell2.skip_add.weight | 1.0000000 | 0.0000000 |
| layer2.1.conv_cell1.conv.weight | 1.0000000 | 0.0000001 |
| layer2.1.conv_cell2.conv.weight | 1.0000000 | 0.0000001 |
| layer3.0.conv_cell1.conv.weight | 1.0000000 | 0.0000001 |
| layer3.0.shortcut.conv.weight | 1.0000000 | 0.0000001 |
| layer3.0.conv_cell2.skip_add.weight | 1.0000000 | 0.0000002 |
| layer3.1.conv_cell1.conv.weight | 1.0000000 | 0.0000005 |
| layer3.1.conv_cell2.conv.weight | 1.0000001 | 0.0000008 |
| conv2.conv.weight | 1.0000001 | 0.0000010 |
| pool.conv.weight | 0.9999999 | 0.0000024 |
| fc.weight | 1.0000000 | 0.0000172 |
+-------------------------------------+--------------+-----------+
7.3.8.10. Deploy Device Check of Hybrid Models¶
Horizon_plugin_pytorch supports constructing and deploying heterogeneous models through fx . This interface checks whether each operator in the model runs on the BPU or the CPU when it is finally deployed.
# from horizon_plugin_pytorch.utils.quant_profiler import check_deploy_device
def check_deploy_device(
model: torch.fx.GraphModule,
print_tabulate: bool = True,
out_dir: Optional[str] = None,
) -> Dict[str, Tuple[str, str]]:
"""Check deploy device(BPU or CPU) of hybrid model.
Arguments:
model: qat or quantized model. MUST be converted by prepare_qat_fx.
print_tabulate (bool, optional): Whether to print the result as tabulate. By default, True.
out_dir: path to save the result .txt file 'deploy_device.txt'. If None, save it in the current directory. Default: None
Returns:
A dict of model deploy infos with schema
* KEY (str): module name
* VALUE (Tuple): (deploy device(BPU or CPU), module type)
"""
7.3.8.10.1. Example¶
import numpy as np
import torch
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn import qat
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.quantization import (
get_default_qat_qconfig,
get_default_qat_out_qconfig,
get_default_calib_qconfig,
prepare_calibration_fx,
prepare_qat_fx,
convert_fx,
)
from horizon_plugin_pytorch.utils.quant_profiler import check_deploy_device
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
class _ConvBlock(nn.Module):
def __init__(self, channels=3):
super().__init__()
self.conv = nn.Conv2d(channels, channels, 1)
self.prelu = torch.nn.PReLU()
def forward(self, input):
x = self.conv(input)
x = self.prelu(x)
return torch.nn.functional.selu(x)
class _SeluModule(nn.Module):
def forward(self, input):
return torch.nn.functional.selu(input)
class HybridModel(nn.Module):
def __init__(self, channels=3):
super().__init__()
self.quant = QuantStub()
self.conv0 = nn.Conv2d(channels, channels, 1)
self.prelu = torch.nn.PReLU()
self.conv1 = _ConvBlock(channels)
self.conv2 = nn.Conv2d(channels, channels, 1)
self.conv3 = nn.Conv2d(channels, channels, 1)
self.conv4 = nn.Conv2d(channels, channels, 1)
self.selu = _SeluModule()
self.dequant = DeQuantStub()
self.identity = torch.nn.Identity()
self.add = FloatFunctional()
def forward(self, input):
x = self.quant(input)
x = self.conv0(x)
x = self.identity(x)
x = self.prelu(x)
x = torch.nn.functional.selu(x)
x = self.conv1(x)
x = self.conv2(x)
x = self.conv3(x)
x = self.identity(x)
y = self.conv4(x)
x = self.add.add(x, y)
x = self.selu(x)
return self.dequant(x)
set_march(March.XXX)
shape = np.random.randint(10, 20, size=4).tolist()
infer_shape = [1] + shape[1:]
infer_data = torch.rand(size=infer_shape)
model = HybridModel(shape[1])
float_res = model(infer_data)
# must use the fx interface
calibration_model = prepare_calibration_fx(
model,
{
"": get_default_calib_qconfig(),
},
hybrid=True,
hybrid_dict={
"module_name": ["conv1.conv", "conv3"],
"module_type": [_SeluModule],
},
)
calibration_model.eval()
for i in range(5):
calibration_model(torch.rand(size=shape))
qat_model = prepare_qat_fx(
calibration_model,
{
"": get_default_qat_qconfig(),
"module_name": [("conv4", get_default_qat_out_qconfig())],
},
hybrid=True,
hybrid_dict={
"module_name": ["conv1.conv", "conv3"],
"module_type": [_SeluModule],
},
)
qat_res = qat_model(infer_data)
check_deploy_device(qat_model)
quantize_model = convert_fx(qat_model)
check_deploy_device(quantize_model)
The results will be shown on the screen and saved in deploy_device.txt.
name deploy device type
------------------------------ --------------- --------
quant CPU module
conv0 BPU module
prelu_input_dequant CPU module
prelu CPU module
selu CPU function
conv1.conv CPU module
conv1.prelu CPU module
selu_1 CPU function
selu_1_activation_post_process CPU module
conv2 BPU module
conv3_input_dequant CPU module
conv3 CPU module
conv3_activation_post_process CPU module
add_1 BPU method
selu_2_input_dequant CPU module
selu_2 CPU function
dequant CPU module
7.3.8.11. Integrated Interface¶
For ease of use and viewing, an integrated interface model_profiler is also provided in horizon_plugin_pytorch, which calls other debugging tools and displays the results in one html page.
# from horizon_plugin_pytorch.utils.quant_profiler import model_profiler
def model_profiler(
model1: torch.nn.Module,
model2: torch.nn.Module,
example_inputs: Any,
mode: str,
out_dir: Optional[str] = None,
kwargs_dict: Optional[dict] = None,
):
"""Profiler the models using debug tools and show result in one page.
This function computes
1) similarity, statistics, weights similarity and shared operators of the given models
2) check unfused operators of the floating-point model and qconfig of the qat model, which controlled by `mode`
The results are shown in one html page named `profiler.html`, which stored in default dir or `out_dir`.
Notes:
1) Only support models compared in any two adjacent stages.
`float vs qat` or `qat vs quantized` is supported, while
`float vs quantized` or `qat vs float` is unsupported.
2) Visual model structures in onnx format and featuremap histogram are not shown in the html file. You can call `export_to_onnx/export_quantized_onnx` and `profile_featuremap` with `with_tensorboard=True`. Customized arguments can also be passed by `kwargs_dict`.
Arguments:
model1: can be float/calibration/qat model
model2: can be calibration/qat/quantized model
example_inputs: model inputs
mode: specify the two models to be compared. Only three modes shown below are supported
"FvsQ": floating-point vs qat. In this mode, `model2` can be either calibration or qat model.
"QvsQ": qat vs quantized.
"CvsQ": calibration vs qat.
out_dir: path to save `profiler.html` and all other result files. If None, results are saved in `horizon_quant_debug` dir in current dir
kwargs_dict: kwargs of debug tools functions in dict format. E.g.
kwargs_dict = {
"featuremap_similarity": {
"similarity_func": Cosine,
},
"profile_featuremap": {
"with_tensorboard": True,
}
...
}
Only support 7 keys, which are the names of the 7 debug functions that will be invoked in this function. The supported keys are:
1) featuremap_similarity
2) get_raw_features
3) profile_featuremap
4) get_module_called_count
5) check_unfused_operations
6) compare_weights
7) check_qconfig
Notes:
1) model and example_inputs must not be defined in kwargs
2) `out_dir` in kwargs will be replaced with `out_dir` in
`model_profiler` arguments
"""
7.3.8.11.1. Example¶
import numpy as np
import pytest
import torch
from torch import nn
from torch.quantization import DeQuantStub, QuantStub
import horizon_plugin_pytorch as D-Robotics
from horizon_plugin_pytorch import nn as horizon_nn
from horizon_plugin_pytorch.march import March, set_march
from horizon_plugin_pytorch.nn.quantized import FloatFunctional
from horizon_plugin_pytorch.qat_mode import QATMode, set_qat_mode
from horizon_plugin_pytorch.quantization import (
convert,
get_default_qat_qconfig,
prepare_qat,
fuse_modules,
)
from horizon_plugin_pytorch.utils.quant_profiler import model_profiler
class Conv2dModule(nn.Module):
def __init__(
self,
in_channels,
out_channels,
kernel_size=1,
stride=1,
padding=0,
dilation=1,
groups=1,
bias=True,
padding_mode="zeros",
):
super().__init__()
self.conv2d = nn.Conv2d(
in_channels,
out_channels,
kernel_size,
stride,
padding,
dilation,
groups,
bias,
padding_mode,
)
self.add = FloatFunctional()
self.bn_mod = nn.BatchNorm2d(out_channels)
self.relu_mod = nn.ReLU()
def forward(self, x, y):
x = self.conv2d(x)
x = self.bn_mod(x)
x = self.add.add(x, y)
x = self.relu_mod(x)
return x
def fuse_model(self):
from horizon_plugin_pytorch.quantization import fuse_modules
fuse_list = ["conv2d", "bn_mod", "add", "relu_mod"]
fuse_modules(
self,
fuse_list,
inplace=True,
)
class TestFuseNet(nn.Module):
def __init__(self, channels) -> None:
super().__init__()
self.convmod1 = Conv2dModule(channels, channels)
self.convmod2 = Conv2dModule(channels, channels)
self.convmod3 = Conv2dModule(channels, channels)
self.shared_conv = nn.Conv2d(channels, channels, 1)
self.bn1 = nn.BatchNorm2d(channels)
self.bn2 = nn.BatchNorm2d(channels)
self.sub = FloatFunctional()
self.relu = nn.ReLU()
def forward(self, x, y):
x = self.convmod1(x, y)
x = self.convmod2(y, x)
x = self.convmod3(x, y)
x = self.shared_conv(x)
x = self.bn1(x)
y = self.shared_conv(y)
y = self.bn2(y)
x = self.sub.sub(x, y)
x = self.relu(x)
return x
def fuse_model(self):
self.convmod1.fuse_model()
self.convmod3.fuse_model()
set_march(March.XXX)
device = torch.device("cpu")
data = torch.arange(1 * 3 * 4 * 4) / 100 + 1
data = data.reshape((1, 3, 4, 4))
data = data.to(torch.float32).to(device)
float_net = TestFuseNet(3).to(device)
float_net(data, data)
float_net.qconfig = horizon.quantization.get_default_calib_qconfig()
calib_net = horizon.quantization.prepare_calibration(float_net).to(
device
)
calib_net(data, data)
calib_net.qconfig = horizon.quantization.get_default_qat_qconfig()
qat_net = prepare_qat(calib_net, inplace=False)
qat_net = qat_net.to(device)
qat_net(data, data)
quantized_net = convert(qat_net)
model_profiler(float_net, qat_net, (data, data), mode="FvsQ")
If the out_dir parameter is not specified, the horizon_quant_debug folder will be generated in the current directory. The results of each debug tool and profiler.html will be saved in this folder.
7.3.8.12. GPU Memory Profiler¶
Plugin provides a GPU memory profile tool to locate memory bottlenet and use checkpoint or saved tensor to save memory.
# from horizon_plugin_pytorch.utils.quant_profiler import show_cuda_memory_consumption
def show_cuda_memory_consumption(
model: torch.nn.Module,
example_inputs: Any,
device: torch.device,
check_leaf_module=None,
out_dir: Optional[str] = None,
file_name: Optional[str] = None,
custom_backward=None,
):
"""
Evaluate memory consumption of a model during forward and backward.
Result will be saved as html file.
Known Issue: If checkpoint is used, some result of backward will named
as 'forward', because during backward the forward hook is called,
rather than backward hook.
Args:
model: The input model.
example_inputs (Any[Tensor]): The input data feed to model.
device: Evaluate on this device.
check_leaf_module: A function to check if a module is leaf. Pass None
to use pre-defined `is_leaf_module`. Defaults to None.
out_dir: path to save the result. If None, will save in the current
directory. Default: None
file_name: result file name. If None, will save result with
name 'mem_info'. Default: None
custom_backward: Run backward by the model ret,
must set retain_graph=False. Defaults to None.
"""
7.3.8.12.1. Example¶
# skip MobilenetV1 definition here
float_net = MobilenetV1()
show_cuda_memory_consumption(float_net, data, torch.device("cuda"))
Will generate following result file in PWD.
mem_info.html
