7.4.1. Quantization API

Eager-mode module fusion (aligned with published X5 autodoc).

horizon_plugin_pytorch.quantization.fuse_modules.fuse_modules(model, modules_to_fuse, inplace=False, fuser_func=<function fuse_known_modules>, fuse_custom_config_dict=None)

Fuses a list of modules into a single module.

Fuses only the following sequence of modules: conv, bn; conv, bn, relu; conv, relu; conv, bn, add; conv, bn, add, relu; conv, add; conv, add, relu; linear, bn; linear, bn, relu; linear, relu; linear, bn, add; linear, bn, add, relu; linear, add; linear, add, relu. For these sequences, the first element in the output module list performs the fused operation. The rest of the elements are set to nn.Identity()

Parameters

  • model – Model containing the modules to be fused

  • modules_to_fuse – list of list of module names to fuse. Can also be a list of strings if there is only a single list of modules to fuse.

  • inplace – bool specifying if fusion happens in place on the model, by default a new model is returned

  • fuser_func – Function that takes in a list of modules and outputs a list of fused modules of the same length. For example, fuser_func([convModule, BNModule]) returns the list [ConvBNModule, nn.Identity()] Defaults to torch.ao.quantization.fuse_known_modules

  • fuse_custom_config_dict –

    custom configuration for fusion

    # Example of fuse_custom_config_dict
fuse_custom_config_dict = {
    # Additional fuser_method mapping
    "additional_fuser_method_mapping": {
        (torch.nn.Conv2d, torch.nn.BatchNorm2d): fuse_conv_bn
    },
}

Returns

model with fused modules. A new copy is created if inplace=True.

Examples

>>> # xdoctest: +SKIP
>>> m = M().eval()
>>> # m is a module containing the sub-modules below
>>> modules_to_fuse = [ ['conv1', 'bn1', 'relu1'],
...                   ['submodule.conv', 'submodule.relu']]
>>> fused_m = fuse_modules(
...             m, modules_to_fuse)
>>> output = fused_m(input)
>>> m = M().eval()
>>> # Alternately provide a single list of modules to fuse
>>> modules_to_fuse = ['conv1', 'bn1', 'relu1']
>>> fused_m = fuse_modules(
...             m, modules_to_fuse)
>>> output = fused_m(input)

Eager-mode quantization prepare / convert (aligned with published X5 autodoc).

horizon_plugin_pytorch.quantization.quantize.convert(module: torch.nn.modules.module.Module, mapping: Optional[Dict[Type[torch.nn.modules.module.Module], Type[torch.nn.modules.module.Module]]] = None, inplace: bool = False, remove_qconfig: bool = True, fast_mode: bool = False, swapable_names=None)

Convert modules.

Convert submodules in input module to a different module according to mapping by calling from_float method on the target module class. And remove qconfig at the end if remove_qconfig is set to True.

Parameters

  • module – input module

  • mapping – a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules

  • inplace – carry out model transformations in-place, the original module is mutated

  • fast_mode – whether to accelerate quantized model forward. If set True, quantized model cannot be compiled

  • swapable_names – names of submodules that can be swapped. Defaults to None.

horizon_plugin_pytorch.quantization.quantize.prepare_calibration(model, inplace=False)

Prepare the model for calibration.

Parameters

  • model – Float model with fused ops

  • inplace – carry out model transformations in-place or not. Defaults to False.

horizon_plugin_pytorch.quantization.quantize.prepare_qat(model: torch.nn.modules.module.Module, mapping: Optional[Dict[Type[torch.nn.modules.module.Module], Type[torch.nn.modules.module.Module]]] = None, inplace: bool = False, optimize_graph: bool = False, hybrid: bool = False, optimize_kwargs: Optional[Dict[str, Tuple]] = None, example_inputs: Any = None, qconfig_setter: Optional[Union[Tuple[horizon_plugin_pytorch.quantization.qconfig_template.QconfigSetterBase, ...], horizon_plugin_pytorch.quantization.qconfig_template.QconfigSetterBase]] = None, verbose: int = 0)

Prepare qat.

Prepare a copy of the model for quantization-aware training and converts it to quantized version.

Quantization configuration should be assigned preemptively to individual submodules in .qconfig attribute.

Parameters

  • model – input model to be modified in-place

  • mapping – dictionary that maps float modules to quantized modules to be replaced.

  • inplace – carry out model transformations in-place, the original module is mutated

  • optimize_graph – whether to do some process on origin model for special purpose. Currently only support using torch.fx to fix cat input scale(only used on Bernoulli)

  • hybrid – whether to generate a hybrid model that some intermediate operation is computed in float. There are some constraints for this functionality now: 1. The hybrid model cannot pass check_model and cannot be compiled. 2. Some quantized operation cannot directly accept input from float operation, user need to manually insert QuantStub.

  • optimize_kwargs –

    a dict for optimize graph with the following format:

    optimize_kwargs = {
    # optional, specify which type of optimization to do. Only
    # support "unify_inputs_scale" now
    "opt_types": ("unify_inputs_scale",),
    # optional, modules start with qualified name to optimize
    "module_prefixes": ("backbone.conv",),
    # optional, modules in these types will be optimize
    "module_types": (horizon.nn.qat.conv2d,),
    # optional, functions to optimize
    "functions": (torch.clamp,),
    # optional, methods to optimize. Only support
    # FloatFunctional methods now
    "methods": ("add",),
}

  • example_inputs – model inputs. It is used to trace model or check model structure.

  • qconfig_setter – Qconfig setter. Only needed when using qconfig template.

  • verbose –

    whether check model structure. it has two levels: 0: do nothing 1: check model structure

    1. if model has shared ops

    2. if model has unfused operations

    3. model quantization config

FX quantization prepare / convert / fuse (aligned with published X5 autodoc).

horizon_plugin_pytorch.quantization.quantize_fx.convert_fx(graph_module: Union[horizon_plugin_pytorch.quantization.fx.graph_module.ObservedGraphModule, horizon_plugin_pytorch.fx.jit_scheme.GraphModule], inplace: bool = False, convert_custom_config_dict: Optional[Dict[str, Any]] = None, _remove_qconfig: bool = True, fast_mode: bool = False) → Union[horizon_plugin_pytorch.quantization.fx.graph_module.QuantizedGraphModule, horizon_plugin_pytorch.fx.jit_scheme.GraphModule]

Convert a calibrated or trained model to a quantized model.

Parameters

  • graph_module – A prepared and calibrated/trained model (GraphModule)

  • inplace – Carry out model transformations in-place, the original module is mutated.

  • convert_custom_config_dict –

    dictionary for custom configurations for convert function:

    convert_custom_config_dict = {
    # We automativally preserve all attributes, this option is
    # just in case and not likely to be used.
    "preserved_attributes": ["preserved_attr"],
}

  • _remove_qconfig – Option to remove the qconfig attributes in the model after convert. for internal use only.

  • fast_mode – whether to accelerate quantized model forward. If set True, quantized model cannot be compiled.

Returns

A quantized model (GraphModule)

Example: convert fx example:

>>> # prepared_model: the model after prepare_fx/prepare_qat_fx and
>>> # calibration/training
>>> quantized_model = convert_fx(prepared_model)
horizon_plugin_pytorch.quantization.quantize_fx.fuse_fx(model: torch.nn.modules.module.Module, fuse_custom_config_dict: Optional[Dict[str, Any]] = None, trace_method: str = 'symbolic', example_inputs: Optional[Any] = None, example_kw_inputs: Optional[Any] = None) → Union[torch.fx.graph_module.GraphModule, horizon_plugin_pytorch.fx.jit_scheme.GraphModule]

Fuse modules like conv+add+bn+relu etc.

Fusion rules are defined in horizon_plugin_pytorch.quantization.fx.fusion_pattern.py

Parameters

  • model – a torch.nn.Module model

  • fuse_custom_config_dict –

    Dictionary for custom configurations for fuse_fx, e.g.

    fuse_custom_config_dict = {
    # We automativally preserve all attributes, this option is
    # just in case and not likely to be used.
    "preserved_attributes": ["preserved_attr"],
}

  • trace_method – method used to get fx graph, availiable options are: ‘symbolic’: Use symbolic trace. ‘jit’: Use jit trace. ‘jit-strip’: Use jit trace and strip the graph outside QuantStub

  • example_inputs – model inputs. It is used to jit trace model or check model structure.

  • example_kw_inputs – model keyword inputs. It is used to trace model when using jit trace method.

Example: fuse_fx example:

>>> from torch.quantization import fuse_fx
>>> m = fuse_fx(m)
horizon_plugin_pytorch.quantization.quantize_fx.prepare_calibration_fx(model, qconfig_dict: Optional[Dict[str, Any]] = None, prepare_custom_config_dict: Optional[Dict[str, Any]] = None, optimize_graph: bool = False, hybrid: bool = False, hybrid_dict: Optional[Dict[str, List]] = None) → horizon_plugin_pytorch.quantization.fx.graph_module.ObservedGraphModule

Prepare the model for calibration.

Parameters

prepare_qat_fx (Same as) –

horizon_plugin_pytorch.quantization.quantize_fx.prepare_qat_fx(model: Union[torch.nn.modules.module.Module, torch.fx.graph_module.GraphModule, horizon_plugin_pytorch.fx.jit_scheme.GraphModule], qconfig_dict: Optional[Dict[str, Any]] = None, prepare_custom_config_dict: Optional[Dict[str, Any]] = None, optimize_graph: bool = False, hybrid: bool = False, hybrid_dict: Optional[Dict[str, List]] = None, opset_version: str = 'hbdk3', example_inputs: Any = None, example_kw_inputs: Any = None, qconfig_setter: Optional[Union[Tuple[horizon_plugin_pytorch.quantization.qconfig_template.QconfigSetterBase, ...], horizon_plugin_pytorch.quantization.qconfig_template.QconfigSetterBase]] = None, trace_method: str = 'symbolic', verbose: int = 0) → Union[horizon_plugin_pytorch.quantization.fx.graph_module.ObservedGraphModule, horizon_plugin_pytorch.fx.jit_scheme.GraphModule]

Prepare a model for quantization aware training.

Parameters

  • model – torch.nn.Module model or GraphModule model (maybe from fuse_fx)

  • qconfig_dict –

    qconfig_dict is a dictionary with the following configurations:

    qconfig_dict = {
    # optional, global config
    "": qconfig,
    # optional, used for module types
    "module_type": [
        (torch.nn.Conv2d, qconfig),
        ...,
    ],
    # optional, used for module names
    "module_name": [
        ("foo.bar", qconfig)
        ...,
    ],
    # priority (in increasing order):
    #   global, module_type, module_name, module.qconfig
    # qconfig == None means quantization should be
    # skipped for anything matching the rule.
    # The qconfig of function or method is the same as the
    # qconfig of its parent module, if it needs to be set
    # separately, please wrap this function as a module.
}

  • prepare_custom_config_dict –

    customization configuration dictionary for quantization tool:

    prepare_custom_config_dict = {
    # We automativally preserve all attributes, this option is
    # just in case and not likely to be used.
    "preserved_attributes": ["preserved_attr"],
}

  • optimize_graph – whether to do some process on origin model for special purpose. Currently only support using torch.fx to fix cat input scale(only used on Bernoulli)

  • hybrid – Whether prepare model in hybrid mode. Default value is False and model runs on BPU completely. It should be True if the model is quantized by model convert or contains some CPU ops. In hybrid mode, ops which aren’t supported by BPU and ops which are specified by the user will run on CPU. How to set qconfig: Qconfig in hybrid mode is the same as qconfig in non-hybrid mode. For BPU op, we should ensure the input of this op is quantized, the activation qconfig of its previous non-quantstub op should not be None even if its previous non-quantstub op is a CPU op. How to specify CPU op: Define CPU module_name or module_type in hybrid_dict.

  • hybrid_dict –

    hybrid_dict is a dictionary to define user-specified CPU op:

    hybrid_dict = {
    # optional, used for module types
    "module_type": [torch.nn.Conv2d, ...],
    # optional, used for module names
    "module_name": ["foo.bar", ...],
}
# priority (in increasing order): module_type, module_name
# To set a function or method as CPU op, wrap it as a module.

  • opset_version – opset_version specifics the version of opset that determines the behavior of hybrid mode. Ops that in the quantized opset will be considered as quantized ops and run on BPU, while ops not in the quantized opset but in the float opset will be marked as hybrid (float) ops and run on CPU. Valid options are “hbdk3” and “hbdk4”.

  • example_inputs – model inputs. It is used to jit trace model or check model structure.

  • example_kw_inputs – model keyword inputs. It is used to trace model when using jit trace method.

  • qconfig_setter – Qconfig setter. Only needed when using qconfig template.

  • trace_method – method used to get fx graph, availiable options are: ‘symbolic’: Use symbolic trace. ‘jit’: Use jit trace. ‘jit-strip’: Use jit trace and strip the graph outside QuantStub

  • verbose –

    whether check model structure. It has three levels: 0: do nothing 1: check qat model structure.

    1. if model has shared ops

    2. if model has unfused operations

    3. model quantization config

Returns

A GraphModule with fake quant modules (configured by qconfig_dict), ready for quantization aware training

Example: prepare_qat_fx example:

>>> import torch
>>> from horizon_plugin_pytorch.quantization import get_default_qat_qconfig
>>> from horizon_plugin_pytorch.quantization import prepare_qat_fx
>>> qconfig = get_default_qat_qconfig()
>>> def train_loop(model, train_data):
...     model.train()
...     for image, target in data_loader:
...         ...
>>> qconfig_dict = {"": qconfig}
>>> prepared_model = prepare_qat_fx(float_model, qconfig_dict)
>>> # Run QAT training
>>> train_loop(prepared_model, train_loop)

FX wrap helper (aligned with published X5 autodoc).

Extended tracer and wrap of torch.fx.

This file defines a inherit tracer of torch.fx.Tracer and a extended wrap to allow wrapping of user-defined Module or method, which help users do some optimization of their own module by torch.fx

horizon_plugin_pytorch.fx.fx_helper.wrap(skip_compile: bool = False)

Extend torch.fx.wrap.

This function can be:

  • 1. called or used as a decorator on a string to register a builtin function as a “leaf function”

  • 1. called or used as a decorator on a function to register this function as a “leaf function”

  • 1. called or used as a decorator on subclass of torch.nn.Module to register this module as a “leaf module”, and register all user defined method in this class as “leaf method”

  • 1. called or used as a decorator on a class method to register it as “leaf method”

Parameters

skip_compile (bool, optional) – Whether wrapped obj is skipped in compile, used by horizon_plugin_pytorch.quantization.fx.split_compilable_model .split_compilable_model. Defaults to False.

Returns

The actural decorator.

Return type:

wrap_inner

Fake-quant operators (aligned with published X5 autodoc).

class horizon_plugin_pytorch.quantization.fake_quantize.FakeQuantize(observer: type = <class 'horizon_plugin_pytorch.quantization.observer.MovingAverageMinMaxObserver'>, saturate: bool = None, in_place: bool = False, compat_mask: bool = True, channel_len: int = 1, fast_training=True, **observer_kwargs)

Simulate the quantize and dequantize operations in training time.

The output of this module is given by

fake_quant_x = clamp(floor(x / scale + 0.5), quant_min, quant_max) * scale # noqa

scale defines the scale factor used for quantization.

zero_point specifies the quantized value to which 0 in floating point maps to

quant_min specifies the minimum allowable quantized value.

quant_max specifies the maximum allowable quantized value.

fake_quant_enabled controls the application of fake quantization on tensors, note that statistics can still be updated.

observer_enabled controls statistics collection on tensors

dtype specifies the quantized dtype that is being emulated with fake-quantization, the allowable values is qint8 and qint16. The values of quant_min and quant_max should be chosen to be consistent with the dtype

Parameters

  • observer – Module for observing statistics on input tensors and calculating scale and zero-point.

  • saturate – Whether zero out the grad for value out of quanti range.

  • in_place – Whether use in place fake quantize.

  • compat_mask – Whether pack the bool mask into bitfield when saturate = True.

  • channel_len – Size of data at channel dim.

  • fast_training – Whether use fast training mode. If True, computing scale and fake quantization will be done in one step.

  • observer_kwargs – Arguments for the observer module

extra_repr()

Set the extra representation of the module

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

set_qparams(scale: Union[torch.Tensor, Sequence, float], zero_point: Optional[Union[torch.Tensor, Sequence, int]] = None)

Set qparams, default symmetric.

classmethod with_args(**kwargs)

Wrapper that allows creation of class factories.

This can be useful when there is a need to create classes with the same constructor arguments, but different instances. Can be used in conjunction with _callable_args

Example

>>> # xdoctest: +SKIP("Undefined vars")
>>> Foo.with_args = classmethod(_with_args)
>>> foo_builder = Foo.with_args(a=3, b=4).with_args(answer=42)
>>> foo_instance1 = foo_builder()
>>> foo_instance2 = foo_builder()
>>> id(foo_instance1) == id(foo_instance2)
False

Moving-average observers (aligned with published X5 autodoc).

class horizon_plugin_pytorch.quantization.observer.MovingAverageMinMaxObserver(averaging_constant=0.01, dtype=None, qscheme=None, quant_min=None, quant_max=None, is_sync_quantize=False, factory_kwargs=None)

Refine this docstring in the future.

Observer module for computing the quantization parameters based on the moving average of the min and max values.

This observer computes the quantization parameters based on the moving averages of minimums and maximums of the incoming tensors. The module records the average minimum and maximum of incoming tensors, and uses this statistic to compute the quantization parameters.

Parameters

  • averaging_constant – Averaging constant for min/max.

  • dtype – Quantized data type

  • qscheme – Quantization scheme to be used, only support per_tensor_symmetric scheme

  • reduce_range – Reduces the range of the quantized data type by 1 bit

  • quant_min – Minimum quantization value.

  • quant_max – Maximum quantization value.

  • is_sync_quantize – Whether use sync quantize

  • factory_kwargs – Arguments for register data buffer

forward(x_orig)

Record the running minimum and maximum of x.

class horizon_plugin_pytorch.quantization.observer.MovingAveragePerChannelMinMaxObserver(averaging_constant=0.01, ch_axis=0, dtype=None, qscheme=None, quant_min=None, quant_max=None, is_sync_quantize=False, factory_kwargs=None)

Refine this docstring in the future.

Observer module for computing the quantization parameters based on the running per channel min and max values.

This observer uses the tensor min/max statistics to compute the per channel quantization parameters. The module records the running minimum and maximum of incoming tensors, and uses this statistic to compute the quantization parameters.

Parameters

  • averaging_constant – Averaging constant for min/max.

  • ch_axis – Channel axis

  • dtype – Quantized data type

  • qscheme – Quantization scheme to be used, Only support per_channel_symmetric

  • quant_min – Minimum quantization value.

  • quant_max – Maximum quantization value.

  • is_sync_quantize – whether use sync quantize

  • factory_kwargs – Arguments for register data buffer

forward(x_orig)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Eager-mode module fusion (aligned with published X5 autodoc).

horizon_plugin_pytorch.quantization.fuse_modules.fuse_known_modules(mod_list, is_qat=False, additional_fuser_method_mapping=None)

Fuse modules.

Return a list of modules that fuses the operations specified in the input module list.

Fuses only the following sequence of modules: conv, bn; conv, bn, relu; conv, relu; conv, bn, add; conv, bn, add, relu; conv, add; conv, add, relu; linear, bn; linear, bn, relu; linear, relu; linear, bn, add; linear, bn, add, relu; linear, add; linear, add, relu. For these sequences, the first element in the output module list performs the fused operation. The rest of the elements are set to nn.Identity()

BPU platform (aligned with published X5 autodoc).

class horizon_plugin_pytorch.march.March(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

BPU platform.

BAYES: Bayes platform

BERNOULLI2: Bernoulli2 platform

BAYES_E: Bayes platform

QConfig factory (aligned with published X5 autodoc).

horizon_plugin_pytorch.quantization.qconfig.get_default_calib_qconfig(dtype='qint8', calib_qkwargs=None, backend='')

Get default calibration qconfig.

Parameters

  • dtype (str) – quantization type, the allowable value is qint8 and qint16

  • calib_qkwargs (dict) – A dict that contains args of CalibFakeQuantize and args of calibration observer.

  • backend (str) – backend implementation

horizon_plugin_pytorch.quantization.qconfig.get_default_qat_out_qconfig(dtype='qint8', weight_fake_quant='fake_quant', weight_qkwargs=None, backend='')

Get default qat out qconfig.

Parameters

  • dtype (str) – quantization type, the allowable value is qint8 and qint16

  • weight_fake_quant (str) – FakeQuantize type of weight, default is fake_quant.Avaliable items is fake_quant, lsq and pact

  • weight_qkwargs (dict) – A dict contain weight Observer type, args of weight FakeQuantize and args of weight Observer.

  • backend (str) – backend implementation

horizon_plugin_pytorch.quantization.qconfig.get_default_qat_qconfig(dtype='qint8', weight_dtype='qint8', activation_fake_quant='fake_quant', weight_fake_quant='fake_quant', activation_qkwargs=None, weight_qkwargs=None, backend='')

Get default qat qconfig.

Parameters

  • dtype (str) – Activation quantization type, the allowable values is qint8 and qint16

  • weight_dtype (str) – Weight quantization type, the allowable values is qint8 and qint16

  • activation_fake_quant (str) – FakeQuantize type of activation, default is fake_quant. Avaliable items is fake_quant, lsq, pact

  • weight_fake_quant (str) – FakeQuantize type of weight, default is fake_quant.Avaliable items is fake_quant, lsq and pact

  • activation_qkwargs (dict) – A dict contain activation Observer type, args of activation FakeQuantize and args of activation Observer.

  • weight_qkwargs (dict) – A dict contain weight Observer type, args of weight FakeQuantize and args of weight Observer.

  • backend (str) – backend implementation

Quantization public API (re-exports used by autodoc).

horizon_plugin_pytorch.quantization.check_model(module: Union[torch.jit._script.ScriptModule, torch.nn.modules.module.Module], example_inputs: tuple, march: Optional[str] = None, input_source: Union[Sequence[str], str] = 'ddr', advice: Optional[int] = None, check_quanti_param: bool = True)

Check if nn.Module or jit.ScriptModule can be compiled by HBDK.

Dump advices for improving performance on BPU.

Parameters

  • module (nn.Module or jit.ScriptModule.) –

  • example_inputs (A tuple of example inputs, in torch.tensor format.) – For jit.trace and shape inference.

  • march (Specify the target march of bpu.) – Valid options are bayes and bernoulli2 and bayes-e. If not provided, use horizon plugin global march.

  • input_source (Specify input features' sources(ddr/resizer/pyramid)) –

  • of (advice (Print HBDK compiler advices for improving the utilization) – the): model on bpu if layers of the model become slow by more than the specified time (in microseconds)

  • check_quanti_param (Check quanti param) –

Returns

0 if pass, otherwise not.

Return type

flag

Return type:

int

horizon_plugin_pytorch.quantization.compile_model(module: Union[torch.jit._script.ScriptModule, torch.nn.modules.module.Module], example_inputs: tuple, hbm: str, march: Optional[str] = None, name: Optional[str] = None, input_source: Union[Sequence[str], str] = 'ddr', input_layout: Optional[str] = None, output_layout: str = 'NCHW', opt: Union[str, int] = 'O2', balance_factor: int = 2, progressbar: bool = True, jobs: int = 16, debug: bool = True, extra_args: Optional[list] = None)

Compile the nn.Module or jit.ScriptModule.

Parameters

  • module (nn.Module or jit.ScriptModule.) –

  • example_inputs (A tuple of example inputs, in torch.tensor format.) – For jit.trace and shape inference.

  • hbm (Specify the output path of hbdk-cc.) –

  • march (Specify the target march of bpu.) – Valid options are bayes and bernoulli2 and bayes-e. If not provided, use horizon plugin global march.

  • name (Name of the model, recorded in hbm.) – Can be obtained by hbdk-disas or hbrtGetModelNamesInHBM in runtime.

  • input_source (Specify input features' sources(ddr/resizer/pyramid)) –

  • input_layout (Specify input layout of all model inputs.) – Available layouts are NHWC, NCHW, BPU_RAW.

  • output_layout (Specify input layout of all model inputs.) – Available layouts are NHWC, NCHW, BPU_RAW.

  • opt (Specify optimization options.) – Available options are O0, O1, O2, O3, ddr, fast, balance.

  • options (balance_factor (Specify the balance ratio when optimization) – is): ‘balance’.

  • progressbar (Show compilation progress to alleviate anxiety.) –

  • compiler (jobs (Specify number of threads launched during) – optimization.): Default is ‘16’. 0 means use all available hardware concurrency.

  • debug (Enable debugging info in hbm.) –

  • extra_args (specify extra args listed in "hbdk-cc -h".) – format in list of string: e.g. [’–ability-entry’, str(entry_value), …]

Returns

0 if pass, otherwise not.

Return type

flag

Return type:

int

horizon_plugin_pytorch.quantization.export_hbir(module: Union[torch.jit._script.ScriptModule, torch.nn.modules.module.Module], example_inputs: tuple, hbir: str, march: Optional[str] = None)

Export the nn.Module or jit.ScriptModule to hbdk3.HBIR.

Parameters

  • module (nn.Module or jit.ScriptModule.) –

  • example_inputs (A tuple of example inputs, in torch.tensor format.) – For jit.trace and shape inference.

  • hbir (Specify the output path of hbir.) –

  • march (Specify march to export hbir.) – Valid options are bayes and bernoulli2 and bayes-e. If not provided, use horizon plugin global march.

Returns

input names and output names

horizon_plugin_pytorch.quantization.perf_model(module: Union[torch.jit._script.ScriptModule, torch.nn.modules.module.Module], example_inputs: tuple, march: Optional[str] = None, out_dir: str = '.', name: Optional[str] = None, hbm: Optional[str] = None, input_source: Union[Sequence[str], str] = 'ddr', input_layout: Optional[str] = None, output_layout: str = 'NCHW', opt: Union[str, int] = 'O3', balance_factor: int = 2, progressbar: bool = True, jobs: int = 16, layer_details: bool = False, extra_args: Optional[list] = None)

Estimate the performance of nn.Module or jit.ScriptModule.

Parameters

  • module (nn.Module or jit.ScriptModule.) –

  • example_inputs (A tuple of example inputs, in torch.tensor format.) – For jit.trace and shape inference.

  • march (Specify the target march of bpu.) – Valid options are bayes and bernoulli2 and bayes-e. If not provided, use horizon plugin global march.

  • performance (out_dir (Specify the output directry to hold the) – results.):

  • name (Name of the model, recorded in hbm.) – Can be obtained by hbdk-disas or hbrtGetModelNamesInHBM in runtime.

  • hbm (Specify the output path of hbdk-cc.) –

  • input_source (Specify input features' sources(ddr/resizer/pyramid)) –

  • input_layout (Specify input layout of all model inputs.) – Available layouts are NHWC, NCHW, BPU_RAW.

  • output_layout (Specify input layout of all model inputs.) – Available layouts are NHWC, NCHW, BPU_RAW.

  • opt (Specify optimization options.) – Available options are O0, O1, O2, O3, ddr, fast, balance.

  • options (balance_factor (Specify the balance ratio when optimization) – is): ‘balance’.

  • progressbar (Show compilation progress to alleviate anxiety.) –

  • compiler (jobs (Specify number of threads launched during) – optimization.): Default is ‘16’. 0 means use all available hardware concurrency.

  • layer_details (show layer performance details. (dev use only)) –

  • extra_args (specify extra args listed in "hbdk-cc -h".) – format in list of string: e.g. [’–ability-entry’, str(entry_value), …]

Returns

Performance details in json dict. Or error code when fail.

horizon_plugin_pytorch.quantization.visualize_model(module: Union[torch.jit._script.ScriptModule, torch.nn.modules.module.Module], example_inputs: tuple, march: Optional[str] = None, save_path: Optional[str] = None, show: bool = True)

Visualize nn.Module or jit.ScriptModule at the view of HBDK.

Parameters

  • module (nn.Module or jit.ScriptModule.) –

  • example_inputs (A tuple of example inputs, in torch.tensor format.) – For jit.trace and shape inference.

  • march (Specify the target march of bpu.) – Valid options are bayes and bernoulli2 and bayes-e. If not provided, use horizon plugin global march.

  • save_path (Specify path to save the plot image.) –

  • show (Display the plotted image via display.) – Make sure X-server is correctly configured.

Returns

None