emloop.models

Classes

  • AbstractModel: Abstract machine learning model which exposes input and output names, run and save methods.
  • Ensemble: Ensemble model facilitates assembling multiple models into one for more accurate predictions.
  • Sequence: Model sequence provides simple abstraction for sequential application of multiple models to input batches.
class emloop.models.AbstractModel(dataset, log_dir, restore_from=None, **kwargs)

Bases: object

Abstract machine learning model which exposes input and output names, run and save methods. AbstractModel implementations are trainable with emloop.MainLoop.

Inheritance diagram of AbstractModel

__init__(dataset, log_dir, restore_from=None, **kwargs)[source]

Model constructor interface.

Additional parameters (currently covered by **kwargs) are passed according to the configuration model section.

Parameters:
  • dataset (Optional[AbstractDataset]) – dataset object
  • log_dir (str) – existing directory in which all output files should be stored
  • restore_from (Optional[str]) – information passed to the model constructor (backend-specific); usually a directory in which the trained model is stored
  • kwargs – configuration section model
input_names

List of model input names.

Return type:Iterable[str]
output_names

List of model output names.

Return type:Iterable[str]
restore_fallback

Return the fully-qualified name of the fallback restore class (e.g. module.submodule.BaseClass).

When restoring a model, emloop tries to use the fallback class if the construction of the model object specified in model configuration section fails.

Return type:str
Returns:fully-qualified name of the fallback restore class
run(batch, train, stream)[source]

Run feed-forward pass with the given batch and return the results as dict.

When train=True, also update parameters.

Parameters:
  • batch (Mapping[str, Sequence[Any]]) – batch to be processed
  • train (bool) – True if this batch should be used for model update, False otherwise
  • stream (StreamWrapper) – stream wrapper (useful for precise buffer management)
Return type:

Mapping[str, Sequence[Any]]

Returns:

results dict

save(name_suffix)[source]

Save the model parameters with the given name_suffix.

Parameters:name_suffix (str) – name suffix to be appended to the saved model
Return type:str
Returns:path to the saved file/dir
class emloop.models.Ensemble(inputs, outputs, aggregation='major_vote', models_root=None, model_paths=None, dataset=None, eager_loading=False, **kwargs)[source]

Bases: models.AbstractModel

Ensemble model facilitates assembling multiple models into one for more accurate predictions.

Warning

Ensemble model can be used for inference only (i.e. no training is supported).

The typical usage is to train multiple (possibly different) models and assemble them with this class.

usage from config
model:
  name: MyEnsemble
  class: emloop.models.Ensemble

  inputs: [images]
  outputs: [predictions]

  models_root: /var/project/models  # will load all the models under this directory
usage from python
import emloop as el

model = el.models.Ensemble(inputs=['images'], outputs=['predictions'], aggregation='mean',
                           models_root='/my/directory/with/models')
# model.run(...)
Inheritance diagram of Ensemble

AGGREGATION_METHODS = ['mean', 'major_vote']

Possible ensemble aggregation methods.

__init__(inputs, outputs, aggregation='major_vote', models_root=None, model_paths=None, dataset=None, eager_loading=False, **kwargs)[source]

Create new Ensemble.

If no models_paths are specified, all the sub-directories of the models_root will be taken as the models to be assembled together.

Parameters:
  • inputs (Sequence[str]) – model input names
  • outputs (Sequence[str]) – model output names
  • aggregation (str) – aggregation method, one of Ensemble.AGGREGATION_METHODS
  • models_root (Optional[str]) – optional root directory of the models to be assembled together
  • model_paths (Optional[Sequence[str]]) – optional list of model directory names/paths
  • dataset (Optional[AbstractDataset]) – optional emloop dataset (will be passed to the assembled models)
  • eager_loading (bool) – load all the models in the constructor
  • kwargs – additional kwargs (unused)
Raises:
_load_models()[source]

Maybe load all the models to be assembled together and save them to the self._models attribute.

Return type:None
input_names

List of model input names.

Return type:Iterable[str]
output_names

List of model output names.

Return type:Iterable[str]
restore_fallback

Ensemble model does not provide restore_fallback.

Return type:None
run(batch, train=False, stream=None)[source]

Run feed-forward pass with the given batch using all the models, aggregate and return the results.

Warning

Ensemble can not be trained.

Parameters:
  • batch (Mapping[str, Sequence[Any]]) – batch to be processed
  • train (bool) – True if this batch should be used for model update, False otherwise
  • stream (Optional[StreamWrapper]) – stream wrapper (useful for precise buffer management)
Return type:

Mapping[str, Sequence[Any]]

Returns:

aggregated results dict

Raises:

ValueError – if the train flag is set to True

save(*args, **kwargs)[source]

Ensemble model can not be saved.

Raises:NotImplementedError – when called
Return type:None
class emloop.models.Sequence(model_paths, models_root=None, dataset=None, eager_loading=False, **_)[source]

Bases: models.AbstractModel

Model sequence provides simple abstraction for sequential application of multiple models to input batches. All the models are fed with the original inputs as well as the outputs of the preceding models. Ultimately, all the model outputs are returned.

Warning

Model sequence can be used for inference only (i.e. no training is supported).

usage from config
model:
  name: MyPipeline
  class: emloop.models.Sequence

  models_root: /var/project/models
  model_paths: [step1, step2, step3]
usage from python
import emloop as el

model = el.models.Sequence(model_paths=['/path/to/step/step1', '/path/to/step/step2'])
# model.run(...)
Inheritance diagram of Sequence

__init__(model_paths, models_root=None, dataset=None, eager_loading=False, **_)[source]

Create new model Sequence.

Parameters:
  • models_root (Optional[str]) – optional root directory of the models to be assembled together
  • model_paths (Sequence[str]) – list of model directory names/paths
  • dataset (Optional[AbstractDataset]) – optional emloop dataset (will be passed to the underlying models)
  • eager_loading (bool) – load all the models in the constructor
_load_models()[source]

Maybe load all the models to be applied and save them to the self._models attribute.

Return type:None
input_names

List of model input names.

Return type:Iterable[str]
output_names

List of model output names.

Return type:Iterable[str]
restore_fallback

Sequence model does not provide restore_fallback.

Return type:None
run(batch, train=False, stream=None)[source]

Run all the models in-order and return accumulated outputs.

N-th model is fed with the original inputs and outputs of all the models that were run before it.

Warning

Sequence model can not be trained.

Parameters:
  • batch (Mapping[str, Sequence[Any]]) – batch to be processed
  • train (bool) – True if this batch should be used for model update, False otherwise
  • stream (Optional[StreamWrapper]) – stream wrapper (useful for precise buffer management)
Return type:

Mapping[str, Sequence[Any]]

Returns:

accumulated model outputs

Raises:

ValueError – if the train flag is set to True

save(*args, **kwargs)[source]

Sequence model can not be saved.

Raises:NotImplementedError – when called
Return type:None