emloop.models

Classes

  • AbstractModel: Abstract machine learning model which exposes input and output names, run and save methods.

  • Ensemble: Ensemble model facilitates assembling multiple models into one for more accurate predictions.

  • Sequence: Model sequence provides simple abstraction for sequential application of multiple models to input batches.

class emloop.models.AbstractModel(dataset, log_dir, restore_from=None, **kwargs)

Bases: object

Abstract machine learning model which exposes input and output names, run and save methods. AbstractModel implementations are trainable with emloop.MainLoop.

Inheritance diagram of AbstractModel

__init__(dataset, log_dir, restore_from=None, **kwargs)[source]

Model constructor interface.

Additional parameters (currently covered by **kwargs) are passed according to the configuration model section.

Parameters
  • dataset (Optional[AbstractDataset]) – dataset object

  • log_dir (str) – existing directory in which all output files should be stored

  • restore_from (Optional[str]) – information passed to the model constructor (backend-specific); usually a directory in which the trained model is stored

  • kwargs – configuration section model

input_names

List of model input names.

Return type

Iterable[str]

output_names

List of model output names.

Return type

Iterable[str]

run(batch, train, stream)[source]

Run feed-forward pass with the given batch and return the results as dict.

When train=True, also update parameters.

Parameters
  • batch (Mapping[str, Sequence[Any]]) – batch to be processed

  • train (bool) – True if this batch should be used for model update, False otherwise

  • stream (StreamWrapper) – stream wrapper (useful for precise buffer management)

Return type

Mapping[str, Sequence[Any]]

Returns

results dict

save(name_suffix)[source]

Save the model parameters with the given name_suffix.

Parameters

name_suffix (str) – name suffix to be appended to the saved model

Return type

str

Returns

path to the saved file/dir

class emloop.models.Ensemble(inputs, outputs, aggregation='major_vote', models_root=None, model_paths=None, dataset=None, eager_loading=False, **kwargs)[source]

Bases: models.AbstractModel

Ensemble model facilitates assembling multiple models into one for more accurate predictions.

Warning

Ensemble model can be used for inference only (i.e. no training is supported).

The typical usage is to train multiple (possibly different) models and assemble them with this class.

usage from config
model:
  name: MyEnsemble
  class: emloop.models.Ensemble

  inputs: [images]
  outputs: [predictions]

  models_root: /var/project/models  # will load all the models under this directory
usage from python
import emloop as el

model = el.models.Ensemble(inputs=['images'], outputs=['predictions'], aggregation='mean',
                           models_root='/my/directory/with/models')
# model.run(...)
Inheritance diagram of Ensemble

AGGREGATION_METHODS = ['mean', 'major_vote']

Possible ensemble aggregation methods.

__init__(inputs, outputs, aggregation='major_vote', models_root=None, model_paths=None, dataset=None, eager_loading=False, **kwargs)[source]

Create new Ensemble.

If no models_paths are specified, all the sub-directories of the models_root will be taken as the models to be assembled together.

Parameters
  • inputs (Sequence[str]) – model input names

  • outputs (Sequence[str]) – model output names

  • aggregation (str) – aggregation method, one of Ensemble.AGGREGATION_METHODS

  • models_root (Optional[str]) – optional root directory of the models to be assembled together

  • model_paths (Optional[Sequence[str]]) – optional list of model directory names/paths

  • dataset (Optional[AbstractDataset]) – optional emloop dataset (will be passed to the assembled models)

  • eager_loading (bool) – load all the models in the constructor

  • kwargs – additional kwargs (unused)

Raises
_load_models()[source]

Maybe load all the models to be assembled together and save them to the self._models attribute.

Return type

None

input_names

List of model input names.

Return type

Iterable[str]

output_names

List of model output names.

Return type

Iterable[str]

run(batch, train=False, stream=None)[source]

Run feed-forward pass with the given batch using all the models, aggregate and return the results.

Warning

Ensemble can not be trained.

Parameters
  • batch (Mapping[str, Sequence[Any]]) – batch to be processed

  • train (bool) – True if this batch should be used for model update, False otherwise

  • stream (Optional[StreamWrapper]) – stream wrapper (useful for precise buffer management)

Return type

Mapping[str, Sequence[Any]]

Returns

aggregated results dict

Raises

ValueError – if the train flag is set to True

save(*args, **kwargs)[source]

Ensemble model can not be saved.

Raises

NotImplementedError – when called

Return type

None

class emloop.models.Sequence(model_paths, models_root=None, dataset=None, eager_loading=False, **_)[source]

Bases: models.AbstractModel

Model sequence provides simple abstraction for sequential application of multiple models to input batches. All the models are fed with the original inputs as well as the outputs of the preceding models. Ultimately, all the model outputs are returned.

Warning

Model sequence can be used for inference only (i.e. no training is supported).

usage from config
model:
  name: MyPipeline
  class: emloop.models.Sequence

  models_root: /var/project/models
  model_paths: [step1, step2, step3]
usage from python
import emloop as el

model = el.models.Sequence(model_paths=['/path/to/step/step1', '/path/to/step/step2'])
# model.run(...)
Inheritance diagram of Sequence

__init__(model_paths, models_root=None, dataset=None, eager_loading=False, **_)[source]

Create new model Sequence.

Parameters
  • models_root (Optional[str]) – optional root directory of the models to be assembled together

  • model_paths (Sequence[str]) – list of model directory names/paths

  • dataset (Optional[AbstractDataset]) – optional emloop dataset (will be passed to the underlying models)

  • eager_loading (bool) – load all the models in the constructor

_load_models()[source]

Maybe load all the models to be applied and save them to the self._models attribute.

Return type

None

input_names

List of model input names.

Return type

Iterable[str]

output_names

List of model output names.

Return type

Iterable[str]

run(batch, train=False, stream=None)[source]

Run all the models in-order and return accumulated outputs.

N-th model is fed with the original inputs and outputs of all the models that were run before it.

Warning

Sequence model can not be trained.

Parameters
  • batch (Mapping[str, Sequence[Any]]) – batch to be processed

  • train (bool) – True if this batch should be used for model update, False otherwise

  • stream (Optional[StreamWrapper]) – stream wrapper (useful for precise buffer management)

Return type

Mapping[str, Sequence[Any]]

Returns

accumulated model outputs

Raises

ValueError – if the train flag is set to True

save(*args, **kwargs)[source]

Sequence model can not be saved.

Raises

NotImplementedError – when called

Return type

None