beyondml.tflow.utils package

Submodules

beyondml.tflow.utils.transformer module

beyondml.tflow.utils.transformer.build_token_position_embedding_block(sequence_length, vocab_size, embed_dim)[source]

Builds a token and position embedding block

Parameters:

  • sequence_length (int) – The length of each sequence

  • vocab_size (int) – The size of the vocabulary used

  • embed_dim (int) – The desired embedding dimension

Returns:

embedding_block – The embedding block, which can be used alone or as a layer in another model

Return type:

TensorFlow keras Functional model

beyondml.tflow.utils.transformer.build_transformer_block(input_shape, embed_dim, num_heads, neurons, dropout_rate=0.1)[source]

Build a Transformer Block

Parameters:

  • input_shape (int or tuple of int) – The input shape for the model to use

  • embed_dim (int) – The dimension of the embedding

  • num_heads (int) – The number of attention heads to use

  • neurons (int) – The number of hidden neurons to use in the hidden layer

  • dropout_rate (float (default 0.1)) – Rate at which dropout is applied

  • value_dim (int or None (default None)) – The dimension to use for the value matrix, if provided

Returns:

transformer_block – The transformer block, which can then be used alone or as a layer in another model

Return type:

TensorFlow keras Functional model

beyondml.tflow.utils.utils module

class beyondml.tflow.utils.utils.ActiveSparsification(performance_cutoff, performance_measure='auto', starting_sparsification=None, max_sparsification=99, sparsification_rate=1, sparsification_patience=10, stopping_delta=0.01, stopping_patience=5, restore_best_weights=True, verbose=1)[source]

Bases: Callback

Keras-compatible callback object which enables active sparsification, allowing for increased sparsification as models train.

on_epoch_end(epoch, logs=None)[source]

Called at the end of an epoch.

Subclasses should override for any actions to run. This function should only be called during TRAIN mode.

Parameters:

  • epoch – Integer, index of epoch.

  • logs – Dict, metric results for this training epoch, and for the validation epoch if validation is performed. Validation result keys are prefixed with val_. For training epoch, the values of the Model’s metrics are returned. Example: {‘loss’: 0.2, ‘accuracy’: 0.7}.

on_train_begin(logs=None)[source]

Called at the beginning of training.

Subclasses should override for any actions to run.

Parameters:

logs – Dict. Currently no data is passed to this argument for this method but that may change in the future.

beyondml.tflow.utils.utils.add_layer_masks(model, additional_custom_objects=None)[source]

Convert a trained model from one that does not have masking weights to one that does have masking weights

Parameters:

  • model (TensorFlow Keras model) – The model to be converted

  • additional_custom_objects (dict or None (default None)) – Additional custom layers to use

Returns:

new_model – The converted model

Return type:

TensorFlow Keras model

beyondml.tflow.utils.utils.get_custom_objects()[source]

Return a dictionary of custom objects (layers) to use when loading models trained using this package

beyondml.tflow.utils.utils.get_task_masking_gradients(model, task_num)[source]

Get the gradients of masking weights within a model

Parameters:

model (TensorFlow Keras model) – The model to retrieve the gradients of

Notes

  • This function should only be run before the model has been trained

    or used to predict. There is an unknown bug related to TensorFlow which is leading to incorrect results after initial training

  • When running this function, randomized input and output data is sent

    through the model to retrieve gradients respective to each task. If the model is compiled using sparse_categorical_crossentropy’ loss, this will break this function’s functionality. As a result, please use `categorical_crossentropy (or even better, mse) before running this function. After retrieving gradients, the model can be recompiled with whatever parameters are desired.

Returns:

gradients – The gradients of the masking weights of the model

Return type:

list of TensorFlow tensors

beyondml.tflow.utils.utils.mask_model(model, percentile, method='gradients', exclusive=True, x=None, y=None)[source]

Mask the multitask model for training respective using the gradients for the tasks at hand

Parameters:

  • model (keras model with MANN masking layers) – The model to be masked

  • percentile (int) – Percentile to use in masking. Any weights less than the percentile value will be made zero

  • method (str (default 'gradients')) – One of either ‘gradients’ or ‘magnitude’ - the method for how to identify weights to mask If method is ‘gradients’, utilizes the gradients with respect to the passed x and y variables to identify the subnetwork to activate for each task If method is ‘magnitude’, uses the magnitude of the weights to identify the subnetwork to activate for each task

  • exclusive (bool (default True)) – Whether to restrict previously-used weight indices for each task. If True, this identifies disjoint subsets of weights within the layer which perform the tasks requested.

  • x (list of np.ndarray or array-like) – The training data input values, ignored if “method” is ‘magnitude’

  • y (list of np.ndarray or array-like) – The training data output values, ignored if “method” is ‘magnitude’

beyondml.tflow.utils.utils.mask_task_weights(model, task_masking_gradients, percentile, respect_previous_tasks=True)[source]
Parameters:

  • model (TensorFlow Keras model) – The model to be masked

  • task_masking_gradients (list of TensorFlow tensors) – The gradients for the specific task requested

  • percentile (int) – The percentile to mask/prune

  • respect_previous_tasks (bool (default True)) – Whether to respect the weights used for previous tasks and not use them for subsequent tasks

Returns:

masked_model – The masked model

Return type:

TensorFlow Keras model

beyondml.tflow.utils.utils.quantize_model(model, dtype='float16', additional_custom_objects=None)[source]

Apply model quantization

Parameters:

  • model (TensorFlow Keras Model) – The model to quantize

  • dtype (str or TensorFlow datatype (default 'float16')) – The datatype to quantize to

  • additional_custom_objects (None or dict (default None)) – Additional custom objects to use to instantiate the model

Returns:

new_model – The quantized model

Return type:

TensorFlow Keras Model

beyondml.tflow.utils.utils.remove_layer_masks(model, additional_custom_objects=None)[source]

Convert a trained model from using Masking layers to using non-masking layers

Parameters:

  • model (TensorFlow Keras model) – The model to be converted

  • additional_custom_objects (dict or None (default None)) – Additional custom layers to use

Returns:

new_model – The converted model

Return type:

TensorFlow Keras model

beyondml.tflow.utils.utils.train_model(model, train_x, train_y, loss, metrics, optimizer, cutoff, batch_size=32, epochs=100, starting_sparsification=0, max_sparsification=99, sparsification_rate=5, sparsification_patience=10, stopping_patience=5)[source]
beyondml.tflow.utils.utils.train_model_iteratively(model, task_gradients, train_x, train_y, validation_split, delta, batch_size, losses, optimizer='adam', metrics=None, starting_pruning=0, pruning_rate=10, patience=5, max_epochs=100)[source]

Train a model iteratively on each task, first obtaining baseline performance on each task and then iteratively training and pruning each task as far back as possible while maintaining acceptable performance on each task

Parameters:

  • model (TensorFlow Keras model) – The model to be trained

  • task_gradients (list of TensorFlow tensors) – Gradients for each task, output from the get_task_masking_gradients function

  • train_x (list of numpy arrays, TensorFlow Datasets, or other) – data types models can train with The input data to use to train on

  • train_y (list of numpy arrays, TensorFlow Datasets, or other) – data types model can train with The output data to use to train on

  • validation_split (float, or list of float) – The proportion of data to use for validation

  • delta (float) – The tolerance between validation losses to be considered “acceptable” performance to continue

  • batch_size (int) – The batch size to train with

  • losses (str, list, or Keras loss function) – The loss or losses to use when training

  • optimizer (str, list, or Keras optimizer) – The optimizer to use when training (default ‘adam’)

  • starting_pruning (int or list of int (default 0)) – The starting pruning rate to use for each task

  • pruning_rate (int or list of int (default [10, 5, 2, 1])) – The pruning rate to use

  • patience (int (default 5)) – The patience for number of epochs to wait for performance to improve sufficiently

  • max_epochs (int or list of int (default 100)) – The maximum number of epochs to use for training each task

Module contents

Some utilities to use when building, loading, and training MANN models