lfcnn.models.sparse_coding package

Subpackages

• lfcnn.models.sparse_coding.utils package
  • Submodules
  • lfcnn.models.sparse_coding.utils.patching module
  • lfcnn.models.sparse_coding.utils.sparse_coding module
  • lfcnn.models.sparse_coding.utils.sparse_tensor_coding module
  • Module contents

Submodules

lfcnn.models.sparse_coding.dictionary_sparse_coding module

Light field sparse coding using online dictionary learning.

class lfcnn.models.sparse_coding.dictionary_sparse_coding.DictionarySparseCoding(overcompleteness, use_mask=False, patch_size_st=(8, 8), patch_size_uv=(5, 5), patch_step_st=(4, 4), patch_step_uv=(4, 4), couple_strength=1e-05, iterations_fista=20, iterations_eigenval=100, efficient_depatch=False, measure_sparsity=True, decomposition='sparse_coding', **kwargs)

Bases: BaseModel

A lfcnn.BaseModel implementing FISTA-based sparse coding. Can be used to train a sparse dictionary using a training dataset (online dictionary learning). But also to use a pre-trained dictionary to reconstruct coded measurements (compressed sensing).

Since light fields are usually too large for full dictionary representation and compressed sensing, the input light field batch is patched in the angular and spatial domain. The coding/reconstruction is then performed on the patches. The full light fields are then recovered from the patches using a de-patch layer.

E.g. a light field batch of shape (b, 9, 9, 32, 32, ch) can be patched to (b*N_patches, 5, 5, 8, 8, ch) using

patch_size_st = [8, 8] (with 50% spatial overlap) patch_step_st = [p - p//2 for p in patch_size_st] patch_size_uv = [5, 5] patch_step_uv = [4, 4] (since 2 * (5, 5) with (1, 1) overlap -> (9, 9) )

See also

lfcnn.models.sparse_coding.utils.sparse_coding.sparse_code_fista() and lfcnn.models.sparse_coding.utils.patching.patch() and lfcnn.models.sparse_coding.utils.patching.depatch()

Parameters:

• overcompleteness – Dictionary overcompleteness. The dictionary will be of shape (N, overcompleteness*N). Here, N denotes the vectorized signal dimension after patching.

• use_mask – Whether to use a coding mask. Used for Compressed Sensing reconstruction of coded input.

• patch_size_st – Spatial patch size (s’, t’)

• patch_size_uv – Angular patch size (u’, v’)

• patch_step_st – Spatial patch step (s_s, s_t)

• patch_step_uv – Angular patch step (s_u, s_v)

• couple_strength – Coupling of the L1-norm term during FISTA sparse coding.

• iterations_fista – Iterations of the FISTA main loop.

• iterations_eigenval – Iterations of the von Mises eigenvalue approximation routine.

• efficient_depatch – Whether to use a more memory efficient depatch implementation. Cannot be used during training. Is only meant for depatching very large batches.

• measure_sparsity – Whether to measure sparsity of coded signal and log it as a metric. This may be very memory consuming when working with very large input batches.

• **kwargs – Passed to model instantiation.

create_model(inputs, augmented_shape=None)

Create the Keras model. Needs to be implemented by the derived class to define the network topology.

Parameters:

• inputs (List[Input]) – List of Keras Inputs. Single or multi inputs supported.

• augmented_shape – The augmented shape as generated by the generator. Can be used to obtain the original light field’s shape, for example the number of subapertures or the number of spectral channels.

Return type:

Model

set_generator_and_reshape()

class lfcnn.models.sparse_coding.dictionary_sparse_coding.SparseCodingLayer(*args, **kwargs)

Bases: Layer

A keras.Layer implementing FISTA-based sparse coding. Can be used to train a sparse dictionary using a training dataset (online dictionary learning). But also to use a pre-trained dictionary to reconstruct coded measurements (compressed sensing).

Input of the layer is assumed to be a batch of light field patches, i.e. of shape (b, N, u’, v’, s’, t’, ch).

Parameters:

• overcompleteness – Dictionary overcompleteness. The dictionary will be of shape (N, overcompleteness*N).

• use_mask – Whether to use a coding mask. Used for Compressed Sensing reconstruction of coded input.

• couple_strength – Coupling of the L1-norm term during FISTA sparse coding.

• iterations_fista – Iterations of the FISTA main loop.

• iterations_eigenval – Iterations of the von Mises eigenvalue approximation routine.

• measure_sparsity – Whether to measure sparsity of coded signal and log it as a metric. This may be very memory consuming when working with very large input batches.

• *args – Passed to keras.Layer instantiation.

• **kwargs –

  Passed to keras.Layer instantiation.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(input)

This is where the layer’s logic lives.

Note here that call() method in tf.keras is little bit different from keras API. In keras API, you can pass support masking for layers as additional arguments. Whereas tf.keras has compute_mask() method to support masking.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

class lfcnn.models.sparse_coding.dictionary_sparse_coding.TensorDecompositionLayer(*args, **kwargs)

Bases: Layer

A keras.Layer implementing FISTA-based sparse coding using Tucker tensor decomposition. Can be used to train a sparse dictionary using a training dataset (online dictionary learning). But also to use a pre-trained dictionary to reconstruct coded measurements (compressed sensing).

Decomposition is performed to separate the angular, spatial, and spectral component of the input light field.

Parameters:

• overcompleteness – Dictionary overcompleteness. The dictionary will be of shape (N, overcompleteness*N).

• use_mask – Whether to use a coding mask. Used for Compressed Sensing reconstruction of coded input.

• couple_strength – Coupling of the L1-norm term during FISTA sparse coding.

• iterations_fista – Iterations of the FISTA main loop.

• iterations_eigenval – Iterations of the von Mises eigenvalue approximation routine.

• measure_sparsity – Whether to measure sparsity of coded signal and log it as a metric. This may be very memory consuming when working with very large input batches.

• vectorized – Whether to perform the sparse decomposition in a fully vectorized way. This way, all light fields in a batch and all patches of each light field are processed in parallel. This is extremely memory demanding and may only be used during training using small light fields. If set to false, the decomposition is performed for each light field seperately, or in slight parallelization when parallel_iterations is not None.

• parallel_iterations – If vectorized is False, defines the number of light field and patches that are processed in parallel. Defaults to 1 (no parallelization) which is the least memory demanding. You may increase this to a value as large as possible for the available memory.

• *args – Passed to keras.Layer instantiation.

• **kwargs –

  Passed to keras.Layer instantiation.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(input)

This is where the layer’s logic lives.

Note here that call() method in tf.keras is little bit different from keras API. In keras API, you can pass support masking for layers as additional arguments. Whereas tf.keras has compute_mask() method to support masking.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

code_serial(x)

Module contents

The LFCNN sparse coding models.

lfcnn.models.sparse_coding.get(model)

Given a model name, returns an lfcnn model instance.

Parameters:

model (str) – Name of the model.

Returns:

Model instance.