Descriptor Generation

We provide the interface DescriptorGenerator to define the high-level behavior for transforming input blob data, in the form of a smqtk_dataprovider.DataElement [1], into a descriptor (feature vector).

This interface also descends from the smqtk_dataprovider.ContentTypeValidator [2] interface to allow implementations the ability to declare what input data content types it can accept for processing. Thus, input DataElement instances must be of a content type that the DescriptorGenerator supports, otherwise an exception is raised when the offending data element is reached.

Descriptors may be generated most simply as numpy.ndarray arrays via the DescriptorGenerator.generate_arrays(). An additional layer of wrapping into a DescriptorElement may be invoked via DescriptorGenerator.generate_elements().

Bundled Implementation Model Details

The DescriptorGenerator interface does not define a model building method, but some implementations require internal models. Below are explanations on how to build or get modes for DescriptorGenerator implementations that require a model.

Caffe 1.0 Default Image Net

The CaffeDescriptorGenerator implementation does not come with a method of training its own models, but requires model files provided by Caffe: the network model file and the image mean binary protobuf file.

The Caffe source tree provides two scripts to download the specific files (relative to the caffe source tree):

# Downloads the network model file
scripts/download_model_binary.py models/bvlc_reference_caffenet

# Downloads the ImageNet mean image binary protobuf file
data/ilsvrc12/get_ilsvrc_aux.sh

These script effectively just download files from a specific source.

If the Caffe source tree is not available, the model files can be downloaded from the following URLs:

Network model: http://dl.caffe.berkeleyvision.org/bvlc_reference_caffenet.caffemodel

Image mean: http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz

Reference

class smqtk_descriptors.interfaces.descriptor_generator.DescriptorGenerator[source]

Base abstract Feature Descriptor interface.

generate_arrays(data_iter: Iterable[DataElement]) → Iterable[ndarray][source]

Generate descriptor vector elements for all input data elements.

Descriptor arrays yielded out will be parallel in association with the data elements input.

Selective Iteration For situations when it is desired to access specific generator returns, like when only one data element is provided in order to get a single array out, it is strongly recommended to expand the returned generator into a sequence type first. For example, expanding out the generator’s returns into a list (list(g.generate_arrays([e]))[0]) is recommended over just getting the “next” element of the returned generator (next(g.generate_arrays([e]))). Expansion into a sequence allows the generator to fully execute, which includes any functionality after the final yield statement in any of the underlying iterators.

Parameters:

data_iter – Iterable of DataElement instances to be described.

Raises:

RuntimeError – Descriptor extraction failure of some kind.
ValueError – Given data element content was not of a valid type with respect to this descriptor generator implementation.

Returns:

Iterator of result numpy.ndarray instances.

generate_elements(data_iter: ~typing.Iterable[~smqtk_dataprovider.interfaces.data_element.DataElement], descr_factory: ~smqtk_descriptors.descriptor_element_factory.DescriptorElementFactory = <smqtk_descriptors.descriptor_element_factory.DescriptorElementFactory object>, overwrite: bool = False) → Generator[DescriptorElement, None, None][source]

Generate DescriptorElement instances for the input data elements, generating new descriptors for those elements that need them, or optionally all input data elements.

Descriptor elements yielded out will be parallel in association with the data elements input. Descriptor element UUIDs are inherited from the data element it was generated from.

Selective Iteration For situations when it is desired to access specific generator returns, like when only one data element is provided in order to get a single element out, it is strongly recommended to expand the returned generator into a sequence type first. For example, expanding out the generator’s returns into a list (list(g.generate_elements([e]))[0]) is recommended over just getting the “next” element of the returned generator (next(g.generate_elements([e]))). Expansion into a sequence allows the generator to fully execute, which includes any functionality after the final yield statement in any of the underlying iterators that may perform required clean-up.

Non-redundant Processing Certain descriptor element implementations, as dictated by the input factory, may be connected to persistent storage in the background. Because of this, some descriptor elements may already “have” a vector on construction. This method, by default, only computes new descriptor vectors for data elements whose associated descriptor element does not report as already containing a vector. If the overwrite flag is True then descriptors are computed for all input data elements and are set to their respective descriptor elements regardless of existing vector storage.

Parameters:

data_iter – Iterable of DataElement instances to be described.
descr_factory – DescriptorElementFactory instance to drive the generation of element instances with some parametrization.
overwrite – By default, if a factory-produced DescriptorElement reports as containing a vector, we do not compute a descriptor again for the associated data element. If this is True, however, we will generate descriptors for all input data elements, overwriting the vectors previously stored in the factory-produces descriptor elements.

Raises:

RuntimeError – Descriptor extraction failure of some kind.
ValueError – Given data element content was not of a valid type with respect to this descriptor generator implementation.
IndexError – Underlying vector-producing generator either under or over produced vectors.

Returns:

Iterator of result DescriptorElement instances. UUIDs of generated DescriptorElement instances will reflect the UUID of the DataElement it was generated from.

generate_one_array(data_elem: DataElement) → ndarray[source]

Convenience wrapper around generate_arrays for the single-input case.

See the documentation for the DescriptorGenerator.generate_arrays() method for more information.

Parameters:

data_elem (smqtk.representation.DataElement) – DataElement instance to be described.

Raises:

RuntimeError – Descriptor extraction failure of some kind.
ValueError – Given data element content was not of a valid type with respect to this descriptor generator implementation.

Returns:

Descriptor vector the given data as a numpy.ndarray instance.

Return type:

numpy.ndarray

generate_one_element(data_elem: ~smqtk_dataprovider.interfaces.data_element.DataElement, descr_factory: ~smqtk_descriptors.descriptor_element_factory.DescriptorElementFactory = <smqtk_descriptors.descriptor_element_factory.DescriptorElementFactory object>, overwrite: bool = False) → DescriptorElement[source]

Convenience wrapper around generate_elements for the single-input case.

See documentation for the DescriptorGenerator.generate_elements() method for more information

Parameters:

data_elem – DataElement instance to be described.
descr_factory – DescriptorElementFactory instance to drive the generation of element instances with some parametrization.
overwrite – By default, if a factory-produced DescriptorElement reports as containing a vector, we do not compute a descriptor again for the associated data element. If this is True, however, we will generate descriptors for all input data elements, overwriting the vectors previously stored in the factory-produces descriptor elements.

Raises:

IndexError – Underlying vector-producing generator either under or over produced vectors.
RuntimeError – Descriptor extraction failure of some kind.
ValueError – Given data element content was not of a valid type with respect to this descriptor generator implementation.

Returns:

Result DescriptorElement instance. UUID of the generated DescriptorElement instance will reflect the UUID of the DataElement it was generated from.

class smqtk_descriptors.impls.descriptor_generator.caffe1.CaffeDescriptorGenerator(network_prototxt: DataElement, network_model: DataElement, image_mean: DataElement | None = None, return_layer: str = 'fc7', batch_size: int = 1, use_gpu: bool = False, gpu_device_id: int = 0, network_is_bgr: bool = True, data_layer: str = 'data', load_truncated_images: bool = False, pixel_rescale: Tuple[float, float] | None = None, input_scale: float | None = None, threads: int | None = None)[source]

Compute images against a Caffe model, extracting a layer as the content descriptor.

Parameters:

network_prototxt – Data element containing the text file defining the network layout.
network_model – Data element containing the trained .caffemodel file to use.
image_mean – Optional data element containing the image mean .binaryproto or .npy file.
return_layer – The label of the layer we take data from to compose output descriptor vector.
batch_size – The maximum number of images to process in one feed forward of the network. This is especially important for GPUs since they can only process a batch that will fit in the GPU memory space.
use_gpu – If Caffe should try to use the GPU
gpu_device_id – Integer ID of the GPU device to use. Only used if use_gpu is True.
network_is_bgr – If the network is expecting BGR format pixels. For example, the BVLC default caffenet does (thus the default is True).
data_layer – String label of the network’s data layer. We assume its ‘data’ by default.
load_truncated_images – If we should be lenient and force loading of truncated image bytes. This is False by default.
pixel_rescale – Re-scale image pixel values before being transformed by caffe (before mean subtraction, etc) into the given tuple (min, max) range. By default, images are loaded in the [0, 255] range. Refer to the image mean being used for desired input pixel scale.
input_scale – Optional floating-point scalar value to scale values of caffe network input data AFTER mean subtraction. This value is directly multiplied against the pixel values.
threads – Optional specific number of threads to use for data loading and pre-processing. If this is None or 0, we introspect the current system thread capacity and use that.

Raises:

AssertionError – Optionally provided image mean protobuf consisted of more than one image, or its shape was neither 1 nor 3 channels.

classmethod from_config(config_dict: Dict, merge_default: bool = True) → T[source]

Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.

This base method is adequate without modification when a class’s constructor argument types are JSON-compliant. If one or more are not, however, this method then needs to be overridden in order to convert from a JSON-compliant stand-in into the more complex object the constructor requires. It is recommended that when complex types are used they also inherit from the Configurable in order to hopefully make easier the conversion to and from JSON-compliant stand-ins.

When this method does need to be overridden, this usually looks like the following pattern:

D = TypeVar("D", bound="MyClass")

class MyClass (Configurable):

    @classmethod
    def from_config(
        cls: Type[D],
        config_dict: Dict,
        merge_default: bool = True
    ) -> D:
        # Perform a shallow copy of the input ``config_dict`` which
        # is important to maintain idempotency.
        config_dict = dict(config_dict)

        # Optionally guarantee default values are present in the
        # configuration dictionary.  This is useful when the
        # configuration dictionary input is partial and the logic
        # contained here wants to use config parameters that may
        # have defaults defined in the constructor.
        if merge_default:
            config_dict = merge_dict(cls.get_default_config(),
                                     config_dict)

        #
        # Perform any overriding of `config_dict` values here.
        #

        # Create and return an instance using the super method.
        return super().from_config(config_dict,
                                   merge_default=merge_default)

Note on type annotations: When defining a sub-class of configurable and override this class method, we will need to defined a new TypeVar that is bound at the new class type. This is because super requires a type to be given that descends from the implementing type. If C is used as defined in this interface module, which is upper-bounded on the base Configurable class, the type analysis will see that we are attempting to invoke super with a type that may not strictly descend from the implementing type (MyClass in the example above), and cause an error during type analysis.

Parameters:

config_dict (dict) – JSON compliant dictionary encapsulating a configuration.
merge_default (bool) – Merge the given configuration on top of the default provided by get_default_config.

Returns:

Constructed instance from the provided config.

get_config() → Dict[str, Any][source]

Return a JSON-compliant dictionary that could be passed to this class’s from_config method to produce an instance with identical configuration.

In the common case, this involves naming the keys of the dictionary based on the initialization argument names as if it were to be passed to the constructor via dictionary expansion.

Returns:: JSON type compliant configuration dictionary.

classmethod get_default_config() → Dict[str, Any][source]

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns:: Default configuration dictionary for the class.
Return type:: dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

classmethod is_usable() → bool[source]

Check whether this class is available for use.

Since certain plugin implementations may require additional dependencies that may not yet be available on the system, or other runtime conditions, this method may be overridden to check for those and return a boolean saying if the implementation is available for usable. When this method returns True, the class is declaring that it should be constructable and usable in the current environment.

By default, this method will return True unless a subclass overrides this class-method with their specific logic.

NOTES:

This should be a class method
When an implementation is deemed not usable, this should emit a
(user) warning, or some other kind of logging, detailing why the implementation is not available for use.

Returns:: Boolean determination of whether this implementation is usable in the current environment.
Return type:: bool

valid_content_types() → Set[str][source]

Returns:: A set valid MIME type content types that this descriptor can handle.