Memory allocation
OpenVX graphs require special memory wrapped in vx_references to operate. In order to avoid unnecessary memory copies GstTIOVX makes use of the standard GStreamer allocation handshaking mechanism. The following subsections describe in more detail this process.
Memory allocation it’s an expensive, non-deterministic process. It is desirable to minimize allocations during runtime. One pattern to achieve this is the well-known Object Pool. GStreamer implements this via the GstBufferPool. By specializing this class, GstTIOVX allocates a set of buffers during pipeline startup and recycles these during normal operation. And, by installing a custom allocator to this pool, the plugin ensures that the memory fulfills all the requirements given by the OpenVX nodes.
In order to create a GStreamer BufferPool, GstTIOVX instantiates it from a GType. This GType corresponds to the specialized BufferPool subclass, which will have a specific implementation for the needs of an input or output of an element.
After the buffer pool is created, the next step is to configure it. It needs it to get the current configuration structure from the buffer pool, then it needs it to configure the parameters and set an appropriate TIOVX allocator. After the configuration structure is done, it can update the configuration.
After the buffer pool is configured, during its activation, it will pre-allocate the configured resources in the buffer pool.
When the buffer pool is activated, acquire-buffer can retrieve a buffer from the buffer pool. Buffers will only be acquired if there is a free buffer in the pool, otherwise it will block until one is available.
Buffers allocated from a buffer pool will automatically be returned to the buffer pool when release_buffer is called and the buffer’s reference counter drops to 0. The buffer pool can be deactivated and when all the buffers are returned to the buffer pool, they will be freed.
The GstTIOVX plugin’s buffer pool implements three custom buffer pools subclasses that inherit from the GstTIOVXBufferPool. These subclasses correspond to the supported type of data the GstBuffers will carry. These are regular video frames, TIOVX tensors, and Bayer raw images. The corresponding subclasses are:
- GstImageBufferPool
- GstTensorBufferPool
- GstRawImageBuffferPool
The hiercharchy of inheritance of the buffer pool classes is as follows:
GstBufferPool
GstTIOVXBufferPool
Gst[custom type of data]BufferPool
Where the "custom type of data" corresponds to the previous custom buffer pool, depending on the type of data, the elements of the plugin will operate with.
The parent class of the GstTIOVXBufferPool. It performs the stages of creation, configuration, activation and releasing of the buffer pool.
The plugin base derivable buffer class that has common methods for all subclasses. This buffer pool class will bind the generic procedures like validating caps, add/free the metadata to the GstBuffer which should be defined by the child classes.
The tensor buffer pool it’s the buffer pool that allocates vx_image, i.e., it handles buffers carrying a regular image data type coming from TIOVX. The inheritance of this class is shown below:
GstBufferPool
GstTIOVXBufferPool
GstImageBufferPool
The GstImageBufferPool defines the caps and metadata with the vx_image data object. It implements functions specific to the allocation of TIOVX image.
See below the method overrides specification for this class:
- Validating caps: Verifies width, height, format and size.
- Adding the meta: The metadata corresponds to the GstMeta structure assigned to each buffer, and has information about the frame’s format, width, height, number of planes, plane offset and plane stride. It also carries a
vx_imageholding the corresponding memory.
The tensor buffer pool it’s the buffer pool that allocates TIOVX tensors, i.e., it handles buffers carrying the multidimensional data object called vx_tensor. The inheritance of this class is shown below:
GstBufferPool
GstTIOVXBufferPool
GstTensorBufferPool
The GstTensorBufferPool defines the caps and metadata with the vx_tensor data object. It implements functions specific to the allocation of TIOVX tensor.
See below the methods overrides specification for this class:
- Validating caps: Verifies number of dimensions, and the data type of the tensor which can be int8, uint8, int16, uint16, int32, uint32 and float32.
- Adding the meta: The metadata corresponds to the GstMeta structure assigned to each buffer, and has the tensor's information about the number of dimensions in the tensor, the size of each dimension, the stride of each dimension, the tensor data type, and tensor full size in memory. It also carries a
vx_tensorholding the corresponding memory.
The raw image buffer pool it’s the buffer pool that allocates for the raw image data, i.e., it handles buffers carrying Bayer frames. The inheritance of this class is shown below:
GstBufferPool
GstTIOVXBufferPool
GstRawImageBufferPool
The GstRawImageBufferPool defines the caps and the metadata with the tivx_raw_image. It implements functions specific to the allocation of TIOVX raw image, and this raw image it's an OpenVX structure that contains the Bayer images.
See below the methods overrides specification for this class:
- Validating caps: Verifies width, height and pixel container Bayer depth so it matches between the caps. Bayer pattern isn’t currently checked since the
tivx_raw_imagedoesn’t currently support this parameter. - Adding the metadata: The metadata corresponds to a GstMeta structure that is assigned to each buffer that has an object type
ti_raw_image; it has the memory and information about how the memory is organized: strides, width, height, bytes per pixel and offsets.
The diagram of the BufferPool structure is detailed below:
In order to share this buffer pool with upstream elements GstTIOVX must participate in the allocator negotiation process. The following diagram details this process. Note that the sequence applies regardless of the element topology, because the pads have been treated as independent actors.
[1] GStreamer. GStreamer Bufferpool. Retrieved October 29, 2021, from https://gstreamer.freedesktop.org/documentation/additional/design/bufferpool.html?gi-language=c.
[2] Valadoc.org. GStreamer BufferPool. Retrieved October 29, 2021, from https://valadoc.org/gstreamer-1.0/Gst.BufferPool.html.