-
+
- \ref subsubsec_file_image_api_low +
- \ref subsubsec_file_image_api_high +
-
+
- \ref subsubsec_file_image_semantics_cbk +
- \ref subsubsec_file_image_semantics_init +
-
+
- \ref subsubsec_file_image_example_read +
- \ref subsubsec_file_image_example_const +
- \ref subsubsec_file_image_example_dp +
- \ref subsubsec_file_image_example_template +
-
+
- \ref subsubsec_file_image_fort_low +
- \ref subsubsec_file_image_fort_high +
Using the Direct Chunk Write Function
+@ref H5DO_UG
Describes another way that chunks can be written to datasets.
@@ -377,7 +419,7 @@ These documents provide additional information for the use and tuning of specifi\ref H5FIM_UG
Describes how to work with HDF5 files in memory. Disk I/O is not required when file images are opened, created, read from, or written to.
diff --git a/doxygen/dox/ViewTools.dox b/doxygen/dox/ViewTools.dox index 43686751bfb..6bb000eb71b 100644 --- a/doxygen/dox/ViewTools.dox +++ b/doxygen/dox/ViewTools.dox @@ -303,6 +303,8 @@ GROUP "/" { } \endcode +\see H5TOOL_DP_UG for more information. + \subsection subsecViewToolsViewContent_h5ls h5ls The h5ls tool by default just displays the objects in the root group. It will not display items in groups beneath the root group unless specified. Useful h5ls options for viewing @@ -355,6 +357,8 @@ The output is shown below: /HDFEOS\ INFORMATION/StructMetadata.0 Dataset {SCALAR} \endcode +\see H5TOOL_LS_UG for more information. + \section secViewToolsViewDset Datasets and Dataset Properties Both h5dump and h5ls can be used to view specific datasets.-
diff --git a/doxygen/examples/FileImageOps.html b/doxygen/examples/FileImageOps.html
deleted file mode 100644
index da7952ab8d9..00000000000
--- a/doxygen/examples/FileImageOps.html
+++ /dev/null
@@ -1,1611 +0,0 @@
-
-
-
-
- 1. Introduction to HDF5 File Image Operations - - -
- 2. C API Call Syntax -
- 2.1. Low-level C API Routines -
- H5Pset_file_image -
- H5Pget_file_image -
- H5Pset_file_image_callbacks -
- H5Pget_file_image_callbacks -
- Virtual File Driver Feature Flags -
- H5Fget_file_image -
- 2.2. High-level C API Routine - -
- 3. C API Call Semantics -
- 3.1. File Image Callback Semantics -
- Buffer Ownership -
- Sharing a File image Buffer with the HDF5 Library -
- File Driver Considerations -
- 3.2. Initial File Image Semantics - -
- 4. Examples -
- 4.1. Reading an In-memory HDF5 File Image -
- 4.2. In-memory HDF5 File Image Construction -
- 4.3. Using HDF5 to Construct and Read a Data Packet -
- 4.4. Using a Template File -
- 5. Java Signatures for File Image Operations API Calls -
- 6. Fortran Signatures for File Image Operations API Calls - -
fapl_idcontains the ID of the target file access property list.
- callbacks_ptrcontains a pointer to an instance of the H5_file_image_callbacks_t structure.
-image_malloccontains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library malloc() call. The parameters of the image_malloc callback are defined as follows:
- sizecontains the size in bytes of the image buffer to allocate.
- file_image_opcontains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2.
- udataholds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
- image_memcpycontains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library memcpy() call except that it returns NULL on failure. Recall that the memcpy C Library routine is defined to return the dest parameter in all cases. The parameters of the image_memcpy callback are defined as follows:
- destcontains the address of the destination buffer.
- srccontains the address of the source buffer.
- sizecontains the number of bytes to copy.
- file_image_opcontains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2.
- udataholds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
- image_realloccontains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library realloc() call. The parameters of the image_realloc callback are defined as follows:
- ptrcontains the pointer to the buffer being reallocated.
- sizecontains the desired size in bytes of the buffer after realloc.
- file_image_opcontains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2.
- udataholds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
- image_freecontains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library free() call except that it will return 0 (SUCCEED) on success and -1 (FAIL) on failure. The parameters of the image_free callback are defined as follows:
- ptrcontains the pointer to the buffer being released.
- file_image_opcontains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2 .
- udataholds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
- udata_copycontains a pointer to a function that (from the perspective of HDF5) allocates a buffer of suitable size, copies the contents of the supplied udata into the new buffer, and returns the address of the new buffer. The function returns NULL on failure. This function is necessary if a non-NULL udata parameter is supplied, so that property lists containing the image callbacks can be copied. If the udata parameter (below) is NULL, then this parameter should be NULL as well. The parameter of the udata_copy callback is defined as follows:
- udatacontains the pointer to the user data block being copied.
- udata_freecontains a pointer to a function that (from the perspective of HDF5) frees a user data block. This function is necessary if a non-NULL udata parameter is supplied so that property lists containing image callbacks can be discarded without a memory leak. If the udata parameter (below) is NULL, this parameter should be NULL as well. The parameter of the udata_free callback is defined as follows:
- udatacontains the pointer to the user data block to be freed.
- udatacontains a pointer value, potentially to user-defined data, that will be passed to the image_malloc, image_memcpy, image_realloc, and image_free callbacks.
-fapl_idcontains the ID of the target file access property list.
- callbacks_ptrcontains a pointer to an instance of the H5_file_image_callbacks_t structure. All fields should be initialized to NULL. See the “H5Pset_file_image_callbacks” section for more information on the H5_file_image_callbacks_t structure.
-- Upon successful return, callbacks_ptr->image_malloc will contain the pointer passed as the image_malloc field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call. -
- Upon successful return, callbacks_ptr->image_memcpy will contain the pointer passed as the image_memcpy field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call. -
- Upon successful return, callbacks_ptr->image_realloc will contain the pointer passed as the image_realloc field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call. -
- Upon successful return, callbacks_ptr->image_free_ptr will contain the pointer passed as the image_free field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call. -
- Upon successful return, callbacks_ptr->udata_copy will contain the pointer passed as the udata_copy field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call. -
- Upon successful return, callbacks_ptr-> udata_free will contain the pointer passed as the udata_free field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call. -
- Upon successful return, callbacks_ptr->udata will contain the pointer passed as the udata field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call. -
file_idcontains the ID of the target file.
- buf_ptrcontains a pointer to the buffer into which the image of the HDF5 file is to be copied. If buf_ptr is NULL, no data will be copied, but the return value will still indicate the buffer size required (or a negative value on error).
- buf_lencontains the size of the supplied buffer.
-buf_ptrcontains a pointer to the supplied initial image. A NULL value is invalid and will cause H5LTopen_file_image to fail.
-buf_lencontains the size of the supplied buffer. A value of 0 is invalid and will cause H5LTopen_file_image to fail.
-flagscontains a set of flags indicating whether the image is to be opened read/write, whether HDF5 is to take control of the buffer, and how long the application promises to maintain the buffer. Possible flags are described in the table below:
-- Indicates that the HDF5 Library should not copy the file image buffer provided, but should use it directly. The HDF5 Library will release the file image when finished. The supplied buffer must have been allocated via a call to the standard C library malloc() or calloc() routines. The HDF5 Library will call free() to release the buffer. In the absence of this flag, the HDF5 Library will copy the buffer provided. The H5LT_FILE_IMAGE_DONT_COPY flag provides an application with the ability to “give ownership” of a file image buffer to the HDF5 Library. -
- The HDF5 Library will modify the buffer on write if the image is opened read/write and the H5LT_FILE_IMAGE_DONT_COPY flag is set. -
- The H5LT_FILE_IMAGE_DONT_RELEASE flag, see below, is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag is set -
- Indicates that the HDF5 Library should not attempt to release the buffer when the file is closed. This implies that the application will tend to this detail and that the application will not discard the buffer until after the file image is closed. -
- Since there is no way to return a changed buffer base address to the application, and since realloc can change this value, calls to realloc() must be barred when this flag is set. As a result, any write that requires an increased buffer size will fail. -
- This flag is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag, see above, is set. -
- If the H5LT_FILE_IMAGE_DONT_COPY flag is set and this flag is not set, the HDF5 Library will release the file image buffer after the file is closed using the standard C library free() routine. -
- Using this flag and the H5LT_FILE_IMAGE_DONT_COPY flag provides a way for the application to specify a buffer that the HDF5 Library can use for opening and accessing as a file image while letting the application retain ownership of the buffer. -
- Construct the image_malloc() call so that it returns the address of the buffer instead of allocating new space. This will keep the library thinking that the buffers are distinct even when they are not. Support this by including the address of the buffer in the user data. As a sanity check, include the buffer’s size in the user data as well, and require image_malloc() to fail if the requested buffer size is unexpected. Finally, include a reference counter in the user data, and increment the reference counter on each call to image_malloc(). -
- Construct the image_memcpy() call so that it does nothing. As a sanity check, make it fail if the source and destination pointers do not match the buffer address in the user data or if the size is unexpected. -
- Construct the image_free() routine so that it does nothing. As a sanity check, make it compare the supplied pointer with the expected pointer in the user data. Also, make it decrement the reference counter and notify the application that the HDF5 Library is done with the buffer when the reference count drops to 0. -
- Construct the image_malloc() call so that it returns the value in the init_ptr field of the user data structure and increments the init_ref_count. As a sanity check, the function should fail if the requested size does not match the init_size field in the user data structure or if any of the modified fields have values other than their initial values. -
- Construct the image_memcpy() call so that it does nothing. As a sanity check, it should be made to fail if the source, destination, and size parameters do not match the init_ptr and init_size fields as appropriate. -
- Construct the image_realloc() call so that it performs a standard realloc. Sanity checking, assuming that the realloc is successful, should be as follows: - -
- If the mod_ptr, mod_size, or mod_ref_count fields of the user data structure still have their initial values, verify that the supplied pointer matches the init_ptr field and that the supplied size does not match the init_size field. Decrement init_ref_count, set mod_ptr equal to the address returned by realloc, set mod_size equal to the supplied size, and set mod_ref_count to 1. -
- If the mod_ptr, mod_size, or mod_ref_count fields of the user data structure are defined, verify that the supplied pointer matches the value of mod_ptr and that the supplied size does not match mod_size. Set mod_ptr equal to the value returned by realloc, and set mod_size equal to the supplied size. -
- Construct the image_free() routine so that it does nothing. Perform sanity checks as follows: -
- If the H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE flag is set, decrement the init_ref_count field of the user data structure. Flag an error if init_ref_count drops below zero. -
- If the H5_FILE_IMAGE_OP_FILE_CLOSE flag is set, check to see if the mod_ptr, mod_size, or mod_ref_count fields of the user data structure have been modified from their initial values. If they have, verify that mod_ref_count contains 1 and then set that field to zero. If they have not been modified, proceed as per the H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE case. -
- H5Fopen() must fail if the target file driver does not set the H5FD_FEAT_ALLOW_FILE_IMAGE flag and a file image is specified in the FAPL. -
- If the target file driver supports the H5FD_FEAT_ALLOW_FILE_IMAGE flag, then H5Fopen() must fail if the file is already open or if a file of the specified name exists. -
- Even if the above constraints are satisfied, H5Fopen() must still fail if the image does not contain a valid (or perhaps just plausibly valid) image of an HDF5 file. In particular, the superblock must be processed, and the file structure be set up accordingly. -
- If the file specified in the H5Fopen() call does not exist, and no initial image is specified in the FAPL, the open must fail because there is no source for the initial image needed by the Core file driver. -
- If the file specified in the H5Fopen() call does exist, and an initial image is specified in the FAPL, the open must fail because the source of the needed initial image is ambiguous: the file image could be taken either from file or from the FAPL. -
- If the file specified in the H5Fopen() call does not exist, and an initial image is specified in the FAPL, the open will succeed. This assumes that the supplied image is valid. Further, if the backing store flag is set, the file specified in the H5Fopen() call will be created, and the contents of the Core file driver’s internal buffer will be written to the new file on flush or close. -
- Introduction -
- Using a File Driver - -
- Implementation of a Driver - -
- Miscellaneous -
H5FD_SEC2--
-This is the default driver which uses Posix file-system functions like
-
readandwriteto perform I/O to a single file. All I/O -requests are unbuffered although the driver does optimize file seeking -operations to some extent. - - H5FD_STDIO-- -This driver uses functions from `stdio.h' to perform buffered I/O -to a single file. - -
H5FD_CORE-- -This driver performs I/O directly to memory and can be used to create small -temporary files that never exist on permanent storage. This type of storage is -generally very fast since the I/O consists only of memory-to-memory copy -operations. - -
H5FD_MPIIO-- -This is the driver of choice for accessing files in parallel using MPI and -MPI-IO. It is only predefined if the library is compiled with parallel I/O -support. - -
H5FD_FAMILY--
-Large format address spaces are partitioned into more manageable pieces and
-sent to separate storage locations using an underlying driver of the user's
-choice. The
h5reparttool can be used to change the sizes of the -family members when stored as files or to convert a family of files to a -single file or vice versa. - - H5FD_SPLIT-- -The format address space is split into meta data and raw data and each is -mapped onto separate storage using underlying drivers of the user's -choice. The meta data storage can be read by itself (for limited -functionality) or both files can be accessed together. -
-
-
-
-
-
-
-
-
-
-
-
-
1. Introduction to HDF5 File Image Operations
- -File image operations allow users to work with HDF5 files in memory in the same ways that users currently work with HDF5 files on disk. Disk I/O is not required when file images are opened, created, read from, or written to.
-An HDF5 file image is an HDF5 file that is held in a buffer in main memory. Setting up a file image in memory involves using either a buffer in the file access property list or a buffer in the Core (aka Memory) file driver.
-The advantage of working with a file in memory is faster access to the data.
-The challenge of working with files in memory buffers is maximizing performance and minimizing memory footprint while working within the constraints of the property list mechanism. This should be a non-issue for small file images, but may be a major issue for large images.
-If invoked with the appropriate flags, the H5LTopen_file_image() high level library call should deal with these challenges in most cases. However, some applications may require the programmer to address these issues directly.
- -1.1. File Image Operations Function Summary
-Functions used in file image operations are listed below.
-| C Function | -Purpose | -Section | -
|---|---|---|
| H5Pset_file_image | -Specifies an initial file image | -2.1.1 | -H5Pget_file_image | -Retrieves a copy of the file image designated for a VFD to use as the initial contents of a file | -2.1.2 | -
| H5Pset_file_image_callbacks | -Manages file image buffer allocation, copying, reallocation, and release | -2.1.3 | - - -H5Pget_file_image_callbacks | -Obtains the current file image callbacks from a file access property list | -2.1.4 | - - -H5Fget_file_image | -Provides a simple way to retrieve a copy of the image of an existing, open file | -2.1.5 | - - -H5LTopen_file_image | -Provides a convenient way to open an initial file image with the Core VFD | -2.1.6 | - -
1.2. Abbreviations
-The following abbreviations are used in this document:
-| Abbreviation | -Explanation | -
|---|---|
| FAPL or fapl | -File Access Property List. In code samples, fapl is used. | -
| VFD | -Virtual File Driver | -
| VFL | -Virtual File Layer | -
1.3. Developer Prerequisites
-Developers who use the file image operations described in this document should be proficient and experienced users of the HDF5 C Library APIs. More specifically, developers should have a working knowledge of property lists, callbacks, and virtual file drivers.
- -1.4. Resources
-See the following for more information.
-The “RFC: File Image Operations” is the primary source for the information in this document.
-The “Alternate File Storage Layouts and Low-level File Drivers” section is in “The HDF5 File” chapter of the HDF5 User’s Guide .
-The H5P_SET_FAPL_CORE function call can be used to modify the file access property list so that the Memory virtual file driver, H5FD_CORE, is used. The Memory file driver is also known as the Core file driver.
-Refer to the Virtual File Layer for more detail. A list of VFL Functions is provided below.
-- -
| C Function | -Purpose | -
|---|---|
| H5Pset_driver | -Sets a file driver | -
| H5Pget_driver_info | -Returns a pointer to file driver information | -
| H5FDregister | -Registers a new file driver as a member of the virtual file driver class | -
| H5FDunregister | -Removes a driver ID from the library | -
| H5FDopen | -Opens a file | -
| H5FDclose | -Closes the file using the driver 'close' callback | -
| H5FDcmp | -Compares the keys of two files using the file driver callback if the files belong to the same driver, otherwise sort the files by driver class pointer value | -
| H5FDquery | -Queries a VFL driver for its feature flags | -
| H5FDalloc | -Allocates memory from the file | -
| H5FDfree | -Frees format addresses in the file | -
| H5FDset_eoa | -Set the end-of-address marker for the file | -
| H5FDget_eoa | -Returns the address of the first byte after the last allocated memory in the file | -
| H5FDget_eof | -Returns the end-of-file address, which is the greater of the end-of-format address and the actual EOF marker | -
| H5FDread | -Reads bytes from the file beginning at the specified address according to the provided data transfer property list | -
| H5FDwrite | -Writes bytes to the file beginning at the specified address according to the provided data transfer property list | -
| H5FDflush | -Notifies driver to flush all cached data | -
2. C API Call Syntax
- -The C API function calls described in this chapter fall into two categories: low-level routines that are part of the main HDF5 C Library and one high-level routine that is part of the “lite” API in the high-level wrapper library. The high-level routine uses the low-level routines and presents frequently requested functionality conveniently packaged for application developers’ use.
- -2.1. Low-level C API Routines
- -The purpose of this section is to describe the low-level C API routines that support file image operations. These routines allow an in-memory image of an HDF5 file to be opened without requiring file system I/O.
-The basic approach to opening an in-memory image of an HDF5 file is to pass the image to the Core file driver, and then tell the Core file driver to open the file. We do this by using the H5Pget/set_file_image calls. These calls allow the user to specify an initial file image.
-A potential problem with the H5Pget/set_file_image calls is the overhead of allocating and copying of large file image buffers. The callback routines enable application programs to avoid this problem. However, the use of these callbacks is complex and potentially hazardous: the particulars are discussed in the semantics and examples chapters below (see section 3.1 and section 4.1 respectively). Fortunately, use of the file image callbacks should seldom be necessary: the H5LTopen_file_image call should address most use cases.
-The property list facility in HDF5 is employed in file image operations. This facility was designed for passing data, not consumable resources, into API calls. The peculiar ways in which the file image allocation callbacks may be used allows us to avoid extending the property list structure to handle consumable resources cleanly and to avoid constructing a new facility for the purpose.
-The sub-sections below describe the low-level C APIs that are used with file image operations.
- -2.1.1. H5Pset_file_image
- -The H5Pset_file_image routine allows an application to provide an image for a file driver to use as the initial contents of the file. This call was designed initially for use with the Core VFD, but it can be used with any VFD that supports using an initial file image when opening a file. See the “Virtual File Driver Feature Flags” section for more information. Calling this routine makes a copy of the provided file image buffer. See the “H5Pset_file_image_callbacks” section for more information.
-The signature of H5Pset_file_image is defined as follows:
-herr_t H5Pset_file_image(hid_t fapl_id, void *buf_ptr, size_t buf_len)
-
-
The parameters of H5Pset_file_image are defined as follows:
-fapl_id contains the ID of the target file access property list. -buf_ptr supplies a pointer to the initial file image, or NULL if no initial file image is desired. -buf_len contains the size of the supplied buffer, or 0 if no initial image is desired. -If either the buf_len parameter is zero, or the buf_ptr parameter is NULL, no file image will be set in the FAPL, and any existing file image buffer in the FAPL will be released. If a buffer is released, the FAPL’s file image buf_len will be set to 0 and buf_ptr will be set to NULL.
-Given the tight interaction between the file image callbacks and the file image, the file image callbacks in a property list cannot be changed while a file image is defined.
-With properly constructed file image callbacks, it is possible to avoid actually copying the file image. The particulars of this are discussed in greater detail in the “C API Call Semantics” chapter and in the “Examples” chapter.
- -2.1.2. H5Pget_file_image
- -The H5Pget_file_image routine allows an application to retrieve a copy of the file image designated for a VFD to use as the initial contents of a file. This routine uses the file image callbacks (if defined) when allocating and loading the buffer to return to the application, or it uses malloc and memcpy if the callbacks are undefined. When malloc and memcpy are used, it will be the caller’s responsibility to discard the returned buffer via a call to free.
-The signature of H5Pget_file_image is defined as follows:
-herr_t H5Pget_file_image(hid_t fapl_id, void **buf_ptr_ptr, size_t *buf_len_ptr)
-
-
The parameters of H5Pget_file_image are defined as follows:
-fapl_id contains the ID of the target file access property list. -
buf_ptr_ptr contains a NULL or a pointer to a void*. If buf_ptr_ptr is not NULL, on successful return, *buf_ptr_ptr will contain a pointer to a copy of the initial image provided in the last call to H5Pset_file_image for the supplied fapl_id. If no initial image has been set, *buf_ptr_ptr will be NULL.
buf_len_ptr contains a NULL or a pointer to size_t. If buf_len_ptr is not NULL, on successful return, *buf_len_ptr will contain the value of the buf_len parameter for the initial image in the supplied fapl_id. If no initial image is set, the value of *buf_len_ptr will be 0.
As with H5Pset_file_image, appropriately defined file image callbacks can allow this function to avoid buffer allocation and memory copy operations.
- -2.1.3. H5Pset_file_image_callbacks
- -The H5Pset_file_image_callbacks API call exists to allow an application to control the management of file image buffers through user defined callbacks. These callbacks will be used in the management of file image buffers in property lists and in select file drivers. These routines are invoked when a new file image buffer is allocated, when an existing file image buffer is copied or resized, or when a file image buffer is released from use. From the perspective of the HDF5 Library, the operations of the image_malloc, image_memcpy, image_realloc, and image_free callbacks must be identical to those of the corresponding C standard library calls (malloc, memcpy, realloc, and free). While the operations must be identical, the file image callbacks have more parameters. The callbacks and their parameters are described below. The return values of image_malloc and image_realloc are identical to the return values of malloc and realloc. However, the return values of image_memcpy and image_free are different than the return values of memcpy and free: the return values of image_memcpy and image_free can also indicate failure. See the “File Image Callback Semantics” section for more information.
-The signature of H5Pset_file_image_callbacks is defined as follows:
- -typedef enum
-{
- H5_FILE_IMAGE_OP_PROPERTY_LIST_SET,
- H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY,
- H5_FILE_IMAGE_OP_PROPERTY_LIST_GET,
- H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE,
- H5_FILE_IMAGE_OP_FILE_OPEN,
- H5_FILE_IMAGE_OP_FILE_RESIZE,
- H5_FILE_IMAGE_OP_FILE_CLOSE
-} H5_file_image_op_t;typedef struct
-{
- void *(*image_malloc)(size_t size, H5_file_image_op_t file_image_op,
- void *udata);
- void *(*image_memcpy)(void *dest, const void *src, size_t size,
- H5_file_image_op_t file_image_op, void *udata);
- void *(*image_realloc)(void *ptr, size_t size,
- H5_file_image_op_t file_image_op, void *udata);
- herr_t (*image_free)(void *ptr, H5_file_image_op_t file_image_op,
- void *udata);
- void *(*udata_copy)(void *udata);
- herr_t (*udata_free)(void *udata);
- void *udata;
-} H5_file_image_callbacks_t;herr_t H5Pset_file_image_callbacks(hid_t fapl_id,
- H5_file_image_callbacks_t *callbacks_ptr)
-
-
The fields of the H5_file_image_callbacks_t structure are defined as follows: - -
-
-
-
-
image_malloc to NULL indicates that the HDF5 Library should invoke the standard C library malloc() routine when allocating file image buffers.- -
-
-
- -
-
-
- -
-
-
image_free to NULL indicates that the HDF5 Library should invoke the standard C library free() routine when releasing file image buffers.- -
-
-
- -
-
-
udata_free returns 0 (SUCCEED) on success and -1 (FAIL) on failure.
-
- -
| Value | -Comments | -
|---|---|
| H5_FILE_IMAGE_OP_PROPERTY_LIST_SET | -This value is passed to the image_malloc and image_memcpy callbacks when an image buffer is being copied while being set in a FAPL | -
| H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY | -This value is passed to the image_malloc and image_memcpy callbacks when an image buffer is being copied when a FAPL is copied | -
| H5_FILE_IMAGE_OP_PROPERTY_LIST_GET | -This value is passed to the image_malloc and image_memcpy callbacks when an image buffer is being copied while being retrieved from a FAPL | -
| H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE | -This value is passed to the image_free callback when an image buffer is being released during a FAPL close operation. | -
| H5_FILE_IMAGE_OP_FILE_OPEN | -This value is passed to the image_malloc and image_memcpy callbacks when an image buffer is copied during a file open operation. While the image being opened will typically be copied from a FAPL, this need not always be the case. An example of an exception is when the Core file driver takes its initial image from a file. | -
| H5_FILE_IMAGE_OP_FILE_RESIZE | -This value is passed to the image_realloc callback when a file driver needs to resize an image buffer. | -
| H5_FILE_IMAGE_OP_FILE_CLOSE | -This value is passed to the image_free callback when an image buffer is being released during a file close operation. | -
In closing our discussion of H5Pset_file_image_callbacks(), we note the interaction between this call and the H5Pget/set_file_image() calls above: since the malloc, memcpy, and free callbacks defined in the instance of H5_file_image_callbacks_t are used by H5Pget/set_file_image(), H5Pset_file_image_callbacks() will fail if a file image is already set in the target property list.
-For more information on writing the file image to disk, set the backing_store parameter. See the H5Pset_fapl_core entry in the HDF5 Reference Manual.
2.1.4. H5Pget_file_image_callbacks
-The H5Pget_file_image_callbacks routine is designed to obtain the current file image callbacks from a file access property list.
-The signature of H5Pget_file_image_callbacks() is defined as follows:
- -herr_t H5Pget_file_image_callbacks(hid_t fapl_id,
- H5_file_image_callbacks_t *callbacks_ptr)
-
-
-
-
2.1.5. Virtual File Driver Feature Flags
- -Implementation of theH5Pget/set_file_image_callbacks() and H5Pget/set_file_image() function calls requires a pair of virtual file driver feature flags. The flags are H5FD_FEAT_ALLOW_FILE_IMAGE and H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS. Both of these are defined in H5FDpublic.h.
-
-The first flag, H5FD_FEAT_ALLOW_FILE_IMAGE, allows a file driver to indicate whether or not it supports file images. A VFD that sets this flag when its ‘query’ callback is invoked indicates that the file image set in the FAPL will be used as the initial contents of a file. Support for setting an initial file image is designed primarily for use with the Core VFD. However, any VFD can indicate support for this feature by setting the flag and copying the image in an appropriate way for the VFD (possibly by writing the image to a file and then opening the file). However, such a VFD need not employ the file image after file open time. In such cases, the VFD will not make an in-memory copy of the file image and will not employ the file image callbacks.
- -File drivers that maintain a copy of the file in memory (only the Core file driver at present) can be constructed to use the initial image callbacks (if defined). Those that do must set the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag, the second flag, when their ‘query’ callbacks are invoked.
- -Thus file drivers that set the H5FD_FEAT_ALLOW_FILE_IMAGE flag but not the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag may read the supplied image from the property list (if present) and use it to initialize the contents of the file. However, they will not discard the image when done, nor will they make any use of any file image callbacks (if defined).
- -If an initial file image appears in a file allocation property list that is used in an H5Fopen() call, and if the underlying file driver does not set the H5FD_FEAT_ALLOW_FILE_IMAGE flag, then the open will fail.
- -If a driver sets both the H5FD_FEAT_ALLOW_FILE_IMAGE flag and the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag, then that driver will allocate a buffer of the required size, copy the contents of the initial image buffer from the file access property list, and then open the copy as if it had just loaded it from file. If the file image allocation callbacks are defined, the driver shall use them for all memory management tasks. Otherwise it will use the standard malloc, memcpy, realloc, and free C library calls for this purpose.
- -If the VFD sets the H5FD_FEAT_ALLOW_FILE_IMAGE flag, and an initial file image is defined by an application, the VFD should ensure that file creation operations (as opposed to file open operations) bypass use of the file image, and create a new, empty file.
- -Finally, it is logically possible that a file driver would set the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag, but not the H5FD_FEAT_ALLOW_FILE_IMAGE flag. While it is hard to think of a situation in which this would be desirable, setting the flags this way will not cause any problems: the two capabilities are logically distinct.
- -2.1.6. H5Fget_file_image
- -The purpose of the H5Fget_file_image routine is to provide a simple way to retrieve a copy of the image of an existing, open file. This routine can be used with files opened using the SEC2 (aka POSIX), STDIO, and Core (aka Memory) VFDs.
- -The signature of H5Fget_file_image is defined as follows:
-ssize_t H5Fget_file_image(hid_t file_id, void *buf_ptr, size_t buf_len)
-
-
If the return value of H5Fget_file_image is a positive value, then the value will be the length of buffer required to store the file image (in other words, the length of the file). A negative value might be returned if the file is too large to store in the supplied buffer or on failure.
- -The current file size can be obtained via a call to H5Fget_filesize(). Note that this function returns the value of the end of file (EOF) and not the end of address space (EOA). While these values are frequently the same, it is possible for the EOF to be larger than the EOA. Since H5Fget_file_image() will only obtain a copy of the file from the beginning of the superblock to the EOA, it will be best to use H5Fget_file_image() to determine the size of the buffer required to contain the image.
- -Other Design Considerations - -Here are some other notes regarding the design and implementation of H5Fget_file_image.
-The H5Fget_file_image call should be part of the high-level library. However, a file driver agnostic implementation of the routine requires access to data structures that are hidden within the HDF5 Library. We chose to implement the call in the library proper rather than expose those data structures.
-There is no reason why the H5Fget_file_image() API call could not work on files opened with any file driver. However, the Family, Multi, and Split file drivers have issues that make the call problematic. At present, files opened with the Family file driver are marked as being created with that file driver in the superblock, and the HDF5 Library refuses to open files so marked with any other file driver. This negates the purpose of the H5Fget_file_image() call. While this mark can be removed from the image, the necessary code is not trivial.
-Thus we will not support the Family file driver in H5Fget_file_image() unless there is demand for it. Files created with the Multi and Split file drivers are also marked in the superblock. In addition, they typically use a very sparse address space. A sparse address space would require the use of an impractically large buffer for an image, and most of the buffer would be empty. So, we see no point in supporting the Multi and Split file drivers in H5Fget_file_image() under any foreseeable circumstances.
- -2.2. High-level C API Routine
- -The H5LTopen_file_image high-level routine encapsulates the capabilities of routines in the main HDF5 Library with conveniently accessible abstractions.
- -2.2.1. H5LTopen_file_image
- -The H5LTopen_file_image routine is designed to provide an easier way to open an initial file image with the Core VFD. Flags to H5LTopen_file_image allow for various file image buffer ownership policies to be requested. See the HDF5 Reference Manual for more information on high-level APIs.
-The signature of H5LTopen_file_image is defined as follows:
- -hid_t H5LTopen_file_image(void *buf_ptr, size_t buf_len, unsigned flags)
-
-
The parameters of H5LTopen_file_image are defined as follows:
--
-
| Value | -Comments | -
|---|---|
| H5LT_FILE_IMAGE_OPEN_RW | -Indicates that the HDF5 Library should open the image read/write instead of the default read-only. | -
| H5LT_FILE_IMAGE_DONT_COPY | -
-
|
-
| H5LT_FILE_IMAGE_DONT_RELEASE | -
-
|
-
The following table is intended to summarize the semantics of the H5LT_FILE_IMAGE_DONT_COPY and H5LT_FILE_IMAGE_DONT_RELEASE flags (shown as “Don’t Copy Flag” and “Don’t Release Flag” respectively in the table):
- -| Don’t Copy Flag | -Don’t Release Flag | -Make Copy of User Supplied Buffer | -Pass User Supplied Buffer to File Driver | -Release User Supplied Buffer When Done | -Permit realloc of Buffer Used by File Driver | -
|---|---|---|---|---|---|
| False | -Don’t care | -True | -False | -False | -True | -
| True | -False | -False | -True | -True | -True | -
| True | -True | -False | -True | -False | -False | -
The return value of H5LTopen_file_image will be a file ID on success or a negative value on failure. The file ID returned should be closed with H5Fclose.
-Note that there is no way currently to specify a “backing store” file name in this definition of H5LTopen_image.
- -3. C API Call Semantics
-The purpose of this chapter is to describe some issues that developers should consider when using file image buffers, property lists, and callback APIs.
- -3.1. File Image Callback Semantics
-The H5Fget/set_file_image_callbacks() API calls allow an application to hook the memory management operations used when allocating, duplicating, and discarding file images in the property list, in the Core file driver, and potentially in any in-memory file driver developed in the future.
-From the perspective of the HDF5 Library, the supplied image_malloc(), image_memcpy(), image_realloc(), and image_free() callback routines must function identically to the C standard library malloc(), memcpy(), realloc(), and free() calls. What happens on the application side can be much more nuanced, particularly with the ability to pass user data to the callbacks. However, whatever the application does with these calls, it must maintain the illusion that the calls have had the expected effect. Maintaining this illusion requires some understanding of how the property list structure works, and what HDF5 will do with the initial images passed to it.
-At the beginning of this document, we talked about the need to work within the constraints of the property list mechanism. When we said “from the perspective of the HDF5 Library…” in the paragraph above, we are making reference to this point.
-The property list mechanism was developed as a way to add parameters to functions without changing the parameter list and breaking existing code. However, it was designed to use only “call by value” semantics, not “call by reference”. The decision to use “call by value” semantics requires that the values of supplied variables be copied into the property list. This has the advantage of simplifying the copying and deletion of property lists. However, if the value to be copied is large (say a 2 GB file image), the overhead can be unacceptable.
-The usual solution to this problem is to use “call by reference” where only a pointer to an object is placed in a parameter list rather than a copy of the object itself. However, use of “call by reference” semantics would greatly complicate the property list mechanism: at a minimum, it would be necessary to maintain reference counts to dynamically allocated objects so that the owner of the object would know when it was safe to free the object.
-After much discussion, we decided that the file image operations calls were sufficiently specialized that it made no sense to rework the property list mechanism to support “call by reference.” Instead we provided the file image callback mechanism to allow the user to implement some version of “call by reference” when needed. It should be noted that we expect this mechanism to be used rarely if at all. For small file images, the copying overhead should be negligible, and for large images, most use cases should be addressed by the H5LTopen_file_image call.
-In the (hopefully) rare event that use of the file image callbacks is necessary, the fundamental point to remember is that the callbacks must be constructed and used in such a way as to maintain the library’s illusion that it is using “call by value” semantics.
-Thus the property list mechanism must think that it is allocating a new buffer and copying the supplied buffer into it when the file image property is set. Similarly, it must think that it is allocating a new buffer and copying the contents of the existing buffer into it when it copies a property list that contains a file image. Likewise, it must think it is de-allocating a buffer when it discards a property list that contains a file image.
-Similar illusions must be maintained when a file image buffer is copied into the Core file driver (or any future driver that uses the file image callbacks) when the file driver re-sizes the buffer containing the image and finally when the driver discards the buffer.
- -3.1.1. Buffer Ownership
-The owner of a file image in a buffer is the party that has the responsibility to discard the file image buffer when it is no longer needed. In this context, the owner is either the HDF5 Library or the application program.
-We implemented the image_* callback facility to allow efficient management of large file images. These facilities can be used to allow sharing of file image buffers between the application and the HDF5 library, and also transfer of ownership in either direction. In such operations, care must be taken to ensure that ownership is clear and that file image buffers are not discarded before all references to them are discarded by the non-owning party.
-Ownership of a file image buffer will only be passed to the application program if the file image callbacks are designed to do this. In such cases, the application program must refrain from freeing the buffer until the library has deleted all references to it. This in turn will happen after all property lists (if any) that refer to the buffer have been discarded, and the file driver (if any) that used the buffer has closed the file and thinks it has discarded the buffer.
- -3.1.2. Sharing a File image Buffer with the HDF5 Library
-As mentioned above, the HDF5 property lists are a mechanism for passing values into HDF5 Library calls. They were created to allow calls to be extended with new parameters without changing the actual API or breaking existing code. They were designed based on the assumption that all new parameters would be “call by value” and not “call by reference.” Having “call by value” parameters means property lists can be copied, reused, and discarded with ease.
-Suppose an application wished to share a file image buffer with the HDF5 Library. This means the library would be allowed to read the file image, but not free it. The file image callbacks might be constructed as follows to share a buffer:
- --
-
For more information on user defined data, see the “H5Pset_file_image_callbacks” section.
- -3.1.3. File Driver Considerations
-When a file image is opened by a driver that sets both the H5FD_FEAT_ALLOW_FILE_IMAGE and the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flags, the driver will allocate a buffer large enough for the initial file image and then copy the image from the property list into this buffer. As processing progresses, the driver will reallocate the image as necessary to increase its size and will eventually discard the image at file close. If defined, the driver will use the file image callbacks for these operations; otherwise, the driver will use the standard C library calls. See the "H5Pset_file_image_callbacks” section for more information.
-As described above, the file image callbacks can be constructed so as to avoid the overhead of buffer allocations and copies while allowing the HDF5 Library to maintain its illusions on the subject. There are two possible complications involving the file driver. The complications are the possibility of reallocation calls from the driver and the possibility of the continued existence of property lists containing references to the buffer.
-Suppose an application wishes to share a file image buffer with the HDF5 Library. The application allows the library to read (and possibly write) the image, but not free it. We must first decide whether the image is to be opened read-only or read/write.
-If the image will be opened read-only (or if we know that any writes will not change the size of the image), the image_realloc() call should never be invoked. Thus the image_realloc() routine can be constructed so as to always fail, and the image_malloc(), image_memcpy(), and image_free() routines can be constructed as described in the section above.
-Suppose, however, that the file image will be opened read/write and may grow during the computation. We must now allow for the base address of the buffer to change due to reallocation calls, and we must employ the user data structure to communicate any change in the buffer base address and size to the application. We pass buffer changes to the application so that the application will be able to eventually free the buffer. To this end, we might define a user data structure as shown in the example below:
- typedef struct udata {
- void *init_ptr;
- size_t init_size;
- int init_ref_count;
- void *mod_ptr;
- size_t mod_size;
- int mod_ref_count;
- }
-Example 1. Using a user data structure to communicate with an application
-We initialize an instance of the structure so that init_ptr points to the buffer to be shared, init_size contains the initial size of the buffer, and all other fields are initialized to either NULL or 0 as indicated by their type. We then pass a pointer to the instance of the user data structure to the HDF5 Library along with allocation callback functions constructed as follows:
- --
-
-
-
-
-
3.2. Initial File Image Semantics
-One can argue whether creating a file with an initial file image is closer to creating a file or opening a file. The consensus seems to be that it is closer to a file open, and thus we shall require that the initial image only be used for calls to H5Fopen().
-Whatever our convention, from an internal perspective, opening a file with an initial file image is a bit of both creating a file and opening a file. Conceptually, we will create a file on disk, write the supplied image to the file, close the file, open the file as an HDF5 file, and then proceed as usual (of course, the Core VFD will not write to the file system unless it is configured to do so). This process is similar to a file create: we are creating a file that did not exist on disk to begin with and writing data to it. Also, we must verify that no file of the supplied name is open. However, this process is also similar to a file open: we must read the superblock and handle the usual file open tasks.
-Implementing the above sequence of actions has a number of implications on the behavior of the H5Fopen() call when an initial file image is supplied:
--
-
As we indicated earlier, if an initial file image appears in the property list of an H5Fcreate() call, it is ignored.
-While the above section on the semantics of the file image callbacks may seem rather gloomy, we get the payback here. The above says everything that needs to be said about initial file image semantics in general. The sub-section below has a few more observations on the Core file driver.
- -3.2.1. Applying Initial File Image Semantics to the Core File Driver
-At present, the Core file driver uses the open() and read() system calls to load an HDF5 file image from the file system into RAM. Further, if the backing_store flag is set in the FAPL entry specifying the use of the Core file driver, the Core file driver’s internal image will be used to overwrite the source file on either flush or close. See the H5Pset_fapl_core entry in the HDF5 Reference Manual for more information.
This results in the following observations. In all cases assume that use of the Core file driver has been specified in the FAPL.
--
-
4. Examples
-The purpose of this chapter is to provide examples of how to read or build an in-memory HDF5 file image.
- -4.1. Reading an In-memory HDF5 File Image
- -The H5Pset_file_image() function call allows the Core file driver to be initialized from an application provided buffer. The following pseudo code illustrates its use:
- -
-<allocate and initialize buf_len and buf>
-<allocate fapl_id>
-<set fapl to use Core file driver>
-H5Pset_file_image(fapl_id, buf, buf_len);
-<discard buf any time after this point>
-<open file>
-<discard fapl any time after this point>
-<read and/or write file as desired, close>
-This solution is easy to code, but the supplied buffer is duplicated twice. The first time is in the call to H5Pset_file_image() when the image is duplicated and the duplicate inserted into the property list. The second time is when the file is opened: the image is copied from the property list into the initial buffer allocated by the Core file driver. This is a non-issue for small images, but this could become a significant performance hit for large images.
-If we want to avoid the extra malloc and memcpycalls, we must decide whether the application should retain ownership of the buffer or pass ownership to the HDF5 Library.
-The following pseudo code illustrates opening the image read -only using the H5LTopen_file_image() routine. In this example, the application retains ownership of the buffer and avoids extra buffer allocations and memcpy calls.
- -
-<allocate and initialize buf_len and buf>
-hid_t file_id;
-unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
-file_id = H5LTopen_file_image(buf, buf_len, flags);
-<read file as desired, and then close>
-<discard buf any time after this point>
-If the application wants to transfer ownership of the buffer to the HDF5 Library, and the standard C library routine free is an acceptable way of discarding it, the above example can be modified as follows:
- -
-<allocate and initialize buf_len and buf>
-hid_t file_id;
-unsigned flags = H5LT_FILE_IMAGE_DONT_COPY;
-file_id = H5LTopen_file_image(buf, buf_len, flags);
-<read file as desired, and then close>
-Again, file access is read-only. Read/write access can be obtained via the H5LTopen_file_image() call, but we will explore that in the section below.
- -4.2. In-memory HDF5 File Image Construction
- -Before the implementation of file image operations, HDF5 supported construction of an image of an HDF5 file in memory with the Core file driver. The H5Fget_file_image() function call allows an application access to the file image without first writing it to disk. See the following code fragment:
- -
-<Open and construct the desired file with the Core file driver>
-H5Fflush(fid);
-size = H5Fget_file_image(fid, NULL, 0);
-buffer_ptr = malloc(size);
-H5Fget_file_image(fid, buffer_ptr, size);
-The use of H5Fget_file_image() may be acceptable for small images. For large images, the cost of the malloc() and memcpy() operations may be excessive. To address this issue, the H5Pset_file_image_callbacks() call allows an application to manage dynamic memory allocation for file images and memory-based file drivers (only the Core file driver at present). The following code fragment illustrates its use. Note that most error checking is omitted for simplicity and that H5Pset_file_image is not used to set the initial file image.
- -
- struct udata_t {
- void * image_ptr;
- size_t image_size;
- } udata = {NULL, 0};
-
-void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- ((struct udata_t *)udata)->image_size = size;
- return(malloc(size));
-}
-
-void *image_memcpy)(void *dest, const void *src, size_t size,
- H5_file_image_op_t file_image_op, void *udata)
-{
- assert(FALSE); /* Should never be invoked in this scenario. */
- return(NULL); /* always fails */
-}
-
-void image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- ((struct udata_t *)udata)->image_size = size;
- return(realloc(ptr, size));
-}
-
-herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE);
- ((struct udata_t *)udata)->image_ptr = ptr;
- return(0); /* if we get here, we must have been successful */
-}
-
-void *udata_copy(void *udata)
-{
- return(udata);
-}
-
-herr_t udata_free(void *udata)
-{
- return(0);
-}
-
-H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
- image_realloc, image_free,
- udata_copy, udata_free,
- (void *)(&udata)};
-
-<allocate fapl_id>
-H5Pset_file_image_callbacks(fapl_id, &callbacks);
-<open core file using fapl_id, write file, close it>
-assert(udata.image_ptr!= NULL);
-/* udata now contains the base address and length of the final version of the core file */
-<use image of file, and then discard it via free()>
-The above code fragment gives the application full ownership of the buffer used by the Core file driver after the file is closed, and it notifies the application that the HDF5 Library is done with the buffer by setting udata.image_ptr to something other than NULL. If read access to the buffer is sufficient, the H5Fget_vfd_handle() call can be used as an alternate solution to get access to the base address of the Core file driver’s buffer.
-The above solution avoids some unnecessary mallocand memcpycalls and should be quite adequate if an image of an HDF5 file is constructed only occasionally. However, if an HDF5 file image must be constructed regularly, and if we can put a strong and tight upper bound on the size of the necessary buffer, then the following pseudo code demonstrates a method of avoiding memory allocation completely. The downside, however, is that buffer is allocated statically. Again, much error checking is omitted for clarity.
- -
-char buf[BIG_ENOUGH];
-struct udata_t {
-void * image_ptr;
-size_t image_size;
-size_t max_image_size;
-int ref_count;
-} udata = {(void *)(&(buf[0]), 0, BIG_ENOUGH, 0};
-
-void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(size <= ((struct udata_t *)udata)->max_image_size);
- assert(((struct udata_t *)udata)->ref_count == 0);
- ((struct udata_t *)udata)->image_size = size;
- (((struct udata_t *)udata)->ref_count)++;
- return((((struct udata_t *)udata)->image_ptr);
-}
-void *image_memcpy)(void *dest, const void *src, size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(FALSE); /* Should never be invoked in this scenario. */
- return(NULL); /* always fails */
-}
-void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(ptr == ((struct udata_t *)udata)->image_ptr);
- assert(size <= ((struct udata_t *)udata)->max_image_size);
- assert(((struct udata_t *)udata)->ref_count == 1);
- ((struct udata_t *)udata)->image_size = size;
- return((((struct udata_t *)udata)->image_ptr);
-}
-herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE);
- assert(ptr == ((struct udata_t *)udata)->image_ptr);
- assert(((struct udata_t *)udata)->ref_count == 1);
- (((struct udata_t *)udata)->ref_count)--;
- return(0); /* if we get here, we must have been successful */
-}
-void *udata_copy(void *udata)
-{
- return(udata);
-}
-herr_t udata_free(void *udata)
-{
- return(0);
-}
-H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
- image_realloc, image_free,
- udata_copy, udata_free,
- (void *)(&udata)};
-/* end of initialization */
-<allocate fapl_id>
-H5Pset_file_image_callbacks(fapl_id, &callbacks);
-<open core file using fapl_id>
-<discard fapl any time after the open>
-<write the file, flush it, and then close it>
-assert(udata.ref_count == 0);
-/* udata now contains the base address and length of the final version of the core file */
-<use the image of the file>
-<reinitialize udata, and repeat the above from the end of initialization onwards to write a new file image>
-If we can further arrange matters so that only the contents of the datasets in the HDF5 file image change, but not the structure of the file itself, we can optimize still further by re-using the image and changing only the contents of the datasets after the initial write to the buffer. The following pseudo code shows how this might be done. Note that the code assumes that buf already contains the image of the HDF5 file whose dataset contents are to be overwritten. Again, much error checking is omitted for clarity. Also, observe that the file image callbacks do not support the H5Pget_file_image() call.
- -
-<buf already defined and loaded with file image>
-<udata already defined and initialized>
-void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(size <= ((struct udata_t *)udata)->max_image_size);
- assert(size == ((struct udata_t *)udata)->image_size);
- assert(((struct udata_t *)udata)->ref_count >= 0);
- ((struct udata_t *)udata)->image_size = size;
- (((struct udata_t *)udata)->ref_count)++;
- return((((struct udata_t *)udata)->image_ptr);
-}
-void *image_memcpy)(void *dest, const void *src, size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(dest == ((struct udata_t *)udata)->image_ptr);
- assert(src == ((struct udata_t *)udata)->image_ptr);
- assert(size <= ((struct udata_t *)udata)->max_image_size);
- assert(size == ((struct udata_t *)udata)->image_size);
- assert(((struct udata_t *)udata)->ref_count >= 1);
- return(dest); /* if we get here, we must have been successful */
-}
-void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- /* One would think that this function is not needed in this scenario, as
- * only the contents of the HDF5 file is being changed, not its size or
- * structure. However, the Core file driver calls realloc() just before
- * close to clip the buffer to the size indicated by the end of the
- * address space.
- *
- * While this call must be supported in this case, the size of
- * the image should never change. Hence the function can limit itself
- * to performing sanity checks, and returning the base address of the
- * statically allocated buffer.
- */
- assert(ptr == ((struct udata_t *)udata)->image_ptr);
- assert(size <= ((struct udata_t *)udata)->max_image_size);
- assert(((struct udata_t *)udata)->ref_count >= 1);
- assert(((struct udata_t *)udata)->image_size == size);
- return((((struct udata_t *)udata)->image_ptr);
-}
-herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
-{
- assert((file_image_op == H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE) ||
- (file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE));
- assert(((struct udata_t *)udata)->ref_count >= 1);
- (((struct udata_t *)udata)->ref_count)--;
- return(0); /* if we get here, we must have been successful */
-}
-void *udata_copy(void *udata)
-{
- return(udata);
-}
-herr_t udata_free(void *udata)
-{
- return(0);
-}
-H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
- image_realloc, image_free,
- udata_copy, udata_free,
- (void *)(&udata)};
-/* end of initialization */
-<allocate fapl_id>
-H5Pset_file_image_callbacks(fapl_id, &callbacks);
-H5Pset_file_image(fapl_id, udata.image_ptr, udata.image_len);
-<open core file using fapl_id>
-<discard fapl any time after the open>
-<overwrite data in datasets in the file, and then close it>
-assert(udata.ref_count == 0);
-/* udata now contains the base address and length of the final version of the core file */
-<use the image of the file>
-<repeat the above from the end of initialization onwards to write new data to datasets in file image>
-Before we go on, we should note that the above pseudo code can be written more compactly, albeit with fewer sanity checks, using the H5LTopen_file_image() call. See the example below:
- -
-<buf already defined and loaded with file image>
-<udata already defined and initialized>
-hid_t file_id;
-unsigned flags = H5LT_FILE_IMAGE_OPEN_RW | H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
-/* end initialization */
-file_id = H5LTopen_file_image(udata.image_ptr, udata.image_len, flags);
-<overwrite data in datasets in the file, and then close it>
-/* udata now contains the base address and length of the final version of the core file */
-<use the image of the file>
-<repeat the above from the end of initialization onwards to write new data to datasets in file image>
-Example 9. Using H5LTopen_file_image where only the datasets change -
The above pseudo code allows updates of a file image about as cheaply as possible. We assume the application has enough RAM for the image and that the HDF5 file structure is constant after the first write.
-While the scenario above is plausible, we will finish this section with a more general scenario. In the pseudo code below, we assume sufficient RAM to retain the HDF5 file image between uses, but we do not assume that the HDF5 file structure remains constant or that we can place a hard pper bound on the image size.
-Since we must use malloc, realloc, and free in this example, and since realloc can change the base address of a buffer, we must maintain two of ptr, size, and ref_count triples in the udata structure. The first triple is for the property list (which will never change the buffer), and the second triple is for the file driver. As shall be seen, this complicates the file image callbacks considerably. Note also that while we do not use H5Pget_file_image() in this example, we do include support for it in the file image callbacks. As usual, much error checking is omitted in favor of clarity.
- -
-struct udata_t {
- void * fapl_image_ptr;
- size_t fapl_image_size;
- int fapl_ref_count;
- void * vfd_image_ptr;
- size_t vfd_image_size;
- nt vfd_ref_count;
- } udata = {NULL, 0, 0, NULL, 0, 0};
-boolean initial_file_open = TRUE;
-
-void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- void * return_value = NULL;
- switch ( file_image_op ) {
- case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET:
- case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY:
- assert(((struct udata_t *)udata)->fapl_image_ptr != NULL);
- assert(((struct udata_t *)udata)->fapl_image_size == size);
- assert(((struct udata_t *)udata)->fapl_ref_count >= 0);
- return_value = ((struct udata_t *)udata)->fapl_image_ptr;
- (((struct udata_t *)udata)->fapl_ref_count)++;
- break;
- case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET:
- assert(((struct udata_t *)udata)->fapl_image_ptr != NULL);
- assert(((struct udata_t *)udata)->vfd_image_size == size);
- assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
- return_value = ((struct udata_t *)udata)->fapl_image_ptr;
- /* don’t increment ref count */
- break;
- case H5_FILE_IMAGE_OP_FILE_OPEN:
- assert(((struct udata_t *)udata)->vfd_image_ptr == NULL);
- assert(((struct udata_t *)udata)->vfd_image_size == 0);
- assert(((struct udata_t *)udata)->vfd_ref_count == 0);
- if (((struct udata_t *)udata)->fapl_image_ptr == NULL ) {
- ((struct udata_t *)udata)->vfd_image_ptr = malloc(size);
- ((struct udata_t *)udata)->vfd_image_size = size;
- } else {
- assert(((struct udata_t *)udata)->fapl_image_size == size);
- assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
- ((struct udata_t *)udata)->vfd_image_ptr = ((struct udata_t *)udata)->fapl_image_ptr;
- ((struct udata_t *)udata)->vfd_image_size = size;
- }
- return_value = ((struct udata_t *)udata)->vfd_image_ptr;
- (((struct udata_t *)udata)->vfd_ref_count)++;
- break;
- default:
- assert(FALSE);
- }
- return(return_value);
-}
-
-void *image_memcpy)(void *dest, const void *src, size_t size,
- H5_file_image_op_t file_image_op, void *udata)
-{
- switch(file_image_op) {
- case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET:
- case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY:
- case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET:
- assert(dest == ((struct udata_t *)udata)->fapl_image_ptr);
- assert(src == ((struct udata_t *)udata)->fapl_image_ptr);
- assert(size == ((struct udata_t *)udata)->fapl_image_size);
- assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
- break;
- case H5_FILE_IMAGE_OP_FILE_OPEN:
- assert(dest == ((struct udata_t *)udata)->vfd_image_ptr);
- assert(src == ((struct udata_t *)udata)->fapl_image_ptr);
- assert(size == ((struct udata_t *)udata)->fapl_image_size);
- assert(size == ((struct udata_t *)udata)->vfd_image_size);
- assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
- assert(((struct udata_t *)udata)->vfd_ref_count == 1);
- break;
- default:
- assert(FALSE);
- break;
- }
- return(dest); /* if we get here, we must have been successful */
-}
-
-void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
-{
- assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr); |
- assert(((struct udata_t *)udata)->vfd_ref_count == 1);
- ((struct udata_t *)udata)->vfd_image_ptr = realloc(ptr, size);
- ((struct udata_t *)udata)->vfd_image_size = size;
- return((((struct udata_t *)udata)->vfd_image_ptr);
-}
-
-herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
-{
- switch(file_image_op) {
- case H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE:
- assert(ptr == ((struct udata_t *)udata)->fapl_image_ptr);
- assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
- (((struct udata_t *)udata)->fapl_ref_count)--;
- break;
- case H5_FILE_IMAGE_OP_FILE_CLOSE:
- assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr);
- assert(((struct udata_t *)udata)->vfd_ref_count == 1);
- (((struct udata_t *)udata)->vfd_ref_count)--;
- break;
- default:
- assert(FALSE);
- break;
- }
- return(0); /* if we get here, we must have been successful */
-}
-
-void *udata_copy(void *udata)
-{
- return(udata);
-}
-
-herr_t udata_free(void *udata)
-{
- return(0);
-}
-H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
- image_realloc, image_free,
- udata_copy, udata_free,
- (void *)(&udata)};
-/* end of initialization */
-<allocate fapl_id>
-H5Pset_file_image_callbacks(fapl_id, &callbacks);
-if ( initial_file_open ) {
- initial_file_open = FALSE;
-} else {
- assert(udata.vfd_image_ptr != NULL);
- assert(udata.vfd_image_size > 0);
- assert(udata.vfd_ref_count == 0);
- assert(udata.fapl_ref_count == 0);
- udata.fapl_image_ptr = udata.vfd_image_ptr;
- udata.fapl_image_size = udata.vfd_image_size;
- udata.vfd_image_ptr = NULL;
- udata.vfd_image_size = 0;
- H5Pset_file_image(fapl_id, udata.fapl_image_ptr, udata.fapl_image_size);
-}
-
-<open core file using fapl_id>
-<discard fapl any time after the open>
-<write/update the file, and then close it>
-assert(udata.fapl_ref_count == 0);
-assert(udata.vfd_ref_count == 0);
-/* udata.vfd_image_ptr and udata.vfd_image_size now contain the base address and length of the final version of the core file */
-<use the image of the file>
-<repeat the above from the end of initialization to modify the file image as needed>
-<free the image when done>
-
The above pseudo code shows how a buffer can be passed back and forth between the application and the HDF5 Library. The code also shows the application having control of the actual allocation, reallocation, and freeing of the buffer.
- -4.3. Using HDF5 to Construct and Read a Data Packet
-Using the file image operations described in this document, we can bundle up data in an image of an HDF5 file on one process, transmit the image to a second process, and then open and read the image on the second process without any mandatory file system I/O.
-We have already demonstrated the construction and reading of such buffers above, but it may be useful to offer an example of the full operation. We do so in the example below using as simple a set of calls as possible. The set of calls in the example has extra buffer allocations. To reduce extra buffer allocations, see the sections above.
-In the following example, we construct an HDF5 file image on process A and then transmit the image to process B where we then open the image and extract the desired data. Note that no file system I/O is performed: all the processing is done in memory with the Core file driver.
- -| *** Process A *** | -*** Process B *** | -
|---|---|
| <Open and construct the desired file with the Core file driver> | -hid_t file_id; | -
| H5Fflush(fid); | -- |
| size = H5Fget_file_image(fid, NULL, 0); | -- |
| buffer_ptr = malloc(size); | -- |
| H5Fget_file_image(fid, buffer_ptr, size); | -- |
| <transmit size> | -<receive size> |
-
| <transmit *buffer_ptr> | -buffer_ptr = malloc(size) | -
| free(buffer_ptr); | -<receive image in *buffer_ptr> | -
| <close core file> | -file_id = H5LTopen_file_image(buf, - buf_len, - H5LT_FILE_IMAGE_DONT_COPY); | -
| - | <read data from file, then close. note that the Core file driver will discard the buffer on close> | -
4.4. Using a Template File
- -After the above examples, an example of the use of a template file might seem anti-climactic. A template file might be used to enforce consistency on file structure between files or in parallel HDF5 to avoid long sequences of collective operations to create the desired groups, datatypes, and possibly datasets. The following pseudo code outlines a potential use:
- -
-<allocate and initialize buf and buflen, with buf containing the desired initial image (which in turn contains the desired group, datatype, and dataset definitions), and buf_len containing the size of buf>
-<allocate fapl_id>
-<set fapl to use desired file driver that supports initial images>
-H5Pset_file_image(fapl_id, buf, buf_len);
-<discard buf any time after this point>
-<open file>
-<discard fapl any time after this point>
-<read and/or write file as desired, close>
-Observe that the above pseudo code includes an unnecessary buffer allocation and copy in the call to H5Pset_file_image(). As we have already discussed ways of avoiding this, we will not address that issue here.
-What is interesting in this case is to consider why the application would find this use case attractive.
-In the serial case, at first glance there seems little reason to use the initial image facility at all. It is easy enough to use standard C calls to duplicate a template file, rename it as desired, and then open it as an HDF5 file.
-However, this assumes that the template file will always be available and in the expected place. This is a questionable assumption for an application that will be widely distributed. Thus, we can at least make an argument for either keeping an image of the template file in the executable or for including code for writing the desired standard definitions to new HDF5 files.
-Assuming the image is relatively small, we can further make an argument for the image in place of the code, as, quite simply, the image should be easier to maintain and modify with an HDF5 file editor.
-However, there remains the question of why one should pass the image to the HDF5 Library instead of writing it directly with standard C calls and then using HDF5 to open it. Other than convenience and a slight reduction in code size, we are hard pressed to offer a reason.
-In contrast, the argument is stronger in the parallel case since group, datatype, and dataset creations are all expensive collective operations. The argument is also weaker: simply copying an existing template file and opening it should lose many of its disadvantages in the HPC context although we would imagine that it is always useful to reduce the number of files in a deployment.
-In closing, we would like to consider one last point. In the parallel case, we would expect template files to be quite large. Parallel HDF5 requires eager space allocation for chunked datasets. For similar reasons, we would expect template files in this context to contain long sequences of zeros with a scattering of metadata here and there. Such files would compress well, and the compressed images would be cheap to distribute across the available processes if necessary. Once distributed, each process could uncompress the image and write to file those sections containing actual data that lay within the section of the file assigned to the process. This approach might be significantly faster than a simple copy as it would allow sparse writes, and thus it might provide a compelling use case for template files. However, this approach would require extending our current API to allow compressed images. We would also have to add the H5Pget/set_image_decompression_callback() API calls. We see no problem in doing this. However, it is beyond the scope of the current effort, and thus we will not pursue the matter further unless there is interest in our doing so.
- -5. Java Signatures for File Image Operations API Calls
- -Potential Java function call signatures for the file image operation APIs are described in this section. These have not yet been implemented, and there are no immediate plans for implementation.
-Note that the H5LTopen_file_image() call is omitted. We have not supported high-level library calls in Java.
- -H5Pset_file_image
-int H5Pset_file_image(int fapl_id, const byte[] buf_ptr);
herr_t H5Pget_file_image(hid_t fapl_id, byte[] buf_ptr_ptr);
public static H5_file_image_op_t
-{
- H5_FILE_IMAGE_OP_PROPERTY_LIST_SET,
- H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY,
- H5_FILE_IMAGE_OP_PROPERTY_LIST_GET,
- H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE,
- H5_FILE_IMAGE_OP_FILE_OPEN,
- H5_FILE_IMAGE_OP_FILE_RESIZE,
- H5_FILE_IMAGE_OP_FILE_CLOSE
-}public interface H5_file_image_malloc_cb extends Callbacks {
- buf[] callback(H5_file_image_op_t file_image_op, CBuserdata udata);
-}public interface H5_file_image_memcpy_cb extends Callbacks {
-buf[] callback(buf[] dest, const buf[] src, H5_file_image_op_t file_image_op, CBuserdata
-udata);
-}public interface H5_file_image_realloc_cb extends Callbacks {
- buf[] callback(buf[] ptr, H5_file_image_op_t file_image_op, CBuserdata udata);
-}public interface H5_file_image_free_cb extends Callbacks {
- void callback(buf[] ptr, H5_file_image_op_t file_image_op, CBuserdata udata);
-}public interface H5_file_udata_copy_cb extends Callbacks {
- buf[] callback(CBuserdata udata);
-}public interface H5_file_udata_free_cb extends Callbacks {
- void callback(CBuserdata udata);
-}public abstract class H5_file_image_callbacks_t
-{
- H5_file_image_malloc_cb image_malloc;
- H5_file_image_memcpy_cb image_memcpy;
- H5_file_image_realloc_cb image_realloc;
- H5_file_image_free_cb image_free;
- H5_file_udata_copy_cb udata_copy;
- H5_file_udata_free_cb udata_free;
- CBuserdata udata;
- public H5_file_image_callbacks_t(
- H5_file_image_malloc_cb image_malloc,
- H5_file_image_memcpy_cb image_memcpy,
- H5_file_image_realloc_cb image_realloc,
- H5_file_image_free_cb image_free,
- H5_file_udata_copy_cb udata_copy,
- H5_file_udata_free_cb udata_free,
- CBuserdata udata) {
- this.image_malloc = image_malloc;
- this.image_memcpy = image_memcpy;
- this.image_realloc = image_realloc;
- this.image_free = image_free;
- this.udata_copy = udata_copy;
- this.udata_free = udata_free;
- this.udata = udata;
- }
-}int H5Pset_file_image_callbacks(int fapl_id, H5_file_image_callbacks_t callbacks_ptr);int H5Pget_file_image_callbacks(int fapl_id, H5_file_image_callbacks_t[] callbacks_ptr);long H5Fget_file_image(int file_id, byte[] buf_ptr);6. Fortran Signatures for File Image Operations API Calls
- -Potential Fortran function call signatures for the file image operation APIs are described in this section. These have not yet been implemented, and there are no immediate plans for implementation.
- -6.1. Low-level Fortran API Routines
- -The Fortran low-level APIs make use of Fortran 2003’s ISO_C_BINDING module in order to achieve portable and standard conforming interoperability with the C APIs. The C pointer (C_PTR) and function pointer (C_FUN_PTR) types are returned from the intrinsic procedures C_LOC(X) and C_FUNLOC(X), respectively, defined in the ISO_C_BINDING module. The argument X is the data or function to which the C pointers point to and must have the TARGET attribute in the calling program. Note that the variable name lengths of the Fortran equivalent of the predefined C constants were shortened to less than 31 characters in order to be Fortran standard compliant.
- -6.1.1. H5Pset_file_image_f
- -The signature of H5Pset_file_image_f is defined as follows:
-SUBROUTINE H5Pset_file_image_f(fapl_id, buf_ptr, buf_len, hdferr)
-
-
The parameters of H5Pset_file_image are defined as follows:
- -INTEGER(hid_t), INTENT(IN):: fapl_id |
-Will contain the ID of the target file access property list. | -
TYPE(C_PTR), INTENT(IN):: buf_ptr |
-Will supply the C pointer to the initial file image or C_NULL_PTR if no initial file image is desired. | -
INTEGER(size_t), INTENT(IN):: buf_len |
-Will contain the size of the supplied buffer or 0 if no initial image is desired. | -
INTEGER, INTENT(OUT) :: hdferr |
-Will return the error status: 0 for success and -1 for failure. | -
6.1.2. H5Pget_file_image_f
-The signature of H5Pget_file_image_f is defined as follows:
-SUBROUTINE H5Pget_file_image_f(fapl_id, buf_ptr, buf_len, hdferr)
-
-
The parameters of H5Pget_file_image_f are defined as follows:
-INTEGER(hid_t), INTENT(IN) :: fapl_id |
-Will contain the ID of the target file access property list | -
TYPE(C_PTR), INTENT(INOUT), VALUE :: buf_ptr |
-Will hold either a C_NULL_PTR or a scalar of type c_ptr. If buf_ptr is not C_NULL_PTR, on successful return, buf_ptr shall contain a C pointer to a copy of the initial image provided in the last call to H5Pset_file_image_f for the supplied fapl_id, or buf_ptr shall contain a C_NULL_PTR if there is no initial image set. The Fortran pointer can be obtained using the intrinsic C_F_POINTER. | -
INTEGER(size_t), INTENT(OUT) :: buf_len |
-Will contain the value of the buffer parameter for the initial image in the supplied fapl_id. The value will be 0 if no initial image is set. | -
INTEGER, INTENT(OUT) :: hdferr |
-Will return the error status: 0 for success and -1 for failure. | -
6.1.3. H5Pset_file_image_callbacks_f
-The signature of H5Pset_file_image_callbacks_f is defined as follows: - -
-INTEGER :: H5_IMAGE_OP_PROPERTY_LIST_SET_F=0,
- H5_IMAGE_OP_PROPERTY_LIST_COPY_F=1,
- H5_IMAGE_OP_PROPERTY_LIST_GET_F=2,
- H5_IMAGE_OP_PROPERTY_LIST_CLOSE_F=3,
- H5_IMAGE_OP_FILE_OPEN_F=4,
- H5_IMAGE_OP_FILE_RESIZE_F=5,
- H5_IMAGE_OP_FILE_CLOSE_F=6
-TYPE, BIND(C) :: H5_file_image_callbacks_t
- TYPE(C_FUN_PTR), VALUE :: image_malloc
- TYPE(C_FUN_PTR), VALUE :: image_memcpy
- TYPE(C_FUN_PTR), VALUE :: image_realloc
- TYPE(C_FUN_PTR), VALUE :: image_free
- TYPE(C_FUN_PTR), VALUE :: udata
- TYPE(C_FUN_PTR), VALUE :: udata_copy
- TYPE(C_FUN_PTR), VALUE :: udata_free
- TYPE(C_PTR), VALUE :: udata
-END TYPE H5_file_image_callbacks_t
-
Fortran Callback APIs
-The Fortran callback APIs are shown below.
-FUNCTION op_func(size, file_image_op, udata,) RESULT(image_malloc) - -
| INTEGER(size_t) :: size | -Will contain the size of the image buffer to allocate in bytes. | -
| INTEGER :: file_image_op | -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. | -
| TYPE(C_PTR), VALUE :: udata | -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. | -
| TYPE(C_FUN_PTR), VALUE :: image_malloc | -Shall contain a pointer to a function with functionality identical to the standard C library memcpy() call. | -
FUNCTION op_func(dest, src, size, & file_image_op, udata) RESULT(image_memcpy) - -
| TYPE(C_PTR), VALUE :: dest | -Will contain the address of the buffer into which to copy. | -
| TYPE(C_PTR), VALUE :: src | -Will contain the address of the buffer from which to copy | -
| INTEGER(size_t) :: size | -Will contain the number of bytes to copy. | -
| INTEGER :: file_image_op | -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. | -
| TYPE(C_PTR), VALUE :: udata | -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. | -
| TYPE(C_FUN_PTR), VALUE :: image_memcpy | -Shall contain a pointer to a function with functionality identical to the standard C library memcpy() call. | -
FUNCTION op_func(ptr, size, & file_image_op, udata) RESULT(image_realloc) -
| TYPE(C_PTR), VALUE :: ptr | -Will contain the pointer to the buffer being reallocated | -
| INTEGER(size_t) :: size | -Will contain the desired size of the buffer after realloc in bytes. | -
| INTEGER :: file_image_op | -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. | -
| TYPE(C_PTR), VALUE :: udata | -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. | -
| TYPE(C_FUN_PTR), VALUE :: image_realloc | -Shall contain a pointer to a unction functionality identical to the standard C library realloc() call. | -
FUNCTION op_func(ptr, file_image_op, udata) RESULT(image_free) -
| TYPE(C_PTR), VALUE :: ptr | -Will contain the pointer to the buffer being released. | -
| INTEGER :: file_image_op | -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. | -
| TYPE(C_PTR), VALUE :: udata | -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. | -
| TYPE(C_PTR), VALUE :: image_free | -Shall contain a pointer to a function with functionality identical to the standard C library free() call | -
FUNCTION op_func(udata) RESULT(udata_copy) -
| TYPE(C_PTR), VALUE :: udata | -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. | -
| TYPE(C_FUN_PTR), VALUE :: udata_copy | -Shall contain a pointer to a function that will allocate a buffer of suitable size, copy the contents of the supplied udata into the new buffer, and return the address of the new buffer. The function will return C_NULL_PTR on failure. | -
FUNCTION op_func(udata) RESULT(udata_free) -
| TYPE(C_PTR), VALUE :: udata | -Shall contain a pointer value, potentially to user-defined data, that will be passed to the image_malloc, image_memcpy, image_realloc, and image_free callbacks. | -
The signature of H5Pset_file_image_callbacks_f is defined as follows:
-SUBROUTINE H5Pset_file_image_callbacks_f(fapl_id, &callbacks_ptr, hdferr) -The parameters are defined as follows:
-INTEGER(hid_t), INTENT(IN) :: fapl_id |
-Will contain the ID of the target file access property list. | -
TYPE(H5_file_image_callbacks_t), INTENT(IN) :: callbacks_ptr |
-Will contain the callback derived type. callbacks_ptr shall contain a pointer to the Fortran function via the intrinsic functions C_LOC(X) and C_FUNLOC(X). | -
INTEGER, INTENT(OUT) :: hdferr |
-Will return the error status: 0 for success and -1 for failure. | -
6.1.4. H5Pget_file_image_callbacks_f
-The H5Pget_file_image_callbacks_f routine is designed to obtain the current file image callbacks from a file access property list.
-The signature is defined as follows
-SUBROUTINE H5Pget_file_image_callbacks_f(fapl_id, callbacks_ptr, hdferr) -The parameters are defined as follows:
-| INTEGER(hid_t), INTENT(IN) :: fapl_id | -Will contain the ID of the target file access property list. | -
| TYPE(H5_file_image_callbacks_t), INTENT(OUT) :: callbacks_ptr | -Will contain the callback derived type. Each member of the derived type shall have the same meaning as its C counterpart. See section 2.1.4 for more information. | -
| INTEGER, INTENT(OUT) :: hdferr | -Will return the error status: 0 for success and -1 for failure. | -
6.1.5. Fortran Virtual File Driver Feature Flags
-Implementation of the H5Pget/set_file_image_callbacks_f() and H5Pget/set_file_image_f() APIs requires a pair of new virtual file driver feature flags:
-H5FD_FEAT_LET_IMAGE_F -H5FD_FEAT_LET_IMAGE_CALLBACK_F -See the “Virtual File Driver Feature Flags” section for more information.
- -6.1.6. H5Fget_file_image_f
-The signature of H5Fget_file_image_f shall be defined as follows:
-SUBROUTINE H5Fget_file_image_f(file_id, buf_ptr, buf_len, hdferr, buf_size) -The parameters of H5Fget_file_image_f are defined as follows:
-| INTEGER(hid_t), INTENT(IN) :: file_id | -Will contain the ID of the target file. | -
| TYPE(C_PTR), INTENT(IN) :: buf_ptr | -Will contain a C pointer to the buffer into which the image of the HDF5 file is to be copied. If buf_ptr is C_NULL_PTR, no data will be copied. | -
| INTEGER(size_t), INTENT(IN) :: buf_len | -Will contain the size in bytes of the supplied buffer. | -
| INTEGER(ssizet_t), INTENT(OUT), OPTIONAL :: buf_size | -Will indicate the buffer size required to store the file image (in other words, the length of the file). If only the buf_size is needed, then buf_ptr should be also be set to C_NULL_PTR | -
| INTEGER, INTENT(OUT) :: hdferr | -Returns the error status: 0 for success and -1 for failure. | -
6.2. High-level Fortran API Routine
- -The new Fortran high-level routine H5LTopen_file_image_f will provide a wrapper for the high-level H5LTopen_file_image function. Consequently, the high-level Fortran API will not be implemented using low-level HDF5 Fortran APIs. - -6.2.1. H5LTopen_file_image_f
-The signature of H5LTopen_file_image_f is defined as follows: -SUBROUTINE H5LTopen_file_image_f(buf_ptr, buf_len, flags, file_id, hdferr) -The parameters of H5LTopen_file_image_f are defined as follows:
-| TYPE(C_PTR), INTENT(IN), VALUE :: buf_ptr | -Will contain a pointer to the supplied initial image. A C_NULL_PTR value is invalid and will cause H5LTopen_file_image_f to fail. | -
| INTEGER(size_t), INTENT(IN) :: buf_len | -Will contain the size of the supplied buffer. A value of 0 is invalid and will cause H5LTopen_file_image_f to fail. | -
| INTEGER, INTENT(IN) :: flags | -Will contain a set of flags indicating whether the image is to be opened read/write, whether HDF5 is to take control of the buffer, and how long the application promises to maintain the buffer. Possible flags are as follows: H5LT_IMAGE_OPEN_RW_F, H5LT_IMAGE_DONT_COPY_F, and H5LT_IMAGE_DONT_RELEASE_F. The C equivalent flags are defined in the “H5LTopen_file_image” section. | -
| INTEGER(hid_t), INTENT(IN) :: file_id | -Will be a file ID on success. | -
| INTEGER, INTENT(OUT) :: hdferr | -Returns the error status: 0 for success and -1 for failure. | -
Initial document, 18 November 1999.
- -Updated on 10/24/00, Quincey Koziol
- -Added the section “Programming Note for C++ Developers Using C -Functions,” 08/23/2012, Mark Evans - - - -
-
-
Table of Contents
--
-
- - -
Introduction
- --The HDF5 file format describes how HDF5 data structures and dataset raw -data are mapped to a linear format address space and the HDF5 -library implements that bidirectional mapping in terms of an -API. However, the HDF5 format specifications do not indicate how -the format address space is mapped onto storage and HDF (version 5 and -earlier) simply mapped the format address space directly onto a single -file by convention. - -
--Since early versions of HDF5 it became apparent that users want the ability to -map the format address space onto different types of storage (a single file, -multiple files, local memory, global memory, network distributed global -memory, a network protocol, etc.) with various types of maps. For -instance, some users want to be able to handle very large format address -spaces on operating systems that support only 2GB files by partitioning the -format address space into equal-sized parts each served by a separate -file. Other users want the same multi-file storage capability but want to -partition the address space according to purpose (raw data in one file, object -headers in another, global heap in a third, etc.) in order to improve I/O -speeds. - -
--In fact, the number of storage variations is probably larger than the -number of methods that the HDF5 team is capable of implementing and -supporting. Therefore, a Virtual File Layer API is being -implemented which will allow application teams or departments to design -and implement their own mapping between the HDF5 format address space -and storage, with each mapping being a separate file driver -(possibly written in terms of other file drivers). The HDF5 team will -provide a small set of useful file drivers which will also serve as -examples for those who which to write their own: - -
--
-
-
Using a File Driver
- --Most application writers will use a driver defined by the HDF5 library or -contributed by another programming team. This chapter describes how existing -drivers are used. - -
- - - -Driver Header Files
- --Each file driver is defined in its own public header file which should -be included by any application which plans to use that driver. The -predefined drivers are in header files whose names begin with -`H5FD' followed by the driver name and `.h'. The `hdf5.h' -header file includes all the predefined driver header files. - -
-
-Once the appropriate header file is included a symbol of the form
-`H5FD_' followed by the upper-case driver name will be the driver
-identification number.(1) However, the
-value may change if the library is closed (e.g., by calling
-H5close) and the symbol is referenced again.
-
-
Creating and Opening Files
- -
-In order to create or open a file one must define the method by which the
-storage is accessed(2) and does so by creating a file access property list(3) which is passed to the H5Fcreate or
-H5Fopen function. A default file access property list is created by
-calling H5Pcreate and then the file driver information is inserted by
-calling a driver initialization function such as H5Pset_fapl_family:
-
-
-hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
-size_t member_size = 100*1024*1024; /*100MB*/
-H5Pset_fapl_family(fapl, member_size, H5P_DEFAULT);
-hid_t file = H5Fcreate("foo%05d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
-H5Pclose(fapl);
-
-
-
-Each file driver will have its own initialization function
-whose name is H5Pset_fapl_ followed by the driver name and which
-takes a file access property list as the first argument followed by
-additional driver-dependent arguments.
-
-
-An alternative to using the driver initialization function is to set the
-driver directly using the H5Pset_driver function.(4) Its second argument is the file driver identifier, which may
-have a different numeric value from run to run depending on the order in which
-the file drivers are registered with the library. The third argument
-encapsulates the additional arguments of the driver initialization
-function. This method only works if the file driver writer has made the
-driver-specific property list structure a public datatype, which is
-often not the case.
-
-
-hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
-static H5FD_family_fapl_t fa = {100*1024*1024, H5P_DEFAULT};
-H5Pset_driver(fapl, H5FD_FAMILY, &fa);
-hid_t file = H5Fcreate("foo.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
-H5Pclose(fapl);
-
-
-
-It is also possible to query the file driver information from a file access
-property list by calling H5Pget_driver to determine the driver and then
-calling a driver-defined query function to obtain the driver information:
-
-
-hid_t driver = H5Pget_driver(fapl);
-if (H5FD_SEC2==driver) {
- /*nothing further to get*/
-} else if (H5FD_FAMILY==driver) {
- hid_t member_fapl;
- haddr_t member_size;
- H5Pget_fapl_family(fapl, &member_size, &member_fapl);
-} else if (....) {
- ....
-}
-
-
-
-
-Performing I/O
- -
-The H5Dread and H5Dwrite functions transfer data between
-application memory and the file. They both take an optional data transfer
-property list which has some general driver-independent properties and
-optional driver-defined properties. An application will typically perform I/O
-in one of three styles via the H5Dread or H5Dwrite function:
-
-
-Like file access properties in the previous section, data transfer properties -can be set using a driver initialization function or a general purpose -function. For example, to set the MPI-IO driver to use independent access for -I/O operations one would say: - -
- --hid_t dxpl = H5Pcreate(H5P_DATA_XFER); -H5Pset_dxpl_mpio(dxpl, H5FD_MPIO_INDEPENDENT); -H5Dread(dataset, type, mspace, fspace, buffer, dxpl); -H5Pclose(dxpl); -- -
-The alternative is to initialize a driver defined C struct and pass it
-to the H5Pset_driver function:
-
-
-hid_t dxpl = H5Pcreate(H5P_DATA_XFER);
-static H5FD_mpio_dxpl_t dx = {H5FD_MPIO_INDEPENDENT};
-H5Pset_driver(dxpl, H5FD_MPIO, &dx);
-H5Dread(dataset, type, mspace, fspace, buffer, dxpl);
-
-
--The transfer property list can be queried in a manner similar to the file -access property list: the driver provides a function (or functions) to return -various information about the transfer property list: - -
- -
-hid_t driver = H5Pget_driver(dxpl);
-if (H5FD_MPIO==driver) {
- H5FD_mpio_xfer_t xfer_mode;
- H5Pget_dxpl_mpio(dxpl, &xfer_mode);
-} else {
- ....
-}
-
-
-
-
-File Driver Interchangeability
- --The HDF5 specifications describe two things: the mapping of data onto a linear -format address space and the C API which performs the mapping. -However, the mapping of the format address space onto storage intentionally -falls outside the scope of the HDF5 specs. This is a direct result of the fact -that it is not generally possible to store information about how to access -storage inside the storage itself. For instance, given only the file name -`/arborea/1225/work/f%03d' the HDF5 library is unable to tell whether the -name refers to a file on the local file system, a family of files on the local -file system, a file on host `arborea' port 1225, a family of files on a -remote system, etc. - -
--Two ways which library could figure out where the storage is located are: -storage access information can be provided by the user, or the library can try -all known file access methods. This implementation uses the former method. - -
--In general, if a file was created with one driver then it isn't possible to -open it with another driver. There are of course exceptions: a file created -with MPIO could probably be opened with the sec2 driver, any file created -by the sec2 driver could be opened as a family of files with one member, -etc. In fact, sometimes a file must not only be opened with the same -driver but also with the same driver properties. The predefined drivers are -written in such a way that specifying the correct driver is sufficient for -opening a file. - -
- - -Implementation of a Driver
- --A driver is simply a collection of functions and data structures which are -registered with the HDF5 library at runtime. The functions fall into these -categories: - -
- --
-
- Functions which operate on modes - -
- Functions which operate on files - -
- Functions which operate on the address space - -
- Functions which operate on data - -
- Functions for driver initialization - -
- Optimization functions - -
Mode Functions
- --Some drivers need information about file access and data transfers which are -very specific to the driver. The information is usually implemented as a pair -of pointers to C structs which are allocated and initialized as part of an -HDF5 property list and passed down to various driver functions. There are two -classes of settings: file access modes that describe how to access the file -through the driver, and data transfer modes which are settings that control -I/O operations. Each file opened by a particular driver may have a different -access mode; each dataset I/O request for a particular file may have a -different data transfer mode. - -
--Since each driver has its own particular requirements for various settings, -each driver is responsible for defining the mode structures that it -needs. Higher layers of the library treat the structures as opaque but must be -able to copy and free them. Thus, the driver provides either the size of the -structure or a pair of function pointers for each of the mode types. - -
--Example: The family driver needs to know how the format address -space is partitioned and the file access property list to use for the -family members. - -
- -
-/* Driver-specific file access properties */
-typedef struct H5FD_family_fapl_t {
- hsize_t memb_size; /*size of each member */
- hid_t memb_fapl_id; /*file access property list of each memb*/
-} H5FD_family_fapl_t;
-
-/* Driver specific data transfer properties */
-typedef struct H5FD_family_dxpl_t {
- hid_t memb_dxpl_id; /*data xfer property list of each memb */
-} H5FD_family_dxpl_t;
-
-
--In order to copy or free one of these structures the member file access -or data transfer properties must also be copied or freed. This is done -by providing a copy and close function for each structure: - -
--Example: The file access property list copy and close functions -for the family driver: - -
- -
-static void *
-H5FD_family_fapl_copy(const void *_old_fa)
-{
- const H5FD_family_fapl_t *old_fa = (const H5FD_family_fapl_t*)_old_fa;
- H5FD_family_fapl_t *new_fa = malloc(sizeof(H5FD_family_fapl_t));
- assert(new_fa);
-
- memcpy(new_fa, old_fa, sizeof(H5FD_family_fapl_t));
- new_fa->memb_fapl_id = H5Pcopy(old_fa->memb_fapl_id);
- return new_fa;
-}
-
-static herr_t
-H5FD_family_fapl_free(void *_fa)
-{
- H5FD_family_fapl_t *fa = (H5FD_family_fapl_t*)_fa;
- H5Pclose(fa->memb_fapl_id);
- free(fa);
- return 0;
-}
-
-
-
-Generally when a file is created or opened the file access properties
-for the driver are copied into the file pointer which is returned and
-they may be modified from their original value (for instance, the file
-family driver modifies the member size property when opening an existing
-family). In order to support the H5Fget_access_plist function the
-driver must provide a fapl_get callback which creates a copy of
-the driver-specific properties based on a particular file.
-
-
-Example: The file family driver copies the member size file -access property list into the return value: - -
- -
-static void *
-H5FD_family_fapl_get(H5FD_t *_file)
-{
- H5FD_family_t *file = (H5FD_family_t*)_file;
- H5FD_family_fapl_t *fa = calloc(1, sizeof(H5FD_family_fapl_t*));
-
- fa->memb_size = file->memb_size;
- fa->memb_fapl_id = H5Pcopy(file->memb_fapl_id);
- return fa;
-}
-
-
-
-
-File Functions
- -
-The higher layers of the library expect files to have a name and allow the
-file to be accessed in various modes. The driver must be able to create a new
-file, replace an existing file, or open an existing file. Opening or creating
-a file should return a handle, a pointer to a specialization of the
-H5FD_t struct, which allows read-only or read-write access and which
-will be passed to the other driver functions as they are
-called.(5)
-
-
-typedef struct {
- /* Public fields */
- H5FD_class_t *cls; /*class data defined below*/
-
- /* Private fields -- driver-defined */
-
-} H5FD_t;
-
-
-
-Example: The family driver requires handles to the underlying
-storage, the size of the members for this particular file (which might be
-different than the member size specified in the file access property list if
-an existing file family is being opened), the name used to open the file in
-case additional members must be created, and the flags to use for creating
-those additional members. The eoa member caches the size of the format
-address space so the family members don't have to be queried in order to find
-it.
-
-
-/* The description of a file belonging to this driver. */
-typedef struct H5FD_family_t {
- H5FD_t pub; /*public stuff, must be first */
- hid_t memb_fapl_id; /*file access property list for members */
- hsize_t memb_size; /*maximum size of each member file */
- int nmembs; /*number of family members */
- int amembs; /*number of member slots allocated */
- H5FD_t **memb; /*dynamic array of member pointers */
- haddr_t eoa; /*end of allocated addresses */
- char *name; /*name generator printf format */
- unsigned flags; /*flags for opening additional members */
-} H5FD_family_t;
-
-
-
-Example: The sec2 driver needs to keep track of the underlying Unix
-file descriptor and also the end of format address space and current Unix file
-size. It also keeps track of the current file position and last operation
-(read, write, or unknown) in order to optimize calls to lseek. The
-device and inode fields are defined on Unix in order to uniquely
-identify the file and will be discussed below.
-
-
-typedef struct H5FD_sec2_t {
- H5FD_t pub; /*public stuff, must be first */
- int fd; /*the unix file */
- haddr_t eoa; /*end of allocated region */
- haddr_t eof; /*end of file; current file size*/
- haddr_t pos; /*current file I/O position */
- int op; /*last operation */
- dev_t device; /*file device number */
- ino_t inode; /*file i-node number */
-} H5FD_sec2_t;
-
-
-
-
-Opening Files
- --All drivers must define a function for opening/creating a file. This -function should have a prototype which is: - -
--
-
-
- Function: static H5FD_t * open (const char *name, unsigned flags, hid_t fapl, haddr_t maxaddr) -
-
-
-
-
-The file name name and file access property list fapl are -the same as were specified in the
H5FcreateorH5Fopen-call. The flags are the same as in those calls also except the -flagH5F_ACC_CREATEis also present if the call was to -H5Fcreateand they are documented in the `H5Fpublic.h' -file. The maxaddr argument is the maximum format address that the -driver should be prepared to handle (the minimum address is always -zero). -
-Example: The sec2 driver opens a Unix file with the requested name -and saves information which uniquely identifies the file (the Unix device -number and inode). - -
- -
-static H5FD_t *
-H5FD_sec2_open(const char *name, unsigned flags, hid_t fapl_id/*unused*/,
- haddr_t maxaddr)
-{
- unsigned o_flags;
- int fd;
- struct stat sb;
- H5FD_sec2_t *file=NULL;
-
- /* Check arguments */
- if (!name || !*name) return NULL;
- if (0==maxaddr || HADDR_UNDEF==maxaddr) return NULL;
- if (ADDR_OVERFLOW(maxaddr)) return NULL;
-
- /* Build the open flags */
- o_flags = (H5F_ACC_RDWR & flags) ? O_RDWR : O_RDONLY;
- if (H5F_ACC_TRUNC & flags) o_flags |= O_TRUNC;
- if (H5F_ACC_CREAT & flags) o_flags |= O_CREAT;
- if (H5F_ACC_EXCL & flags) o_flags |= O_EXCL;
-
- /* Open the file */
- if ((fd=open(name, o_flags, 0666))<0) return NULL;
- if (fstat(fd, &sb)<0) {
- close(fd);
- return NULL;
- }
-
- /* Create the new file struct */
- file = calloc(1, sizeof(H5FD_sec2_t));
- file->fd = fd;
- file->eof = sb.st_size;
- file->pos = HADDR_UNDEF;
- file->op = OP_UNKNOWN;
- file->device = sb.st_dev;
- file->inode = sb.st_ino;
-
- return (H5FD_t*)file;
-}
-
-
-
-
-Closing Files
- --Closing a file simply means that all cached data should be flushed to the next -lower layer, the file should be closed at the next lower layer, and all -file-related data structures should be freed. All information needed by the -close function is already present in the file handle. - -
--
-
-
- Function: static herr_t close (H5FD_t *file) -
-
-
-
-
-The file argument is the handle which was returned by the
open-function, and thecloseshould free only memory associated with the -driver-specific part of the handle (the public parts will have already been released by HDF5's virtual file layer). -
-Example: The sec2 driver just closes the underlying Unix file, -making sure that the actual file size is the same as that known to the -library by writing a zero to the last file position it hasn't been -written by some previous operation (which happens in the same code which -flushes the file contents and is shown below). - -
- -
-static herr_t
-H5FD_sec2_close(H5FD_t *_file)
-{
- H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
-
- if (H5FD_sec2_flush(_file)<0) return -1;
- if (close(file->fd)<0) return -1;
- free(file);
- return 0;
-}
-
-
-
-
-File Keys
- --Occasionally an application will attempt to open a single file more than one -time in order to obtain multiple handles to the file. HDF5 allows the files to -share information(6) but in order to -accomplish this HDF5 must be able to tell when two names refer to the same -file. It does this by associating a driver-defined key with each file opened -by a driver and comparing the key for an open request with the keys for all -other files currently open by the same driver. - -
--
-
-
- Function: const int cmp (const H5FD_t *f1, const H5FD_t *f2) -
-
-
-
-
-The driver may provide a function which compares two files f1 and -f2 belonging to the same driver and returns a negative, positive, or -zero value a la the
strcmpfunction.(7) If this -function is not provided then HDF5 assumes that all calls to theopen-callback return unique files regardless of the arguments and it is up to the -application to avoid doing this if that assumption is incorrect. -
-Each time a file is opened the library calls the cmp function to
-compare that file with all other files currently open by the same driver and
-if one of them matches (at most one can match) then the file which was just
-opened is closed and the previously opened file is used instead.
-
-
-Opening a file twice with incompatible flags will result in failure. For -instance, opening a file with the truncate flag is a two step process which -first opens the file without truncation so keys can be compared, and if no -matching file is found already open then the file is closed and immediately -reopened with the truncation flag set (if a matching file is already open then -the truncating open will fail). - -
--Example: The sec2 driver uses the Unix device and i-node as the -key. They were initialized when the file was opened. - -
- -
-static int
-H5FD_sec2_cmp(const H5FD_t *_f1, const H5FD_t *_f2)
-{
- const H5FD_sec2_t *f1 = (const H5FD_sec2_t*)_f1;
- const H5FD_sec2_t *f2 = (const H5FD_sec2_t*)_f2;
-
- if (f1->device < f2->device) return -1;
- if (f1->device > f2->device) return 1;
-
- if (f1->inode < f2->inode) return -1;
- if (f1->inode > f2->inode) return 1;
-
- return 0;
-}
-
-
-
-
-Saving Modes Across Opens
- --Some drivers may also need to store certain information in the file superblock -in order to be able to reliably open the file at a later date. This is done by -three functions: one to determine how much space will be necessary to store -the information in the superblock, one to encode the information, and one to -decode the information. These functions are optional, but if any one is -defined then the other two must also be defined. - -
--
-
-
- Function: static hsize_t sb_size (H5FD_t *file) -
- -
- Function: static herr_t sb_encode (H5FD_t *file, char *name, unsigned char *buf) -
- -
- Function: static herr_t sb_decode (H5FD_t *file, const char *name, const unsigned char *buf) -
-
-
-
-
-The
-sb_sizefunction returns the number of bytes necessary to encode -information needed later if the file is reopened. Thesb_encode-function encodes information from the file into buffer buf -allocated by the caller. It also writes an 8-character (plus null -termination) into thenameargument, which should be a unique -identification for the driver. Thesb_decodefunction looks at -the name - -- decodes -data from the buffer buf and updates the file argument with the new information, -advancing *p in the process. -
-The part of this which is somewhat tricky is that the file must be readable -before the superblock information is decoded. File access modes fall outside -the scope of the HDF5 file format, but they are placed inside the boot block -for convenience.(8) - -
--Example: To be written later. - -
- - -Address Space Functions
- --HDF5 does not assume that a file is a linear address space of bytes. Instead, -the library will call functions to allocate and free portions of the HDF5 -format address space, which in turn map onto functions in the file driver to -allocate and free portions of file address space. The library tells the file -driver how much format address space it wants to allocate and the driver -decides what format address to use and how that format address is mapped onto -the file address space. Usually the format address is chosen so that the file -address can be calculated in constant time for data I/O operations (which are -always specified by format addresses). - -
- - - -Userblock and Superblock
- --The HDF5 format allows an optional userblock to appear before the actual HDF5 -data in such a way that if the userblock is sucked out of the file and -everything remaining is shifted downward in the file address space, then the -file is still a valid HDF5 file. The userblock size can be zero or any -multiple of two greater than or equal to 512 and the file superblock begins -immediately after the userblock. - -
--HDF5 allocates space for the userblock and superblock by calling an -allocation function defined below, which must return a chunk of memory at -format address zero on the first call. - -
- - -Allocation of Format Regions
- --The library makes many types of allocation requests: - -
--
-
-
H5FD_MEM_SUPER-- -An allocation request for the userblock and/or superblock. -
H5FD_MEM_BTREE-- -An allocation request for a node of a B-tree. -
H5FD_MEM_DRAW-- -An allocation request for the raw data of a dataset. -
H5FD_MEM_META-- -An allocation request for the raw data of a dataset which -the user has indicated will be relatively small. -
H5FD_MEM_GROUP-- -An allocation request for a group leaf node (internal nodes of the group tree -are allocated as H5MF_BTREE). -
H5FD_MEM_GHEAP-- -An allocation request for a global heap collection. Global heaps are used to -store certain types of references such as dataset region references. The set -of all global heap collections can become quite large. -
H5FD_MEM_LHEAP-- -An allocation request for a local heap. Local heaps are used to store the -names which are members of a group. The combined size of all local heaps is a -function of the number of object names in the file. -
H5FD_MEM_OHDR-- -An allocation request for (part of) an object header. Object headers are -relatively small and include meta information about objects (like the data -space and type of a dataset) and attributes. -
-When a chunk of memory is freed the library adds it to a free list and
-allocation requests are satisfied from the free list before requesting memory
-from the file driver. Each type of allocation request enumerated above has its
-own free list, but the file driver can specify that certain object types can
-share a free list. It does so by providing an array which maps a request type
-to a free list. If any value of the map is H5MF_DEFAULT (zero) then the
-object's own free list is used. The special value H5MF_NOLIST indicates
-that the library should not attempt to maintain a free list for that
-particular object type, instead calling the file driver each time an object of
-that type is freed.
-
-
-Mappings predefined in the `H5FDpublic.h' file are: -
-
-
-
H5FD_FLMAP_SINGLE-- -All memory usage types are mapped to a single free list. -
H5FD_FLMAP_DICHOTOMY-- -Memory usage is segregated into meta data and raw data for the purposes of -memory management. -
H5FD_FLMAP_DEFAULT-- -Each memory usage type has its own free list. -
-Example: To make a map that manages object headers on one free list
-and everything else on another free list one might initialize the map with the
-following code: (the use of H5FD_MEM_SUPER is arbitrary)
-
-
-H5FD_mem_t mt, map[H5FD_MEM_NTYPES];
-
-for (mt=0; mt<H5FD_MEM_NTYPES; mt++) {
- map[mt] = (H5FD_MEM_OHDR==mt) ? mt : H5FD_MEM_SUPER;
-}
-
-
--If an allocation request cannot be satisfied from the free list then one of -two things happen. If the driver defines an allocation callback then it is -used to allocate space; otherwise new memory is allocated from the end of the -format address space by incrementing the end-of-address marker. - -
--
-
-
- Function: static haddr_t alloc (H5FD_t *file, H5MF_type_t type, hsize_t size) -
-
-
-
-
-The file argument is the file from which space is to be allocated, -type is the type of memory being requested (from the list above) without -being mapped according to the freelist map and size is the number of -bytes being requested. The library is allowed to allocate large chunks of -storage and manage them in a layer above the file driver (although the current -library doesn't do that). The allocation function should return a format -address for the first byte allocated. The allocated region extends from that -address for size bytes. If the request cannot be honored then the -undefined address value is returned (
HADDR_UNDEF). The first call to -this function for a file which has never had memory allocated must -return a format address of zero orHADDR_UNDEFsince this is how the -library allocates space for the userblock and/or superblock. -
-Example: To be written later. - -
- - -Freeing Format Regions
- -
-When the library is finished using a certain region of the format address
-space it will return the space to the free list according to the type of
-memory being freed and the free list map described above. If the free list has
-been disabled for a particular memory usage type (according to the free list
-map) and the driver defines a free callback then it will be
-invoked. The free callback is also invoked for all entries on the free
-list when the file is closed.
-
-
-
-
-
- Function: static herr_t free (H5FD_t *file, H5MF_type_t type, haddr_t addr, hsize_t size) -
-
-
-
-
-The file argument is the file for which space is being freed; type -is the type of object being freed (from the list above) without being mapped -according to the freelist map; addr is the first format address to free; -and size is the size in bytes of the region being freed. The region -being freed may refer to just part of the region originally allocated and/or -may cross allocation boundaries provided all regions being freed have the same -usage type. However, the library will never attempt to free regions which have -already been freed or which have never been allocated. -
-A driver may choose to not define the free function, in which case
-format addresses will be leaked. This isn't normally a huge problem since the
-library contains a simple free list of its own and freeing parts of the format
-address space is not a common occurrence.
-
-
-Example: To be written later. - -
- - -Querying Address Range
- --Each file driver must have some mechanism for setting and querying the end of -address, or EOA, marker. The EOA marker is the first format address -after the last format address ever allocated. If the last part of the -allocated address range is freed then the driver may optionally decrease the -eoa marker. - -
--
-
-
- Function: static haddr_t get_eoa (H5FD_t *file) -
-
-
-
-
-This function returns the current value of the EOA marker for the specified -file. -
-Example: The sec2 driver just returns the current eoa marker value -which is cached in the file structure: - -
- -
-static haddr_t
-H5FD_sec2_get_eoa(H5FD_t *_file)
-{
- H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
- return file->eoa;
-}
-
-
-
-The eoa marker is initially zero when a file is opened and the library may set
-it to some other value shortly after the file is opened (after the superblock
-is read and the saved eoa marker is determined) or when allocating additional
-memory in the absence of an alloc callback (described above).
-
-
-Example: The sec2 driver simply caches the eoa marker in the file -structure and does not extend the underlying Unix file. When the file is -flushed or closed then the Unix file size is extended to match the eoa marker. - -
- -
-static herr_t
-H5FD_sec2_set_eoa(H5FD_t *_file, haddr_t addr)
-{
- H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
- file->eoa = addr;
- return 0;
-}
-
-
-
-
-Data Functions
- --These functions operate on data, transferring a region of the format address -space between memory and files. - -
- - - -Contiguous I/O Functions
- --A driver must specify two functions to transfer data from the library to the -file and vice versa. - -
--
-
-
- Function: static herr_t read (H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, void *buf) -
- -
- Function: static herr_t write (H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, const void *buf) -
-
-
-
-
-The
readfunction reads data from file file beginning at address -addr and continuing for size bytes into the buffer buf -supplied by the caller. Thewritefunction transfers data in the -opposite direction. Both functions take a data transfer property list -dxpl which indicates the fine points of how the data is to be -transferred and which comes directly from theH5Dreador -H5Dwritefunction. Both functions receive type of -data being written, which may allow a driver to tune it's behavior for -different kinds of data. -
-Both functions should return a negative value if they fail to transfer the -requested data, or non-negative if they succeed. The library will never -attempt to read from unallocated regions of the format address space. - -
-
-Example: The sec2 driver just makes system calls. It tries not to
-call lseek if the current operation is the same as the previous
-operation and the file position is correct. It also fills the output buffer
-with zeros when reading between the current EOF and EOA markers and restarts
-system calls which were interrupted.
-
-
-static herr_t
-H5FD_sec2_read(H5FD_t *_file, H5FD_mem_t type/*unused*/, hid_t dxpl_id/*unused*/,
- haddr_t addr, hsize_t size, void *buf/*out*/)
-{
- H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
- ssize_t nbytes;
-
- assert(file && file->pub.cls);
- assert(buf);
-
- /* Check for overflow conditions */
- if (REGION_OVERFLOW(addr, size)) return -1;
- if (addr+size>file->eoa) return -1;
-
- /* Seek to the correct location */
- if ((addr!=file->pos || OP_READ!=file->op) &&
- file_seek(file->fd, (file_offset_t)addr, SEEK_SET)<0) {
- file->pos = HADDR_UNDEF;
- file->op = OP_UNKNOWN;
- return -1;
- }
-
- /*
- * Read data, being careful of interrupted system calls, partial results,
- * and the end of the file.
- */
- while (size>0) {
- do nbytes = read(file->fd, buf, size);
- while (-1==nbytes && EINTR==errno);
- if (-1==nbytes) {
- /* error */
- file->pos = HADDR_UNDEF;
- file->op = OP_UNKNOWN;
- return -1;
- }
- if (0==nbytes) {
- /* end of file but not end of format address space */
- memset(buf, 0, size);
- size = 0;
- }
- assert(nbytes>=0);
- assert((hsize_t)nbytes<=size);
- size -= (hsize_t)nbytes;
- addr += (haddr_t)nbytes;
- buf = (char*)buf + nbytes;
- }
-
- /* Update current position */
- file->pos = addr;
- file->op = OP_READ;
- return 0;
-}
-
-
-
-Example: The sec2 write callback is similar except it updates
-the file EOF marker when extending the file.
-
-
Flushing Cached Data
- -
-Some drivers may desire to cache data in memory in order to make larger I/O
-requests to the underlying file and thus improving bandwidth. Such drivers
-should register a cache flushing function so that the library can insure that
-data has been flushed out of the drivers in response to the application
-calling H5Fflush.
-
-
-
- - --Example: The sec2 driver doesn't cache any data but it also doesn't -extend the Unix file as aggressively as it should. Therefore, when finalizing a -file it should write a zero to the last byte of the allocated region so that -when reopening the file later the EOF marker will be at least as large as the -EOA marker saved in the superblock (otherwise HDF5 will refuse to open the -file, claiming that the data appears to be truncated). - -
- -
-static herr_t
-H5FD_sec2_flush(H5FD_t *_file)
-{
- H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
-
- if (file->eoa>file->eof) {
- if (-1==file_seek(file->fd, file->eoa-1, SEEK_SET)) return -1;
- if (write(file->fd, "", 1)!=1) return -1;
- file->eof = file->eoa;
- file->pos = file->eoa;
- file->op = OP_WRITE;
- }
-
- return 0;
-}
-
-
-
-
-Optimization Functions
- --The library is capable of performing several generic optimizations on I/O, but -these types of optimizations may not be appropriate for a given VFL driver. -
- --Each driver may provide a query function to allow the library to query whether -to enable these optimizations. If a driver lacks a query function, the library -will disable all types of optimizations which can be queried. -
- --
-
-
- Function: static herr_t query (const H5FD_t *file, unsigned long *flags) -
-
-
-
-This function is called by the library to query which optimizations to enable -for I/O to this driver. These are the flags which are currently defined: - -
-
-
- H5FD_FEAT_AGGREGATE_METADATA (0x00000001) -
- Defining the H5FD_FEAT_AGGREGATE_METADATA for a VFL driver means that -the library will attempt to allocate a larger block for metadata and -then sub-allocate each metadata request from that larger block. -
- H5FD_FEAT_ACCUMULATE_METADATA (0x00000002) -
- Defining the H5FD_FEAT_ACCUMULATE_METADATA for a VFL driver means that -the library will attempt to cache metadata as it is written to the file -and build up a larger block of metadata to eventually pass to the VFL -'write' routine. -
- H5FD_FEAT_DATA_SIEVE (0x00000004) -
- Defining the H5FD_FEAT_DATA_SIEVE for a VFL driver means that
-the library will attempt to cache raw data as it is read from/written to
-a file in a "data sieve" buffer. See Rajeev Thakur's papers:
-
-
-
- http://www.mcs.anl.gov/~thakur/papers/romio-coll.ps.gz
-
- http://www.mcs.anl.gov/~thakur/papers/mpio-high-perf.ps.gz -
-
-
- http://www.mcs.anl.gov/~thakur/papers/romio-coll.ps.gz
-
-
-
Registration of a Driver
- --Before a driver can be used the HDF5 library needs to be told of its -existence. This is done by registering the driver, which results in a driver -identification number. Instead of passing many arguments to the registration -function, the driver information is entered into a structure and the address -of the structure is passed to the registration function where it is -copied. This allows the HDF5 API to be extended while providing backward -compatibility at the source level. - -
--
-
-
- Function: hid_t H5FDregister (H5FD_class_t *cls) -
-
-
-
-
-The driver described by struct cls is registered with the library and an -ID number for the driver is returned. -
-The H5FD_class_t type is a struct with the following fields:
-
-
-
-
-
const char *name-- -A pointer to a constant, null-terminated driver name to be used for debugging -purposes. -
size_t fapl_size-- -The size in bytes of the file access mode structure or zero if the driver -supplies a copy function or doesn't define the structure. -
void *(*fapl_copy)(const void *fapl)--
-An optional function which copies a driver-defined file access mode structure.
-This field takes precedence over
fm_sizewhen both are defined. - void (*fapl_free)(void *fapl)--
-An optional function to free the driver-defined file access mode structure. If
-null, then the library calls the C
freefunction to free the -structure. - size_t dxpl_size-- -The size in bytes of the data transfer mode structure or zero if the driver -supplies a copy function or doesn't define the structure. -
void *(*dxpl_copy)(const void *dxpl)--
-An optional function which copies a driver-defined data transfer mode
-structure. This field takes precedence over
xm_sizewhen both are -defined. - void (*dxpl_free)(void *dxpl)--
-An optional function to free the driver-defined data transfer mode
-structure. If null, then the library calls the C
freefunction to -free the structure. - H5FD_t *(*open)(const char *name, unsigned flags, hid_t fapl, haddr_t maxaddr)-- -The function which opens or creates a new file. -
herr_t (*close)(H5FD_t *file)-- -The function which ends access to a file. -
int (*cmp)(const H5FD_t *f1, const H5FD_t *f2)-- -An optional function to determine whether two open files have the same key. If -this function is not present then the library assumes that two files will -never be the same. -
int (*query)(const H5FD_t *f, unsigned long *flags)-- -An optional function to determine which library optimizations a driver can -support. -
haddr_t (*alloc)(H5FD_t *file, H5FD_mem_t type, hsize_t size)-- -An optional function to allocate space in the file. -
herr_t (*free)(H5FD_t *file, H5FD_mem_t type, haddr_t addr, hsize_t size)-- -An optional function to free space in the file. -
haddr_t (*get_eoa)(H5FD_t *file)-- -A function to query how much of the format address space has been allocated. -
herr_t (*set_eoa)(H5FD_t *file, haddr_t)-- -A function to set the end of address space. -
haddr_t (*get_eof)(H5FD_t *file)-- -A function to return the current end-of-file marker value. -
herr_t (*read)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, void *buffer)-- -A function to read data from a file. -
herr_t (*write)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, const void *buffer)-- -A function to write data to a file. -
herr_t (*flush)(H5FD_t *file)-- -A function which flushes cached data to the file. -
H5FD_mem_t fl_map[H5FD_MEM_NTYPES]-- -An array which maps a file allocation request type to a free list. -
-Example: The sec2 driver would be registered as: - -
- -
-static const H5FD_class_t H5FD_sec2_g = {
- "sec2", /*name */
- MAXADDR, /*maxaddr */
- NULL, /*sb_size */
- NULL, /*sb_encode */
- NULL, /*sb_decode */
- 0, /*fapl_size */
- NULL, /*fapl_get */
- NULL, /*fapl_copy */
- NULL, /*fapl_free */
- 0, /*dxpl_size */
- NULL, /*dxpl_copy */
- NULL, /*dxpl_free */
- H5FD_sec2_open, /*open */
- H5FD_sec2_close, /*close */
- H5FD_sec2_cmp, /*cmp */
- H5FD_sec2_query, /*query */
- NULL, /*alloc */
- NULL, /*free */
- H5FD_sec2_get_eoa, /*get_eoa */
- H5FD_sec2_set_eoa, /*set_eoa */
- H5FD_sec2_get_eof, /*get_eof */
- H5FD_sec2_read, /*read */
- H5FD_sec2_write, /*write */
- H5FD_sec2_flush, /*flush */
- H5FD_FLMAP_SINGLE, /*fl_map */
-};
-
-hid_t
-H5FD_sec2_init(void)
-{
- if (!H5FD_SEC2_g) {
- H5FD_SEC2_g = H5FDregister(&H5FD_sec2_g);
- }
- return H5FD_SEC2_g;
-}
-
-
--A driver can be removed from the library by unregistering it - -
--
-
-
- Function: herr_t H5Dunregister (hid_t driver) -
- -Where driver is the ID number returned when the driver was registered. -
-Unregistering a driver makes it unusable for creating new file access or data -transfer property lists but doesn't affect any property lists or files that -already use that driver. - -
- - - - -Programming Note -for C++ Developers Using C Functions
- -If a C routine that takes a function pointer as an argument is -called from within C++ code, the C routine should be returned from -normally.
- -Examples of this kind of routine include callbacks such as
-H5Pset_elink_cb and H5Pset_type_conv_cb
-and functions such as H5Tconvert and
-H5Ewalk2.
Exiting the routine in its normal fashion allows the HDF5 C -Library to clean up its work properly. In other words, if the C++ -application jumps out of the routine back to the C++ -“catch” statement, the library is not given the -opportunity to close any temporary data structures that were set -up when the routine was called. The C++ application should save -some state as the routine is started so that any problem that -occurs might be diagnosed.
- - - - - - - -Querying Driver Information
- --
-
-
- Function: void * H5Pget_driver_data (hid_t fapl) -
- -
- Function: void * H5Pget_driver_data (hid_t fxpl) -
-
-
-
-
-This function is intended to be used by driver functions, not applications. -It returns a pointer directly into the file access property list -
faplwhich is a copy of the driver's file access mode originally -provided to theH5Pset_driverfunction. If its argument is a data -transfer property listfxplthen it returns a pointer to the -driver-specific data transfer information instead. -
Miscellaneous
- -
-The various private H5F_low_* functions will be replaced by public
-H5FD* functions so they can be called from drivers.
-
-
-All private functions H5F_addr_* which operate on addresses will be
-renamed as public functions by removing the first underscore so they can be
-called by drivers.
-
-
-The haddr_t address data type will be passed by value throughout the
-library. The original intent was that this type would eventually be a union of
-file address types for the various drivers and may become quite large, but
-that was back when drivers were part of HDF5. It will become an alias for an
-unsigned integer type (32 or 64 bits depending on how the library was
-configured).
-
-
-The various H5F*.c driver files will be renamed H5FD*.c and each
-will have a corresponding header file. All driver functions except the
-initializer and API will be declared static.
-
-
-This documentation didn't cover optimization functions which would be useful -to drivers like MPI-IO. Some drivers may be able to perform data pipeline -operations more efficiently than HDF5 and need to be given a chance to -override those parts of the pipeline. The pipeline would be designed to call -various H5FD optimization functions at various points which return one of -three values: the operation is not implemented by the driver, the operation is -implemented but failed in a non-recoverable manner, the operation is -implemented and succeeded. - -
--Various parts of HDF5 check the only the top-level file driver and do -something special if it is the MPI-IO driver. However, we might want to be -able to put the MPI-IO driver under other drivers such as the raw part of a -split driver or under a debug driver whose sole purpose is to accumulate -statistics as it passes all requests through to the MPI-IO driver. Therefore -we will probably need a function which takes a format address and or object -type and returns the driver which would have been used at the lowest level to -process the request. - -
- --
Footnotes
-(1)
-The driver name is by convention and might -not apply to drivers which are not distributed with HDF5. -
(2)
-The access method also indicates how to translate -the storage name to a storage server such as a file, network protocol, or -memory. -
(3)
-The term -"file access property list" is a misnomer since storage isn't -required to be a file. -
(4)
-This -function is overloaded to operate on data transfer property lists also, as -described below. -
(5)
-Read-only access is only appropriate when opening an existing -file. -
(6)
-For instance, writing data to one handle will cause -the data to be immediately visible on the other handle. -
(7)
-The ordering is -arbitrary as long as it's consistent within a particular file driver. -
(8)
-File access modes do not describe data, but rather -describe how the HDF5 format address space is mapped to the underlying -file(s). Thus, in general the mapping must be known before the file superblock -can be read. However, the user usually knows enough about the mapping for the -superblock to be readable and once the superblock is read the library can fill -in the missing parts of the mapping. -
- - - - - diff --git a/doxygen/hdf5_footer.html b/doxygen/hdf5_footer.html index 520f3f57291..6023e109739 100644 --- a/doxygen/hdf5_footer.html +++ b/doxygen/hdf5_footer.html @@ -3,18 +3,15 @@
-
$navpath
-
- $generatedby
-
-
$doxygenversion
+ - $generatedby
$doxygenversion
-$generatedby -
- $doxygenversion
+$generatedby $doxygenversion
+