Graceful Shutdown storage-users service

modules/ROOT/pages/depl-examples/bare-metal.adoc

@@ -28,6 +28,8 @@ This guide was tested on a Debian 11 (bullseye) host but should work on any othe
 
 NOTE: The embedded IDM service provides a minimal LDAP service for Infinite Scale and does not replace a LDAP server. See the xref:{s-path}/idm.adoc[IDM Service Configuration] for details.
 
+NOTE: Consider to extend the configuration described based on the information given in xref:{s-path}/storage-users.adoc#graceful-shutdown[Graceful Shutdown].
+
 == Requirements
 
 * A supported filesystem according the xref:prerequisites/prerequisites.adoc#filesystems-and-shared-storage[ocis prerequisites,window=_blank].

modules/ROOT/pages/deployment/services/s-list/storage-users.adoc

@@ -15,9 +15,25 @@
 
 * Storage-Users listens on port 9157 by default.
 
-== Manage Unfinished Uploads
+== Deprecated Metadata Backend
+
+Starting with Infinite Scale version 3.0.0, the default backend for metadata switched to messagepack. If the setting `STORAGE_USERS_OCIS_METADATA_BACKEND` has not been defined manually, the backend will be migrated to `messagepack` automatically. Although it is still possible to manually configure `xattrs`, this setting should not be used anymore as it will be removed in a later version.
+
+== Graceful Shutdown
+
+Starting with Infinite Scale version 3.1, you can define a graceful shutdown period for the `storage-users` service.
+
+IMPORTANT: The graceful shutdown period is only applicable if the `storage-users` service runs as standalone service. It does not apply if the `storage-users` service runs as part of the single binary or as single Docker environment. To build an environment where the `storage-users` service runs as a standalone service, you must start two instances, one _without_ the `storage-users` service and one _only with_ the the `storage-users` service. Note that both instances must be able to communicate on the same network. 
+
+When hard-stopping Infinite Scale, for example with the `kill <pid>` command (SIGKILL), it is possible and likely that not all data from the decomposedfs (metadata) has been written to the storage which may result in an inconsistent decomposedfs. When gracefully shutting down Infinite Scale, using a command like SIGTERM, the process will no longer accept any write requests from _other_ services and will try to write the internal open  requests which can take an undefined duration based on many factors. To mitigate that situation, the following things have been implemented:
+
+*   With the value of the environment variable `STORAGE_USERS_GRACEFUL_SHUTDOWN_TIMEOUT`, the `storage-users` service will delay its shutdown giving it time to finalize writing necessary data. This delay can be necessary if there is a lot of data to be saved and/or if storage access/thruput is slow. In such a case you would receive an error log entry informing you that not all data could be saved in time. To prevent such occurrences, you must increase the default value.
 
-// referencing: [oCIS FS] clean up aborted uploads https://github.com/owncloud/ocis/issues/2622
+*   If a shutdown error has been logged, the command-line maintenance tool xref:maintenance/commands/commands.adoc#inspect-and-manipulate-node-metadata[Inspect and Manipulate Node Metadata] can help to fix the issue. Please contact support for details. 
+
+== CLI Commands
+
+=== Manage Unfinished Uploads
 
 When using Infinite Scale as user storage, a directory named `storage/users/uploads` can be found in the Infinite Scale data folder. This is an intermediate directory based on {tus-url}[TUS] which is an open protocol for resumable uploads. Each upload consists of a _blob_ and a _blob.info_ file. Note that the term _blob_ is just a placeholder.
 
@@ -48,7 +64,7 @@ COMMANDS:
    clean    Clean up leftovers from expired uploads
 ----
 
-=== Command Examples
+==== Command Examples
 
 Command to identify incomplete uploads::
 +
@@ -82,9 +98,7 @@ Cleaned uploads:
 ----
 --
 
-== Purge Expired Space Trash-Bin Items
-
-// referencing: https://github.com/owncloud/ocis/pull/5500
+=== Purge Expired Space Trash-Bin Items
 
 This command is about purging old trash-bin items of `project` spaces (spaces that have been created manually) and `personal` spaces.
 
@@ -115,10 +129,6 @@ include::partial$multi-location/caching-list.adoc[]
 // you must not have one or more blank lines here as it breaks continuous numbering!
 . When using `redis-sentinel`, the Redis master to use is configured via `STORAGE_USERS_STAT_CACHE_STORE_NODES`, `STORAGE_USERS_FILEMETADATA_CACHE_STORE_NODES` and `STORAGE_USERS_ID_CACHE_STORE_NODES` in the form of `<sentinel-host>:<sentinel-port>/<redis-master>` like `10.10.0.200:26379/mymaster`.
 
-== Deprecated Metadata Backend
-
-Starting with Infinite Scale version 3.0.0, the default backend for metadata switched to messagepack. If the setting `STORAGE_USERS_OCIS_METADATA_BACKEND` has not been defined manually, the backend will be migrated to `messagepack` automatically. Though still possible to manually configure `xattrs`, this setting should not be used anymore as it will be removed in a later version.
-
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[]