Skip to content

Configuration Storage

Jump to bottom
Denis Garus edited this page Apr 27, 2023 · 9 revisions

Storage

Artipie "Storage" is an abstraction on top of multiple key-value storage providers. Artipie supports:

The Storage is used for storing repository data, proxy/mirrors repository caching and for Artipie configuration. Such storage can be configured in various config files and in various sections of config files, but storage configuration structure is always the same: each storage is defined by storage yaml key, mandatory type parameter and provider dependent configuration parameters.

File System storage

The file system storage uses file-system as a back-end for binary key-value mapping - it save blobs in files using file paths as a keys. It requires the root path to be configured with path parameters.

Example:

storage:
  type: fs
  path: /var/artipie

S3 storage

Artipie supports any S3-compatible cloud storage (e.g. AWS, Digital-Ocean, GCE). The type of S3 storage is s3. Supported settings for S3 storage are:

  • bucket (string, required) - bucket name
  • region (string, optional) - bucket region name
  • endpoint (string, optional) - S3 API provider URL, default is standard AWS S3 endpoint
  • credentials (map, optional):
    • type (string, required inside the credentials map) - authentication type, one of: basic
    • accessKeyId (string, required inside the credentials map) - access API key ID
    • secretAccessKey (string, required inside the credentials map) - secret key

Example:

storage:
  type: s3
  bucket: artipie
  region: east
  endpoint: https://minio.selfhosted/s3
  credentials:
    type: basic
    accessKeyId: asagn8as8f81
    secretAccessKey: 9889sg8nas8ng

Configure the credentials that should be used to authenticate with AWS.
The default provider will attempt to identify the credentials automatically using the following checks:

  • Java System Properties - aws.accessKeyId and aws.secretKey
  • Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  • Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
  • Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set and security manager has permission to access the variable.
  • Instance profile credentials delivered through the Amazon EC2 metadata service

Etcd storage

Etcd storage uses etcd cluster as a back-end. It may be useful for configuration storage of Artipie server. This kind of storage require blobs to be smaller than 10Mb. Storage type is etcd, other parameters are:

  • endpoints (string list, required) - the list of valid cluster endpoints
  • timeout (number, optional) - connection timeout in milliseconds

Example:

storage:
  type: etcd
  endpoints:
    - http://node1.ectd.local:2379
    - http://node2.etcd.local:2379
  timeout: 5000

In memory storage

In-memory storage is not persistent, it exists only while Artipie process is alive and is used in Artipie tests, check the implementation for more details. There is no possibility to use in memory storage from configuration, it's for unit and integration tests only.

Custom storage

Artipie users have an option to implement and use a custom storage. If you want to make your storage, you need to define asto-core dependency in pom file of your project:

<dependency>
   <groupId>com.artipie</groupId>
   <artifactId>asto-core</artifactId>
   <version>...</version>
</dependency>

On the next step, you have to implement interface Storage to host data in the place which you need. The interface StorageFactory is responsible for creating a new storage instance. You have to implement this interface and mark the implementation with annotation ArtipieStorageFactory. This annotation helps Artipie to find factory classes and provides a name of storage type. Storage type name must be unique in the scope of one Artipie server. In the case of type name conflict, Artipie will throw an exception on a start-up stage. Storage configuration is represented by an interface StorageConfig. Currently, this interface has the single implementation YamlStorageConfig that allows to define configuration as yaml file. It's also possible to use an own implementation of StorageConfig.

To start Artipie with a custom storage, you have to:

  • provide a file with configuration of storage; this is done the same way as for other storage types.
  • put a jar file that contains implementation classes and all needed libraries to classpath.

If logging is switched to info level, you should see the following log record:

Initiated storage factory [type={your-storage-type}, class={your-storage-factory-class-name}]

You can study a storage implementation based on Redis java client Redisson as a good example.

Storage Verification tests

The asto-core module contains verification tests which are designed to confirm correctness of a storage implementation. A storage verification test class has to extend the StorageWhiteboxVerification and implement the newStorage method, which returns a new instance of storage to check.

Note: To avoid possible mutual affects between tests methods, test environment should be returned to initial state before each test.

Note: If storage can't be used with SubStorage, you should override newBaseForSubStorage or(and) newBaseForRootSubStorage method(s) to return Optional.empty().

If a storage implementation passes all verification tests, it can be used by Artipie server to host binary data or configuration.

Storage Aliases

Artipie has special configuration item for storage aliases: _storages.yaml file located in configuration root. This file can define storages with names, then repository configuration file can use these names (or aliases) instead of full storage description. It allows to avoid duplicates and to hide real storage configuration or credentials from repository maintainers: Artipie server administrator can configure storage and provide alias to users, and users will set this alias for repositories instead of full configuration.

# _storages.yaml
storages:
  default: # storage alias name to use in repository configs
    type: fs
    path: /var/artipie/data
  remote_s3: # storage alias name to use in repository configs
     type: s3
     bucket: artipie
     region: east
     endpoint: https://minio.selfhosted/s3
     credentials:
        type: basic
        accessKeyId: asagn8as8f81
        secretAccessKey: 9889sg8nas8ng
# repository configuration
repo:
  type: file
  storage: default
# or
repo:
   type: maven
   storage: remote_s3
Clone this wiki locally