Skip to content

Commit

Permalink
MGMT-8571: Deploy the image service as a stateful set (#3067)
Browse files Browse the repository at this point in the history 
* MGMT-8571: add optional image storage to agent service config

* MGMT-8571: add image storage to operator deploy script and docs

* MGMT-8571: Refactor monitoring into a separate method

This also stops trying to monitor the image service as a deployment and
reduces the number of `Get` calls by only fetching each deployment once
per loop rather than re-fetching it for each condition we want to check.

* MGMT-8571: Deploy the image service as a stateful set

Previously, when deploying the image service as a deployment, we were
writing the template images to the pod filesystem. This isn't good for
performance or the health of the node we're running on.

Because of this the image service now will run as a stateful set and
request a PV for the data directory. If information for a new PVC has
been provided in the agentserviceconfig. If no PVC information was
provided an emptydir volume is used.

Changing the storage template of a stateful set requires that we delete
and recreate the entire statefulset. This means that, in this case, we
can't use controllerutil.CreateOrUpdate.

This commit creates a separate function to reconcile just this
statefulset which has a section very similar to create or update but with
some additional logic around when we can do a normal update and when we
need to recreate the statefulset.

This also adds some logic to remove the old image service deployment.
This is required for upgrade. Note the specific error handling which
ensures that we only delete the deployment if we successfully
reconciled (created, most likely) the statefulset to replace it.

A finalizer is added to the statefulset to ensure we remove the PVC as
we need to support resizing the volume which will mean claiming a new
PV.

A finalizer is also added to the agentserviceconfig object to ensure we
clean up the image-service PVCs in the case that the agentserviceconfig
is deleted directly.

https://issues.redhat.com/browse/MGMT-8571
  • Loading branch information
@carbonin
carbonin authored Mar 10, 2022
1 parent 2f81d62 commit c77aa81
Show file tree
Hide file tree
Showing 13 changed files with 1,239 additions and 163 deletions.
10 changes: 8 additions & 2 deletions api/v1beta1/agentserviceconfig_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,12 @@ type AgentServiceConfigSpec struct {
// With respect to the resource requests, minimum 10GiB is recommended.
//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Storage for database"
DatabaseStorage corev1.PersistentVolumeClaimSpec `json:"databaseStorage"`
// ImageStorage defines the spec of the PersistentVolumeClaim to be
// created for each replica of the image service.
// If a PersistentVolumeClaim is provided 2GiB per OSImage entry is required
// +optional
//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Storage for images"
ImageStorage *corev1.PersistentVolumeClaimSpec `json:"imageStorage"`
// MirrorRegistryRef is the reference to the configmap that contains mirror registry configuration
// In case no configuration is need, this field will be nil. ConfigMap must contain to entries:
// ca-bundle.crt - hold the contents of mirror registry certificate/s
Expand Down Expand Up @@ -116,8 +122,8 @@ const (
ReasonIngressCertFailure string = "IngressCertFailure"
// ReasonConfigFailure when there was a failure configuring/deploying the assisted-service configmap.
ReasonConfigFailure string = "ConfigFailure"
// ReasonImageHandlerDeploymentFailure when there was a failure configuring/deploying the assisted-image-service deployment.
ReasonImageHandlerDeploymentFailure string = "ImageHandlerDeploymentFailure"
// ReasonImageHandlerStatefulSetFailure when there was a failure configuring/deploying the assisted-image-service stateful set.
ReasonImageHandlerStatefulSetFailure string = "ImageHandlerStatefulSetFailure"
// ReasonDeploymentFailure when there was a failure configuring/deploying the assisted-service deployment.
ReasonDeploymentFailure string = "DeploymentFailure"
// ReasonStorageFailure when there was a failure configuring/deploying the validating webhook.
Expand Down
5 changes: 5 additions & 0 deletions api/v1beta1/zz_generated.deepcopy.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

161 changes: 161 additions & 0 deletions config/crd/bases/agent-install.openshift.io_agentserviceconfigs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,167 @@ spec:
backing this claim.
type: string
type: object
imageStorage:
description: ImageStorage defines the spec of the PersistentVolumeClaim
to be created for each replica of the image service. If a PersistentVolumeClaim
is provided 2GiB per OSImage entry is required
properties:
accessModes:
description: 'AccessModes contains the desired access modes the
volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1'
items:
type: string
type: array
dataSource:
description: 'This field can be used to specify either: * An existing
VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot)
* An existing PVC (PersistentVolumeClaim) If the provisioner
or an external controller can support the specified data source,
it will create a new volume based on the contents of the specified
data source. If the AnyVolumeDataSource feature gate is enabled,
this field will always have the same contents as the DataSourceRef
field.'
properties:
apiGroup:
description: APIGroup is the group for the resource being
referenced. If APIGroup is not specified, the specified
Kind must be in the core API group. For any other third-party
types, APIGroup is required.
type: string
kind:
description: Kind is the type of resource being referenced
type: string
name:
description: Name is the name of resource being referenced
type: string
required:
- kind
- name
type: object
dataSourceRef:
description: 'Specifies the object from which to populate the
volume with data, if a non-empty volume is desired. This may
be any local object from a non-empty API group (non core object)
or a PersistentVolumeClaim object. When this field is specified,
volume binding will only succeed if the type of the specified
object matches some installed volume populator or dynamic provisioner.
This field will replace the functionality of the DataSource
field and as such if both fields are non-empty, they must have
the same value. For backwards compatibility, both fields (DataSource
and DataSourceRef) will be set to the same value automatically
if one of them is empty and the other is non-empty. There are
two important differences between DataSource and DataSourceRef:
* While DataSource only allows two specific types of objects,
DataSourceRef allows any non-core object, as well as PersistentVolumeClaim
objects. * While DataSource ignores disallowed values (dropping
them), DataSourceRef preserves all values, and generates an
error if a disallowed value is specified. (Alpha) Using this
field requires the AnyVolumeDataSource feature gate to be enabled.'
properties:
apiGroup:
description: APIGroup is the group for the resource being
referenced. If APIGroup is not specified, the specified
Kind must be in the core API group. For any other third-party
types, APIGroup is required.
type: string
kind:
description: Kind is the type of resource being referenced
type: string
name:
description: Name is the name of resource being referenced
type: string
required:
- kind
- name
type: object
resources:
description: 'Resources represents the minimum resources the volume
should have. If RecoverVolumeExpansionFailure feature is enabled
users are allowed to specify resource requirements that are
lower than previous value but must still be higher than capacity
recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources'
properties:
limits:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Limits describes the maximum amount of compute
resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
type: object
requests:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Requests describes the minimum amount of compute
resources required. If Requests is omitted for a container,
it defaults to Limits if that is explicitly specified, otherwise
to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
type: object
type: object
selector:
description: A label query over volumes to consider for binding.
properties:
matchExpressions:
description: matchExpressions is a list of label selector
requirements. The requirements are ANDed.
items:
description: A label selector requirement is a selector
that contains values, a key, and an operator that relates
the key and values.
properties:
key:
description: key is the label key that the selector
applies to.
type: string
operator:
description: operator represents a key's relationship
to a set of values. Valid operators are In, NotIn,
Exists and DoesNotExist.
type: string
values:
description: values is an array of string values. If
the operator is In or NotIn, the values array must
be non-empty. If the operator is Exists or DoesNotExist,
the values array must be empty. This array is replaced
during a strategic merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: matchLabels is a map of {key,value} pairs. A
single {key,value} in the matchLabels map is equivalent
to an element of matchExpressions, whose key field is "key",
the operator is "In", and the values array contains only
"value". The requirements are ANDed.
type: object
type: object
storageClassName:
description: 'Name of the StorageClass required by the claim.
More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1'
type: string
volumeMode:
description: volumeMode defines what type of volume is required
by the claim. Value of Filesystem is implied when not included
in claim spec.
type: string
volumeName:
description: VolumeName is the binding reference to the PersistentVolume
backing this claim.
type: string
type: object
mirrorRegistryRef:
description: 'MirrorRegistryRef is the reference to the configmap
that contains mirror registry configuration In case no configuration
Expand Down
Loading

0 comments on commit c77aa81

Please sign in to comment.