Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase the default PVC size for logstash #7540

Merged
merged 9 commits into from
Feb 9, 2024
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ WARNING: Volume support for Logstash is a breaking change to earlier versions of
[discrete]
== Specifying the volume claim settings

By default, a PersistentVolume called `logstash-data` is created, that maps to `/usr/share/logstash/data` for persistent storage, typically used for storage from plugins. The `logstash-data` volume claim is, by default, a small (1Gi) volume, using the standard StorageClass of your Kubernetes cluster, but can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data`.
By default, a PersistentVolume called `logstash-data` is created, that maps to `/usr/share/logstash/data` for persistent storage, typically used for storage from plugins. The `logstash-data` volume claim is, by default, a 1.5Gi volume, using the standard StorageClass of your Kubernetes cluster, but can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data`.
kaisecheng marked this conversation as resolved.
Show resolved Hide resolved

For production workloads, you should define your own volume claim template with the desired storage capacity and (optionally) the Kubernetes link:https://kubernetes.io/docs/concepts/storage/storage-classes/[storage class] to associate with the persistent volume. To override this volume claim for `data` usages, the name of this volume claim must be `logstash-data`.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be a good idea to add extra information on using persistent volumes, something like:

In particular, when enabling the persistent queue feature, using dead letter queues, or certain plugins that make heavy use of temporary storage, the default volume size will likely be insufficient for production workloads and you should increase the storage capacity, or consider creating separate volumes for these use cases.

Maybe just before the Separate storage section?

Expand Down
18 changes: 9 additions & 9 deletions pkg/controller/logstash/logstash_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -303,7 +303,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -432,7 +432,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -550,7 +550,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -650,15 +650,15 @@ func TestReconcileLogstash_Resize(t *testing.T) {
}{
{
name: "Cannot increase storage with fixed storage class",
initialCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "3Gi",
storageClass: fixedStorageClass,
wantErr: true,
},
{
name: "Cannot decrease storage with resizable storage class",
initialCapacity: "3Gi",
desiredCapacity: "1Gi",
desiredCapacity: "1.5Gi",
storageClass: resizableStorageClass,
wantErr: true,
},
Expand All @@ -671,14 +671,14 @@ func TestReconcileLogstash_Resize(t *testing.T) {
},
{
name: "Nothing happens when keeping the storage the same with resizable storage class",
initialCapacity: "1Gi",
desiredCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "1.5Gi",
storageClass: resizableStorageClass,
wantErr: false,
},
{
name: "Can successfully resize the storage with resizable storage class",
initialCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "3Gi",
storageClass: resizableStorageClass,
extraVerify: func(r ReconcileLogstash, desiredCapacity string) (reconcile.Result, error) {
Expand Down
2 changes: 1 addition & 1 deletion pkg/controller/logstash/volume/defaults.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ var (
ContainerMountPath: ConfigMountPath,
}

DefaultPersistentVolumeSize = resource.MustParse("1Gi")
DefaultPersistentVolumeSize = resource.MustParse("1.5Gi")

// DefaultDataVolumeClaim is the default data volume claim for Logstash pods.
// We default to a 1GB persistent volume, using the default storage class.
kaisecheng marked this conversation as resolved.
Show resolved Hide resolved
Expand Down