Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase the default PVC size for logstash #7540

Merged
merged 9 commits into from
Feb 9, 2024
Merged

Increase the default PVC size for logstash #7540

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c934b7a
Increase the default PVC size for logstash
kaisecheng Feb 8, 2024
f3d0c9c
update unit tests
kaisecheng Feb 8, 2024
23f25aa
update doc
kaisecheng Feb 8, 2024
9bbf3fa
Update docs/orchestrating-elastic-stack-applications/logstash.asciidoc
kaisecheng Feb 8, 2024
b1e09c9
Update docs/orchestrating-elastic-stack-applications/logstash.asciidoc
kaisecheng Feb 8, 2024
9caa231
Update docs/orchestrating-elastic-stack-applications/logstash.asciidoc
kaisecheng Feb 8, 2024
3536e39
Update docs/orchestrating-elastic-stack-applications/logstash.asciidoc
kaisecheng Feb 9, 2024
b8671c0
Update docs/orchestrating-elastic-stack-applications/logstash.asciidoc
kaisecheng Feb 9, 2024
43cd3e0
Update pkg/controller/logstash/volume/defaults.go
kaisecheng Feb 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/orchestrating-elastic-stack-applications/logstash.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ WARNING: Volume support for Logstash is a breaking change to earlier versions of
[discrete]
== Specifying the volume claim settings

By default, a PersistentVolume called `logstash-data` is created, that maps to `/usr/share/logstash/data` for persistent storage, typically used for storage from plugins. The `logstash-data` volume claim is, by default, a small (1Gi) volume, using the standard StorageClass of your Kubernetes cluster, but can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data`.
By default, a PersistentVolume called `logstash-data` is created, that maps to `/usr/share/logstash/data` for persistent storage, typically used for storage from plugins. The `logstash-data` volume claim is, by default, a 1.5Gi volume, using the standard StorageClass of your Kubernetes cluster, but can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data`.
kaisecheng marked this conversation as resolved.
Show resolved Hide resolved

For production workloads, you should define your own volume claim template with the desired storage capacity and (optionally) the Kubernetes link:https://kubernetes.io/docs/concepts/storage/storage-classes/[storage class] to associate with the persistent volume. To override this volume claim for `data` usages, the name of this volume claim must be `logstash-data`.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be a good idea to add extra information on using persistent volumes, something like:

In particular, when enabling the persistent queue feature, using dead letter queues, or certain plugins that make heavy use of temporary storage, the default volume size will likely be insufficient for production workloads and you should increase the storage capacity, or consider creating separate volumes for these use cases.

Maybe just before the Separate storage section?

Expand All @@ -326,8 +326,9 @@ spec:
storage: 2Gi
----

In particular, when enabling the persistent queue (PQ) feature, using dead letter queues (DLQ), or certain plugins that make heavy use of temporary storage, the default volume size will likely be insufficient for production workloads and you should increase the storage capacity, or consider creating separate volumes for these use cases.
kaisecheng marked this conversation as resolved.
Show resolved Hide resolved

Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.
Separate storage can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.
kaisecheng marked this conversation as resolved.
Show resolved Hide resolved

This example shows how to setup separate storage for a PQ:

Expand Down
18 changes: 9 additions & 9 deletions pkg/controller/logstash/logstash_controller_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -303,7 +303,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -432,7 +432,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -550,7 +550,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -650,15 +650,15 @@ func TestReconcileLogstash_Resize(t *testing.T) {
}{
{
name: "Cannot increase storage with fixed storage class",
initialCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "3Gi",
storageClass: fixedStorageClass,
wantErr: true,
},
{
name: "Cannot decrease storage with resizable storage class",
initialCapacity: "3Gi",
desiredCapacity: "1Gi",
desiredCapacity: "1.5Gi",
storageClass: resizableStorageClass,
wantErr: true,
},
Expand All @@ -671,14 +671,14 @@ func TestReconcileLogstash_Resize(t *testing.T) {
},
{
name: "Nothing happens when keeping the storage the same with resizable storage class",
initialCapacity: "1Gi",
desiredCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "1.5Gi",
storageClass: resizableStorageClass,
wantErr: false,
},
{
name: "Can successfully resize the storage with resizable storage class",
initialCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "3Gi",
storageClass: resizableStorageClass,
extraVerify: func(r ReconcileLogstash, desiredCapacity string) (reconcile.Result, error) {
Expand Down
2 changes: 1 addition & 1 deletion pkg/controller/logstash/volume/defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ var (
ContainerMountPath: ConfigMountPath,
}

DefaultPersistentVolumeSize = resource.MustParse("1Gi")
DefaultPersistentVolumeSize = resource.MustParse("1.5Gi")

// DefaultDataVolumeClaim is the default data volume claim for Logstash pods.
// We default to a 1GB persistent volume, using the default storage class.
kaisecheng marked this conversation as resolved.
Show resolved Hide resolved
Expand Down