Decompress GZIP'd user data #1762

cartermckinnon · 2024-04-11T19:51:28Z

Issue #, if available:

Description of changes:

Adds support for GZIP-compressed user data.

The following scenarios are supported:

User data that consists of a NodeConfig that is compressed with GZIP.
User data that is a multi-part MIME document that is compressed with GZIP.
User data that is a multi-part MIME document containing parts that are individually GZIP'd (and have the Content-Encoding: gzip header).
A combination of 2 and 3.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

ndbaker1

the change LGTM. doesn't seem like we are going to adopt decompressing individual parts, so can we do a doc update before merging?

cartermckinnon · 2024-05-16T00:37:32Z

doesn't seem like we are going to adopt decompressing individual parts

@ndbaker1 latest rev adds support for this, PTAL.

cartermckinnon · 2024-05-16T00:38:30Z

nodeadm/internal/configprovider/userdata_test.go

I rewrote the unit tests for this config provider; they're more verbose but I think being explicit about the input is more clear and lets us more easily add test cases in the future.

Did you do any integration testing here? Or is that covered by the existing integration tests?

i like the change, thanks for skipping over the junk to setup mime documents and testing the actual provider interface

nodeadm/internal/configprovider/userdata.go

mmerkes · 2024-05-16T19:03:59Z

nodeadm/internal/configprovider/userdata.go

+				if part.Header.Get(contentEncodingHeader) == "gzip" {
+					decompressedNodeConfigPart, err := decompressIfGZIP(nodeConfigPart)
+					if err != nil {
+						return nil, err
+					}
+					nodeConfigPart = decompressedNodeConfigPart
+				}


Why do you need to make a check that it's gzip and then gracefully handle it not being gzip?

wouldn't just calling decompressIfGZIP in all cases work better since its fallible in constant time and safe either way?

To not hit this branch you'd have to set the nodeConfig mediaType but forget to add the gzip encoding header, which im sure would only happen on accident. maybe handling it explicitly is more "proper", but not sure if its necessary

MIME semantics dictate that if the section is GZIP'd, this header needs to be there. If a user (or library) is creating MIME documents that use GZIP but omit this header, that should fail because it's not conformant to the spec.

Generally, folks are only using GZIP compression via a library which should implement this properly -- folks aren't hand-writing and hand-compressing their MIME docs.

Turns out cloudinit doesn't care what the spec says, and just tries to base64-decode and gzip-decompress pretty much anything it handles at any scope. It's more important for us to match the cloudinit behavior than to follow the spec, for better or worse. Updated in latest rev

mmerkes · 2024-05-16T19:04:48Z

nodeadm/internal/configprovider/userdata_test.go

Did you do any integration testing here? Or is that covered by the existing integration tests?

ndbaker1 · 2024-05-16T19:45:22Z

nodeadm/internal/configprovider/userdata.go

+				if part.Header.Get(contentEncodingHeader) == "gzip" {
+					decompressedNodeConfigPart, err := decompressIfGZIP(nodeConfigPart)
+					if err != nil {
+						return nil, err
+					}
+					nodeConfigPart = decompressedNodeConfigPart
+				}


wouldn't just calling decompressIfGZIP in all cases work better since its fallible in constant time and safe either way?

To not hit this branch you'd have to set the nodeConfig mediaType but forget to add the gzip encoding header, which im sure would only happen on accident. maybe handling it explicitly is more "proper", but not sure if its necessary

nodeadm/internal/configprovider/userdata.go

ndbaker1 · 2024-05-16T19:51:30Z

nodeadm/internal/configprovider/userdata_test.go

i like the change, thanks for skipping over the junk to setup mime documents and testing the actual provider interface

nodeadm/internal/configprovider/userdata.go

cartermckinnon · 2024-05-16T20:26:09Z

/ci
+workflow:os_distros al2023

github-actions · 2024-05-16T20:26:41Z

@cartermckinnon roger that! I've dispatched a workflow. 👍

github-actions · 2024-05-16T21:24:10Z

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.23 / al2023	success ✅	failure ❌
1.24 / al2023	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅

cartermckinnon · 2024-05-16T21:28:01Z

One flake:

[Fail] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] [It] updates the published spec when one version gets renamed [Conformance]

Not related

cartermckinnon · 2024-05-22T21:14:00Z

/ci
+workflow:os_distros al2023

github-actions · 2024-05-22T21:14:36Z

@cartermckinnon roger that! I've dispatched a workflow. 👍

github-actions · 2024-05-22T22:13:10Z

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.23 / al2023	success ✅	success ✅
1.24 / al2023	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅

ndbaker1 · 2024-05-23T00:28:57Z

nodeadm/internal/configprovider/userdata.go

+				if err != nil {
+					return nil, err
+				}
+				nodeConfigPart, err = decompressIfGZIP(nodeConfigPart)
+				if err != nil {
+					return nil, err


nit: theres a custom error message around the calls above but not here :(

the more i think on this, should we be more tolerant and break on these errors so that we can try the remaining parts? maybe not if theres just a simple mistake and the user would prefer it doesn't continue if their config is not included

ndbaker1 · 2024-05-23T00:34:20Z

nodeadm/internal/configprovider/userdata.go

+	decodedLen, err := e.Decode(decodedData, data)
+	if err != nil {
+		return data, nil


so none of the possible errors here end up mattering?

oh i guess it can only return CorruptInputError 🤔

ndbaker1

lgtm as-is, one nit comment

ndbaker1 · 2024-05-23T00:54:42Z

nodeadm/internal/configprovider/userdata.go

+	decodedLen, err := e.Decode(decodedData, data)
+	if err != nil {
+		return data, nil


oh i guess it can only return CorruptInputError 🤔

cartermckinnon force-pushed the gzip-userdata branch 3 times, most recently from 176e80f to 706489f Compare April 11, 2024 19:57

cartermckinnon mentioned this pull request Apr 11, 2024

nodeadm doesn't support userdata content-encoded with gzip #1734

Closed

ndbaker1 reviewed Apr 30, 2024

View reviewed changes

cartermckinnon force-pushed the gzip-userdata branch 2 times, most recently from 3fceead to d741996 Compare May 16, 2024 00:32

cartermckinnon commented May 16, 2024

View reviewed changes

mmerkes reviewed May 16, 2024

View reviewed changes

ndbaker1 reviewed May 16, 2024

View reviewed changes

awslabs deleted a comment from github-actions bot May 16, 2024

cartermckinnon force-pushed the gzip-userdata branch 2 times, most recently from 4936741 to 2900e9d Compare May 22, 2024 21:09

Decompress GZIP'd user data

8071860

cartermckinnon force-pushed the gzip-userdata branch from 2900e9d to 8071860 Compare May 22, 2024 21:10

ndbaker1 reviewed May 23, 2024

View reviewed changes

ndbaker1 approved these changes May 23, 2024

View reviewed changes

cartermckinnon merged commit d87c6c4 into main May 23, 2024
10 checks passed

cartermckinnon deleted the gzip-userdata branch May 23, 2024 19:21

atmosx pushed a commit to gathertown/amazon-eks-ami that referenced this pull request Jun 18, 2024

Decompress GZIP'd user data (awslabs#1762)

1653dca

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decompress GZIP'd user data #1762

Decompress GZIP'd user data #1762

cartermckinnon commented Apr 11, 2024 •

edited

Loading

ndbaker1 left a comment

cartermckinnon commented May 16, 2024

cartermckinnon May 16, 2024 •

edited

Loading

mmerkes May 16, 2024

ndbaker1 May 16, 2024

mmerkes May 16, 2024

ndbaker1 May 16, 2024

cartermckinnon May 16, 2024

cartermckinnon May 22, 2024

mmerkes May 16, 2024

ndbaker1 May 16, 2024

ndbaker1 May 16, 2024

cartermckinnon commented May 16, 2024

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

cartermckinnon commented May 16, 2024 •

edited

Loading

cartermckinnon commented May 22, 2024

github-actions bot commented May 22, 2024

github-actions bot commented May 22, 2024

ndbaker1 May 23, 2024

ndbaker1 May 23, 2024 •

edited

Loading

ndbaker1 May 23, 2024

ndbaker1 May 23, 2024

ndbaker1 left a comment

ndbaker1 May 23, 2024

Decompress GZIP'd user data #1762

Decompress GZIP'd user data #1762

Conversation

cartermckinnon commented Apr 11, 2024 • edited Loading

ndbaker1 left a comment

Choose a reason for hiding this comment

cartermckinnon commented May 16, 2024

cartermckinnon May 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cartermckinnon commented May 16, 2024

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

cartermckinnon commented May 16, 2024 • edited Loading

cartermckinnon commented May 22, 2024

github-actions bot commented May 22, 2024

github-actions bot commented May 22, 2024

Choose a reason for hiding this comment

ndbaker1 May 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndbaker1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cartermckinnon commented Apr 11, 2024 •

edited

Loading

cartermckinnon May 16, 2024 •

edited

Loading

cartermckinnon commented May 16, 2024 •

edited

Loading

ndbaker1 May 23, 2024 •

edited

Loading