-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This commit modifies how the buffer queue manages chunks in order to reduce memory consumption. Previously, we were appending uncompressed data to a single buffer. On flush, we would copy the data out and process it in a worker. The worker would be responsible for compressing the data and posting the data over HTTP. This commit applies two improvements: - compress data on write to queue. This drastically reduces memory consumption. - pass buffer to channel on flush, rather than copying to intermediate buffer. There is no justification for copying data from one buffer to another. In a simple test with an 8MB GZIP compressed AWS Config change notification, our memory consumption went down from ~47MB to ~6MB.
- Loading branch information
Showing
9 changed files
with
144 additions
and
42 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
# Forwarder HTTP test utility | ||
|
||
This utility exercises the Forwarder lambda code for the case where the backend is an HTTP server. | ||
|
||
At a bare minimum, you must provide a list of input files via the command line args: | ||
|
||
``` | ||
go run main.go ./my-test.json ./another-file.csv | ||
``` | ||
|
||
The output of the program will be a directory containing all request bodies | ||
sent to the HTTP server. Each output filename will be named according to the | ||
hash of contents. This is useful for verifying code changes do not affect how | ||
data is chunked. | ||
|
||
When processing files, the code will apply the same presets that the Forwarder | ||
lambda does. You may have to use `content-type` or `content-encoding` flags to | ||
fake either object attribute, since neither is a property of the local | ||
filesystem. For example, to test an AWS Config file, you would have to set `content-encoding`: | ||
|
||
``` | ||
go run main.go \ | ||
-content-encoding=gzip \ | ||
./123456789012_Config_us-west-2_ConfigHistory_AWS::SSM::ManagedInstanceInventory_20240607T130841Z_20240607T190342Z_1.json.gz | ||
``` | ||
|
||
In the lambda case, `content-encoding` is already set in S3. In the local | ||
testing case, we must configure it manually. | ||
|
||
## Profiling | ||
|
||
To dump a profile of the executed code, set `-profile`: | ||
|
||
``` | ||
go run main.go \ | ||
-profile=mem \ | ||
... | ||
``` | ||
|
||
You can then explore the file through `go tool`, e.g: | ||
``` | ||
go tool pprof -http=:8080 forwarder-post/mem.pprof | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters