-
Notifications
You must be signed in to change notification settings - Fork 334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[lambda][rule] some s3 optimizations #126
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The biggest benefit will actually be improving memory efficiency. Looks good, but you still have to sadly truncate the files.
yield line.rstrip() | ||
|
||
# remove the file | ||
os.remove(downloaded_s3_object) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't work as expected in Lambda - you have to truncate the file first to avoid using all of the disk space
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm that's strange and doesn't make much sense, but okay. I will add the truncate back.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, Jack and I both independently observed the phenomenon. os.remove
is really os.unlink
, and presumably Lambda's version of unlinking
is not the same as a traditional OS
Args: | ||
downloaded_s3_object (string): A full path to the downloaded file. | ||
|
||
Returns: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yields:
(not returns)
140b654
to
d3611db
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
…ack a truncate call due to aws bug
d3611db
to
ce14db3
Compare
to @airbnb/streamalert-maintainers
size: small
changes