Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: allow the instances to send metrics #3067

Merged
merged 1 commit into from
Mar 16, 2023
Merged

Conversation

rgoomar
Copy link
Contributor

@rgoomar rgoomar commented Mar 15, 2023

This is needed in order to be able to send metrics around the disk and memory. Those were getting a 403 until this update was made.

Example error in the Cloudwatch agent logs:

not authorized to perform: cloudwatch:PutMetricData because no permissions boundary allows the cloudwatch:PutMetricData action
        status code: 403, request id: 

This is needed in order to be able to send metrics around the disk and memory. Those were getting a 403 until this update was made.
@npalm npalm changed the title fix(cloudwatch): allow the instances to send metrics fix: allow the instances to send metrics Mar 16, 2023
@npalm
Copy link
Member

npalm commented Mar 16, 2023

Just wondering, the change is only adding the permissions to the boundary condition. Assume yout attach a policy to the instance profile as well? Can you share the script you use to push memory metrics?

@rgoomar
Copy link
Contributor Author

rgoomar commented Mar 16, 2023

Just wondering, the change is only adding the permissions to the boundary condition. Assume yout attach a policy to the instance profile as well? Can you share the script you use to push memory metrics?

@npalm

This is actually just fixing the behavior that should happen. The runners add a policy that has cloudwatch:PutMetricsData which is helpful if you set enable_cloudwatch_agent flag. So, it will have the correct IAM policy in place along with the CW agent on the runner although if you look at the cloudwatch agent logs just from those default values, it fails to actually send over metrics due to a 403 error from the permissions boundaries.

All I did was go into the console and manually update the role with this permission boundary change and metrics like memory usage on the host starting sending properly.

@npalm npalm merged commit 55c40ff into philips-labs:main Mar 16, 2023
@rgoomar rgoomar deleted the patch-1 branch March 16, 2023 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants