Collect cloud metadata #826
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
7 tasks
sethmlarson
reviewed
Jun 16, 2020
|
Leaving a thought here: A lot of the JSON response value collecting is fragile to any change in the metadata API response structure. If any field on the JSON response were to change then all fields wouldn't be detected for that provider. Obviously unlikely to happen but something I thought of. |
|
@sethmlarson Yes, that's definitely a concern. Unfortunately we don't have an easy way to do cross-cloud integration tests to test against the actual endpoints, though we may add that in the future. Luckily, these APIs tend to be very stable because they're heavily relied on. |
sethmlarson
reviewed
Jun 16, 2020
|
@beniwohli This has now been tested against apm-server 7.8.0 and the latest nightly. Working in both places (and I do see the metadata in the nightly). Ready for final review! |
beniwohli
approved these changes
Jun 26, 2020
5 tasks
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Sep 30, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Oct 1, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to axw/apm-agent-go
that referenced
this pull request
Oct 1, 2020
Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
axw
added a commit
to elastic/apm-agent-go
that referenced
this pull request
Oct 2, 2020
* Fetch cloud metadata Based on the Python reference implementation: elastic/apm-agent-python#826 By default we attempt to sniff metadata for all cloud providers. This can be overridden by setting the environment variable ELASTIC_APM_CLOUD_PROVIDER to one of "none", "aws", "azure", or "gcp". We set a short (100ms) socket connect timeout, and a slightly longer overall timeout (1s) for fetching the cloud metadata. In tests we disable cloud metadata fetching by default. There are currently no functional tests, as that would rely on us running in a known cloud environment.
romulorosa
pushed a commit
to romulorosa/apm-agent-python
that referenced
this pull request
Oct 15, 2020
* Add cloud metadata skeleton * Add config value * Add aws metadata collection * Use resp instead of r * Add GCP metadata * Add Azure metadata * Refactor headers * Use older metadata service version to guarantee availability * Add configuration docs * Use a single GCP metadata call with recursive=true * Use urllib3 properly * Fix some copy pasta * Normalize variable names * Pop zone if empty in azure * Remove provider guessing * Token step doesn't appear necessary for AWS * Just kidding, token needed for IMDSv2 * Actually import cloud * Add a socket call to avoid urllib3 logs * Add tests * Add changelog * Don't bother with tcp handshake for google metadata * Add logs for missing metadata when provider is defined * Move metadata generation to post-thread-start * Update tests to allow metadata to be generated in transport * For GCP IDs to string * Consolidate ret * We don't need warning counts, just check for the warning we care about * Fix header encodings test timing * Turn off cloud metadata for tests * Typo * We do need a short sleep here * Remove unused import
romulorosa
pushed a commit
to romulorosa/apm-agent-python
that referenced
this pull request
Oct 15, 2020
* Add cloud metadata skeleton * Add config value * Add aws metadata collection * Use resp instead of r * Add GCP metadata * Add Azure metadata * Refactor headers * Use older metadata service version to guarantee availability * Add configuration docs * Use a single GCP metadata call with recursive=true * Use urllib3 properly * Fix some copy pasta * Normalize variable names * Pop zone if empty in azure * Remove provider guessing * Token step doesn't appear necessary for AWS * Just kidding, token needed for IMDSv2 * Actually import cloud * Add a socket call to avoid urllib3 logs * Add tests * Add changelog * Don't bother with tcp handshake for google metadata * Add logs for missing metadata when provider is defined * Move metadata generation to post-thread-start * Update tests to allow metadata to be generated in transport * For GCP IDs to string * Consolidate ret * We don't need warning counts, just check for the warning we care about * Fix header encodings test timing * Turn off cloud metadata for tests * Typo * We do need a short sleep here * Remove unused import
beniwohli
pushed a commit
to beniwohli/apm-agent-python
that referenced
this pull request
Sep 14, 2021
* Add cloud metadata skeleton * Add config value * Add aws metadata collection * Use resp instead of r * Add GCP metadata * Add Azure metadata * Refactor headers * Use older metadata service version to guarantee availability * Add configuration docs * Use a single GCP metadata call with recursive=true * Use urllib3 properly * Fix some copy pasta * Normalize variable names * Pop zone if empty in azure * Remove provider guessing * Token step doesn't appear necessary for AWS * Just kidding, token needed for IMDSv2 * Actually import cloud * Add a socket call to avoid urllib3 logs * Add tests * Add changelog * Don't bother with tcp handshake for google metadata * Add logs for missing metadata when provider is defined * Move metadata generation to post-thread-start * Update tests to allow metadata to be generated in transport * For GCP IDs to string * Consolidate ret * We don't need warning counts, just check for the warning we care about * Fix header encodings test timing * Turn off cloud metadata for tests * Typo * We do need a short sleep here * Remove unused import
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this pull request do?
Collects cloud metadata
Remaining work
Add provider guessingRelated issues
Closes #822