Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions #19423

Merged
merged 3 commits into from
Jun 30, 2020

Conversation

kwinstonix
Copy link
Contributor

@kwinstonix kwinstonix commented Jun 26, 2020

What does this PR do?

This PR is to add a parameter aws_partition for metricbeat aws cloudwatch module, so metricbeat can support aws china and aws us-gov account.

Why is it important?

Because there are three partitions aws, aws-cn, aws-us-gov of aws regions, now metricbeat can only support aws global region, here is a hardcode.

// Get IAM account name
awsConfig.Region = "us-east-1"
svcIam := iam.New(awscommon.EnrichAWSConfigWithEndpoint(
config.AWSConfig.Endpoint, "iam", "", awsConfig))
req := svcIam.ListAccountAliasesRequest(&iam.ListAccountAliasesInput{})

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. add aws_partition param in aws module config
metricbeat.modules:
  - module: aws
    aws_partition: aws-cn
    period: 300s
    access_key_id: 
    secret_access_key:
  1. run metricbeat , open debug log, then you can see it can retrive metrics from correct regions

Related issues

Use cases

  • no side affect
  • user can use metricbeat to get metrics from aws china and aws us-gov account except aws global account

Screenshots

aws global account, set aws_partition: aws in aws module config or remove the param

image

aws china account, set aws_partition: aws-cn in aws module config

image

@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 26, 2020
@kwinstonix kwinstonix changed the title add params aws_partition to support other aws region: aws-cn, aws-u… [MetricBeat] add params aws_partition to support other aws region: aws-cn, aws-u… Jun 26, 2020
@kwinstonix kwinstonix changed the title [MetricBeat] add params aws_partition to support other aws region: aws-cn, aws-u… [MetricBeat] add params aws_partition to support: aws-cn, aws-us-gov region Jun 26, 2020
@kwinstonix kwinstonix changed the title [MetricBeat] add params aws_partition to support: aws-cn, aws-us-gov region [MetricBeat] add param aws_partition to support: aws-cn, aws-us-gov region Jun 26, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 26, 2020

❕ Build Aborted

The PR is not allowed to run in the CI yet

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19423 updated]

  • Reason: The PR is not allowed to run in the CI yet

  • Start Time: 2020-06-26T11:17:48.971+0000

  • Duration: 3 min 50 sec

  • Commit: 5b6dc38

Steps errors

Expand to view the steps failures

  • Name: Error signal
    • Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permission

    • Duration: 0 min 0 sec

    • Start Time: 2020-06-26T11:20:35.163+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-26T11:20:36.739Z] Stage "Elastic Agent x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.740Z] Stage "Elastic Agent x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.741Z] Stage "Elastic Agent Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.742Z] Stage "Filebeat oss" skipped due to earlier failure(s)
[2020-06-26T11:20:36.742Z] Stage "Filebeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.743Z] Stage "Filebeat Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.744Z] Stage "Filebeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.744Z] Stage "Filebeat Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.745Z] Stage "Filebeat x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.746Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.746Z] Stage "Auditbeat oss Linux" skipped due to earlier failure(s)
[2020-06-26T11:20:36.747Z] Stage "Auditbeat crosscompile" skipped due to earlier failure(s)
[2020-06-26T11:20:36.748Z] Stage "Auditbeat oss Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.748Z] Stage "Auditbeat oss Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.749Z] Stage "Auditbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.750Z] Stage "Auditbeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.751Z] Stage "Auditbeat x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.751Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.752Z] Stage "Libbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.753Z] Stage "Metricbeat OSS Unit tests" skipped due to earlier failure(s)
[2020-06-26T11:20:36.753Z] Stage "Metricbeat OSS Integration tests" skipped due to earlier failure(s)
[2020-06-26T11:20:36.754Z] Stage "Metricbeat Python integration tests" skipped due to earlier failure(s)
[2020-06-26T11:20:36.755Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.755Z] Stage "Metricbeat crosscompile" skipped due to earlier failure(s)
[2020-06-26T11:20:36.756Z] Stage "Metricbeat Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.757Z] Stage "Metricbeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.758Z] Stage "Metricbeat Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.758Z] Stage "Metricbeat x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.759Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.759Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.760Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.760Z] Stage "Winlogbeat Windows x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.761Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.762Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.762Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:36.763Z] Stage "Kubernetes" skipped due to earlier failure(s)
[2020-06-26T11:20:36.835Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.836Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.837Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.838Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.839Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.839Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.840Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.841Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.842Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:37.322Z] Failed in branch Elastic Agent x-pack
[2020-06-26T11:20:37.322Z] Failed in branch Elastic Agent x-pack Windows
[2020-06-26T11:20:37.323Z] Failed in branch Elastic Agent Mac OS X
[2020-06-26T11:20:37.324Z] Failed in branch Filebeat oss
[2020-06-26T11:20:37.324Z] Failed in branch Filebeat x-pack
[2020-06-26T11:20:37.325Z] Failed in branch Filebeat Mac OS X
[2020-06-26T11:20:37.325Z] Failed in branch Filebeat x-pack Mac OS X
[2020-06-26T11:20:37.326Z] Failed in branch Filebeat Windows
[2020-06-26T11:20:37.327Z] Failed in branch Filebeat x-pack Windows
[2020-06-26T11:20:37.327Z] Failed in branch Auditbeat oss Linux
[2020-06-26T11:20:37.328Z] Failed in branch Auditbeat crosscompile
[2020-06-26T11:20:37.328Z] Failed in branch Auditbeat oss Mac OS X
[2020-06-26T11:20:37.329Z] Failed in branch Auditbeat oss Windows
[2020-06-26T11:20:37.329Z] Failed in branch Auditbeat x-pack
[2020-06-26T11:20:37.330Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-06-26T11:20:37.330Z] Failed in branch Auditbeat x-pack Windows
[2020-06-26T11:20:37.331Z] Failed in branch Libbeat x-pack
[2020-06-26T11:20:37.331Z] Failed in branch Metricbeat OSS Unit tests
[2020-06-26T11:20:37.332Z] Failed in branch Metricbeat OSS Integration tests
[2020-06-26T11:20:37.332Z] Failed in branch Metricbeat Python integration tests
[2020-06-26T11:20:37.333Z] Failed in branch Metricbeat crosscompile
[2020-06-26T11:20:37.334Z] Failed in branch Metricbeat Mac OS X
[2020-06-26T11:20:37.334Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-06-26T11:20:37.335Z] Failed in branch Metricbeat Windows
[2020-06-26T11:20:37.335Z] Failed in branch Metricbeat x-pack Windows
[2020-06-26T11:20:37.336Z] Failed in branch Winlogbeat Windows x-pack
[2020-06-26T11:20:37.336Z] Failed in branch Kubernetes
[2020-06-26T11:20:37.572Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.573Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.574Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:37.575Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.576Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.577Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:37.630Z] Failed in branch Packetbeat
[2020-06-26T11:20:37.631Z] Failed in branch dockerlogbeat
[2020-06-26T11:20:37.632Z] Failed in branch Journalbeat
[2020-06-26T11:20:37.816Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.817Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.818Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.819Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:37.855Z] Failed in branch Metricbeat x-pack
[2020-06-26T11:20:37.856Z] Failed in branch Winlogbeat
[2020-06-26T11:20:38.017Z] Failed in branch Heartbeat
[2020-06-26T11:20:38.018Z] Failed in branch Libbeat
[2020-06-26T11:20:38.018Z] Failed in branch Functionbeat
[2020-06-26T11:20:38.019Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:38.091Z] Failed in branch Generators
[2020-06-26T11:20:38.386Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19423
[2020-06-26T11:20:38.484Z] [INFO] getVaultSecret: Getting secrets
[2020-06-26T11:20:38.542Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-26T11:20:39.275Z] + chmod 755 generate-build-data.sh
[2020-06-26T11:20:39.275Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4 ABORTED 170045
[2020-06-26T11:20:39.275Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4/steps/?limit=10000 -o steps-info.json
[2020-06-26T11:20:39.826Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4/tests/?status=FAILED -o tests-errors.json
[2020-06-26T11:20:39.826Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4/log/ -o pipeline-log.txt

@kwinstonix kwinstonix changed the title [MetricBeat] add param aws_partition to support: aws-cn, aws-us-gov region [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions Jun 26, 2020
@kwinstonix kwinstonix marked this pull request as ready for review June 26, 2020 07:55
@kwinstonix kwinstonix force-pushed the add_aws_partition_param branch from 55b3d1b to 5b6dc38 Compare June 26, 2020 11:17
@kaiyan-sheng kaiyan-sheng self-assigned this Jun 26, 2020
@kaiyan-sheng kaiyan-sheng added the Team:Platforms Label for the Integrations - Platforms team label Jun 26, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 26, 2020
@kaiyan-sheng
Copy link
Contributor

jenkins, test this please

@kaiyan-sheng
Copy link
Contributor

Thanks for working on both changes!! Sorry for the delay, I will review them today!!

Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally and it works. Thank you for the contribution! I will merge this PR and get it backported.

@kaiyan-sheng
Copy link
Contributor

run tests please

@kaiyan-sheng kaiyan-sheng merged commit 1ea8c84 into elastic:master Jun 30, 2020
kaiyan-sheng added a commit that referenced this pull request Jun 30, 2020
kaiyan-sheng added a commit that referenced this pull request Jul 1, 2020
kaiyan-sheng added a commit that referenced this pull request Jul 1, 2020
…support aws-cn, aws-us-gov regions (#19519)

* [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions (#19423)

* add params `aws_partition` to support other aws region: aws-cn, aws-us-gov

(cherry picked from commit 1ea8c84)
Co-authored-by: martin <kwinstonix@users.noreply.github.com>
v1v added a commit to v1v/beats that referenced this pull request Jul 2, 2020
…ne-beats

* upstream/master: (105 commits)
  ci: enable packaging job (elastic#19536)
  ci: disable upstream trigger on PRs for the packaging job (elastic#19490)
  Implement memlog on-disk handling (elastic#19408)
  fix go.mod for PR elastic#19423 (elastic#19521)
  [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions (elastic#19423)
  Input v2 stateless manager (elastic#19406)
  Input v2 compatibility layer (elastic#19401)
  [Elastic Agent] Fix artifact downloading to allow endpoint-security to be downloaded (elastic#19503)
  fix: ignore target changes on scans (elastic#19510)
  Add more helpers to pipeline/testing package (elastic#19405)
  Report dependencies in CSV format (elastic#19506)
  [Filebeat] Fix reference leak in TCP and Unix socket inputs (elastic#19459)
  Cursor input skeleton (elastic#19378)
  Add changelog. (elastic#19495)
  [DOC] Typo in Kerberos (elastic#19265)
  Remove accidentally commited unused NOTICE template (elastic#19485)
  [Elastic Agent] Support the install, control, and uninstall of Endpoint (elastic#19248)
  [Filebeat][httpjson] Add split_events_by config setting (elastic#19246)
  ci: disabling packaging job until we fix it (elastic#19481)
  Fix golang.org/x/tools to release1.13 (elastic#19478)
  ...
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…regions (elastic#19423)

* add params `aws_partition` to support other aws region: aws-cn, aws-us-gov
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Platforms Label for the Integrations - Platforms team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Metricbeat] Unable to get account id in China regions
4 participants