[ci][terraform][aws] tags with metadata #3126

v1v · 2022-04-19T17:12:32Z

What does this PR do?

Standardise the below tags/labels in the terraform resources for AWS/GCP:

environment => static value
repo. => static value
branch => dynamic value
build => dynamic value
created_date => dynamic value

Why is it important?

Help with tearing down any of the ephemeral resources which were not successfully removed as part of the system tests.

Implementation details

Those tag/labels are lowercase based to be GCP/AWS compliance
environment=CI will allow to filter all those cloud resources which were created in the CI
While created_date will help to filter those which were older than X days or X hours, since it's Unix epoch time based.
branch and build will help to identify what build in the CI and what PR, Branch or Tag created those resources.

Further details

BUILD_ID
The current build ID, identical to BUILD_NUMBER for builds created in 1.597+, but a YYYY-MM-DD_hh-mm-ss timestamp for older builds.

BRANCH_NAME
For a multibranch project, this will be set to the name of the branch being built, for example in case you wish to deploy to production from master but not from feature branches; if corresponding to some kind of change request, the name is generally arbitrary (refer to CHANGE_ID and CHANGE_TARGET).

Test

Issues

Requires elastic/elastic-package#792

elasticmachine · 2022-04-19T17:59:46Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-04-25T15:18:36.507+0000
Duration: 94 min 10 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4604
Skipped	7
Total	4611

Steps errors

Expand to view the steps failures

`Sleep`

Took 0 min 8 sec . View more details here
Description: 5

`Boot up the Elastic stack`

Took 4 min 42 sec . View more details here
Description: ../../build/elastic-package stack up -d -v --version 7.14.0

`Boot up the Elastic stack`

Took 3 min 29 sec . View more details here
Description: ../../build/elastic-package stack up -d -v --version 8.1.0

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

.ci/Jenkinsfile

mtojek

I'm setting a blocker on this to merge the elastic/elastic-package#792 first, then iterate on this one.

mtojek

As this PR changed the implementation of elastic-package, let's first release a new version. It's just about Git tagging the latest commit in the main branch.

v1v · 2022-04-25T08:53:59Z

I just merged the PR for the elastic-package and created the tag https://github.com/elastic/elastic-package/releases/tag/v0.47.0

mtojek · 2022-04-25T09:31:16Z

Version bump PR here: #3182

v1v · 2022-04-25T10:25:26Z

/test

v1v · 2022-04-25T11:43:23Z

.ci/Jenkinsfile

@@ -258,7 +259,13 @@ def withCloudTestEnv(Closure body) {
    [var: "AWS_SECRET_ACCESS_KEY", password: aws.secret_key],
  ])
  withEnvMask(vars: maskedVars) {
-    body()
+    withEnv(["TF_VAR_BRANCH=${env.BRANCH_NAME.toLowerCase()}",


For some reason these changes are not consumed:

terraform_1 | + tags = { terraform_1 | + "Name" = "elastic-package-test-33598" terraform_1 | } terraform_1 | + tags_all = { terraform_1 | + "Name" = "elastic-package-test-33598" terraform_1 | + "branch" = "unknown-branch" terraform_1 | + "build" = "unknown-build" terraform_1 | + "created_date" = "unknown-date" terraform_1 | + "environment" = "unknown-environment" terraform_1 | + "repo" = "unknown-repo-name" terraform_1 | }

Somehow some env variables are not populated from the CI:

terraform_1 | + tags = { terraform_1 | + "Name" = "elastic-package-test-17719" terraform_1 | } terraform_1 | + tags_all = { terraform_1 | + "Name" = "elastic-package-test-17719" terraform_1 | + "branch" = "unknown" terraform_1 | + "build" = "6" terraform_1 | + "created_date" = "unknown" terraform_1 | + "environment" = "unknown" terraform_1 | + "repo" = "integrations"

I guess, the elastic-packages has a higher precendence:

https://github.com/elastic/elastic-package/pull/792/files#diff-1d4bd2c1ad56472b4dc4c97566e6ce9dc00bda82eb59cf14d1ab8d5c930a1fe0R9-R14

I would start with investigating why the releasing job for v0.47.0 failed: https://beats-ci.elastic.co/job/Ingest-manager/job/elastic-package/job/v0.47.0/

@v1v It looks like we need a fix:

terraform_1 | Plan: 1 to add, 0 to change, 0 to destroy. terraform_1 | google_compute_instance.default: Creating... terraform_1 | terraform_1 | Error: Error creating instance: googleapi: Error 400: Invalid value for field 'resource.labels': ''. Label value 'v0.47.0' violates format constraints. The value can only contain lowercase letters, numeric characters, underscores and dashes. The value can be at most 63 characters long. International characters are allowed., invalid terraform_1 | terraform_1 | with google_compute_instance.default, terraform_1 | on compute.tf line 7, in resource "google_compute_instance" "default": terraform_1 | 7: resource "google_compute_instance" "default" { terraform_1 | terraform_1 | + echo 'Terraform definitions applied.' terraform_1 | Terraform definitions applied. terraform_1 | + set +x

Fixed already added to the elastic-package and also here, so the new metadata is populated now

terraform_1 | + tags = { terraform_1 | + "Name" = "elastic-package-test-73355" terraform_1 | } terraform_1 | + tags_all = { terraform_1 | + "Name" = "elastic-package-test-73355" terraform_1 | + "branch" = "pr-3126" terraform_1 | + "build" = "8" terraform_1 | + "created_date" = "1650891241341" terraform_1 | + "environment" = "ci" terraform_1 | + "repo" = "integrations" terraform_1 | }

mtojek · 2022-04-25T11:43:40Z

@v1v Please merge the main branch, so that this PR will pick the last bumped version. Part of the implementation is in elastic-package and that PR was merged seconds ago.

v1v · 2022-04-25T11:45:36Z

@Mergifyio rebase

mergify · 2022-04-25T11:45:56Z

rebase

✅ Branch has been successfully rebased

…ations into feature/labels-cloud-ephemeral-components * 'feature/labels-cloud-ephemeral-components' of github.com:v1v/integrations: (23 commits) use branch standardise labels/tags Update .ci/Jenkinsfile [ci][terraform][aws] tags with metadata Bump github.com/elastic/elastic-package from 0.46.0 to 0.47.0 (elastic#3182) Spring boot package [Memory - data stream] (elastic#2979) [cisco_ise] Add Cisco ISE package (elastic#2855) [apache_spark][executor] Add Apache Spark package with Executor data stream (elastic#2943) [apache_spark][driver] Add Apache Spark package with Driver data stream (elastic#2945) cisco_duo: simplify grok expression for handling ports (elastic#3170) cisco_duo: fix handling of IP addresses with port numbers (elastic#3117) [sophos] Various improvements and log samples from 18.5 (elastic#3127) [cisco_asa] fix visualizations (elastic#3146) [apache_spark][application] Add Apache Spark package with Application data stream (elastic#2941) Fix pagination bug that skipped events when more than one page is present. (elastic#3140) Cis k8s name migration (elastic#3113) remove exported fields; these will be added to Kibana docs instead (elastic#3093) Exclude ecosystem as owner of subdirectories of packages (elastic#3132) [awsfargate] bump package version (elastic#3130) add first csp rule template (elastic#3081) ...

.ci/Jenkinsfile

Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>

mtojek

Just double-checking, system tests for GCP are not there yet, right?

v1v · 2022-04-25T16:36:48Z

Just double-checking, system tests for GCP are not there yet, right?

I could not find any other references to terraform but only in the aws package

mtojek · 2022-04-26T07:50:20Z

I checked these errors and they're unrelated. I guess that you can merge this PR.

[ci][terraform][aws] tags with metadata

5737f58

v1v added Team:Automation Label for the Observability productivity team automation labels Apr 19, 2022

v1v self-assigned this Apr 19, 2022

v1v commented Apr 20, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

Update .ci/Jenkinsfile

b0540d2

v1v mentioned this pull request Apr 20, 2022

Helper step to get the current build time elastic/apm-pipeline-library#1663

Merged

mtojek suggested changes Apr 20, 2022

View reviewed changes

v1v added 2 commits April 21, 2022 14:28

standardise labels/tags

c83e6ba

use branch

48cae03

v1v marked this pull request as ready for review April 21, 2022 13:35

v1v requested review from a team as code owners April 21, 2022 13:35

v1v requested a review from a team April 21, 2022 13:35

mtojek reviewed Apr 21, 2022

View reviewed changes

kuisathaverat approved these changes Apr 21, 2022

View reviewed changes

v1v commented Apr 25, 2022

View reviewed changes

v1v and others added 4 commits April 25, 2022 11:45

[ci][terraform][aws] tags with metadata

e5a6095

Update .ci/Jenkinsfile

738b239

standardise labels/tags

8201787

use branch

811a011

v1v force-pushed the feature/labels-cloud-ephemeral-components branch from 48cae03 to 811a011 Compare April 25, 2022 11:45

v1v added 2 commits April 25, 2022 13:38

Env variables are the ones defined in elastic/elastic-package#792

d20bc25

ci: replace special chars

f883573

mtojek self-requested a review April 25, 2022 14:55

mtojek reviewed Apr 25, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

Update .ci/Jenkinsfile

6ad53f8

Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>

mtojek approved these changes Apr 25, 2022

View reviewed changes

v1v merged commit 8ad42d5 into elastic:main Apr 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci][terraform][aws] tags with metadata #3126

[ci][terraform][aws] tags with metadata #3126

v1v commented Apr 19, 2022 •

edited

Loading

elasticmachine commented Apr 19, 2022 •

edited

Loading

Build stats

Test stats 🧪

`Sleep`

`Boot up the Elastic stack`

`Boot up the Elastic stack`

mtojek left a comment

mtojek left a comment •

edited

Loading

v1v commented Apr 25, 2022

mtojek commented Apr 25, 2022

v1v commented Apr 25, 2022

v1v Apr 25, 2022

v1v Apr 25, 2022

mtojek Apr 25, 2022

mtojek Apr 25, 2022

v1v Apr 25, 2022

mtojek commented Apr 25, 2022 •

edited

Loading

v1v commented Apr 25, 2022

mergify bot commented Apr 25, 2022

mtojek left a comment

v1v commented Apr 25, 2022

mtojek commented Apr 26, 2022

[ci][terraform][aws] tags with metadata #3126

[ci][terraform][aws] tags with metadata #3126

Conversation

v1v commented Apr 19, 2022 • edited Loading

What does this PR do?

Why is it important?

Implementation details

Further details

Test

Issues

elasticmachine commented Apr 19, 2022 • edited Loading

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

Sleep

Boot up the Elastic stack

Boot up the Elastic stack

🤖 GitHub comments

mtojek left a comment

Choose a reason for hiding this comment

mtojek left a comment • edited Loading

Choose a reason for hiding this comment

v1v commented Apr 25, 2022

mtojek commented Apr 25, 2022

v1v commented Apr 25, 2022

v1v Apr 25, 2022

Choose a reason for hiding this comment

v1v Apr 25, 2022

Choose a reason for hiding this comment

mtojek Apr 25, 2022

Choose a reason for hiding this comment

mtojek Apr 25, 2022

Choose a reason for hiding this comment

v1v Apr 25, 2022

Choose a reason for hiding this comment

mtojek commented Apr 25, 2022 • edited Loading

v1v commented Apr 25, 2022

mergify bot commented Apr 25, 2022

✅ Branch has been successfully rebased

mtojek left a comment

Choose a reason for hiding this comment

v1v commented Apr 25, 2022

mtojek commented Apr 26, 2022

v1v commented Apr 19, 2022 •

edited

Loading

elasticmachine commented Apr 19, 2022 •

edited

Loading

`Sleep`

`Boot up the Elastic stack`

`Boot up the Elastic stack`

mtojek left a comment •

edited

Loading

mtojek commented Apr 25, 2022 •

edited

Loading