feat: Security improvements, add option to disable userdata logging

* chore(release): 0.17.0 [skip ci] * Adding support for new workflow_job event. ([#1019](philips-labs#1019)) ([a74e10b](philips-labs@a74e10b)) * chore(release): 0.18.0 [skip ci] * add format checking for lambdas in CI ([#899](philips-labs#899)) ([#1080](philips-labs#1080)) ([ae9c277](philips-labs@ae9c277)) * add option to overwrite / disable egress [#748](philips-labs#748) ([#1112](philips-labs#1112)) ([9c2548d](philips-labs@9c2548d)) * replace depcrated 'request' dependency by 'node-fetch' ([#903](philips-labs#903)) ([#1082](philips-labs#1082)) ([fb51756](philips-labs@fb51756)) * chore(release): 0.18.1 [skip ci] * webhook labels for `workflow_job` ([#1133](philips-labs#1133)) ([4b39fb9](philips-labs@4b39fb9)) * chore(release): 0.19.0 [skip ci] * **scale-down:** Update Owner Logic ([#1065](philips-labs#1065)) ([ba2536b](philips-labs@ba2536b)), closes [#2](philips-labs#2) * explicit set region for downloading runner distribution from S3 ([#1204](philips-labs#1204)) ([439fb1b](philips-labs@439fb1b)) * upgrade jest ([#1219](philips-labs#1219)) ([c8b8139](philips-labs@c8b8139)) * use dynamic block to ignore null market opts ([#1202](philips-labs#1202)) ([df9bd78](philips-labs@df9bd78)) * use dynamic block to ignore null market opts ([#1202](philips-labs#1202)) ([06a5598](philips-labs@06a5598)) * **logging:** Additional Logging ([#1135](philips-labs#1135)) ([f7f194d](philips-labs@f7f194d)) * **scale-down:** Clearing cache between runs ([#1164](philips-labs#1164)) ([e72227b](philips-labs@e72227b)) * chore(release): 0.19.1 [skip ci] * `instance_types` from a Set to a List, so instance order preference is preserved ([#1154](philips-labs#1154)) ([150d227](philips-labs@150d227)) * chore(release): 0.20.0 [skip ci] * Add option to disable SSL verification support for GitHub Enterprise Server ([#1216](philips-labs#1216)) ([3c3ef19](philips-labs@3c3ef19)), closes [#1207](philips-labs#1207) * chore(release): 0.20.1 [skip ci] * Upgrade lambda runtime to node 14.x ([#1203](philips-labs#1203)) ([570949a](philips-labs@570949a)) * **webhook:** remove node fetch ([ca14ac5](philips-labs@ca14ac5)) * **webhook:** replace node-fetch by axios [#1247](philips-labs#1247) ([80fff4b](philips-labs@80fff4b)) * added more detailed logging for scaling up and down ([#1222](philips-labs#1222)) ([9aa7456](philips-labs@9aa7456)) * chore(release): 0.21.0 [skip ci] * Ignore github managed labels and add check disable option ([#1244](philips-labs#1244)) ([859fa38](philips-labs@859fa38)) * remove unused app client since SSH key is used to secure app authorization ([#1223](philips-labs#1223)) ([4cb5cf1](philips-labs@4cb5cf1)) * upgrade Terraform version of module 1.0.x ([philips-labs#1254](philips-labs#1254)) ([2a817dc](philips-labs@2a817dc)) * chore(release): 0.21.1 [skip ci] * **logging:** Adjusting scale logging messages and levels ([philips-labs#1286](philips-labs#1286)) ([665e1a6](philips-labs@665e1a6)) * **logging:** Adjusting webhook logs and levels ([philips-labs#1287](philips-labs#1287)) ([9df5fb8](philips-labs@9df5fb8)) * Update launch template to use metadata service v2 ([philips-labs#1278](philips-labs#1278)) ([ef16287](philips-labs@ef16287)) * chore(release): 0.22.0 [skip ci] * adding message retention seconds ([philips-labs#1354](philips-labs#1354)) ([a19929f](philips-labs@a19929f)) * adding var for tags for ec2s ([philips-labs#1357](philips-labs#1357)) ([31cf02d](philips-labs@31cf02d)) * add validation to distribution_bucket_name variable ([philips-labs#1356](philips-labs#1356)) ([6522317](philips-labs@6522317)) * chore(release): 0.23.0 [skip ci] * add option to format logging in JSON for lambdas ([#1228](philips-labs#1228)) ([a250b96](philips-labs@a250b96)) * add option to specify SSE config for dist bucket ([philips-labs#1324](philips-labs#1324)) ([ae84302](philips-labs@ae84302)) * reducing verbosity of role and profile ([philips-labs#1358](philips-labs#1358)) ([922ef99](philips-labs@922ef99)) * chore(release): 0.23.1 [skip ci] * configurable metadata options for runners ([philips-labs#1377](philips-labs#1377)) ([f37df23](philips-labs@f37df23)) * chore(release): 0.24.0 [skip ci] * support single line for app private key ([philips-labs#1368](philips-labs#1368)) ([14183ac](philips-labs@14183ac)) * update return codes, no error code for job that are ignored ([philips-labs#1381](philips-labs#1381)) ([f9f705f](philips-labs@f9f705f)) * chore(release): 0.25.0 [skip ci] * Add option to configure concurrent running scale up lambda ([philips-labs#1415](philips-labs#1415)) ([23ee630](philips-labs@23ee630)) * clean up non used variables in examples ([philips-labs#1416](philips-labs#1416)) ([fe65a5f](philips-labs@fe65a5f)) * chore(release): 0.25.1 [skip ci] * Add required providers to module ssm ([philips-labs#1423](philips-labs#1423)) ([5b68b7b](philips-labs@5b68b7b)) * chore(release): 0.25.2 [skip ci] * add logging context to runner lambda ([philips-labs#1399](philips-labs#1399)) ([0ba0930](philips-labs@0ba0930)) * **logging:** Add context to webhook logs ([philips-labs#1401](philips-labs#1401)) ([8094576](philips-labs@8094576)) * chore(release): 0.26.0 [skip ci] * Add hooks for prebuilt images (AMI), including amazon linux packer example ([philips-labs#1444](philips-labs#1444)) ([060daac](philips-labs@060daac)) * add runners binaries bucket as terraform output ([5809fee](philips-labs@5809fee)) * chore(release): 0.26.1 [skip ci] * Download lambda ([philips-labs#1480](philips-labs#1480)) ([f1b99d9](philips-labs@f1b99d9)) * **syncer:** Add tests, coverage report, and refactor lambda / naming ([philips-labs#1478](philips-labs#1478)) ([8266442](philips-labs@8266442)) * install_config_runner -> install_runner ([philips-labs#1479](philips-labs#1479)) ([de5b93f](philips-labs@de5b93f)) * chore(release): 0.27.0 [skip ci] * add windows support ([philips-labs#1476](philips-labs#1476)) ([dbba705](philips-labs@dbba705)) * chore(release): 0.27.1 [skip ci] * add --preserve-env to start-runner.sh to enable RUNNER_ALLOW_RUNASROOT ([philips-labs#1537](philips-labs#1537)) ([1cd9cd3](philips-labs@1cd9cd3)) * remove export from install script. ([philips-labs#1538](philips-labs#1538)) ([d32ca1b](philips-labs@d32ca1b)) * chore(release): 0.27.2 [skip ci] * Dowload lambda see [philips-labs#1541](philips-labs#1541) for details. ([philips-labs#1542](philips-labs#1542)) ([7cb73c8](philips-labs@7cb73c8)) * chore(release): 0.28.0 [skip ci] * add option ephemeral runners ([philips-labs#1374](philips-labs#1374)) ([2f323d6](philips-labs@2f323d6)), closes [philips-labs#1399](philips-labs#1399) [philips-labs#1444](philips-labs#1444) * Change default location of runner to `/opt` and fix Ubuntu example ([philips-labs#1572](philips-labs#1572)) ([77f350b](philips-labs@77f350b)) * Replace run instance API by create fleet API ([philips-labs#1556](philips-labs#1556)) ([27e974d](philips-labs@27e974d)) * Support t4g Graviton instance type ([philips-labs#1561](philips-labs#1561)) ([3fa5896](philips-labs@3fa5896)) * Add config for windows ami ([philips-labs#1525](philips-labs#1525)) ([7907984](philips-labs@7907984)) * chore(release): 0.29.0 [skip ci] * Strict label check and replace disable_check_wokflow_job_labels by opt in enable_workflow_job_labels_check ([philips-labs#1591](philips-labs#1591)) ([405b11d](philips-labs@405b11d)) * chore(release): 0.30.0 [skip ci] * Add scheduled / pull based scaling for org level runners ([philips-labs#1577](philips-labs#1577)) ([8197432](philips-labs@8197432)) * chore(release): 0.30.1 [skip ci] * **runnrs:** Pool runners to allow multiple pool_config objects ([philips-labs#1621](philips-labs#1621)) ([c9c7c69](philips-labs@c9c7c69)) * chore(release): 0.31.0 [skip ci] * **packer:** add vars and minor clean up ([philips-labs#1611](philips-labs#1611)) ([1c897a4](philips-labs@1c897a4)) * **webhook:** depcrated warning on ts-jest mocked ([philips-labs#1615](philips-labs#1615)) ([56c1ece](philips-labs@56c1ece)) * chore(release): 0.32.0 [skip ci] * **runner:** Replace patch by install ICU package for ARM runners ([philips-labs#1624](philips-labs#1624)) ([74cfa51](philips-labs@74cfa51)) * **images:** use new runner install location ([philips-labs#1628](philips-labs#1628)) ([36c1bf5](philips-labs@36c1bf5)) * **packer:** Add missing RUNNER_ARCHITECTURE for amazn-linux2 ([philips-labs#1647](philips-labs#1647)) ([ec497a2](philips-labs@ec497a2)) * chore(release): 0.33.0 [skip ci] * **images:** Added ubuntu-focual example packer configuration ([philips-labs#1644](philips-labs#1644)) ([997b171](philips-labs@997b171)) * **examples:** Update AMI filter ([philips-labs#1673](philips-labs#1673)) ([39c019c](philips-labs@39c019c)) * chore(release): 0.34.0 [skip ci] * Add output image id used in launch template ([philips-labs#1676](philips-labs#1676)) ([a49fab4](philips-labs@a49fab4)) * chore(release): 0.34.1 [skip ci] * **syncer:** Fix for windows binaries in action runner syncer ([philips-labs#1716](philips-labs#1716)) ([63e0e27](philips-labs@63e0e27)) * chore(release): 0.34.2 [skip ci] * Limit AWS Terraform Provider to 3.* ([philips-labs#1741](philips-labs#1741)) ([0cf2b5d](philips-labs@0cf2b5d)) * **runner:** Cannot disable cloudwatch agent ([philips-labs#1738](philips-labs#1738)) ([0f798ca](philips-labs@0f798ca)) * chore(release): 0.35.0 [skip ci] * Parameterise delete_on_termination ([philips-labs#1758](philips-labs#1758)) ([6282351](philips-labs@6282351)), closes [philips-labs#1745](philips-labs#1745) * **runner:** Ability to disable default runner security group creation ([philips-labs#1718](philips-labs#1718)) ([94779f8](philips-labs@94779f8)) * chore(release): 0.36.0 [skip ci] * **runner:** Add option to disable auto update ([philips-labs#1791](philips-labs#1791)) ([c2a834f](philips-labs@c2a834f)) * chore(release): 0.37.0 [skip ci] * Add associate_public_ip_address variable to windows AMI too ([philips-labs#1819](philips-labs#1819)) ([0b8e1fc](philips-labs@0b8e1fc)), closes [/github.com/philips-labs/pull/1816#issuecomment-1060650668](https://github.com/philips-labs//github.com/philips-labs/terraform-aws-github-runner/pull/1816/issues/issuecomment-1060650668) * Add associate_public_ip_address variable ([philips-labs#1816](philips-labs#1816)) ([052e9f8](philips-labs@052e9f8)) * Add option for KMS encryption for cloudwatch log groups ([philips-labs#1833](philips-labs#1833)) ([3f1a67f](philips-labs@3f1a67f)) * Add SQS queue resource policy to improve security ([philips-labs#1798](philips-labs#1798)) ([96def9a](philips-labs@96def9a)) * Add Support for Alternative Partitions in ARNs (like govcloud) ([philips-labs#1815](philips-labs#1815)) ([0ba06c8](philips-labs@0ba06c8)) * Add variable to specify custom commands while building the AMI ([philips-labs#1838](philips-labs#1838)) ([8f9c342](philips-labs@8f9c342)) * Autoupdate should be disabled by default ([philips-labs#1797](philips-labs#1797)) ([828bed6](philips-labs@828bed6)) * Create SQS DLQ policy only if DLQ is created ([philips-labs#1839](philips-labs#1839)) ([c88a005](philips-labs@c88a005)) * Upgrade Amazon base AMI to Amazon Linux 2 kernel 5x ([philips-labs#1812](philips-labs#1812)) ([9aa5532](philips-labs@9aa5532)) * chore(release): 0.38.0 [skip ci] * Add option for ephemeral to check builds status before scaling ([philips-labs#1854](philips-labs#1854)) ([7eb0bda](philips-labs@7eb0bda)) * Retention days was used instead of kms key id for pool ([philips-labs#1855](philips-labs#1855)) ([aa29d93](philips-labs@aa29d93)) * chore(release): 0.39.0 [skip ci] * Add possibility to create multiple ebs ([philips-labs#1845](philips-labs#1845)) ([7a2ca0d](philips-labs@7a2ca0d)) * Don't delete busy runners ([philips-labs#1832](philips-labs#1832)) ([0e9b083](philips-labs@0e9b083)) * chore(release): 0.40.0 [skip ci] * Support multi runner process support for runner scale down. ([philips-labs#1859](philips-labs#1859)) ([3658d6a](philips-labs@3658d6a)) * Set the minimal AWS provider to 3.50 ([philips-labs#1937](philips-labs#1937)) ([16095d8](philips-labs@16095d8)) * chore(release): 0.40.1 [skip ci] * Avoid non semantic commontes can be merged. ([philips-labs#1969](philips-labs#1969)) ([ad1c872](philips-labs@ad1c872)) * chore(release): 0.40.2 [skip ci] * Outputs for pool need to account for complexity ([philips-labs#1970](philips-labs#1970)) ([2d92906](philips-labs@2d92906)) * chore(release): 0.40.3 [skip ci] * Volume size is ingored ([philips-labs#2014](philips-labs#2014)) ([b733248](philips-labs@b733248)), closes [philips-labs#1954](philips-labs#1954) * chore(release): 0.40.4 [skip ci] * Wrong block device mapping ([philips-labs#2019](philips-labs#2019)) ([c42a467](philips-labs@c42a467)) * chore(release): 1.0.0 [skip ci] * var.volume_size replaced by var.block_device_mappings * The module is upgraded to AWS Terraform provider 4.x * Improve syncer s3 kms encryption ([38ed5be](philips-labs@38ed5be)) * Remove var.volume_size in favour of var.block_device_mappings ([4e97048](philips-labs@4e97048)) * Support AWS 4.x Terraform provider ([philips-labs#1739](philips-labs#1739)) ([cfb6da2](philips-labs@cfb6da2)) * Wrong block device mapping ([philips-labs#2019](philips-labs#2019)) ([185ef20](philips-labs@185ef20)) * chore(release): 1.1.0 [skip ci] * Add option to enable detailed monitoring for runner launch template ([philips-labs#2024](philips-labs#2024)) ([e73a267](philips-labs@e73a267)) * chore(release): 1.1.1 [skip ci] * **runner:** Don't treat the string "false" as true. ([philips-labs#2051](philips-labs#2051)) ([b67c7dc](philips-labs@b67c7dc)) * chore(release): 1.2.0 [skip ci] * Replace environment variable by prefix ([philips-labs#1858](philips-labs#1858)) ([e2f9a27](philips-labs@e2f9a27)) * docs: fix hyperlinks in the Terraform Registry documentation (philips-labs#2085) This makes the hyperlink correct in the Terraform Registry documentation * chore(release): 1.3.0 [skip ci] * Support arm64 lambda functions ([philips-labs#2121](philips-labs#2121)) ([9e2a7b6](philips-labs@9e2a7b6)) * Support Node16 for AWS Lambda ([philips-labs#2073](philips-labs#2073)) ([68a2014](philips-labs@68a2014)) * replaced old environment variable ([philips-labs#2146](philips-labs#2146)) ([f2072f7](philips-labs@f2072f7)) * set explicit permissions on s3 for syncer lambda ([philips-labs#2145](philips-labs#2145)) ([aa7edd1](philips-labs@aa7edd1)) * set kms key on aws_s3_object when encryption is enabled ([philips-labs#2147](philips-labs#2147)) ([b4dc706](philips-labs@b4dc706)) * chore(release): 1.4.0 [skip ci] * Add option to match some of the labes instead of all [philips-labs#2122](philips-labs#2122) ([philips-labs#2123](philips-labs#2123)) ([c5e3c21](philips-labs@c5e3c21)) * don't apply extra labels unless defined ([philips-labs#2181](philips-labs#2181)) ([c0b11bb](philips-labs@c0b11bb)) * Remove asterik in permission for runner lambda to describe instances ([9b9da03](philips-labs@9b9da03)) * chore(release): 1.4.1 [skip ci] * added server_side_encryption key to download trigger for distribution ([philips-labs#2207](philips-labs#2207)) ([404e3b6](philips-labs@404e3b6)) * chore(release): 1.5.0 [skip ci] * Add ubuntu-jammy example image based on existing ubuntu-focal ([philips-labs#2102](philips-labs#2102)) ([486ae91](philips-labs@486ae91)) * **images:** avoid wrong AMI could be selected for ubuntu focal ([philips-labs#2214](philips-labs#2214)) ([76be94b](philips-labs@76be94b)) * chore(release): 1.6.0 [skip ci] * Add options extra option to ebs block device mapping ([philips-labs#2052](philips-labs#2052)) ([7cd2524](philips-labs@7cd2524)) * Enable node16 default ([philips-labs#2074](philips-labs#2074)) ([58aa5ed](philips-labs@58aa5ed)) * Incorrect path of Runner logs ([philips-labs#2233](philips-labs#2233)) ([98eff98](philips-labs@98eff98)) * Preventing that lambda webhook fails when it tries to process an installation_repositories event ([philips-labs#2288](philips-labs#2288)) ([8656c83](philips-labs@8656c83)) * Update ubuntu example to fix /opt/hostedtoolcache ([philips-labs#2302](philips-labs#2302)) ([8eea748](philips-labs@8eea748)) * Webhook lambda misleading log ([philips-labs#2291](philips-labs#2291)) ([c6275f9](philips-labs@c6275f9)) * chore(release): 1.7.0 [skip ci] * Webhook accept jobs where not all labels are provided in job. ([philips-labs#2209](philips-labs#2209)) ([6d9116f](philips-labs@6d9116f)) * Ignore case for runner labels. ([philips-labs#2315](philips-labs#2315)) ([014985a](philips-labs@014985a)) * chore(release): 1.8.0 [skip ci] * Add option to disable lambda to sync runner binaries ([philips-labs#2314](philips-labs#2314)) ([9f7d32d](philips-labs@9f7d32d)) * **examples:** Upgrading ubuntu example to 22.04 ([philips-labs#2250](philips-labs#2250)) ([d4b7650](philips-labs@d4b7650)), closes [philips-labs#2103](philips-labs#2103) * chore(release): 1.8.1 [skip ci] * **runners:** Pass allocation strategy ([philips-labs#2345](philips-labs#2345)) ([68d3445](philips-labs@68d3445)) * chore(release): 1.9.0 [skip ci] * Add option to enable access log for API gateway ([philips-labs#2387](philips-labs#2387)) ([fcd9fba](philips-labs@fcd9fba)) * add s3_location_runner_distribution var as expandable for userdata ([philips-labs#2371](philips-labs#2371)) ([05fe737](philips-labs@05fe737)) * Encrypted data at REST on SQS by default ([philips-labs#2431](philips-labs#2431)) ([7f3f4bf](philips-labs@7f3f4bf)) * **images:** Allow passing instance type when building windows image ([philips-labs#2369](philips-labs#2369)) ([eca23bf](philips-labs@eca23bf)) * **runners:** Fetch instance environment tag though metadata ([philips-labs#2346](philips-labs#2346)) ([27db290](philips-labs@27db290)) * **runners:** Set the default Windows AMI to Server 2022 ([philips-labs#2325](philips-labs#2325)) ([78e99d1](philips-labs@78e99d1)) * chore(release): 1.9.1 [skip ci] * **webhook:** Use `x-hub-signature-256` header as default ([philips-labs#2434](philips-labs#2434)) ([9c3e495](philips-labs@9c3e495)) * chore(release): 1.10.0 [skip ci] * Download runner release via latest release API ([philips-labs#2455](philips-labs#2455)) ([e75e092](philips-labs@e75e092)) * fix: Execute runner in own process, mask token in logs * Add option to disable user_data logging * Enforcing debug is disabled, and introduce option to enable debug logging. * add section related to security considerations * add section related to security considerations Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net> Co-authored-by: Derek Crosson <derekcrosson18@gmail.com>
enverus-cts · Oct 11, 2022 · 9a9e2ee · 9a9e2ee
1 parent e197cbd
commit 9a9e2ee
Show file tree

Hide file tree

Showing 12 changed files with 624 additions and 10 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
diff --git a/README.md b/README.md
@@ -26,6 +26,7 @@ This [Terraform](https://www.terraform.io/) module creates the required infrastr
 - [Sub modules](#sub-modules)
   - [ARM64 configuration for submodules](#arm64-configuration-for-submodules)
 - [Debugging](#debugging)
+- [Security Consideration](#security-consideration)
 - [Requirements](#requirements)
 - [Providers](#providers)
 - [Modules](#modules)
@@ -352,6 +353,14 @@ In case the setup does not work as intended follow the trace of events:
 - Once an EC2 instance is running, you can connect to it in the EC2 user interface using Session Manager (use `enable_ssm_on_runners = true`). Check the user data script using `cat /var/log/user-data.log`. By default several log files of the instances are streamed to AWS CloudWatch, look for a log group named `<environment>/runners`. In the log group you should see at least the log streams for the user data installation and runner agent.
 - Registered instances should show up in the Settings - Actions page of the repository or organization (depending on the installation mode).
 
+## Security Consideration
+
+This module creates resources in your AWS infrastructure, and EC2 instances for hosting the self-hosted runners on-demand. IAM permissions are set to a minimal level, and could be further limit by using permission boundaries. Instances permissions are limit to retrieve and delete the registration token, access the instance own tags, and terminate the instance itself.
+
+The examples are using standard AMI's for different operation systems. Instances are not hardened, and sudo operation are not blocked. To provide an out of the box working expierence by default the module installs and configure the runner. However secrets are not hard coded, they finally end up in the memory of the instances. You can harden the instance by providing your own AMI and overwriting the cloud-init script.
+
+We welcome any improvement to the standard module to make the default as secure as possible, in the end it remains your responsibility to keep your environment secure.
+
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Requirements
 
@@ -409,6 +418,7 @@ In case the setup does not work as intended follow the trace of events:
 | <a name="input_enable_runner_binaries_syncer"></a> [enable\_runner\_binaries\_syncer](#input\_enable\_runner\_binaries\_syncer) | Option to disable the lambda to sync GitHub runner distribution, useful when using a pre-build AMI. | `bool` | `true` | no |
 | <a name="input_enable_runner_detailed_monitoring"></a> [enable\_runner\_detailed\_monitoring](#input\_enable\_runner\_detailed\_monitoring) | Should detailed monitoring be enabled for the runner. Set this to true if you want to use detailed monitoring. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html for details. | `bool` | `false` | no |
 | <a name="input_enable_ssm_on_runners"></a> [enable\_ssm\_on\_runners](#input\_enable\_ssm\_on\_runners) | Enable to allow access the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. | `bool` | `false` | no |
+| <a name="input_enable_user_data_debug_logging_runner"></a> [enable\_user\_data\_debug\_logging\_runner](#input\_enable\_user\_data\_debug\_logging\_runner) | Option to enable debug logging for user-data, this logs all secrets as well. | `bool` | `false` | no |
 | <a name="input_enabled_userdata"></a> [enabled\_userdata](#input\_enabled\_userdata) | Should the userdata script be enabled for the runner. Set this to false if you are using your own prebuilt AMI. | `bool` | `true` | no |
 | <a name="input_environment"></a> [environment](#input\_environment) | A name that identifies the environment, used as prefix and for tagging. | `string` | `null` | no |
 | <a name="input_fifo_build_queue"></a> [fifo\_build\_queue](#input\_fifo\_build\_queue) | Enable a FIFO queue to remain the order of events received by the webhook. Suggest to set to true for repo level runners. | `bool` | `false` | no |

diff --git a/examples/arm64/README.md b/examples/arm64/README.md
@@ -4,7 +4,7 @@ This module shows how to create GitHub action runners using AWS Graviton instanc
 
 ## Usages
 
-Steps for the full setup, such as creating a GitHub app can be found in the root module's [README](../../README.md). First download the Lambda releases from GitHub. Alternatively you can build the lambdas locally with Node or Docker, there is a simple build script in `<root>/.ci/build.sh`. In the `main.tf` you can simply remove the location of the lambda zip files, the default location will work in this case.
+Steps for the full setup, such as creating a GitHub app can be found in the root module's [README](https://github.com/philips-labs/terraform-aws-github-runner). First download the Lambda releases from GitHub. Alternatively you can build the lambdas locally with Node or Docker, there is a simple build script in `<root>/.ci/build.sh`. In the `main.tf` you can simply remove the location of the lambda zip files, the default location will work in this case.
 
 > Ensure you have set the version in `lambdas-download/main.tf` for running the example. The version needs to be set to a GitHub release version, see https://github.com/philips-labs/terraform-aws-github-runner/releases
 
@@ -15,7 +15,7 @@ terraform apply
 cd ..
 ```
 
-Before running Terraform, ensure the GitHub app is configured. See the [configuration details](../../README.md#usages) for more details.
+Before running Terraform, ensure the GitHub app is configured. See the [configuration details](https://github.com/philips-labs/terraform-aws-github-runner#usages) for more details.
 
 ```bash
 terraform init

diff --git a/examples/ubuntu/main.tf b/examples/ubuntu/main.tf
@@ -32,7 +32,7 @@ module "runners" {
   # runners_lambda_zip                = "lambdas-download/runners.zip"
 
   enable_organization_runners = false
-  runner_extra_labels         = "ubuntu,example"
+  runner_extra_labels         = "default,example"
 
   # enable access to the runners via SSM
   enable_ssm_on_runners = true
@@ -102,4 +102,6 @@ module "runners" {
   #   idleCount = 1
   # }]
 
+  # Enable logging all commands of user_data, secrets will be logged!!!
+  # enable_user_data_debug_logging_runner = true
 }
diff --git a/examples/ubuntu/templates/user-data.sh b/examples/ubuntu/templates/user-data.sh
@@ -1,6 +1,17 @@
-#!/bin/bash -x
+#!/bin/bash
 exec > >(tee /var/log/user-data.log | logger -t user-data -s 2>/dev/console) 2>&1
 
+
+# AWS suggest to create a log for debug purpose based on https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-log-user-data/
+# As side effect all command, set +x disable debugging explicitly.
+#
+# An alternative for masking tokens could be: exec > >(sed 's/--token\ [^ ]* /--token\ *** /g' > /var/log/user-data.log) 2>&1
+set +x
+
+%{ if enable_debug_logging }
+set -x
+%{ endif }
+
 ${pre_install}
 
 # Install AWS CLI

diff --git a/main.tf b/main.tf
@@ -203,12 +203,13 @@ module "runners" {
   role_path                 = var.role_path
   role_permissions_boundary = var.role_permissions_boundary
 
-  enabled_userdata      = var.enabled_userdata
-  userdata_template     = var.userdata_template
-  userdata_pre_install  = var.userdata_pre_install
-  userdata_post_install = var.userdata_post_install
-  key_name              = var.key_name
-  runner_ec2_tags       = var.runner_ec2_tags
+  enabled_userdata               = var.enabled_userdata
+  enable_user_data_debug_logging = var.enable_user_data_debug_logging_runner
+  userdata_template              = var.userdata_template
+  userdata_pre_install           = var.userdata_pre_install
+  userdata_post_install          = var.userdata_post_install
+  key_name                       = var.key_name
+  runner_ec2_tags                = var.runner_ec2_tags
 
   create_service_linked_role_spot = var.create_service_linked_role_spot
 

diff --git a/modules/runners/README.md b/modules/runners/README.md
@@ -130,6 +130,7 @@ yarn run dist
 | <a name="input_enable_runner_binaries_syncer"></a> [enable\_runner\_binaries\_syncer](#input\_enable\_runner\_binaries\_syncer) | Option to disable the lambda to sync GitHub runner distribution, useful when using a pre-build AMI. | `bool` | `true` | no |
 | <a name="input_enable_runner_detailed_monitoring"></a> [enable\_runner\_detailed\_monitoring](#input\_enable\_runner\_detailed\_monitoring) | Enable detailed monitoring for runners | `bool` | `false` | no |
 | <a name="input_enable_ssm_on_runners"></a> [enable\_ssm\_on\_runners](#input\_enable\_ssm\_on\_runners) | Enable to allow access to the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. | `bool` | n/a | yes |
+| <a name="input_enable_user_data_debug_logging"></a> [enable\_user\_data\_debug\_logging](#input\_enable\_user\_data\_debug\_logging) | Option to enable debug logging for user-data, this logs all secrets as well. | `bool` | `false` | no |
 | <a name="input_enabled_userdata"></a> [enabled\_userdata](#input\_enabled\_userdata) | Should the userdata script be enabled for the runner. Set this to false if you are using your own prebuilt AMI | `bool` | `true` | no |
 | <a name="input_environment"></a> [environment](#input\_environment) | A name that identifies the environment, used as prefix and for tagging. | `string` | `null` | no |
 | <a name="input_ghes_ssl_verify"></a> [ghes\_ssl\_verify](#input\_ghes\_ssl\_verify) | GitHub Enterprise SSL verification. Set to 'false' when custom certificate (chains) is used for GitHub Enterprise Server (insecure). | `bool` | `true` | no |

diff --git a/modules/runners/main.tf b/modules/runners/main.tf
@@ -133,6 +133,7 @@ resource "aws_launch_template" "runner" {
   }
 
   user_data = var.enabled_userdata ? base64encode(templatefile(local.userdata_template, {
+    enable_debug_logging            = var.enable_user_data_debug_logging
     s3_location_runner_distribution = local.s3_location_runner_distribution
     pre_install                     = var.userdata_pre_install
     install_runner = templatefile(local.userdata_install_runner[var.runner_os], {

diff --git a/modules/runners/templates/start-runner.sh b/modules/runners/templates/start-runner.sh
@@ -84,6 +84,8 @@ echo "Starting runner after $(awk '{print int($1/3600)":"int(($1%3600)/60)":"int
 echo "Starting the runner as user $run_as"
 
 if [[ $agent_mode = "ephemeral" ]]; then
+
+cat >/opt/start-runner-service.sh <<-EOF
   echo "Starting the runner in ephemeral mode"
   sudo --preserve-env=RUNNER_ALLOW_RUNASROOT -u "$run_as" -- ./run.sh
   echo "Runner has finished"
@@ -92,6 +94,11 @@ if [[ $agent_mode = "ephemeral" ]]; then
   systemctl stop amazon-cloudwatch-agent.service
   echo "Terminating instance"
   aws ec2 terminate-instances --instance-ids "$instance_id" --region "$region"
+EOF
+  chmod 755 /opt/start-runner-service.sh
+  # Starting the runner via a own process to ensure this process terminates
+  nohup /opt/start-runner-service.sh &
+
 else
   echo "Installing the runner as a service"
   ./svc.sh install "$run_as"

diff --git a/modules/runners/templates/user-data.sh b/modules/runners/templates/user-data.sh
@@ -1,6 +1,18 @@
 #!/bin/bash -e
+
 exec > >(tee /var/log/user-data.log | logger -t user-data -s 2>/dev/console) 2>&1
 
+# AWS suggest to create a log for debug purpose based on https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-log-user-data/
+# As side effect all command, set +x disable debugging explicitly.
+#
+# An alternative for masking tokens could be: exec > >(sed 's/--token\ [^ ]* /--token\ *** /g' > /var/log/user-data.log) 2>&1
+
+set +x
+
+%{ if enable_debug_logging }
+set -x
+%{ endif }
+
 ${pre_install}
 
 yum update -y

diff --git a/modules/runners/variables.tf b/modules/runners/variables.tf
@@ -571,3 +571,9 @@ variable "enable_runner_binaries_syncer" {
   type        = bool
   default     = true
 }
+
+variable "enable_user_data_debug_logging" {
+  description = "Option to enable debug logging for user-data, this logs all secrets as well."
+  type        = bool
+  default     = false
+}
diff --git a/variables.tf b/variables.tf
@@ -726,3 +726,8 @@ variable "queue_encryption" {
   }
 }
 
+variable "enable_user_data_debug_logging_runner" {
+  description = "Option to enable debug logging for user-data, this logs all secrets as well."
+  type        = bool
+  default     = false
+}