Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(runner): Add option to disable auto update #1791

Merged
merged 8 commits into from
Feb 25, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -400,9 +400,10 @@ In case the setup does not work as intended follow the trace of events:
| <a name="input_cloudwatch_config"></a> [cloudwatch\_config](#input\_cloudwatch\_config) | (optional) Replaces the module default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details. | `string` | `null` | no |
| <a name="input_create_service_linked_role_spot"></a> [create\_service\_linked\_role\_spot](#input\_create\_service\_linked\_role\_spot) | (optional) create the serviced linked role for spot instances that is required by the scale-up lambda. | `bool` | `false` | no |
| <a name="input_delay_webhook_event"></a> [delay\_webhook\_event](#input\_delay\_webhook\_event) | The number of seconds the event accepted by the webhook is invisible on the queue before the scale up lambda will receive the event. | `number` | `30` | no |
| <a name="input_disable_runner_autoupdate"></a> [disable\_runner\_autoupdate](#input\_disable\_runner\_autoupdate) | Disable the auto update of the github runner agent. Be-aware there is a grace period of 30 days, see also the [GitHub article](https://github.blog/changelog/2022-02-01-github-actions-self-hosted-runners-can-now-disable-automatic-updates/) | `bool` | `false` | no |
| <a name="input_enable_cloudwatch_agent"></a> [enable\_cloudwatch\_agent](#input\_enable\_cloudwatch\_agent) | Enabling the cloudwatch agent on the ec2 runner instances, the runner contains default config. Configuration can be overridden via `cloudwatch_config`. | `bool` | `true` | no |
| <a name="input_enable_ephemeral_runners"></a> [enable\_ephemeral\_runners](#input\_enable\_ephemeral\_runners) | Enable ephemeral runners, runners will only be used once. | `bool` | `false` | no |
| <a name="input_enable_managed_runner_security_group"></a> [enable\_managed\_runner\_security\_group](#inputenable\_managed\_runner\_security\_group) | Enabling the default managed security group creation. Unmanaged security groups can be specified via `runner_additional_security_group_ids`. | `bool` | `true` | no |
| <a name="input_enable_managed_runner_security_group"></a> [enable\_managed\_runner\_security\_group](#input\_enable\_managed\_runner\_security\_group) | Enabling the default managed security group creation. Unmanaged security groups can be specified via `runner_additional_security_group_ids`. | `bool` | `true` | no |
| <a name="input_enable_organization_runners"></a> [enable\_organization\_runners](#input\_enable\_organization\_runners) | Register runners to organization, instead of repo level | `bool` | `false` | no |
| <a name="input_enable_ssm_on_runners"></a> [enable\_ssm\_on\_runners](#input\_enable\_ssm\_on\_runners) | Enable to allow access the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. | `bool` | `false` | no |
| <a name="input_enabled_userdata"></a> [enabled\_userdata](#input\_enabled\_userdata) | Should the userdata script be enabled for the runner. Set this to false if you are using your own prebuilt AMI | `bool` | `true` | no |
Expand Down Expand Up @@ -454,7 +455,7 @@ In case the setup does not work as intended follow the trace of events:
| <a name="input_runner_iam_role_managed_policy_arns"></a> [runner\_iam\_role\_managed\_policy\_arns](#input\_runner\_iam\_role\_managed\_policy\_arns) | Attach AWS or customer-managed IAM policies (by ARN) to the runner IAM role | `list(string)` | `[]` | no |
| <a name="input_runner_log_files"></a> [runner\_log\_files](#input\_runner\_log\_files) | (optional) Replaces the module default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details. | <pre>list(object({<br> log_group_name = string<br> prefix_log_group = bool<br> file_path = string<br> log_stream_name = string<br> }))</pre> | `null` | no |
| <a name="input_runner_metadata_options"></a> [runner\_metadata\_options](#input\_runner\_metadata\_options) | Metadata options for the ec2 runner instances. | `map(any)` | <pre>{<br> "http_endpoint": "enabled",<br> "http_put_response_hop_limit": 1,<br> "http_tokens": "optional"<br>}</pre> | no |
| <a name="input_runner_os"></a> [runner\_os](#input\_runner\_os) | The Operating System to use for GitHub Actions Runners (linux,win) | `string` | `"linux"` | no |
| <a name="input_runner_os"></a> [runner\_os](#input\_runner\_os) | The EC2 Operating System type to use for action runner instances (linux,windows). | `string` | `"linux"` | no |
| <a name="input_runner_run_as"></a> [runner\_run\_as](#input\_runner\_run\_as) | Run the GitHub actions agent as user. | `string` | `"ec2-user"` | no |
| <a name="input_runners_lambda_s3_key"></a> [runners\_lambda\_s3\_key](#input\_runners\_lambda\_s3\_key) | S3 key for runners lambda function. Required if using S3 bucket to specify lambdas. | `any` | `null` | no |
| <a name="input_runners_lambda_s3_object_version"></a> [runners\_lambda\_s3\_object\_version](#input\_runners\_lambda\_s3\_object\_version) | S3 object version for runners lambda function. Useful if S3 versioning is enabled on source bucket. | `any` | `null` | no |
Expand Down
1 change: 1 addition & 0 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ module "runners" {
github_app_parameters = local.github_app_parameters
enable_organization_runners = var.enable_organization_runners
enable_ephemeral_runners = var.enable_ephemeral_runners
disable_runner_autoupdate = var.disable_runner_autoupdate
enable_managed_runner_security_group = var.enable_managed_runner_security_group
scale_down_schedule_expression = var.scale_down_schedule_expression
minimum_running_time_in_minutes = var.minimum_running_time_in_minutes
Expand Down
2 changes: 1 addition & 1 deletion modules/runner-binaries-syncer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ No modules.
| <a name="input_role_permissions_boundary"></a> [role\_permissions\_boundary](#input\_role\_permissions\_boundary) | Permissions boundary that will be added to the created role for the lambda. | `string` | `null` | no |
| <a name="input_runner_allow_prerelease_binaries"></a> [runner\_allow\_prerelease\_binaries](#input\_runner\_allow\_prerelease\_binaries) | Allow the runners to update to prerelease binaries. | `bool` | `false` | no |
| <a name="input_runner_architecture"></a> [runner\_architecture](#input\_runner\_architecture) | The platform architecture of the runner instance\_type. | `string` | `"x64"` | no |
| <a name="input_runner_os"></a> [runner\_os](#input\_runner\_os) | The operating system for the runner instance (linux, win), defaults to 'linux' | `string` | `"linux"` | no |
| <a name="input_runner_os"></a> [runner\_os](#input\_runner\_os) | The EC2 Operating System type to use for action runner instances (linux,windows). | `string` | `"linux"` | no |
| <a name="input_server_side_encryption_configuration"></a> [server\_side\_encryption\_configuration](#input\_server\_side\_encryption\_configuration) | Map containing server-side encryption configuration. | `any` | `{}` | no |
| <a name="input_syncer_lambda_s3_key"></a> [syncer\_lambda\_s3\_key](#input\_syncer\_lambda\_s3\_key) | S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas. | `any` | `null` | no |
| <a name="input_syncer_lambda_s3_object_version"></a> [syncer\_lambda\_s3\_object\_version](#input\_syncer\_lambda\_s3\_object\_version) | S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket. | `any` | `null` | no |
Expand Down
5 changes: 3 additions & 2 deletions modules/runners/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,11 @@ yarn run dist
| <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | The EC2 instance block device configuration. Takes the following keys: `device_name`, `delete_on_termination`, `volume_type`, `volume_size`, `encrypted`, `iops` | `map(string)` | `{}` | no |
| <a name="input_cloudwatch_config"></a> [cloudwatch\_config](#input\_cloudwatch\_config) | (optional) Replaces the module default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details. | `string` | `null` | no |
| <a name="input_create_service_linked_role_spot"></a> [create\_service\_linked\_role\_spot](#input\_create\_service\_linked\_role\_spot) | (optional) create the service linked role for spot instances that is required by the scale-up lambda. | `bool` | `false` | no |
| <a name="input_disable_runner_autoupdate"></a> [disable\_runner\_autoupdate](#input\_disable\_runner\_autoupdate) | Disable the auto update of the github runner agent. Be-aware there is a grace period of 30 days, see also the [GitHub article](https://github.blog/changelog/2022-02-01-github-actions-self-hosted-runners-can-now-disable-automatic-updates/) | `bool` | `true` | no |
| <a name="input_egress_rules"></a> [egress\_rules](#input\_egress\_rules) | List of egress rules for the GitHub runner instances. | <pre>list(object({<br> cidr_blocks = list(string)<br> ipv6_cidr_blocks = list(string)<br> prefix_list_ids = list(string)<br> from_port = number<br> protocol = string<br> security_groups = list(string)<br> self = bool<br> to_port = number<br> description = string<br> }))</pre> | <pre>[<br> {<br> "cidr_blocks": [<br> "0.0.0.0/0"<br> ],<br> "description": null,<br> "from_port": 0,<br> "ipv6_cidr_blocks": [<br> "::/0"<br> ],<br> "prefix_list_ids": null,<br> "protocol": "-1",<br> "security_groups": null,<br> "self": null,<br> "to_port": 0<br> }<br>]</pre> | no |
| <a name="input_enable_cloudwatch_agent"></a> [enable\_cloudwatch\_agent](#input\_enable\_cloudwatch\_agent) | Enabling the cloudwatch agent on the ec2 runner instances, the runner contains default config. Configuration can be overridden via `cloudwatch_config`. | `bool` | `true` | no |
| <a name="input_enable_ephemeral_runners"></a> [enable\_ephemeral\_runners](#input\_enable\_ephemeral\_runners) | Enable ephemeral runners, runners will only be used once. | `bool` | `false` | no |
| <a name="input_enable_managed_runner_security_group"></a> [enable\_managed\_runner\_security\_group](#inputenable\_managed\_runner\_security\_group) | Enabling the default managed security group creation. Unmanaged security groups can be specified via `runner_additional_security_group_ids`. | `bool` | `true` | no |
| <a name="input_enable_managed_runner_security_group"></a> [enable\_managed\_runner\_security\_group](#input\_enable\_managed\_runner\_security\_group) | Enabling the default managed security group creation. Unmanaged security groups can be specified via `runner_additional_security_group_ids`. | `bool` | `true` | no |
| <a name="input_enable_organization_runners"></a> [enable\_organization\_runners](#input\_enable\_organization\_runners) | n/a | `bool` | n/a | yes |
| <a name="input_enable_ssm_on_runners"></a> [enable\_ssm\_on\_runners](#input\_enable\_ssm\_on\_runners) | Enable to allow access to the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. | `bool` | n/a | yes |
| <a name="input_enabled_userdata"></a> [enabled\_userdata](#input\_enabled\_userdata) | Should the userdata script be enabled for the runner. Set this to false if you are using your own prebuilt AMI | `bool` | `true` | no |
Expand Down Expand Up @@ -167,7 +168,7 @@ yarn run dist
| <a name="input_runner_group_name"></a> [runner\_group\_name](#input\_runner\_group\_name) | Name of the runner group. | `string` | `"Default"` | no |
| <a name="input_runner_iam_role_managed_policy_arns"></a> [runner\_iam\_role\_managed\_policy\_arns](#input\_runner\_iam\_role\_managed\_policy\_arns) | Attach AWS or customer-managed IAM policies (by ARN) to the runner IAM role | `list(string)` | `[]` | no |
| <a name="input_runner_log_files"></a> [runner\_log\_files](#input\_runner\_log\_files) | (optional) List of logfiles to send to CloudWatch, will only be used if `enable_cloudwatch_agent` is set to true. Object description: `log_group_name`: Name of the log group, `prefix_log_group`: If true, the log group name will be prefixed with `/github-self-hosted-runners/<var.environment>`, `file_path`: path to the log file, `log_stream_name`: name of the log stream. | <pre>list(object({<br> log_group_name = string<br> prefix_log_group = bool<br> file_path = string<br> log_stream_name = string<br> }))</pre> | `null` | no |
| <a name="input_runner_os"></a> [runner\_os](#input\_runner\_os) | The EC2 Operating System type to use for action runner instances (linux,win). | `string` | `"linux"` | no |
| <a name="input_runner_os"></a> [runner\_os](#input\_runner\_os) | The EC2 Operating System type to use for action runner instances (linux,windows). | `string` | `"linux"` | no |
| <a name="input_runner_run_as"></a> [runner\_run\_as](#input\_runner\_run\_as) | Run the GitHub actions agent as user. | `string` | `"ec2-user"` | no |
| <a name="input_runners_lambda_s3_key"></a> [runners\_lambda\_s3\_key](#input\_runners\_lambda\_s3\_key) | S3 key for runners lambda function. Required if using S3 bucket to specify lambdas. | `any` | `null` | no |
| <a name="input_runners_lambda_s3_object_version"></a> [runners\_lambda\_s3\_object\_version](#input\_runners\_lambda\_s3\_object\_version) | S3 object version for runners lambda function. Useful if S3 versioning is enabled on source bucket. | `any` | `null` | no |
Expand Down
11 changes: 10 additions & 1 deletion modules/runners/lambdas/runners/src/pool/pool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ export async function adjust(event: PoolEvent): Promise<void> {
const instanceTypes = process.env.INSTANCE_TYPES.split(',');
const instanceTargetTargetCapacityType = process.env.INSTANCE_TARGET_CAPACITY_TYPE;
const ephemeral = yn(process.env.ENABLE_EPHEMERAL_RUNNERS, { default: false });
const disableAutoUpdate = yn(process.env.DISABLE_RUNNER_AUTOUPDATE, { default: false });
const launchTemplateName = process.env.LAUNCH_TEMPLATE_NAME;
const instanceMaxSpotPrice = process.env.INSTANCE_MAX_SPOT_PRICE;
const instanceAllocationStrategy = process.env.INSTANCE_ALLOCATION_STRATEGY || 'lowest-price'; // same as AWS default
Expand Down Expand Up @@ -60,7 +61,15 @@ export async function adjust(event: PoolEvent): Promise<void> {
if (topUp > 0) {
logger.info(`The pool will be topped up with ${topUp} runners.`);
await createRunners(
{ ephemeral, ghesBaseUrl, runnerExtraLabels, runnerGroup, runnerOwner, runnerType: 'Org' },
{
ephemeral,
ghesBaseUrl,
runnerExtraLabels,
runnerGroup,
runnerOwner,
runnerType: 'Org',
disableAutoUpdate: disableAutoUpdate,
},
{
ec2instanceCriteria: {
instanceTypes,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,13 @@ describe('scaleUp with public GH', () => {
expect(createRunner).toBeCalledWith(expectedRunnerParams);
});

it('disable auto update on the runner.', async () => {
process.env.DISABLE_RUNNER_AUTOUPDATE = 'true';
await scaleUpModule.scaleUp('aws:sqs', TEST_DATA);
expectedRunnerParams.runnerServiceConfig = expectedRunnerParams.runnerServiceConfig + ` --disableupdate`;
expect(createRunner).toBeCalledWith(expectedRunnerParams);
});

it('Scaling error should cause reject so retry can be triggered.', async () => {
process.env.RUNNERS_MAXIMUM_COUNT = '1';
process.env.ENABLE_EPHEMERAL_RUNNERS = 'true';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ interface CreateGitHubRunnerConfig {
runnerGroup: string | undefined;
runnerOwner: string;
runnerType: 'Org' | 'Repo';
disableAutoUpdate: boolean;
}

interface CreateEC2RunnerConfig {
Expand All @@ -40,7 +41,8 @@ function generateRunnerServiceConfig(githubRunnerConfig: CreateGitHubRunnerConfi
githubRunnerConfig.runnerGroup !== undefined ? `--runnergroup ${githubRunnerConfig.runnerGroup} ` : '';
const configBaseUrl = githubRunnerConfig.ghesBaseUrl ? githubRunnerConfig.ghesBaseUrl : 'https://github.com';
const ephemeralArgument = githubRunnerConfig.ephemeral ? '--ephemeral ' : '';
const runnerArgs = `--token ${token} ${labelsArgument}${ephemeralArgument}`;
const disableUpdateArgument = githubRunnerConfig.disableAutoUpdate ? '--disableupdate ' : '';
const runnerArgs = `--token ${token} ${labelsArgument}${ephemeralArgument}${disableUpdateArgument}`;
return githubRunnerConfig.runnerType === 'Org'
? `--url ${configBaseUrl}/${githubRunnerConfig.runnerOwner} ${runnerArgs}${runnerGroupArgument}`.trim()
: `--url ${configBaseUrl}/${githubRunnerConfig.runnerOwner} ${runnerArgs}`.trim();
Expand Down Expand Up @@ -141,6 +143,7 @@ export async function scaleUp(eventSource: string, payload: ActionRequestMessage
const instanceTypes = process.env.INSTANCE_TYPES.split(',');
const instanceTargetTargetCapacityType = process.env.INSTANCE_TARGET_CAPACITY_TYPE;
const ephemeralEnabled = yn(process.env.ENABLE_EPHEMERAL_RUNNERS, { default: false });
const disableAutoUpdate = yn(process.env.DISABLE_RUNNER_AUTOUPDATE, { default: false });
const launchTemplateName = process.env.LAUNCH_TEMPLATE_NAME;
const instanceMaxSpotPrice = process.env.INSTANCE_MAX_SPOT_PRICE;
const instanceAllocationStrategy = process.env.INSTANCE_ALLOCATION_STRATEGY || 'lowest-price'; // same as AWS default
Expand Down Expand Up @@ -195,6 +198,7 @@ export async function scaleUp(eventSource: string, payload: ActionRequestMessage
runnerGroup,
runnerOwner,
runnerType,
disableAutoUpdate,
},
{
ec2instanceCriteria: {
Expand Down
13 changes: 7 additions & 6 deletions modules/runners/pool.tf
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,13 @@ module "pool" {
role_path = local.role_path
role_permissions_boundary = var.role_permissions_boundary
runner = {
ephemeral = var.enable_ephemeral_runners
extra_labels = var.runner_extra_labels
launch_template = aws_launch_template.runner
group_name = var.runner_group_name
pool_owner = var.pool_runner_owner
role = aws_iam_role.runner
disable_runner_autoupdate = var.disable_runner_autoupdate
ephemeral = var.enable_ephemeral_runners
extra_labels = var.runner_extra_labels
launch_template = aws_launch_template.runner
group_name = var.runner_group_name
pool_owner = var.pool_runner_owner
role = aws_iam_role.runner
}
subnet_ids = var.subnet_ids
tags = local.tags
Expand Down
13 changes: 7 additions & 6 deletions modules/runners/pool/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,14 @@ resource "aws_lambda_function" "pool" {

environment {
variables = {
RUNNER_OWNER = var.config.runner.pool_owner
DISABLE_RUNNER_AUTOUPDATE = var.config.runner.disable_runner_autoupdate
ENABLE_EPHEMERAL_RUNNERS = var.config.runner.ephemeral
ENVIRONMENT = var.config.environment
GHES_URL = var.config.ghes.url
INSTANCE_ALLOCATION_STRATEGY = var.config.instance_allocation_strategy
INSTANCE_MAX_SPOT_PRICE = var.config.instance_max_spot_price
INSTANCE_TARGET_CAPACITY_TYPE = var.config.instance_target_capacity_type
INSTANCE_TYPES = join(",", var.config.instance_types)
LAUNCH_TEMPLATE_NAME = var.config.runner.launch_template.name
LOG_LEVEL = var.config.lambda.log_level
LOG_TYPE = var.config.lambda.log_type
Expand All @@ -27,12 +32,8 @@ resource "aws_lambda_function" "pool" {
PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.config.github_app_parameters.key_base64.name
RUNNER_EXTRA_LABELS = var.config.runner.extra_labels
RUNNER_GROUP_NAME = var.config.runner.group_name
RUNNER_OWNER = var.config.runner.pool_owner
SUBNET_IDS = join(",", var.config.subnet_ids)
ENABLE_EPHEMERAL_RUNNERS = var.config.runner.ephemeral
INSTANCE_TYPES = join(",", var.config.instance_types)
INSTANCE_TARGET_CAPACITY_TYPE = var.config.instance_target_capacity_type
INSTANCE_MAX_SPOT_PRICE = var.config.instance_max_spot_price
INSTANCE_ALLOCATION_STRATEGY = var.config.instance_allocation_strategy
}
}

Expand Down
Loading