Cloudflare access for @nodejs/web-infra

[MattIPv4]

👋 Currently only the build WG has access to make changes to the Cloudflare config. While in theory this is fine, and the web-infra team could collaborate with them on any changes needed, it has become apparent recently that there is perhaps not enough coverage within the build WG to ensure that changes can be made in a timely manner.

As an example of this, we've had intermittent 500 issues with Node.js downloads for a while now, and this has been exacerbated in part by a slight misconfiguration in Cloudflare that is causing some 500 responses to be cached. nodejs/build#3473 was opened over a month ago to make the needed changes, but as of writing this still hasn't been actioned, and is still causing impact to users as reported in nodejs/nodejs.org#5818 similar, and the web-infra team is unable to do anything but wait for the build WG.

I think it'd be beneficial to explore granting the web-infra team access to Cloudflare to make configuration changes alongside the build WG, as the web-infra team has been formed to help look after all the infra relating to nodejs.org (Cloudflare, Vercel, etc.). This'd allow the web-infra team to more directly respond to and fix issues reported such as this 500 caching error without having to loop in the build WG, as well as increasing the bus factor for access to Cloudflare as it would seem this is very limited within the build WG, leading to the month+ wait for this current change request.

cc @nodejs/tsc
cc @nodejs/build
cc @nodejs/web-infra

[mhdawson]

The build team has been cautious in terms of giving access to key infrastructure. While I understand the desire to make changes more quickly, part of that caution is to avoid emergency situations where the build team has to jump into fix things when they have not planned time to be able to do that.

There are discussions with the foundation in terms of taking over management of some/all of the Node.js infra and I think moving forward with that is a good way to address the current frustrations. It should provide an SLA for requests as well as people who are paid to respond quickly when changes cause issues.

I think having the Linux IT team take over the the Website infra including the downloads, along with Cloudflare would be a good first step in terms of the Foundation helping with Node.js infra and would be good to prioritize in terms of addressing @ovflowd frustrations as well.

[mhdawson]

@bensternthal do you have any idea of the timeline for when the Linux IT team would be able to start working on managing the Website/cloudflare infra and related discussion with the build WG?

@UlisesGascon did some initial work on using Teraform to manage our cloudflare configuration and doing that might be a great way were requests could be done through PRs and then the Linux IT would be the team that would land those PRs once there were approvals, and be ready to do rollbacks if needed.

[UlisesGascon]

@UlisesGascon did some initial work on using Teraform to manage our cloudflare configuration and doing that might be a great way were requests could be done through PRs and then the Linux IT would be the team that would land those PRs once there were approvals, and be ready to do rollbacks if needed.

Yes, I think Terraform is the way to go here. It will allow us to be faster and safer when making changes in Cloudflare. I can focus on this as a priority once I am back from holidays. We just need to agree first within the Build team that we are confident with the new way of using infrastructure secrets in the Github actions and and how to trigger the changes, etc.., as this is a new tool for the project/team. 👍.

This was the major blocker until now for Terraform adoption nodejs/build#3391.

[bensternthal]

@mhdawson Right now the IT team is blocked because they do not have any access to the node accounts for github, jenkins, and cloudflare. Is there anything you can do to help here? They just need read only access to complete their audit.

[mhdawson]

@bensternthal

I don't remember the issue/slack conversation but for github they should already have read access to most things in the repo, if they could be specific about what they need to see that cannot today that would be great.

For Jenkins they should already have read access to most of the CI, jobs etc. Again if we could be more specific about what they don't have access to that's needed that would help us add specific persmissions in the Jenkings config.

For cloudflare I think I can figure that out. I believe we need to add a cloudflare id with red only privs. What is the id that we should add?

[richardlau]

For Jenkins they should already have read access to most of the CI, jobs etc. Again if we could be more specific about what they don't have access to that's needed that would help us add specific persmissions in the Jenkings config.

I think you need to be a collaborator to have read access to the job configs.
There's an issue tracking Jenkins access for LF IT: nodejs/build#3444
And another for Cloudflare access: nodejs/build#3445
Both of those are not in the scope of this issue which is for the web-infra team. We discussed that in this week's Build WG call and our preferred route of expanding write access to Cloudflare is to progress the Terraform work and enable changes to be PR-able into a GH repo.

[mhdawson]

@richardlau thanks for the links to those issues.

@bensternthal can you add the id that we need to add for read-only cloudflare access into nodejs/build#3445

[bensternthal]

@mhdawson much thanks for the help! The account to add is hostmaster+openjs@linuxfoundation.org

Mentioning @vvalderrv for visibility

[mhdawson]

We took off the agenda as the plan is to control access to cloudflare through terafrom, so until we do that there is nothing to discuss in the build WG meetings.

[ovflowd]

Closing this for the time being, until we have LFX infrastructure set in place.