The mirrors.jenkins-ci.org is missing some necessary metadata files, which prevents it from being added as an apt/yum repo

[ak1ra-komj]

Problem with this page

https://mirrors.jenkins-ci.org/

Expected behavior

mirrors.jenkins-ci.org can be correctly added as an apt/yum repo.

Actual behavior

The apt/yum repo for mirrors.jenkins-ci.org is missing some necessary metadata files, which prevents it from being added correctly as an apt/yum repo.

For example, following the instructions in debian-stable/ will configure the debian apt repo correctly, but the corresponding mirror mirrors.jenkins-ci.org is missing some of the necessary metadata files compared to pkg.jenkins.io, in the following two files, the latter does not exist.

• https://pkg.jenkins.io/debian-stable/binary/Release.gpg
• https://mirrors.jenkins-ci.org/debian-stable/binary/Release.gpg

The mirrors run by third-party are upstream from mirrors.jenkins-ci.org, which prevents all the rest of the mirrors from being added as an apt/yum repo, an issue mentioned in tuna/issues jenkins-infra/jenkins.io#308, but they believe that the problem should be solved upstream.

Possible solution

Do a full sync for mirrors.jenkins-ci.org with pkg.jenkins.io as the upstream.

[dduportal]

Hi @ak1ra-komj , what is the reason for using the mirrors (which, by the way, should be using the get.jenkins.io domain)?

The official documentation states that https://pkg.jenkins.io/redhat-stable/ should be used as a YUM repository. The mirrors are only designed to host the packages only.

[ak1ra-komj]

Hi @ak1ra-komj, what is the reason for using the mirrors (which, by the way, should be using the get.jenkins.io domain)?

The official documentation states that https://pkg.jenkins.io/redhat-stable/ should be used as a YUM repository. The mirrors are only designed to host the packages only.

Accessing pkg.jenkins.io directly from mainland China has connectivity issues, and although I observed that pkg.jenkins.io uses fastly's CDN, fastly's edge nodes are still poorly accessible in mainland China.

Instead of using mirrors.jenkins-ci.org directly, I failed in my attempt to use one of the mirrors listed in Mirrorstats, which are upstream from mirrors.jenkins-ci.org.

If the mirrors are only designed to host the packages, what is the point of hosting mirrors for .deb/.rpm packages if the package manager can not use them? Direct download and installation is an option, but not recommended.

[dduportal]

(transferred the issue the infrastructure issue tracker as it is not related to the jenkins.io website itself)

[dduportal]

Accessing pkg.jenkins.io directly from mainland China has connectivity issues, and although I observed that pkg.jenkins.io uses fastly's CDN, but fastly's edge nodes are still poorly accessible in mainland China.

As I stated earlier, today (but it has to change int eh future), only pkg.jenkins.io is supported as a source for YUM packages.

I'm really sorry that Fastly's nodes are poorly accessible in China. We can and should improve the situation but it requires help. It's not as simple as "run a sync upstream mirror" as there are multiple security mechanisme at stake here to protect end users and avoid man in the middle scenarii.

There are multiple different challenges in your problem, I'll try to separate them one by one to see what can we do on short term, and what requires more work to help you.

Instead of using mirrors.jenkins-ci.org directly, I failed in my attempt to use one of the mirrors listed in Mirrorstats, which are upstream from mirrors.jenkins-ci.org.

Interesting result. What error did you have? I'm interested in what is the problem with the tsinghua.edu.cn mirror (which is expected to be available in mainland China)?

We've asked for sponsors to host mirrors in China since years and never had any feeback until recently: tsinghua.edu.cn is the only mirror that is generously maintained by sponsor): if you know any uniersity, company,organization who would be interested in helping us by running a mirror to improve the life of our China-located users, that would be awesome!

If the mirrors are only designed to host the packages only, what is the point to host mirrors for .deb/.rpm packages if it can not be use by package manager? Direct download and install is an option, but not recommended.

The main idea (at least initially) is to decrease the cost of hosting our own mirror because Jenkins is a community with limited funding. Outbound bandwidth is a source of great cost: adding packages to the mirror grid made sense to avoid the hugh bandwidth (package are binary files which weight a lot of Megabytes).

On the other end, the package index for YUM, APT, etc. need some kind of control (due to metadata and their signature) in case a package has to be removed due to security issue (not mentioning signature of content): this is the initial reason why the design makes the pkg.jenkins.io to be the initial source of truth, and then delegates the "heavy" download to get.jenkins.io. This is a strategy built in on most of the package mirrors when facing large-scale download issues.

That being explained, we had recent issues about the YUM repositories (#3183 and #3338) that shows we should work or challenge these initial assertions to work on the yum mirroring.
One of the main challenge regarding putting the whole YUM repo in the mirror grid (index and metadatas included) is that we cannot guarantuee the retention time of a given package as the mirrors are not managed by us.

=> There is definitievely something to be done to help you, but it requires some work.

A good first step would be to find a sponsor to help us setup an infrastructure inside China.

[ak1ra-komj]

Hi, @dduportal, thank you for such a detailed reply, I'll add some information here,

only pkg.jenkins.io is supported as a source for YUM packages.

I know that pkg.jenkins.io is currently the only supported apt/yum repo.

Interesting result. What error did you have? I'm interested in what is the problem with the tsinghua.edu.cn mirror

The TUNA mirrors sync is fine, it's just missing some necessary metadata files to be used as an apt/yum repo, as I said at the beginning. This can be reproduced in the following way.

In Debian 11, for example, when trying to replace https://pkg.jenkins.io/ with https://mirrors.tuna.tsinghua.edu.cn/jenkins/, it returns "The repository xxx does not have a Release file".

The first apt update attempt in the example below also shows a domain resolving failure when connecting directly to pkg.jenkins.io from mainland China.

root@2d35c1ad8482:~# gpg /usr/share/keyrings/jenkins-keyring.asc
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa4096 2023-03-27 [SC] [expires: 2026-03-26]
      63667EE74BBA1F0A08A698725BA31D57EF5975CA
uid           Jenkins Project <jenkinsci-board@googlegroups.com>
sub   rsa4096 2023-03-27 [E] [expires: 2026-03-26]

root@2d35c1ad8482:~# cat /etc/apt/sources.list.d/jenkins.list
deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/

root@2d35c1ad8482:~# apt update
... truncated ...
Err:10 https://pkg.jenkins.io/debian-stable binary/ InRelease
  Temporary failure resolving 'pkg.jenkins.io'
Fetched 8800 kB in 20s (438 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch https://pkg.jenkins.io/debian-stable/binary/InRelease  Temporary failure resolving 'pkg.jenkins.io'
W: Some index files failed to download. They have been ignored, or old ones used instead.

root@2d35c1ad8482:~# sed -i 's%https://pkg.jenkins.io/%https://mirrors.tuna.tsinghua.edu.cn/jenkins/%' /etc/apt/sources.list.d/jenkins.list
root@2d35c1ad8482:~# cat /etc/apt/sources.list.d/jenkins.list
deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://mirrors.tuna.tsinghua.edu.cn/jenkins/debian-stable binary/

root@2d35c1ad8482:~# apt update
... truncated ...
Ign:1 https://mirrors.tuna.tsinghua.edu.cn/jenkins/debian-stable binary/ InRelease
Err:3 https://mirrors.tuna.tsinghua.edu.cn/jenkins/debian-stable binary/ Release
  404  Not Found [IP: 101.6.15.130 443]
Reading package lists... Done
E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/jenkins/debian-stable binary/ Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

[ak1ra-komj]

...It's not as simple as "run a sync upstream mirror" as there are multiple security mechanisms at stake here to protect end users and avoid man-in-the-middle scenarios.
One of the main challenges regarding putting the whole YUM repo in the mirror grid (index and metadata included) is that we cannot guarantee the retention time of a given package as the mirrors are not managed by us.

I also considered the possibility of "man-in-the-middle" attacks on mirrors before submitting this issue. Comparing mirror sites across Linux distributions, the public key used to sign packages pre-installed in the OS solves the initial "trust" problem, whereas Jenkins pkg does not, and still requires a public key from somewhere, which is a real risk of MITM if it is a "third party" such as a mirror site. There's no good solution to this problem unless you require the public key to be retrieved from the jenkins.io domain.

As for the frequency of syncing, I think it should be the user's responsibility to choose a mirror site that is updated frequently enough. Currently mirrors status already has a last sync time, so users have a reason not to choose a mirror that doesn't update frequently enough.

...if you know any university, company, or organization who would be interested in helping us by running a mirror to improve the life of our China-located users, that would be awesome!

Interestingly, I found on MirrorZ.org that more than just TUNA mirrors are currently synchronizing with mirrors.jenkins-ci.org. They use a common upstream rsync://rsync.osuosl.org/jenkins/

[dduportal]

Thanks @ak1ra-komj for the feedbacks and explanations, it helps a lot!

We've discussed this subject during today's Jenkins Infrastructure weekly meeting:

• We need to extend the subject "improve China access to Jenkins downloads" to a broader community audience than only the infra team. For that we need a sponsor to help us (someone to work in pair with the infra team in China). Do you happen to know someone interested in that (or do you want to be that person)?
• We are going to check with Fastly if they have plans for covering China network. We are also studying alternative and/or complements such as Cloudflare. Please note this won't be immediate given our limited time (and the huge backlog we already have today) unless we get serious hands-on help
• The issue In reference to issue #3183 #3338 could help (if we can work on this soon) as it could ends up in defining full synced package repositories mirrors everywhere (and might lead to removing pkg.jenkins.io from fastly at all). Again it's an idea which needs to be worked on so no immediate actionnable.

Does it make sense for you?

[dduportal]

Hi @ak1ra-komj , just a quick check to share the status with you: we are working hard on #2649 that should help for both problems (update center index in China and the package indexes which are incomplete in the mirrors).

Please note it is a hard prerequisite: we need it before any other actions are started.

Once done, we see the following potential improvements:

• Update Center: we'll be able to host ourselves a copy of the index in the Cloudflare China network. We plan to use the same HTTP redirector system as get.jenkins.io to ensure that any request from China's network would be redirected to the Cloudflare China's mirror. As we (Jenkins infrasrtuctre will control these mirrors (different network than get.jenkins.io), that will drastically improve the situation for serving the update center. Does it make sense?

• Packages: Same idea: using a mirror redirector at pkg.jenkins.io to serve package index (while keeping heavy downloads to get.jenkins.io) would help providing different copies close to your network location.

[ak1ra-komj]

Hi @dduportal, sorry for the delay of reply,

Thanks to you and the jenkins-infra team for all your hard work, however, I noticed that Cloudflare China Network Access requires Enterprise plans in order to use it, so I'm not sure if this will cost the jenkins-infra team more than they budgeted for?

[dduportal]

Hi @dduportal, sorry for the delay of reply,

Thanks to you and the jenkins-infra team for all your hard work, however, I noticed that Cloudflare China Network Access requires Enterprise plans in order to use it, so I'm not sure if this will cost the jenkins-infra team more than they budgeted for?

We plan to ask for Open Source sponsorship. Otherwise we'll have to look to another mirror in China (even one we could host ourselves).

Good catch, thanks for the feedback!

[lemeurherve]

For the record, I've just applied to their sponsorship program for the Jenkins project via https://www.cloudflare.com/lp/oss-sponsorship/