Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate linux builder OS from CentOS 7 to RockyLinux 8 #381

Merged
merged 12 commits into from
Sep 2, 2024

Conversation

wuhuizuo
Copy link
Contributor

Fixes #379

@ti-chi-bot ti-chi-bot bot requested a review from purelind August 13, 2024 14:06
Copy link

ti-chi-bot bot commented Aug 13, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

This pull request is titled "Migrate linux builder OS from CentOS 7 to RockyLinux 8". The main changes in this pull request are updating the base image for the Dockerfiles in the cd/builders directory from centos:7.9.2009 to quay.io/rockylinux/rockylinux:8.10.20240528. The PR description indicates that this change is addressing issue #379.

There do not appear to be any obvious problems with this pull request. The changes are straightforward and appear to be well-contained within the affected Dockerfiles.

However, there are a couple of suggestions for improvement:

  • The Dockerfiles could benefit from some additional documentation to explain the reasoning behind the migration from CentOS 7 to RockyLinux 8, as well as any potential implications for users of the resulting images.
  • The tidb-dashboard Dockerfile includes an update to Node.js version 16.x, which may introduce breaking changes for users who rely on specific dependencies or features. It may be worth considering a more conservative update strategy, such as moving to Node.js version 14.x initially and allowing users to opt-in to 16.x if desired.

Overall, this pull request appears to be a straightforward and well-executed migration from CentOS 7 to RockyLinux 8.

@ti-chi-bot ti-chi-bot bot added the size/L label Aug 13, 2024
Copy link

ti-chi-bot bot commented Aug 13, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

This pull request migrates the Linux builder OS from CentOS 7 to RockyLinux 8 for several Dockerfiles. It replaces the CentOS base image with the RockyLinux base image for each Dockerfile. The new image is used to build the binaries for TiDB, TiKV, TiFlash, PD, TiDB Dashboard, and TiFlow. It also updates the Golang version from 1.21.6 to 1.21.13 for the PD Dockerfile.

Overall, this pull request looks good, and the changes seem straightforward. However, there are a few potential problems that need to be addressed before merging this pull request.

First, the new OS version may affect the dependencies required by the projects. The dependencies may be different from those used in CentOS 7. Therefore, it is necessary to ensure that the new dependencies are compatible with the projects.

Second, the build process may take longer due to the new base image's size or different package management tools. The build time should be monitored to ensure that it does not significantly increase.

Third, the new image may introduce new security vulnerabilities that were not present in CentOS 7. Therefore, it is essential to perform a security audit on the new image.

Finally, we need to ensure that the Dockerfiles are backward compatible with the CentOS 7 base image. If any issues arise, we should fix them before merging this pull request.

To address these potential problems, I suggest the following:

  1. Review the dependencies required by the projects and ensure that they are compatible with the new OS version.

  2. Monitor the build process and optimize it if necessary.

  3. Perform a security audit on the new image.

  4. Test backward compatibility with the CentOS 7 base image and fix any issues that arise.

Once these issues are addressed, this pull request should be ready for merge.

@wuhuizuo
Copy link
Contributor Author

/just msb-tidb

Fixes #379

Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Copy link

ti-chi-bot bot commented Aug 13, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request proposes to migrate the base operating system of the builder images from CentOS 7 to Rocky Linux 8. The changes are made to the Dockerfiles of several builder images by updating the base image to use Rocky Linux 8 and making minor adjustments to the packages that are installed.

Potential problems:

  • The changes may introduce compatibility issues with the existing codebase, especially if there are any dependencies that are not compatible with Rocky Linux 8.
  • The changes may cause issues with the tooling used in the CI/CD pipeline if they are not compatible with Rocky Linux 8.

Fixing suggestions:

  • Test the changes thoroughly to ensure compatibility with the existing codebase and tooling.
  • Use a phased approach to the migration, starting with a small subset of the builder images and gradually expanding to cover all images.
  • Monitor the CI/CD pipeline closely after the changes are deployed to catch any issues early on.

Overall, the changes proposed in the pull request seem reasonable, but caution should be exercised to ensure a smooth transition.

@wuhuizuo
Copy link
Contributor Author

/just msb-tidb

2 similar comments
@wuhuizuo
Copy link
Contributor Author

/just msb-tidb

@wuhuizuo
Copy link
Contributor Author

/just msb-tidb

Signed-off-by: wuhuizuo <wuhuizuo@126.com>

Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Copy link

ti-chi-bot bot commented Aug 14, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request titled Migrate linux builder OS from CentOS 7 to RockyLinux 8 is aimed at migrating the builder operating system from CentOS 7 to RockyLinux 8. This migration is important as CentOS 7 has reached its end of life.

Key Changes:

  1. The operating system in various Dockerfiles has been changed from CentOS 7 to RockyLinux 8.
  2. The repository for RockyLinux 8 has been set to quay.io/rockylinux/rockylinux:8.10.20240528.
  3. The package installation commands have been changed to use dnf, the default package manager in RockyLinux, instead of yum.
  4. The conditions in the Github Action file have been changed to look for commands in the comment body instead of the issue body.
  5. The Golang version has been updated in a couple of Dockerfiles.
  6. The Tikv Dockerfile for FIPS has been updated to export JEMALLOC_SYS_WITH_LG_PAGE=16 if the architecture is aarch64.
  7. The tiflash/Dockerfile now has labels added.

Potential Problems:

  1. No checks are made to ensure that all required dependencies are available in the new OS image. Some packages may not be available in the new OS or may behave differently.
  2. The JEMALLOC_SYS_WITH_LG_PAGE=16 environment variable is set only for aarch64 architecture. If this is required for other architectures, it could cause problems.
  3. The changed conditions in the Github Action file now only look for commands in comments. If commands are still provided in the issue body, they would be ignored.

Fixing Suggestions:

  1. Test the build process thoroughly to make sure all dependencies are properly resolved and the build process works as expected in the new OS.
  2. If the JEMALLOC_SYS_WITH_LG_PAGE=16 environment variable is required for other architectures, modify the script accordingly.
  3. Update the documentation to reflect the change in how commands are processed by the Github Action.

…ntos 7 to rocky linux 8

The ones are compatible with centos 7
- ng-monitoring
- tidb
- tiflow

The ones are NOT compatible with centos 7
- pd
- tidb-dashboard

Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Copy link

ti-chi-bot bot commented Aug 14, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

This pull request aims to migrate the Linux builder OS from CentOS 7 to RockyLinux 8. Here are the key changes:

  1. The base OS in the Dockerfiles is changed from CentOS 7 to RockyLinux 8.
  2. The yum package manager commands have been replaced with dnf package manager commands.
  3. The golang version and pnpm version in some Dockerfiles have been updated.
  4. The action triggers in Github Actions workflow are updated to only run on issue comment created, and the tests are triggered based on the comment body content.
  5. Some changes in build steps and artifacts in the package.yaml.tmpl file.

Here are potential problems and suggestions:

  1. Missing Repository: The new base image, quay.io/rockylinux/rockylinux:8.10.20240528, might not be available or accessible. The accessibility and existence of the repository should be confirmed.
  2. Dependency Issues: The migration from CentOS 7 to RockyLinux 8 may cause dependency issues due to the changes in the underlying system libraries and packages. Therefore, it's essential to test the new system thoroughly.
  3. The Golang version has been changed in the Dockerfiles. This might lead to unexpected behavior if the application hasn't been tested with the new version.
  4. The PR doesn't contain any test results, it's recommended to include this information to ensure all test cases are passed after changes.
  5. The Github Actions only trigger on issue comment created, which reduces the chance of the actions running accidentally. However, if there is any need to trigger the actions on other events like issue comment edited or deleted, this change will prevent it.
  6. The change in the package.yaml.tmpl file might affect the building process or the built artifacts. Make sure the changes align with the building and deploying process.

Overall, I recommend that the changes are thoroughly tested to ensure stability and compatibility with the new OS and the updated dependencies. Make sure the changes in the Github Actions and package.yaml.tmpl file align with your CI/CD process.

…om centos 7 to rocky linux 8

use the ubi edition base image.
@ti-chi-bot ti-chi-bot bot added size/XL and removed size/L labels Aug 14, 2024
Copy link

ti-chi-bot bot commented Aug 14, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

Title: Migrate linux builder OS from CentOS 7 to RockyLinux 8

Key Changes:

  1. The base builder image is changed from CentOS 7 to RockyLinux 8.
  2. The builder profiles devtoolset8, devtoolset9, and devtoolset10 are removed from the GitHub workflow files pull-cd-builder-images.yaml and release-cd-builder-images.yaml.
  3. In the pull-cd-builder-images2.yaml workflow file, the event types are changed from [created, edited, deleted] to [created]. The command to checkout the pull request is changed from hub pr checkout to gh pr checkout. The conditions to run tests for different modules are altered to check the comment body rather than the issue body.
  4. In various Dockerfile files, the base image is changed from CentOS to RockyLinux, and several other modifications are made such as changing the installation of packages to use dnf instead of yum, updating versions of some packages, and altering several build commands.
  5. In skaffold.yaml, the patches for devtoolset8, devtoolset9, and devtoolset10 are removed.
  6. In packages.yaml.tmpl, the build step for fips profile is updated to set JEMALLOC_SYS_WITH_LG_PAGE=16 on aarch64 and the Dockerfile for fips profile is changed to another file.

Potential Problems:

  1. Since the base OS is changed to Rocky Linux, there may be compatibility issues with the existing code or dependencies.
  2. The removal of the devtoolset profiles may affect the building process if any part of the code relies on these profiles.
  3. The changes in the GitHub workflows may affect the CI/CD process.

Fixing Suggestions:

  1. Thoroughly test the new base OS to ensure all dependencies and packages work as expected.
  2. Check whether the removed devtoolset profiles are used anywhere in the code. If they are, provide an alternative or reintroduce these profiles.
  3. Verify that the modified GitHub workflows are functioning as expected.

Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Copy link

ti-chi-bot bot commented Aug 14, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

This PR is about migrating the Linux builder OS from CentOS 7 to Rocky Linux 8.

Key changes of this PR include:

  1. Changes in OS in different Dockerfiles from CentOS 7 to Rocky Linux 8.
  2. Removal of devtoolset from .github/workflows/pull-cd-builder-images.yaml and .github/workflows/release-cd-builder-images.yaml.
  3. Changes to the Github Actions workflow to update how the actions trigger and handle issue comments.
  4. Changes to the builder images and the build steps in the Dockerfiles.
  5. Update of some software versions like Node.js, pnpm, and Golang in some Dockerfiles.
  6. Changes in the skaffold.yaml file by removing the devtoolset related profiles.
  7. Changes in the packages.yaml.tmpl file, mainly regarding the Docker image building process.

Potential problems:

  1. Packages that depend on devtoolset might face compatibility issues since it is removed.
  2. The new OS and updated software versions might have compatibility issues with the existing system.
  3. The changes in Github Actions workflow might affect the CI/CD process.
  4. Changes in the Dockerfile might affect the build process and the built images.
  5. The changes in packages.yaml.tmpl might affect the package building process.

Fixing suggestions:

  1. Test the build process thoroughly to ensure the removal of devtoolset doesn't cause any issues.
  2. Test the system thoroughly with the new OS and the updated software versions.
  3. Monitor the Github Actions workflow to ensure it works as expected.
  4. Test the build process and the built images thoroughly to ensure they work as expected.
  5. Test the changes in packages.yaml.tmpl to ensure the package building process works as expected.

@wuhuizuo
Copy link
Contributor Author

/just msb-tikv

@wuhuizuo
Copy link
Contributor Author

Copy link

ti-chi-bot bot commented Aug 14, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

Key Changes:

  1. The builder OS for Linux is migrated from CentOS 7 to Rocky Linux 8.
  2. The devtoolset8, devtoolset9, and devtoolset10 profiles are removed from the GitHub Actions jobs.
  3. The comment event types are adjusted to only consider 'created' events.
  4. The hub pr checkout command is replaced by gh pr checkout.
  5. The conditions for starting the tests for each module in the GitHub Actions workflow are changed to check the comment body instead of the issue body.
  6. The Dockerfiles for various builders are updated to use the new Rocky Linux base image and corresponding package manager commands. The labels in the Dockerfiles are also updated.
  7. The 'tiflow' Dockerfile is updated to build with FIPS support.

Potential Problems:

  1. The change from CentOS 7 to Rocky Linux 8 might introduce incompatibilities or unexpected behavior due to differences in package versions, system libraries, or other platform-specific details.
  2. The removal of devtoolset8, devtoolset9, and devtoolset10 profiles might impact any components that depend on these specific developer toolsets.
  3. The change to consider only 'created' comment events might miss edits or deletions that were previously triggering actions.
  4. The command gh pr checkout requires the GitHub CLI to be installed and authenticated. If it's not set up, this command will fail.
  5. Modifying the Dockerfile might introduce issues with the docker image, such as missing dependencies or incompatibilities with the base image.

Fixing Suggestions:

  1. Thoroughly test the new Rocky Linux environment to ensure that all components function as expected.
  2. Verify that no components are adversely affected by the removal of devtoolset8, devtoolset9, and devtoolset10 profiles.
  3. If edits or deletions of comments should trigger actions, adjust the event types accordingly.
  4. Make sure that GitHub CLI is installed and authenticated in the environment where the gh pr checkout command is run.
  5. Test the docker build and run processes thoroughly to ensure all components work as expected inside the Docker container.

Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Copy link

ti-chi-bot bot commented Aug 15, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

Key Changes:

  1. The builder OS for the Linux environment is being migrated from CentOS 7 to RockyLinux 8.
  2. The CentOS 7 references in the Dockerfiles for ng-monitoring, pd, tidb-dashboard, tidb-operator, tidb, tiflash, tiflow, and tikv have been replaced with RockyLinux 8.
  3. Some of the packages installed in the Dockerfiles have been updated.
  4. The builder profiles for devtoolset8, devtoolset9, and devtoolset10 have been removed from the GitHub workflows and Skaffold configuration.
  5. The changes to the GitHub workflows also include adjustments to when certain jobs are triggered, and how the PR is checked out.
  6. Other minor updates and cleanups have also been made to the Dockerfiles and workflows.

Potential Problems:

  1. The new base OS, RockyLinux 8, could potentially lead to compatibility issues if certain packages or libraries relied on by the project are not available or have different versions in RockyLinux 8 compared to CentOS 7. It is important to test the newly built images thoroughly in the target environment.
  2. The removal of the devtoolset builder profiles could impact any processes that rely on these specific toolsets. If these were used for specific build or compilation processes, their absence could lead to build failures or compatibility issues.
  3. The changes to the GitHub workflows could lead to unexpected behavior in the CI/CD processes. For example, the changes to when certain jobs are triggered could impact the order of operations or dependencies between jobs. The change from 'hub pr checkout' to 'gh pr checkout' may affect how the PR is checked out if the environment or permissions are not configured correctly for the 'gh' command.

Fixing Suggestions:

  1. Thoroughly test the new images built with RockyLinux 8 in the target environment to ensure compatibility and correct operation. Address any compatibility issues that arise.
  2. If the devtoolset profiles were necessary, consider adding them back or finding suitable replacements in the RockyLinux environment.
  3. Verify the changes to the GitHub workflows to ensure they perform as expected. Test the workflows in different scenarios to confirm correct operation. Verify that the 'gh' command is properly configured in the GitHub actions environment.

Fixes #379

Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Copy link

ti-chi-bot bot commented Aug 21, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

This pull request titled Migrate linux builder OS from CentOS 7 to RockyLinux 8 is about changing the operating system on which the builder is running.

Key changes include:

  1. Modification of the Github workflows (pull-cd-builder-images.yaml, pull-cd-builder-images2.yaml, and release-cd-builder-images.yaml), where the strategy matrix got updated and devtoolset related settings got removed.
  2. Modification of Dockerfiles for various components (ng-monitoring, pd, tidb-dashboard, tidb-operator, tidb, tiflash, tiflow, tikv) where the base image was changed from centos:7.9.2009 to rockylinux:8.10.20240528 and concomitant changes in the package installation commands and build instructions.
  3. Modification of skaffold.yaml where devtoolset related settings got removed.
  4. Update in packages/packages.yaml.tmpl where tikv dockerfile is changed to tikv-fips.Dockerfile when the profile is fips.

Potential problems:

  1. The build process might fail due to differences in the package manager or available packages in the new OS.
  2. There could be compatibility problems with the new OS. The applications might behave differently on the new OS due to differences in system libraries or system services.
  3. The removal of devtoolset might affect the build process if it was required for any component.

Fixing suggestions:

  1. Make sure that all necessary packages are installed correctly in the new OS. If any package is missing or not available, find an alternative or add the necessary repositories to the package manager.
  2. Perform thorough testing to ensure that all applications work as expected on the new OS.
  3. If devtoolset was necessary, try to find a way to install it in the new OS, or find an alternative.

Copy link

ti-chi-bot bot commented Aug 27, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request titled "Migrate linux builder OS from CentOS 7 to RockyLinux 8" aims to update the OS for the Linux builder from CentOS 7 to RockyLinux 8. Below are the main changes:

  1. Updating the GitHub workflow files: The pull request modifies '.github/workflows/pull-cd-builder-images.yaml' and '.github/workflows/release-cd-builder-images.yaml'. The specific 'devtoolset' profiles are removed from the matrix strategy, and the 'builder-others-slow' module is added back.

  2. Updating Dockerfiles: The base image for all Dockerfiles in this PR is changed from CentOS 7 to RockyLinux 8. In addition, there are changes to the installation of various dependencies, and changes in the build process in some Dockerfiles.

Potential problems:

  1. Missing verification of new base image: While the PR updates the base image, there is no explicit verification to ensure that all dependencies and applications are working as expected with the new OS.

  2. Updating the workflow configurations: The PR makes significant changes to the GitHub workflow files. These changes need to be tested thoroughly to ensure they don't break the existing CI/CD processes.

  3. Inconsistencies in the Dockerfiles: There are changes in the build process in some Dockerfiles but not in others. This can potentially lead to inconsistencies in the built images.

Fixing suggestions:

  1. Testing: After updating the base image, thorough testing should be performed to ensure all applications and dependencies are working as expected. This can be done through unit tests, integration tests, and manual testing if necessary.

  2. Thorough review of the workflow changes: A thorough review should be performed on the changes in the GitHub workflow files to ensure they don't break existing processes.

  3. Consistent Dockerfiles: Ensure that the build processes in all Dockerfiles are consistent to avoid potential issues in the future. If there are changes necessary in some Dockerfiles, consider if these changes should be applied to all Dockerfiles.

@ti-chi-bot ti-chi-bot bot removed the needs-rebase label Aug 28, 2024
…c++ install into /usr/lib64

Signed-off-by: wuhuizuo <wuhuizuo@126.com>
Copy link

ti-chi-bot bot commented Sep 2, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

The pull request aims to migrate the Linux builder OS from CentOS 7 to RockyLinux 8 in the CI/CD pipeline.

Key Changes:

  • The CentOS 7 based images in Dockerfiles are replaced with RockyLinux 8.
  • The installation of necessary packages in Dockerfiles are updated to use the dnf package manager.
  • The devtoolset-8, devtoolset-9, and devtoolset-10 specific profiles in the GitHub actions workflow file and skaffold.yaml file are removed.
  • The builder profile of the matrix configuration in the GitHub actions workflow file is updated from local-docker to devtoolset8,local-docker.
  • The nodejs and pnpm versions are updated in the Dockerfile.
  • The Golang version is updated to v1.21.13 in some Dockerfiles.
  • The builder-others-slow module is added to the matrix configuration in the GitHub actions workflow file.

Potential Problems:

  • The removed devtoolset specific profiles might have been used for testing the build with different GCC versions in CentOS. This might lead to less coverage in testing.
  • Updating the nodejs and pnpm versions might introduce compatibility issues with the existing code.
  • The Golang version update to v1.21.13 might introduce compatibility issues with the existing code.
  • The addition of the builder-others-slow module might increase the total build time in the CI/CD pipeline.

Fixing Suggestions:

  • If the removed devtoolset profiles are necessary, consider finding a similar solution for RockyLinux.
  • Ensure that the updated nodejs and pnpm versions are compatible with the existing code.
  • Test the compatibility of the updated Golang version with the existing code.
  • If the addition of the builder-others-slow module increases the total build time significantly, consider removing it or optimizing the build process.

@wuhuizuo
Copy link
Contributor Author

wuhuizuo commented Sep 2, 2024

We should merge it after we create a new repo release.

@wuhuizuo
Copy link
Contributor Author

wuhuizuo commented Sep 2, 2024

/approve

Copy link

ti-chi-bot bot commented Sep 2, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Sep 2, 2024
@ti-chi-bot ti-chi-bot bot merged commit 8a12cd0 into main Sep 2, 2024
19 checks passed
@ti-chi-bot ti-chi-bot bot deleted the wuhuizuo/issue379 branch September 2, 2024 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate linux builder OS from CentOS 7 to RockyLinux 8
1 participant