Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[metricbeat][mysql] Add SSL support #37997

Merged
merged 17 commits into from
Mar 15, 2024
Merged

Conversation

gpop63
Copy link
Contributor

@gpop63 gpop63 commented Feb 13, 2024

Overview

Adds SSL support for following metricsets:

  • query
  • performance
  • status
  • galera_status

The query metricset uses a different SQL client (helper/sql). Others use the client from the module itself.

Added a config struct as part of this change.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

genroot.sh

OPENSSL_ROOT_CA=$1

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl genrsa 2048 > certs/root-ca-key.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl req -new -x509 -nodes -days 3600 \
        -subj "${OPENSSL_ROOT_CA}" \
        -key /certs/root-ca-key.pem -out /certs/root-ca.pem

genserver.sh

OPENSSL_SERVER=$1

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl req -newkey rsa:2048 -days 3600 -nodes \
        -subj "${OPENSSL_SERVER}" \
        -keyout /certs/server-key.pem -out /certs/server-req.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl rsa -in /certs/server-key.pem -out /certs/server-key.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl x509 -req -in /certs/server-req.pem -days 3600 \
        -CA /certs/root-ca.pem -CAkey /certs/root-ca-key.pem \
        -set_serial 01 -out /certs/server-cert.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl verify -CAfile /certs/root-ca.pem /certs/server-cert.pem

genclient.sh

OPENSSL_CLIENT=$1

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl req -newkey rsa:2048 -days 3600 -nodes \
        -subj "${OPENSSL_CLIENT}" \
        -keyout /certs/client-key.pem -out /certs/client-req.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl rsa -in /certs/client-key.pem -out /certs/client-key.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl x509 -req -in /certs/client-req.pem -days 3600 \
        -CA /certs/root-ca.pem -CAkey /certs/root-ca-key.pem \
        -set_serial 01 -out /certs/client-cert.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl verify -CAfile /certs/root-ca.pem /certs/client-cert.pem

gencerts.sh

mkdir -p certs

OPENSSL_SUBJ="/C=US/ST=California/L=Santa Clara"
OPENSSL_CA="${OPENSSL_SUBJ}/CN=fake-CA"
OPENSSL_SERVER="${OPENSSL_SUBJ}/CN=fake-server"
OPENSSL_CLIENT="${OPENSSL_SUBJ}/CN=fake-client"

sh ./genroot.sh "${OPENSSL_CA}"
sh ./genserver.sh "${OPENSSL_SERVER}"
sh ./genclient.sh "${OPENSSL_CLIENT}"

Then run sh +x gencerts.sh to generate required certificates.

Update certificates ownership and permissions so the mysql container can use them.

sudo chown :docker certs/*
sudo chmod 755 certs/*
docker-compose.up.yml

version: '3.8'
services:
  mysql:
    image: mysql/mysql-server:8.0.21
    command: [ "mysqld",
                    "--character-set-server=utf8mb4",
                    "--collation-server=utf8mb4_unicode_ci",
                    "--bind-address=0.0.0.0",
                    "--require_secure_transport=ON",
                    "--ssl-ca=/etc/certs/root-ca.pem",
                    "--ssl-cert=/etc/certs/server-cert.pem",
                    "--ssl-key=/etc/certs/server-key.pem",
                    "--default_authentication_plugin=mysql_native_password" ]
    environment:
      MYSQL_ROOT_PASSWORD: "example"
      MYSQL_ROOT_HOST: "%"
    restart: always
    volumes:
      - type: bind
        source: ./certs
        target: /etc/certs/
    ports:
      - 3306:3306

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 13, 2024
@mergify mergify bot assigned gpop63 Feb 13, 2024
Copy link
Contributor

mergify bot commented Feb 13, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @gpop63? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@gpop63 gpop63 added the Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team label Feb 13, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 13, 2024
@elasticmachine
Copy link
Collaborator

elasticmachine commented Feb 13, 2024

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2024-03-12T08:56:27.566+0000

  • Duration: 55 min 49 sec

Test stats 🧪

Test Results
Failed 0
Passed 4567
Skipped 905
Total 5472

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@gpop63 gpop63 changed the title [metricbeat] [mysq] Add SSL support [metricbeat] [mysql] Add SSL support Feb 14, 2024
@gpop63 gpop63 marked this pull request as ready for review February 14, 2024 10:29
@gpop63 gpop63 requested review from a team as code owners February 14, 2024 10:29
Copy link
Member

@shmsr shmsr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look more or less good only but I feel I need to do another comprehensive review. I'll do it ASAP.

metricbeat/module/mysql/mysql.go Show resolved Hide resolved
metricbeat/module/mysql/config.go Outdated Show resolved Hide resolved
metricbeat/module/mysql/mysql.go Outdated Show resolved Hide resolved
@shmsr
Copy link
Member

shmsr commented Feb 14, 2024

Can you share the minimum steps to test the changes made in the "How to test this PR locally" section of the PR description?

@pierrehilbert pierrehilbert added the Team:Elastic-Agent Label for the Agent team label Feb 14, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved
metricbeat/module/mysql/config.go Outdated Show resolved Hide resolved
gpop63 and others added 3 commits February 14, 2024 20:48
@gpop63
Copy link
Contributor Author

gpop63 commented Feb 15, 2024

/test

@@ -31,7 +31,7 @@ import (
func TestNewDB(t *testing.T) {
service := compose.EnsureUp(t, "mysql")

db, err := NewDB(GetMySQLEnvDSN(service.Host()))
db, err := NewDB(GetMySQLEnvDSN(service.Host()), nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we introduce a test that validates the added SSL support?

@shmsr shmsr changed the title [metricbeat] [mysql] Add SSL support [metricbeat][mysql] Add SSL support Feb 16, 2024
Copy link
Member

@shmsr shmsr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good. Are you planning to add integration tests (what Denis also asked)?

@gpop63 gpop63 force-pushed the add_mysql_tls_support branch from 4fe8058 to 162a21c Compare February 27, 2024 22:07
@gpop63 gpop63 requested a review from shmsr March 4, 2024 14:08
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @gpop63

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @gpop63

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @gpop63

@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 12, 2024

@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 12, 2024

💔 Build Failed

Failed CI Steps

History

cc @gpop63

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @gpop63

@gpop63 gpop63 merged commit 9172e9c into elastic:main Mar 15, 2024
40 of 47 checks passed
#ssl.certificate: "/etc/pki/client/cert.crt"

# Client certificate key file
#ssl.key: "/etc/pki/client/cert.key"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pasted text should've had a new line character at the end. The missing character led to the broken documentation. See the fixing PR #38367

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Team:Elastic-Agent Label for the Agent team Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Metricbeat] Add TLS/SSL configuration options in the MySQL module
6 participants