MDBF-829 - Update MariaDB Server cnf #655

RazvanLiviuVarzaru · 2024-11-27T14:56:33Z

Main feature: MDBF-791 - Run Production BuildBot services in docker containers

In this Pull Request:

separate configs for PROD and DEV.
Updated docker-compose to choose the right one based on environment.
TODO: maybe a templating engine in the future to de-duplicate configuration settings.

Q:
-> @fauust how Restic backup will work for containers?
-> @grooverdan Couldn't find anything for slow query log in containers --> stdout (to be inspected with docker logs via journald driver). Anything like that possible? I've mounted a volume for these logs.

Source of Truth:
@fauust / @grooverdan For PROD configuration, well, my source of truth was hz-bbm1.
Here's the current prod layout, if you have any suggestions or I missed something, please let me know.

:/etc/mysql$ tree
.
├── conf.d
│   ├── mysql.cnf -> this is empty
│   └── mysqldump.cnf -> Settings described in [A]
├── debian.cnf -> I guess we dont need it: "THIS FILE IS OBSOLETE. STOP USING IT IF POSSIBLE"
├── debian-start -> Same as above
├── mariadb.cnf -> Settings described in [B]
├── mariadb.conf.d
│   ├── 50-client.cnf -> Nothing set here
│   ├── 50-mysql-clients.cnf -> Nothing set here
│   ├── 50-mysqld_safe.cnf ->  Settings described in [C], not included.
│   ├── 50-server.cnf -> Setting described in [D], not included.
│   ├── 60-galera.cnf -> Nothing set here
│   └── 99-enable-encryption.cnf.preset
│       └── enable_encryption.preset -> Not used
├── my.cnf -> mariadb.cnf
└── my.cnf.fallback -> Guess this is a backup file, only contains !includedir /etc/mysql/conf.d/

 [A] - Duplicate of [B]
    [mysqldump]
    quick
    quote-names
    max_allowed_packet	= 16M

[B]
    # Ansible managed

    [mariadb]

    # Basic settings
    user                  = mysql
    pid-file              = /run/mysqld/mysqld.pid
    socket                = /run/mysqld/mysqld.sock
    basedir               = /usr
    datadir               = /var/lib/mysql
    tmpdir                = /tmp
    lc-messages-dir       = /usr/share/mysql
    lc_messages           = en_US
    skip-external-locking
    port                  = 3306
    bind-address          = 0.0.0.0

    # Fine tuning
    max_connections         = 300
    connect_timeout         = 5
    wait_timeout            = 600
    max_allowed_packet      = 32M
    thread_cache_size       = 128
    sort_buffer_size        = 128M
    bulk_insert_buffer_size = 128M
    tmp_table_size          = 32M
    max_heap_table_size     = 32M
    skip-name-resolve       = 1
    innodb_log_file_size    = 2G

    # Logging
    # warning log-basename is defined later for replication
    # some log option may be ignored.
    # see: https://mariadb.com/kb/en/mysqld-options/#-log-basename
    # log_error = /var/log/mysql/error.log
    slow_query_log
    slow_query_log_file    = /var/log/mysql/mariadb-slow.log
    long_query_time        = 2
    log_slow_verbosity     = query_plan,explain
    log_error              = /var/log/mysql/mariadb.err.log

    # Query cache
    query_cache_size        = 16M

    # Character sets
    character-set-server = utf8mb4
    collation-server     = utf8mb4_general_ci

    # InnoDB
    # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
    # Read the manual for more InnoDB related options. There are many!
    innodb_buffer_pool_size = 40G

    # Replication
    server-id = 1
    log-basename = mariadb

    log_bin
    expire_logs_days = 5
    max_binlog_size = 100M
    binlog_format = MIXED
    # the following permits to simplify the process of moving a replica to a
    # primary node role by ensuring that replication is not started on primary node
    skip-slave-start

    [mysqldump]
    quick
    quote-names
    max_allowed_packet = 16M

[C]:
    [mysqld_safe]
    nice = 0
    skip_log_error
    syslog

[D]: Duplicate of [B] except expire_logs_days value
    pid-file                = /run/mysqld/mysqld.pid
    basedir                 = /usr
    expire_logs_days        = 10
    character-set-server  = utf8mb4
    collation-server      = utf8mb4_general_ci

fauust · 2024-11-27T15:14:59Z

-> @fauust how Restic backup will work for containers?

See example below:

faust@hz-downloads:~$ cat /etc/cron.d/docker-db-backup 
# backup db from the docker container
00 02 * * * root docker exec prod-db mariadb-backup --user=root --password=password --backup --stream=xbstream 2>/var/log/mariadb-backup.log >/backup/mariadb.xb || cat /var/log/mariadb-backup.log

And then, restic will backup /backup/mariadb.xb directory.

fauust

I am not sure that I understand how the dev configuration of mariadb is done, what is sure is that if it's a symlink we clearly can not afford same setting on DEV, way less RAM for instance...

docker-compose/docker-compose.yaml

fauust · 2024-11-27T15:29:21Z

Source of Truth:
@fauust / @grooverdan For PROD configuration, well, my source of truth was hz-bbm1.

The source of truth is:

default MDBF 10.11 configuration (on Debian);
with defaults https://github.com/fauust/ansible-role-mariadb/ ansible deployment and https://gitlab.com/mariadb/sysadmin/-/blob/main/ansible/host_vars/hz-bbm1.yml?ref_type=heads#L241-284.

RazvanLiviuVarzaru · 2024-11-27T15:30:24Z

-> @fauust how Restic backup will work for containers?

See example below:
faust@hz-downloads:~$ cat /etc/cron.d/docker-db-backup 
# backup db from the docker container
00 02 * * * root docker exec prod-db mariadb-backup --user=root --password=password --backup --stream=xbstream 2>/var/log/mariadb-backup.log >/backup/mariadb.xb || cat /var/log/mariadb-backup.log
And then, restic will backup /backup/mariadb.xb directory.

@fauust Don't we need a volume mount for backup?

RazvanLiviuVarzaru · 2024-11-27T15:33:34Z

I am not sure that I understand how the dev configuration of mariadb is done, what is sure is that if it's a symlink we clearly can not afford same setting on DEV, way less RAM for instance...

This is why there are 2 folders now, two separate configs.

RazvanLiviuVarzaru · 2024-11-27T15:36:29Z

@fauust I've pinned the server version here: 6b68cd2

So we control how minor upgrades are rolled out.
This is preventing a sudden pull of a newer version during a restart of the services.

fauust · 2024-11-27T15:42:44Z

@fauust Don't we need a volume mount for backup?

Nope, it use STDOUT (or SDTERR, ~~I can't remember exactly since there is a specific thing with mariadb-backup with regard to this~~ confirmed, see below). You can try on the DEV platform.

https://perconadev.atlassian.net/browse/PXB-1469

docker-compose/docker-compose.yaml

docker-compose/mariadb-config/prod/mariadb.cnf

docker-compose/generate-config.py

grooverdan · 2024-11-28T02:59:37Z

Source of Truth:
@fauust / @grooverdan For PROD configuration, well, my source of truth was hz-bbm1.

The source of truth is:
* default MDBF 10.11 configuration (on Debian);

Note https://github.com/MariaDB/server/blob/10.11/debian/additions/mariadb.conf.d/50-server.cnf is the current default.

* with defaults https://github.com/fauust/ansible-role-mariadb/ ansible deployment and https://gitlab.com/mariadb/sysadmin/-/blob/main/ansible/host_vars/hz-bbm1.yml?ref_type=heads#L241-284.

RazvanLiviuVarzaru · 2024-11-28T12:24:00Z

Source of Truth:
@fauust / @grooverdan For PROD configuration, well, my source of truth was hz-bbm1.

The source of truth is:
* default MDBF 10.11 configuration (on Debian);
Note https://github.com/MariaDB/server/blob/10.11/debian/additions/mariadb.conf.d/50-server.cnf is the current default.
* with defaults https://github.com/fauust/ansible-role-mariadb/ ansible deployment and https://gitlab.com/mariadb/sysadmin/-/blob/main/ansible/host_vars/hz-bbm1.yml?ref_type=heads#L241-284.

@grooverdan Does the fact that this configuration file is included (by /etc/mysql/mariadb.cnf), has any impact on us if we declare non-default values in conf.d/mariadb.cnf ?

For example we have global variables defined in conf.d/mariadb.cnf for all:
-> bind-address
-> expire_logs_days
-> character-set-server
-> collation-server

Except pid-file and basedir which I don't think matter much if they are default.

grooverdan · 2024-11-28T20:12:14Z

@grooverdan Does the fact that this configuration file is included (by /etc/mysql/mariadb.cnf), has any impact on us if we declare non-default values in conf.d/mariadb.cnf ?

For example we have global variables defined in conf.d/mariadb.cnf for all: -> bind-address -> expire_logs_days -> character-set-server -> collation-server

Except pid-file and basedir which I don't think matter much if they are default.

Your config file will take effect as its read last, just validating to be sure:

$ cat d/mariadb.cnf 
[mariadb]
expire-logs-days=30
(base) 
/tmp 
$ podman run -v ./d://etc/mysql/conf.d/:z mariadb:10.11 my_print_defaults --mariadbd
--socket=/run/mysqld/mysqld.sock
--host-cache-size=0
--skip-name-resolve
--pid-file=/run/mysqld/mysqld.pid
--basedir=/usr
--expire_logs_days=10
--character-set-server=utf8mb4
--collation-server=utf8mb4_general_ci
--expire-logs-days=30
(base) 
/tmp 
$ podman run -v ./d://etc/mysql/conf.d/:z mariadb:10.11 --help --verbose | grep expire
2024-11-28 20:04:41 0 [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text
  --binlog-expire-logs-seconds=# 
                      binlog_expire_logs_seconds seconds; It and
                      expire_logs_days are linked, such that changes in one are
  --disconnect-on-expired-password 
  --expire-logs-days=# 
                      expire_logs_days days; It and binlog_expire_logs_seconds
binlog-expire-logs-seconds                                   2592000
disconnect-on-expired-password                               FALSE
expire-logs-days                                             30

Noting that charset/collation is the default (for this image, changes in ~ 11.5/11.6). bind-address is the real default, listen to all interfaces, and not the debian default, because listen to all is the only sane default value for a container.

RazvanLiviuVarzaru · 2024-11-29T09:33:00Z

@grooverdan @fauust
I've implemented Daniel's suggestions here:
f2232f2

Please let me know if there is anything else to cover so I can merge rebase and merge this PR.

fauust · 2024-11-29T10:48:15Z

docker-compose/generate-config.py

@@ -41,10 +41,10 @@
      - MARIADB_AUTO_UPGRADE=1
    network_mode: host
    healthcheck:
-      test: ['CMD', "mariadb-admin", "--password=password", "--protocol", "tcp", "ping"]
+      test: ['CMD', "healthcheck.sh", "--connect", "--innodb_initialized"]
    volumes:
      - ./mariadb:/var/lib/mysql:rw


This should probably go to /srv/mariadb or somewhere else but should be outside docker-compose IMO.

Changed in 05c7c5f

I don't think we need to update rsync.exclude, we're deploying in /srv/{dev,prod}.

What's the approach to make this switch on DEV?
My plan is:
-> stop mariadb container
-> copy the datadir to /srv/mariadb/
-> merge this PR

Sounds OK?
@fauust

grooverdan · 2024-11-29T21:23:44Z

@grooverdan @fauust I've implemented Daniel's suggestions here: f2232f2

Please let me know if there is anything else to cover so I can merge rebase and merge this PR.

From previous:

All options that aren't innodb, log_slow_query, or replication based are either default or potentially harmful (like query_cache_size).

your max_connections is ok.

so removing, port, bind-address, connection_timeout through to skip-name-resolved, query_cache_size, [mysqldump]

max_allowed_packet and sort_buffer_size are above defaults, so conservativity you could keep those.

RazvanLiviuVarzaru · 2024-12-02T14:40:44Z

@grooverdan @fauust I've implemented Daniel's suggestions here: f2232f2
Please let me know if there is anything else to cover so I can merge rebase and merge this PR.

From previous:

All options that aren't innodb, log_slow_query, or replication based are either default or potentially harmful (like query_cache_size).

your max_connections is ok.

so removing, port, bind-address, connection_timeout through to skip-name-resolved, query_cache_size, [mysqldump]

max_allowed_packet and sort_buffer_size are above defaults, so conservativity you could keep those.

Solved in 6db3b51.
Let me know if it's OK. Thanks!

RazvanLiviuVarzaru · 2024-12-02T14:43:44Z

@fauust @grooverdan
I've also added a change to old_mode, to allow emoji's.
41e21fd

Details in: https://mariadb.zulipchat.com/#narrow/channel/236699-Buildbot/topic/MDBF-824.20tarball.20builder.20is.20on.20now.20EOL.20Debian.2010/near/485630218

docker-compose/mariadb-config/prod/mariadb.cnf

grooverdan · 2024-12-03T01:31:27Z

@grooverdan @fauust I've implemented Daniel's suggestions here: f2232f2
Please let me know if there is anything else to cover so I can merge rebase and merge this PR.

so removing, connection_timeout through to skip-name-resolved,..

max_allowed_packet and sort_buffer_size are above defaults, so conservativity you could keep those.

Solved in 6db3b51. Let me know if it's OK. Thanks!

though to was referring to everything in-between.

Can commit for now and look at query/system performance after the bunch of changes, including below, are deployed.

@fauust @grooverdan I've also added a change to old_mode, to allow emoji's. 41e21fd

Details in: https://mariadb.zulipchat.com/#narrow/channel/236699-Buildbot/topic/MDBF-824.20tarball.20builder.20is.20on.20now.20EOL.20Debian.2010/near/485630218

As you are changing the character set, may as well change the collation too at the same time collation-server=utf8mb4_uca1400_ai_ci.

Separate configs for PROD and DEV. - updated docker-compose to choose the right one based on environment. - old_mode = '' to allow utf8mb4 on the connection. Emoji's can now be inserted into buildbot.changes.comments - cleaned PROD configuration where default/harmful options were present. - pinned MariaDB Server version to 10.11.10 to be in control of minor upgrades. - switched to standard healthcheck script for containers - moved datadir to /srv/mariadb

RazvanLiviuVarzaru requested review from fauust, cvicentiu and grooverdan November 27, 2024 14:56

fauust reviewed Nov 27, 2024

View reviewed changes

docker-compose/docker-compose.yaml Outdated Show resolved Hide resolved

docker-compose/docker-compose.yaml Show resolved Hide resolved

grooverdan reviewed Nov 28, 2024

View reviewed changes

fauust reviewed Nov 29, 2024

View reviewed changes

fauust approved these changes Dec 2, 2024

View reviewed changes

docker-compose/mariadb-config/prod/mariadb.cnf Show resolved Hide resolved

RazvanLiviuVarzaru force-pushed the migration/mariadb-cnf branch from 05c7c5f to 35979cd Compare December 4, 2024 08:05

RazvanLiviuVarzaru force-pushed the migration/mariadb-cnf branch from 35979cd to 62483d0 Compare December 4, 2024 08:50

RazvanLiviuVarzaru merged commit 6e0999d into MariaDB:dev Dec 4, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MDBF-829 - Update MariaDB Server cnf #655

MDBF-829 - Update MariaDB Server cnf #655

RazvanLiviuVarzaru commented Nov 27, 2024 •

edited

Loading

fauust commented Nov 27, 2024

fauust left a comment

fauust commented Nov 27, 2024

RazvanLiviuVarzaru commented Nov 27, 2024

RazvanLiviuVarzaru commented Nov 27, 2024

RazvanLiviuVarzaru commented Nov 27, 2024 •

edited

Loading

fauust commented Nov 27, 2024 •

edited

Loading

grooverdan commented Nov 28, 2024

RazvanLiviuVarzaru commented Nov 28, 2024 •

edited

Loading

grooverdan commented Nov 28, 2024

RazvanLiviuVarzaru commented Nov 29, 2024

fauust Nov 29, 2024

RazvanLiviuVarzaru Dec 2, 2024 •

edited

Loading

grooverdan commented Nov 29, 2024

RazvanLiviuVarzaru commented Dec 2, 2024

RazvanLiviuVarzaru commented Dec 2, 2024

grooverdan commented Dec 3, 2024

MDBF-829 - Update MariaDB Server cnf #655

MDBF-829 - Update MariaDB Server cnf #655

Conversation

RazvanLiviuVarzaru commented Nov 27, 2024 • edited Loading

fauust commented Nov 27, 2024

fauust left a comment

Choose a reason for hiding this comment

fauust commented Nov 27, 2024

RazvanLiviuVarzaru commented Nov 27, 2024

RazvanLiviuVarzaru commented Nov 27, 2024

RazvanLiviuVarzaru commented Nov 27, 2024 • edited Loading

fauust commented Nov 27, 2024 • edited Loading

grooverdan commented Nov 28, 2024

RazvanLiviuVarzaru commented Nov 28, 2024 • edited Loading

grooverdan commented Nov 28, 2024

RazvanLiviuVarzaru commented Nov 29, 2024

fauust Nov 29, 2024

Choose a reason for hiding this comment

RazvanLiviuVarzaru Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

grooverdan commented Nov 29, 2024

RazvanLiviuVarzaru commented Dec 2, 2024

RazvanLiviuVarzaru commented Dec 2, 2024

grooverdan commented Dec 3, 2024

RazvanLiviuVarzaru commented Nov 27, 2024 •

edited

Loading

RazvanLiviuVarzaru commented Nov 27, 2024 •

edited

Loading

fauust commented Nov 27, 2024 •

edited

Loading

RazvanLiviuVarzaru commented Nov 28, 2024 •

edited

Loading

RazvanLiviuVarzaru Dec 2, 2024 •

edited

Loading