Merge pull request #1 from eliba/remote-improvements

Remote improvements

README.md

@@ -1,39 +1,56 @@
 # Description
-This role configures automated backups for remote machines which cannot reach the backup server directly, e.g. due to firewalls, NAT, etc. The backup server must be able to reach the remote client via SSH.
+This role configures automated backups for remote backup clients which cannot reach the backup server directly, e.g. due to firewalls, NAT, etc. The backup server must be able to reach the remote client via SSH.
 
-For this to work, an SSH connection is initiated by a systemd service `restic-remote-backup@.service` on the backup server each night, forwarding the `rest-server` port to the remote client and then invoking the `restic-backup.service` there.
+This role configures both restic remote clients (machines that need files backed up) and the restic REST server (which hosts the restic repositories). Once this role is deployed, the backup server starts a systemd service `restic-remote-backup@<remote-client>.service`, which initiates an SSH connection, forwarding the backup server's port to the remote restic client via SSH and then invoking the `restic-backup.service` there.
+
+<details><p><summary>Flowchart</summary>
+
+```mermaid
+sequenceDiagram
+    Backup Server->>+Backup Server: systemctl start restic-remote-backup@Client.service
+    Backup Server->>+Remote Client: Open SSH tunnel, forward REST server port 3000,<br>execute sudo systemctl start restic-backup.service
+
+    Remote Client->>+restic-backup.service: Starting restic-backup.service - Restic backup...
+    restic-backup.service->>restic-backup.service: restic init ...<br>restic backup ...
+    restic-backup.service->>Remote Client: restic: open repository at rest:http(s)://user:pass@Server:3000/Client
+    Remote Client->>Remote Client: "Server" always resolves to IP 127.0.0.1
+    restic-backup.service->>Backup Server: using parent snapshot d289ee09<br>...<br>snapshot fa15be6f saved
+    restic-backup.service->>-Remote Client: restic-backup.service: Deactivated successfully.
+    Remote Client->>-Backup Server: Finished restic-backup.service - Restic backup.
+    Backup Server->>-Backup Server: restic-remote-backup@Client.service: Succeeded.
+```
+</p></details>
+
+You can regard this role as kind of an extension to the `fifty2technology.restic_client` role, as they are both integrated into each other, but it once made more sense to unload the `restic_client` role and separate the "remote client" functionality from it. It is also coupled with the `fifty2technology.restic_server` role, since the backup server needs to be configured to connect to remote clients - this means it can only work with clients with a [restic REST server](https://github.com/restic/rest-server) account configured as a repository.
 
 Once a week (on saturday night) the backup server also connects to each remote client and runs the `restic-prune.service`.
 
-For monitoring purposes, the backup server connects to each remote machine once every day via `restic-remote-exporter@.service` to gather metrics about the latest backup snapshot. This cannot be done by the client itself since it requires access to the restic repository on the server.
+For monitoring purposes and if configured, the backup server connects to each remote machine once every day via `restic-remote-exporter@.service` to gather metrics about the latest backup snapshot. This cannot be done by the client itself since it requires access to the restic repository on the server.
 
 # Requirements
-* The variable `restic_client_is_remote` must be set to `true` to enable remote backups.
+* The variable `restic_client_is_remote` must be set to `true` to enable remote backups. It must also be set during `restic_client` role deployment so it configures and activates systemd services correctly.
 * `backup_server` must be set to the `inventory_hostname` of the backup server, e.g. `backupserver.example.com`.
-* `backup_server_password` must contain the login credentials for the backup server.
-* `backup_server_become_password` must contain the privilege escalation password for the unprivileged user on the backup server.
+* `backup_server_password` must contain the login credentials for the backup server if SSH password-authentication is used. With key-based authentication, this can be left undefined (will default to empty string).
+* `backup_server_become_password` must contain the privilege escalation password for the unprivileged user on the backup server. The same as above applies here in case of key-based authentication.
 
-The `backup_server`\* variables are necessary so that `delegate_to` tasks to the backup server are run correctly. It shouldn't be necessary if you use SSH login with keys and/or as privileged user.
+The `backup_server_*password` variables are necessary for password-only authentication, so that `delegate_to` tasks to the backup server are run correctly. It is not necessary if you use SSH login with keys and/or as privileged user.
 
 # Role Variables
 All variables which can be overridden are stored in defaults/main.yml file as well as in table below.
 
 | Name | Default value | Description |
 | ------ | ------ | ----- |
-| `restic_remote_server_authorized_key` | "" | **Required** SSH public key of the backup server. This key will be put into the client's `~/.ssh/authorized_keys` to enable the server to connect. Only used when `restic_client_is_remote: true`. |
-| `restic_remote_client_rest_password` | `"{{ restic_client_rest_password \| default( lookup('passwordstore', 'IT/backup/auth/' + inventory_hostname) ) }}"` | Read REST authentication password from the passwordstore. If it does not exist yet, let the relevant task fail. This password should have been generated by the `restic_client` role, which is a prerequisite for this role. |
 | `restic_remote_server_user` | `"{{ restic_server_user \| default('rest-server') }}"` | Username to use for initiating SSH connections as from server to client |
 | `restic_remote_server_group` | `"{{ restic_server_group \| default('rest-server') }}"` | Groupname used to assert that a SSH private/public keypair exists with correct ownership on the server. The keypair should exist already (created in `restic_server` role). |
 | `restic_remote_client_port` | `"{{ restic_server_listen_port \| default('3000') }}"` | Port to use on client side to terminate SSH tunnel coming from the backup server. Set to a port not used by something else on the client. Defaults to the same port the `restic_server` role uses to let the REST server listen on. |
 | `restic_remote_client_user` | `"{{ restic_client_user \| default('restic') }}"` | Username used by the server to connect to the client via SSH. The server then starts the `restic-backup.service` as this user via `sudo systemctl start restic-backup.service`. |
+| `restic_remote_ssh_port` | 22 | SSH port of the client, which will be used by the backup server to connect. This role does not configure 'Port' in SSHD config! |
 | `restic_remote_client_group` | `"{{ restic_client_group \| default('restic') }}"` | Groupname getting granted `sudo` privileges on client to start systemd services `restic-backup.service` and `restic-prune.service`. The client's user must be in this group. |
-| `restic_remote_client_home` | `"{{ restic_client_home \| default('/home/' + restic_remote_client_user) }}"` | Construct config directory from `restic_client` defaults. Required for placing the `repository` file to the correct location. |
-| `restic_remote_client_config_dir` | `"{{ restic_client_config_dir \| default(restic_remote_client_home + '/.config/restic') }}"` | See comment of `restic_remote_client_home` above |
 | `restic_remote_client_is_remote` | `"{{ restic_client_is_remote \| default(false) }}"` | Repeat from `restic_client` role to check whether client should be configured to be backed up remotely by the backup server. |
 | `restic_remote_no_log` | true | Do not show sensible content when printed by `ansible-playbook` runs. Set to `false` for debugging e.g. repository file templating. |
 
 # Dependencies
-The `restic_client` role should be run on targets beforehand, and the `restic_server` role for the restic rest-server(s) must be deployed already. After the `restic_server` role was run and generated a SSH public key, this key must be specified in variable `restic_remote_server_authorized_key` (Also see README of `restic_server` role).
+The `restic_client` role must be run on targets beforehand, and the `restic_server` role for the restic rest-server must be deployed already.
 
 # Example Playbook
 ```
@@ -42,25 +59,21 @@ The `restic_client` role should be run on targets beforehand, and the `restic_se
   become: true
 
   vars:
+    restic_client_is_remote: true
     backup_server: backupserver.example.com
-    backup_server_password: supersecurepassword
-    backup_server_become_password: supersecurebecomepassword
 
   roles:
+    - role: restic_client
     - role: restic_remote
 ```
 
 ## Testing with Molecule
-Testing with molecule is partly manual, due to the limited automation process. A simple `molecule test` will not work, since it is still necessary to specify the backupserver's SSH public key (generated by the `restic_server` role) mid-run.
-
 Example how to fully test the whole `restic_remote` role from deploy to a remote backup trigger:
-1. Run `molecule create`
-   1. This will deploy a restic client via `restic_client` role and a restic rest-server via `restic_server` role.
-   2. In the `restic_server` run output, find the freshly generated SSH public key of the backupserver
-   3. Put this key into `molecule/default/converge.yml` in the `restic_remote_server_authorized_key` variable.
-2. Run `molecule converge`
-3. Run `molecule login -h backupserver` to login to the backupserver container.
-   1. Run `systemctl start restic-remote-backup@restic_remote_debian11.service` to test if the backup is working.
+1. Run `molecule converge`
+2. Run `molecule login -h backupserver` to login to the backupserver container.
+   1. Run `systemctl start restic-remote-backup@restic_remote_debian12.service` to test if the backup is working.
    2. Run `journalctl -fe` to check for possible errors.
-4. Run `molecule login -h restic_remote_debian11`
-   1. Run `journalctl -fe` to check for possible errors.
+3. Run `molecule login -h restic_remote_debian12`
+   1. Run `journalctl -ft restic` to check for possible errors.
+
+Dependig on the container environment, molecule containers might not be able to resolve each others hostnames, breaking the SSH connection attempts in the `restic-remote-backup@<remote-client>.service`. In this case, add the `restic_remote_debian12` hostname to `/etc/hosts` manually on the `backupserver`. In a productive setup, hostnames must be resolvable via DNS.

defaults/main.yml

@@ -1,20 +1,5 @@
 ---
 
-###
-# SSH public key of the backup server. This key will be put into the client's
-# '~/.ssh/authorized_keys' to enable the server to connect. Only used when
-# 'restic_client_is_remote: true'.
-###
-restic_remote_server_authorized_key: ""
-
-###
-# Read REST authentication password from the passwordstore.
-# If it does not exist yet, let the relevant task fail. This password should
-# have been generated by the 'restic_client' role, which is a prerequisite for
-# this role.
-###
-restic_remote_client_rest_password: "{{ restic_client_rest_password | default(lookup('passwordstore', 'IT/backup/auth/' + inventory_hostname)) }}"
-
 ###
 # Username to use for initiating SSH connections as from server to client
 ###
@@ -30,29 +15,31 @@ restic_remote_server_group: "{{ restic_server_group | default('rest-server') }}"
 # Port to use on client side to terminate SSH tunnel coming from the backup
 # server. Set to a port not used by something else on the client. Defaults to
 # the same port the 'restic_server' role uses to let the REST server listen on.
+# Becomes SSH_REMOTE_LISTEN_PORT on the server when initiating the remote backup.
 ###
 restic_remote_client_port: "{{ restic_server_listen_port | default('3000') }}"
 
 ###
 # Username to use for SSH connections from server to client, and user which
 # starts the `restic-backup.service` via sudo on the client.
+# Becomes SSH_USER on the server when initiating the remote backup.
 ###
 restic_remote_client_user: "{{ restic_client_user | default('restic') }}"
 
+###
+# SSH port of the client, which will be used by the backup server to connect.
+# Becomes SSH_PORT on the server when initiating the remote backup.
+# This role does not configure 'Port' in SSHD config!
+###
+restic_remote_ssh_port: "22"
+
 ###
 # Groupname getting granted `sudo` privileges on client to start systemd
 # services `restic-backup.service` and `restic-prune.service`. The client's
 # user must be in this group.
 ###
 restic_remote_client_group: "{{ restic_client_group | default('restic') }}"
 
-###
-# Construct config directory from 'restic_client' defaults. Required for
-# placing the 'repository' file to the correct location.
-###
-restic_remote_client_home: "{{ restic_client_home | default('/home/' + restic_remote_client_user) }}"
-restic_remote_client_config_dir: "{{ restic_client_config_dir | default(restic_remote_client_home + '/.config/restic') }}"
-
 ###
 # Repeat from 'restic_client' role to check whether client should be configured
 # to be backed up remotely by the backup server.

molecule/default/converge.yml

@@ -6,10 +6,7 @@
 
   vars:
     restic_client_is_remote: true
-    backup_server: backupserver
-    backup_server_password: ""
-    backup_server_become_password: ""
-    restic_remote_server_authorized_key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... rest-server@backupserver
+    backup_server: "{{ groups['backupservers'][0] }}"
 
   roles:
     - role: fifty2technology.restic_remote

molecule/default/molecule.yml

@@ -6,17 +6,34 @@ driver:
   name: podman
 platforms:
   - name: backupserver
-    image: "geerlingguy/docker-debian11-ansible:latest"
+    image: "geerlingguy/docker-debian12-ansible:latest"
     command: /sbin/init
     tmpfs:
       - /run
       - /tmp
     volumes:
       - /sys/fs/cgroup:/sys/fs/cgroup:ro
     network: "restic_remote_network"
-
-  - name: restic_remote_${MOLECULE_DISTRO:-debian11}
-    image: "geerlingguy/docker-${MOLECULE_DISTRO:-debian11}-ansible:latest"
+  - name: restic_remote_debian12
+    image: "geerlingguy/docker-debian12-ansible:latest"
+    command: /sbin/init
+    tmpfs:
+      - /run
+      - /tmp
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:ro
+    network: "restic_remote_network"
+  - name: restic_remote_debian12_ssh_port
+    image: "geerlingguy/docker-debian12-ansible:latest"
+    command: /sbin/init
+    tmpfs:
+      - /run
+      - /tmp
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:ro
+    network: "restic_remote_network"
+  - name: restic_remote_debian12_client_port
+    image: "geerlingguy/docker-debian12-ansible:latest"
     command: /sbin/init
     tmpfs:
       - /run
@@ -33,5 +50,41 @@ provisioner:
           backupservers:
             hosts:
               backupserver:
+          backupclients:
+            restic_remote_debian12:
+            restic_remote_debian12_client_port:
+            restic_remote_debian12_ssh_port:
+    group_vars:
+      all:
+        restic_server_listen_port: 3000
+    host_vars:
+      ###
+      # * 'our_rest_password' variable is needed to template the custom REST
+      #   repository of this molecule test in 'restic_client' role AND the
+      #   'post_tasks' in 'prepare.yml' where the REST accounts are added to
+      #   the REST backup server.
+      # * 'restic_client_encryption_password' is only required for 'restic_client' role.
+      # * 'restic_remote_client_port' is used by 'restic_client' to template
+      #   'repository-REST.j2', AND used by 'restic_remote' to set client-
+      #   specific env-file on backup server (if not defined, will default to
+      #   'restic_server_listen_port' for both roles).
+      # * 'restic_remote_ssh_port' is only used by 'restic_remote'.
+      ###
+      restic_remote_debian12:
+        our_rest_password: HsWDmQJKFbmOCqI5KfvufiELsltWCzWu
+        restic_client_encryption_password: 2S0xRfeAYdl0tptDfELLLZyD8mC3p7H2veV8rTc2uWRodvIEThkm
+      restic_remote_debian12_client_port:
+        our_rest_password: Ghm62NzJJzVEHo4eJ.Y04tNWxlMAaqA.
+        restic_client_encryption_password: CrLLuGLiR@YBO%TqwgKRVI,fqss5sZsKYfc!%DZfueo_kWeg^Q8e
+        restic_remote_client_port: 4711
+      restic_remote_debian12_ssh_port:
+        our_rest_password: YWgZniJobptX6_44MreYL0gVxeSrHLlA
+        restic_client_encryption_password: OSfFesEUn9IO1hFkAd_4aERxwkRmoCKdrykhON5Agp9sbssDGTfi
+        restic_remote_ssh_port: 2222
 verifier:
   name: ansible
+scenario:
+  converge_sequence:
+    - create
+    - prepare
+    - converge

molecule/default/prepare.yml

@@ -7,8 +7,9 @@
     - hardware
 
   vars:
-    backup_server: backupserver
+    restic_client_repository: templates/repository-REST.j2
     restic_client_use_lvm: false  # doesn't work inside container, would use host's LVM
+    restic_client_is_remote: true
 
   pre_tasks:
     - name: Force update package cache (Debian)
@@ -21,13 +22,11 @@
   roles:
     - role: fifty2technology.restic_client
 
+
 - name: Prepare a backup server
   hosts: backupservers
   gather_facts: true
 
-  vars:
-    backup_server: backupserver
-
   pre_tasks:
     - name: Force update package cache (Debian)
       ansible.builtin.apt:
@@ -38,3 +37,25 @@
 
   roles:
     - role: fifty2technology.restic_server
+
+  post_tasks:
+    ###
+    # This task ensures all auth credentials are present in the backup servers
+    # '.htpasswd' file. It's read from host_vars, preferrably this should be
+    # read through Ansible Vault or any other reasonable secure lookup mechanism.
+    # The .htpasswd file specification does not allow dots in usernames, therefore
+    # we change them to underscores here.
+    # Ref: https://httpd.apache.org/docs/2.4/programs/htpasswd.html#restrictions
+    ###
+    - name: Add rest-server accounts
+      community.general.htpasswd:
+        path: "{{ restic_server_backup_path }}/.htpasswd"
+        name: "{{ item.inventory_hostname | regex_replace('\\.', '_') }}"
+        password: "{{ item.our_rest_password }}"
+        crypt_scheme: bcrypt
+        state: present
+        owner: "{{ restic_server_user }}"
+        group: "{{ restic_server_group }}"
+        mode: '0600'
+      become: true
+      loop: "{{ groups['backupclients'] }}"

molecule/default/templates/repository-REST.j2

@@ -0,0 +1,13 @@
+{#- Dots are not allowed in .htpasswd usernames, so we replace them with '_'.
+    Also see 'tasks/add_rest_user.yml' in 'restic_server' role and
+    https://httpd.apache.org/docs/2.4/programs/htpasswd.html#restrictions. #}
+{% set _rest_username            = inventory_hostname | regex_replace('\\.', '_') %}
+
+{#- Note that REST passwords cannot contain special characters in order to not
+    get confused with field delimiters in the 'repository' string. #}
+{% set _rest_password            = our_rest_password %}
+
+{% set _backup_server            = groups['backupservers'][0] %}
+{% set _backup_server_port       = restic_remote_client_port | default(restic_server_listen_port) -%}
+
+rest:http://{{ _rest_username }}:{{ _rest_password }}@{{ _backup_server }}:{{ _backup_server_port }}/{{ _rest_username }}

requirements.txt

@@ -1,3 +1,5 @@
 ansible
 ansible-lint
 yamllint
+molecule
+molecule-plugins[podman]==23.5.0  # fixes https://github.com/ansible-community/molecule-plugins/issues/242

tasks/configure_client.yml

@@ -4,7 +4,7 @@
   ansible.posix.authorized_key:
     user: "{{ restic_remote_client_user }}"
     state: present
-    key: "{{ restic_remote_server_authorized_key }}"
+    key: "{{ _restic_remote_server_authorized_key.content | b64decode }}"
   become: true
 
 ###
@@ -37,15 +37,3 @@
     group: root
     mode: '0640'
   become: true
-
-###
-# 'no_log' for all tasks that include a secret
-###
-- name: Template restic remote repository path (to REST server via incoming SSH)
-  ansible.builtin.template:
-    src: templates/remote-repository.j2
-    dest: "{{ restic_remote_client_config_dir }}/repository"
-    owner: "{{ restic_remote_client_user }}"
-    group: "{{ restic_remote_client_group }}"
-    mode: '0400'
-  no_log: "{{ restic_remote_no_log }}"