Netdata monitoring of docker in docker no longer possible

[Trufax]

Describe the bug
Iam running netdata in docker to monitor the host and other docker containers. Once i run the ansible script for OS_hardening the netdata docker container can no longer monitor other docker containers. No traffic si shown for the other containers.

Expected behavior
The netdata docker container should still be able to monitor traffic for other docker containers via the docker api proxy container.

Actual behavior

Once the OS hardening scirpt was run it is no longer possible for netdata to monitor the traffic of other docker containers.

Example Playbook

- hosts: 10.XX.XX.XX
  remote_user: myuser
  become: yes
  collections:
    - devsec.hardening
  roles:
  - role: os_hardening
    vars:
      sysctl_overwrite:
       # Enable IPv4 traffic forwarding. (OS Hardening) required for Docker
        net.ipv4.ip_forward: 1
     

Example docker compose of netdata and proxy

version: '3'
services:
  netdata:
    image: netdata/netdata
    container_name: netdata
    hostname: netdatatest # set to fqdn of host
    ports:
      - 19999:19999
    restart: unless-stopped
    cap_add:
      - SYS_PTRACE
    security_opt:
      - apparmor:unconfined
    volumes:
      - ./netdataconfig/netdata:/etc/netdata
      - netdatalib:/var/lib/netdata
      - netdatacache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
    environment:
      - DOCKER_HOST=http://proxy:2375
      - VIRTUALIZATION:xen

  proxy:
    image: tecnativa/docker-socket-proxy
    hostname: proxy
    privileged: true
    volumes: 
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      CONTAINERS: 1
      NETWORKS: 1


volumes:
  netdataconfig:
  netdatalib:
  netdatacache:

OS / Environment

Ubuntu 20.10

Ansible Version

ansible 2.10.5
  config file = None
  configured module search path = ['/home/myuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.9.1+ (default, Jan 20 2021, 14:49:22) [GCC 10.2.1 20210110]

Role Version

latest

Additional context
Iam not sure if its really a bug in the scipt. It might very be that the scirpt simply disables a feature which is used by netdata docker container to monitor other docker containers. Any ideas which setting could it be ?

[schurzi]

Hey @Trufax, I don't think, this is a bug. After all, we are hardening the server ;)
I think the source of your problem could be hidepid_option in our config. With this we hide all other processes from user processes. Since netdata is running in Docker and trying to read /proc this seems like the problem. if you set this option to 0 it should work again. Can you test this?

[Trufax]

Hey thanks for the reply! I also think its a setting but at least only changing "hidepid_option: 0" did not do the trick unfrotunatelly. I created a fresh installation und setup netdata, i run the below playbook, rebooted and network traffic was no longer updated for the containers. Any other idea what setting it could be ?

- hosts: testbox
  remote_user: myuser
  become: yes
  collections:
    - devsec.hardening
  roles:
  - role: ssh_hardening
    vars:
      sftp_enabled: true
      ssh_print_motd: ture
      ssh_print_pam_motd: true
      ssh_print_last_log: true
  - role: os_hardening
    vars:
      sysctl_overwrite:
        net.ipv4.ip_forward: 1
        hidepid_option: 0

[schurzi]

I'm not directly aware of any other option, that might produce this kind of problem. While looking at your code, I noticed, that the indentation of hidepid_optionis wrong. It is not a sysctl_overwrite

This looks more correct:

- hosts: testbox
  remote_user: myuser
  become: yes
  collections:
    - devsec.hardening
  roles:
  - role: ssh_hardening
    vars:
      sftp_enabled: true
      ssh_print_motd: ture
      ssh_print_pam_motd: true
      ssh_print_last_log: true
  - role: os_hardening
    vars:
      sysctl_overwrite:
        net.ipv4.ip_forward: 1
      hidepid_option: 0

[Trufax]

Yay you solved it !! I am new to the YAML Syntax so i did not know this could be the issue. Thank you very much !!

Should ansible throw an error if its wrong formated or will it just ignore it ?

[schurzi]

nice :)

Well, thats one of the main problems with YAML. It thinks hidepid_option is an element in sysctl_overwrite when you indent it to the level below. It does not know, that this in not used or refferenced anywhere. Also the missing hidepid_option on the "root" level is not mandatory, since we define an default. So this kind of problem can emerge in many cases. The best practice is to always check indentation and see if problems with missing variables come from there.