Recap

recap is a system status reporting tool. A reporting script that generates reports of various information about the server.

Contribution

Contribution guidelines can be found in CONTRIBUTING.md

Dependencies

• bash >= 4
• coreutils
• gawk
• grep
• iotop
• iproute/iproute2
• elinks
• procps
• psmisc
• sysstat >= 9

Versioning

recap is following the x.y.z versioning as defined below:

• x (major) - Changes that prevent at least some rolling upgrades.
• y (minor) - Changes that don't break any rolling upgrades but require closer user attention for example configuration defaults, function behavior, tools used to produce reports, among others.
• z (patch) - Changes that are backward-compatible including features and/or bug fixes.

Installation

It is highly recommended to make use of a package to install recap is the easiest way to keep it updated whenever there is a new release.

Fedora

recap is available from the main Fedora repository (spec file).

dnf install recap

RHEL/CentOS

recap is available from the EPEL repository (spec file).

yum install recap

Debian

Currently only available in testing and unstable. For other releases see the options to build a deb package or install from source.

The official Debian files are available in this repository

Ubuntu

At the moment there is no public repository for Ubuntu, two options are available, build a deb package or install manually, see instructions down below.

Build a deb package

This repository contains the official Debian files required to build a deb package.

These steps used to be used to build the deb package, use them as a guide:

# Install all the packages required for building the package
apt-get update
apt-get install debhelper devscripts git -y

## For Ubuntu:
apt-get install equivs -y

# Create the working dir:
mkdir recap
cd recap

# Get the Debian configs
git init
git remote add origin https://github.com/jkirk/recap.git
git fetch --no-tags origin
git checkout -qf FETCH_HEAD
git submodule update --init --recursive
export LATEST=$( git log --format="%h" --no-merges -1 )

# Build dependencies
echo "yes" | mk-build-deps --install --remove debian/control

# Get upstream recap code
git checkout --orphan upstream
git reset --hard
git remote add upstream https://github.com/rackerlabs/recap.git
git fetch -t upstream
latest_tag=$( git tag | tail -1 )
git archive ${latest_tag} -o ../recap_${latest_tag}.orig.tar.gz
tar -zxf ../recap_${latest_tag}.orig.tar.gz
git fetch --no-tags origin
git checkout ${LATEST} -- debian

# Build the package
debuild -us -uc --lintian-opts --profile debian

# Package will be created in ../recap_${latest_tag}-<RELEASE>_all.deb
# RELEASE comes from the changelog in the Debian repository.

Manual

1. Install the required dependencies.
2. Clone this repository: git clone https://github.com/rackerlabs/recap.git
3. Change into the new directory: cd recap
4. Install the program: sudo make install

The information captured will be found in log files in the /var/log/recap/ directory.

About the locations of the scripts

• The default location of the install is "/" it can be overridden with DESTDIR.
• The scripts, man pages and docs are installed under ""/usr/local" by default, this can be overridden with PREFIX. Main scripts are installed on in "./sbin" by default, this can be overriden with BINDIR.
• The core scripts and the plugins are installed on top of PREFIX in "./recap/plugins-available" by default, this can be overridden with LIBDIR

The following example is a common location for most of the distributions, this will install recap under /usr:

$ sudo make PREFIX="/usr" install

This other example will install recap under your homedirectory but using the default locations for the script, i.e. under "~/usr/local":

$ make DESTDIR="~" install

The Makefile scripts attempts to detect systemd if so, the install option will install the systemd unit files. The install will not enable the timers, but it will show the commands required to enable each of the timers. When systemd is not detected the cronjobs will be installed.

Is up to each package distribution to follow their own best practices regarding enabling/disabling the timers on install/remove of the package.

Ansible

An ansible playbook could be used to install recap from a git repository. The playbook is located in tools under ansible_recap.yml the playbook can be used to install it on Red Hat based and Debian based distros. Or to uninstall it defining the uninstall variable.

Variables

• repo - The location of the repository, default: https://github.com/rackerlabs/recap.git.
• ref - The reference to use this could be a branch, a tag or commit, default: master.
• binpath - The value of BINPATH, default: /sbin.
• destdir - The value of DESTDIR, default: "".
• prefix - The value of PREFIX, default: /usr.
• tmp_install_dir - The location where the cloned repo will be placed, default: /tmp/recap.
• uninstall - Then this is defined it will remove recap, default: undefined.
• enable_plugins - To enable the global plugin configuration default: false.
• plugin_list - A list of plugins to enable, from the plugin-available directory, default: all.

Install (default)

Install the stable version of recap:

ansible-playbook tools/ansible_recap.yml

Install the development version of recap:

ansible-playbook tools/ansible_recap.yml -e ref=development

Install branch foo from a different repository:

ansible-playbook tools/ansible_recap.yml -e ref=foo -e repo=https://github.com/bar/recap.git

Install recap with BINPATH in /bin:

ansible-playbook tools/ansible_recap.yml -e binpath=/bin

Install recap with all plugins enabled:

ansible-playbook tools/ansible_recap.yml -e enable_plugins=true

Install recap with a list of plugins to enable:

ansible-playbook tools/ansible_recap.yml \
  -e enable_plugins=true \
  -e '{"plugin_list":[docker_top,redis]}'

Uninstall

Uninstall recap from the default path:

ansible-playbook tools/ansible_recap.yml -e uninstall=yes

Uninstall recap from a custom location:

ansible-playbook tools/ansible_recap.yml -e uninstall=yes -e destdir=/tmp/test

Cron/Timers and Configuration

Timers(systemd)

Multiple unit files are available to make use of timers, here the default schedules for the recap scripts:

• recap (default every 10min)
• recap-onboot (runs at boot time)
• recaplog (default: Once a day 1am)

Enabling timers

Each one of the timers can be enabled with:

sudo systemctl enable recap.timer --now"
sudo systemctl enable recaplog.timer --now"
sudo systemctl enable recap-onboot.timer --now"

Disabling timers

Each one of the timers can be disabled with:

sudo systemctl disable recap.timer --now"
sudo systemctl disable recaplog.timer --now"
sudo systemctl disable recap-onboot.timer --now"

Cron

The cron file (/etc/cron.d/recap) is used to determine the execution time of recap and recaplog. By default the cron execution for recap is enabled to run every 10 min. and recaplog is expected to run every day at 1 am, but those can be adjusted as needed.

Configuration

The following variables are commented out with the defaults values in the configuration file /etc/recap.conf which can be overridden.

Settings shared by recap scripts

• BASEDIR - Directory where the logs are saved.

  Default: BASEDIR="/var/log/recap"

• LIBDIR - Directory where the libraries/functions are located.

  The default value depends on the PREFIX used when installing, the default PREFIX on the Makefile is /usr/local, then:

  Default: LIBDIR="/usr/local/lib/recap"

  But packages use /usr as the PREFIX, then through a package it is expected to be:

  Default: LIBDIR="/usr/lib/recap"

Settings used only by recaplog

• LOG_COMPRESS - Enable or disable log compression.

  Default: LOG_COMPRESS=1

• LOG_EXPIRY - Log files will be deleted after LOG_EXPIRY days

  Default: LOG_EXPIRY=15

Settings used only by recap

• MAILTO - Send a report to the email defined.

  Default: MAILTO=""

• MIN_FREE_SPACE - Minimum free space (in MB) required in ${BASEDIR} to run recap, a value of 0 deactivates this check.

  Default: MIN_FREE_SPACE=0

Reports

These are the type of reports generated and their dependencies.

fdisk

• USEFDISK - Generates logs from "fdisk ${OPTS_FDISK}"

  Default: USEFDISK="no"

mysql

• USEMYSQL - Generates logs from "mysqladmin status"

  Makes use of DOTMYDOTCNF.

  Required by: USEMYSQLPROCESSLIST, USEINNODB

  Default: USEMYSQL="no"

• USEMYSQLPROCESSLIST - Generates logs from "mysqladmin processlist"

  Makes use of DOTMYDOTCNF and MYSQL_PROCESS_LIST

  Requires: USEMYSQL

  Default: USEMYSQLPROCESSLIST="no"

• USEINNODB - Generates logs from "mysql show engine innodb status"

  Makes use of DOTMYDOTCNF

  Requires: USEMYSQL

  Default: USEINNODB="no"

netstat

• USENETSTAT - Generates network stats from "ss ${OPTS_NETSTAT}"

  Required by: USENETSTATSUM

  Default: USENETSTAT="yes"

• USENETSTATSUM - Generates logs from "nstat ${OPTS_NETSTAT_SUM}".

  Requires: USENETSTAT

  Default: USENETSTATSUM="no"

ps

• USEPS - Generates logs from "ps"

  Options can be modified in OTPS_PS

  Default: USEPS="yes"

pstree

• USEPSTREE - Generates logs from pstree

  Options can be modified in OTPS_PSTREE

  Default: USEPSTREE="no"

resources

• USERESOURCES - Generates "resources"(uptime, free, vmstat, iostat, iotop) log

  Required by: USEDF, USESLAB, USESAR, USESARQ, USESARR

  Default: USERESOURCES="yes"

• USEDF - Generates logs from df

  Requires: USERESOURCES

  Options can be modified in OPTS_DF

  Default: USEDF="yes"

• USESLAB - Generates logs from the slab table.

  Requires: USERESOURCES

  Default: USESLAB="no"

• USERSAR - Generates logs from sar.

  Requires: USERESOURCES

  Default: USESAR="yes"

• USESARQ - Generates logs from "sar -q" (logs queue length, load data)

  Requires: USERESOURCES

  Default: USESARQ="no"

• USESARR - Generates logs from"sar -r" (logs memory data)

  Requires: USERESOURCES

  Default: USESARR="no"

Options

Options used by the tools generating the reports

• DOTMYDOTCNF - Defines the path to the mysql client configuration file

  Required by: USEMYSQL, USEMYSQLPROCESSLIST, USEINNODB

  Default: DOTMYDOTCNF="/root/.my.cnf"

• MYSQL_PROCESS_LIST - Format to display MySQL process list, options are "table" or "vertical".

  Required by: USEMYSQLPROCESSLIST

  Default: MYSQL_PROCESS_LIST="table"

• OPTS_DF - df options

  Required by: USEDF

  Default: OPTS_DF="-x nfs"

• OPTS_FDISK - Option used by USEFDISK.

  Required by: USEFDISK

  Default: OPTS_FDISK="-l"

• OPTS_FREE - free options

  Required by: USEFREE

  Default: OPTS_FREE=""

• OPTS_IOSTAT - iostat options

  Required by: USERESOURCES

  Default: OPTS_IOSTAT="-t -x 1 3"

• OPTS_IOTOP - iotop options

  Required by: USERESOURCES

  Default: OPTS_IOTOP="-b -o -t -n 3"

• OPTS_NETSTAT - ss options

  Required by: USENETSTAT

  Default: OPTS_NETSTAT="-atunp"

• OPTS_NETSTAT_SUM - nstat options

  Required by: USENETSTATSUM

  Default: OPTS_NETSTAT_SUM="-a"

• OPTS_PS - ps options

  Required by: USEPS

  Default: OPTS_PS="auxfww"

• OPTS_PSTREE - pstree options

  Required by: USEPSTREE

  Default: OPTS_PSTREE="-p"

• OPTS_VMSTAT - vmstat options

  Required by: USERESOURCES

  Default: OPTS_VMSTAT="-S M 1 3"

Plugins

• USEPLUGINS - Enable/Disable plugins usage.

  Default: USEPLUGINS=no

Plugins are stored in the plugin directory, defined by LIBDIR/plugins-available

Default: /usr/lib/local/recap/plugin-available

Enabling plugins

To enable plugins, the following is required to:

• Setting USEPLUGINS="yes" in /etc/recap.conf
• Symlinking plugins-enabled/plugin_name to plugins-available/plugin_name

Naming conventions:

• Plugin scripts can be named in any way. It's desired that they describe the purpose of the plugin in one word. When multiple words are required use underscores (_). Don't use extensions or dates (e.g. YYYYMMDD) in the plugin name. Some examples:

  • Good names for plugins
    • redis
    • memcache
    • docker_images
    • memcache_13
  • Bad names for plugins
    • johndoe_apache (not very descriptive)
    • myplugin (non explicit)
    • test.sh (non explicit, using extension)
    • recap-plugin (non explicit, using hyphens)
    • Sendmail (CamelCase)
    • redis.bak (extension)
    • ms sql (space between words)
    • reports_20202020 (use of a date)

• Allowed naming convention for plugin OPTIONS in /etc/recap.conf: PLUGIN_OPTS__<OPT_NAME> Some examples:

  • Good plugin option names:
    • PLUGIN_OPTS_MEMCACHE_PROTO
    • PLUGIN_OPTS_AWS_KEY
    • PLUGIN_OPTS_REDIS_PORT
    • PLUGIN_OPTS_DOCKER_HUB_URL
  • Bad plugin option names:
    • plugin_opts_my_plugin (lower case)
    • PLUGIN_OPTS_MY_VARIABLE (lacking plugin reference)
    • PLUGIN_OPTS_DOCKER_port (CamelCase)
    • PLUGIN-OPTS-NTP (using hyphens instead of underscores, missing the option)

• Inside the plugin file/script it is expected only functions. recap will only call one function: print_<plugin_name> where plugin_name must match the name of the file.

• Optionally, other functions can be defined to create different entries in the log. Those other functions could be controlled by plugin variables (PLUGIN_OPTS__<OPT_NAME>). Those variables are set in /etc/recap.conf and conditionally called from the main plugin function plugin_name

• Any plugin variable defined must have a default value.

• The plugins are expected to follow some of the practices followed in recap. Please refer to CONTRIBUTING.md

• A template of a plugin is provided in doc/plugin_template

Changelog & Contributions

Information about changes and contributors is documented in the CHANGELOG.md

License

recap is licensed under the GNU General Public License v2.0