intro.org

Configuration files tools

The directory contains a python module to manage a hierarchy of configuration files and tools to extract values from such configuration files. The tools can be used from shell scripts. Alternatively, a python program can use the module directlly.

The format is that interpreted by the configparser module: a fairly basic INI file, consisting of key-value pairs. The format allows multiline values: whitespace at the beginning of a line makes it a continuation of the previous line with the whitespace squashed down to a single space. Comments start with a hash character #. Empty lines are ignored and can be used for readability.

Here is a basic example consisting of two sections:

[foo]
var1 = val1

[bar]
var1 = val1-bar
var2 = val2

The separator between the key and the value can be either an `equals’ sign or a colon. Trailing white space in the key and leading white space in the value are stripped (and the continuation line rule above prohibits leading space in the key as well).

Installation

In this directory, say (as root)

python3 setup.py install

Set-up

The base configuration file in the sequence is specified by an env variable:

export CONFIG=/path/to/top-level/config/file/foo.conf

or it can be specified explicitly using the –config (or its short form -C) option on any program that uses this module.

The base configuration file can specify a search path and a sequence of config files to be loaded as follows:

[config]
path = dir1, dir2, dir3
files = foo.conf, bar.conf

The paths can be absolute or relative. Relative paths are relative to the directory of the base config file. In this case, the following files will be read in sequence:

base config file
dir1/foo.conf
dir1/bar.conf
dir2/foo.conf
dir2/bar.conf
dir3/foo.conf
dir3/bar.conf

The first instance of a given key in a given section in this sequence of files determines the value of the key.

An empty base configuration file (which must exist) is equivalent to the following:

[config]
path = ., ../config, ../../config
files = foo.conf

where foo.conf is the basename of the base config file.

Example usage in shell scripts and Makefiles

The tool getconf.py can be used to get a value from a section (or a sequence of sections) in a config file hierarchy (as described above):

$ getconf.py var1 foo
val1
$ getconf.py var1 bar foo
val1-bar
$ getconf.py var1 foo bar
val1

This can be used in shell scripts:

foovar1=$(getconf.py var1 foo)
barvar1=$(getconf.py var1 bar)

or in Makefiles:

foovar1 = $(shell getconf.py var1 foo)

Example usage in python scripts

The getconf.py program itself is a good example of how to use this module in python:

#! /usr/bin/env python3

import sys
import os
import configtools
from optparse import make_option

def main(conf, args, opts):

    if opts.dump:
        conf.write(sys.stdout)
        return 0

    if opts.all:
        for sec in args:
            if conf.has_section(sec):
                print("[%s]" % (sec))
                items = conf.items(sec)
                items.sort()
                for (n, v) in items:
                    print("%s = %s" % (n, v))
                print()
        return 0

    option = args[0]
    for sec in args[1:]:
        if conf.has_section(sec):
            if conf.has_option(sec, option):
                if opts.list:
                    print(conf.get(sec, option).replace(',', ' ').replace('\n', ' ').replace('  ', ' '))
                else:
                    print(conf.get(sec, option))
                return 0
    return 1

options = [
    make_option("-a", "--all", action="store_true", dest="all", help="print all items in section"),
    make_option("-d", "--dump", action="store_true", dest="dump", help="print everything"),
    make_option("-l", "--list", action="store_true", dest="list", help="print it as a shell list, translating commas to spaces"),
    make_option("-L", "--listfiles", action="store_true", dest="listfiles", help="print the list of config files"),
]

if __name__ == '__main__':
    opts, args = configtools.parse_args(options)
    conf, files = configtools.init(opts)
    if opts.listfiles:
        files.reverse()
        print(files)
        sys.exit(0)
    status = main(conf, args, opts)
    sys.exit(status)

After the initialization calls, we get a SafeConfigParser object conf, through which we can obtain the values of keys using conf.get(section, option). The module provides a few conveniences:

• range expansion: foo[1-3] can be used as a value to mean “foo1, foo2, foo3”. Only one range can be specified per value: foo[1-3][0-9] does not work - only the first range is expanded[fn:1]. Nesting is also not allowed, is not currently detected and can give confusing results.
• hosts can be specified in the config file as follows:

[hosts]
clients = gprfc0[57, 59]
servers = gprfs00[3-8]

and can be extracted in various ways using gethosts.py.

Some notes

• A list of values is specified in comma-separated form. That is only a convention, not a hard-and-fast rule, but it is strongly recommended nevertheless. Changing from CSV to a space-separated list is easily done by specifying the --list option to getconf.py (see above). The convention is also used by the gethosts.py script.
• The configparser module allows substitutions by using the SafeConfigParser class (and the configtools module takes advantage of that).
• To see the documentation of the module, say

  $ python3
  import configparser
  help(configparser)
      

  or on the web:

  https://docs.python.org/3/library/configparser.html

• This is python3-based. There is an equivalent ConfigParser module in Python 2.7 (but note the spelling).

Footnotes

[fn:1] if necessary, that limitation gives an ugly way to get a literal ‘foo[1-3]’ by specifying ‘foo[][1-3]’.