diff --git a/README.rst b/README.rst
deleted file mode 100644
index b8210919..00000000
--- a/README.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-.. include:: docs/README.rst
-
diff --git a/docs/AUTHORS.rst b/docs/AUTHORS.rst
new file mode 100644
index 00000000..81eb0183
--- /dev/null
+++ b/docs/AUTHORS.rst
@@ -0,0 +1,73 @@
+.. role:: raw-html-m2r(raw)
+ :format: html
+
+
+Authors
+=======
+
+This list is sorted by the number of commits per contributor in *descending* order.
+
+.. list-table::
+ :header-rows: 1
+
+ * - Avatar
+ - Contributor
+ - Contributions
+ * - :raw-html-m2r:`
`
+ - `@myii `_
+ - 56
+ * - :raw-html-m2r:`
`
+ - `@aboe76 `_
+ - 17
+ * - :raw-html-m2r:`
`
+ - `@javierbertoli `_
+ - 8
+ * - :raw-html-m2r:`
`
+ - `@gravyboat `_
+ - 6
+ * - :raw-html-m2r:`
`
+ - `@evvers `_
+ - 4
+ * - :raw-html-m2r:`
`
+ - `@nmadhok `_
+ - 3
+ * - :raw-html-m2r:`
`
+ - `@alxwr `_
+ - 2
+ * - :raw-html-m2r:`
`
+ - `@vutny `_
+ - 2
+ * - :raw-html-m2r:`
`
+ - `@daks `_
+ - 2
+ * - :raw-html-m2r:`
`
+ - `@k-hamza `_
+ - 2
+ * - :raw-html-m2r:`
`
+ - `@puneetk `_
+ - 2
+ * - :raw-html-m2r:`
`
+ - `@andygabby `_
+ - 1
+ * - :raw-html-m2r:`
`
+ - `@Jokipii `_
+ - 1
+ * - :raw-html-m2r:`
`
+ - `@duk3luk3 `_
+ - 1
+ * - :raw-html-m2r:`
`
+ - `@marco-m `_
+ - 1
+ * - :raw-html-m2r:`
`
+ - `@whiteinge `_
+ - 1
+ * - :raw-html-m2r:`
`
+ - `@sroegner `_
+ - 1
+ * - :raw-html-m2r:`
`
+ - `@babilen5 `_
+ - 1
+ * - :raw-html-m2r:`
`
+ - `@GMAzrael `_
+ - 1
+
diff --git a/docs/CHANGELOG.rst b/docs/CHANGELOG.rst
new file mode 100644
index 00000000..c91271f4
--- /dev/null
+++ b/docs/CHANGELOG.rst
@@ -0,0 +1,481 @@
+
+Changelog
+=========
+
+`1.0.0 `_ (2019-02-28)
+----------------------------------------------------------------------------------------------------------
+
+Code Refactoring
+^^^^^^^^^^^^^^^^
+
+
+* **components:** split components into separate subdirs (\ `d957055 `_\ ), closes `/github.com/saltstack-formulas/template-formula/pull/48#pullrequestreview-207182085 `_ `/github.com/saltstack-formulas/template-formula/pull/48#discussion_r259805312 `_
+* **include+require:** use variable for duplicate values (\ `4443518 `_\ )
+* **pkg:** change to ``package`` instead (\ `2cd82e5 `_\ ), closes `/github.com/saltstack-formulas/template-formula/pull/48#discussion_r259951123 `_
+* **pkg:** move ``pkg`` related components into separate directory (\ `c21f82b `_\ )
+* **states:** set state IDs based on a dependable structure (\ `6690ee6 `_\ ), closes `/github.com/saltstack-formulas/template-formula/pull/48#discussion_r259953473 `_ `/github.com/saltstack-formulas/template-formula/pull/48#discussion_r259956996 `_
+* **topdir:** use for ``include`` and ``require`` except ``init.sls`` (\ `a218e91 `_\ )
+* **tpldir:** use ``topdir`` globally in place of ``tpldir`` (\ `2838bc9 `_\ )
+* **tplroot:** use ``tplroot`` instead of ``topdir`` to match ``tpldata`` (\ `b7356b0 `_\ )
+
+Continuous Integration
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **kitchen:** specify ``image`` explicitly for each platform (\ `b25fbdc `_\ )
+* **kitchen+travis:** use ``debian:jessie-backports`` as ``debian-8`` (\ `1b9d249 `_\ ), closes `#50 `_ `/github.com/saltstack/salt-pack/issues/657#issuecomment-467932962 `_
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **components:** update for separation of ``pkg``\ , ``config`` & ``service`` (\ `726fcab `_\ )
+* **readme:** add suggested improvement to ``template.service.clean`` (\ `bf1039c `_\ )
+* **readme:** fix typos (\ `007159a `_\ )
+
+Features
+^^^^^^^^
+
+
+* **pkg:** add ``clean`` states (\ `422c7ac `_\ )
+* **pkg:** use ``require`` requisite between ``pkg`` states (\ `6e7141b `_\ ), closes `/github.com/saltstack/salt/blob/0c78d7dc894058988d171a28a11bd4a9dbf60266/salt/utils/jinja.py#L120 `_ `/github.com/saltstack/salt/blob/0c78d7dc894058988d171a28a11bd4a9dbf60266/salt/utils/templates.py#L145 `_ `/github.com/saltstack/salt/issues/10838#issuecomment-391718086 `_
+
+Reverts
+^^^^^^^
+
+
+* **kitchen+travis:** disable ``debian-8`` due to ``2019.2`` bug (\ `e8f0f7e `_\ )
+
+BREAKING CHANGES
+^^^^^^^^^^^^^^^^
+
+
+* **states:** Wholesale state ID changes will break implementations
+ that are relying on the previous state IDs for requisite purposes.
+* **pkg:** Changing the ``pkg`` directory to ``package`` will break
+ implementations that are depending on ``pkg`` for ``include`` or ``sls``\ -based
+ requisite purposes.
+
+`0.7.6 `_ (2019-02-27)
+----------------------------------------------------------------------------------------------------------
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **yaml:** os*.yaml map files needs at least an empty dict (\ `dd99750 `_\ )
+
+`0.7.5 `_ (2019-02-27)
+----------------------------------------------------------------------------------------------------------
+
+Bug Fixes
+^^^^^^^^^
+
+
+* **pillar:** fix ``os_family`` typo (\ `3f89c12 `_\ )
+* **tofs:** update comments in ``files_switch`` macro for new method (\ `3fa3640 `_\ )
+
+Code Refactoring
+^^^^^^^^^^^^^^^^
+
+
+* **macros:** use ``tplroot`` instead of ``topdir`` to match ``tpldata`` (\ `923b459 `_\ )
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **tofs:** add more sub-headings to ease document navigation (\ `2c5dc21 `_\ )
+* **tofs:** apply language formatting to source code blocks (\ `0638413 `_\ )
+* **tofs:** explain how all parts of the ``source`` can be customised (\ `2f82eb5 `_\ ), closes `#44 `_
+* **tofs:** improve general use of language (\ `5105d29 `_\ )
+* **tofs:** update the ``files_switch`` section for the updated macro (\ `788f732 `_\ )
+* **tofs:** use ``{%-`` for all Jinja statements (\ `4348df8 `_\ )
+
+`0.7.4 `_ (2019-02-27)
+----------------------------------------------------------------------------------------------------------
+
+Continuous Integration
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **kitchen:** check for repos updates before trying package installation (\ `b632383 `_\ )
+* **kitchen+travis:** disable ``debian-8`` due to ``2019.2`` installation bug (\ `178c710 `_\ )
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **contributing:** separate ``BREAKING CHANGE`` under its own heading (\ `ee053d7 `_\ )
+
+`0.7.3 `_ (2019-02-25)
+----------------------------------------------------------------------------------------------------------
+
+Bug Fixes
+^^^^^^^^^
+
+
+* **tofs:** use ``tpldir`` derivative ``topdir`` for pillar (config) paths (\ `5e9df00 `_\ )
+
+`0.7.2 `_ (2019-02-24)
+----------------------------------------------------------------------------------------------------------
+
+Code Refactoring
+^^^^^^^^^^^^^^^^
+
+
+* **tpldir:** use ``tpldir`` or derivatives to make formula portable (\ `52d03d8 `_\ ), closes `#22 `_
+
+Continuous Integration
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **kitchen:** improve comments about ``opensuse`` problems encountered (\ `c246939 `_\ )
+* **travis:** prevent ``release`` stage running for PRs (\ `3a072c7 `_\ ), closes `/travis-ci.com/saltstack-formulas/template-formula/jobs/180068519#L466 `_ `/github.com/saltstack-formulas/template-formula/pull/42#issuecomment-466446324 `_
+
+`0.7.1 `_ (2019-02-24)
+----------------------------------------------------------------------------------------------------------
+
+Continuous Integration
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **kitchen:** use ``salt-minion`` version of ``opensuse`` to ensure tests run (\ `99b073a `_\ )
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **changelog:** remove erroneous "closes" used by ``semantic-release`` (\ `be4571d `_\ )
+
+`0.7.0 `_ (2019-02-23)
+----------------------------------------------------------------------------------------------------------
+
+Features
+^^^^^^^^
+
+
+* **tofs:** implement backwards-compatible TOFSv2 for configurability (\ `068a94d `_\ )
+
+`0.6.0 `_ (2019-02-23)
+----------------------------------------------------------------------------------------------------------
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **contributing:** add basic introductory text before the TOC (\ `45ccaf6 `_\ )
+* **contributing:** modify quoted heading to prevent TOC inclusion (\ `abcb6ef `_\ )
+* **readme:** convert note into a heading (\ `5f2d789 `_\ )
+
+Features
+^^^^^^^^
+
+
+* **toc:** use ``markdown-toc`` directly to update inline (\ `a5bae1e `_\ )
+
+`0.5.0 `_ (2019-02-23)
+----------------------------------------------------------------------------------------------------------
+
+Features
+^^^^^^^^
+
+
+* **kitchen+travis:** add ``opensuse-leap`` after resolving issues (\ `7614a3c `_\ )
+* **kitchen+travis:** conduct tests on a wider range of platforms (\ `1348078 `_\ )
+
+Tests
+^^^^^
+
+
+* **inspec:** update ``supports`` for all platforms added (\ `42f93b3 `_\ )
+
+`0.4.0 `_ (2019-02-23)
+----------------------------------------------------------------------------------------------------------
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **contributing:** centre-align version bump columns in table (\ `a238cae `_\ )
+
+Features
+^^^^^^^^
+
+
+* **authors:** update automatically alongside ``semantic-release`` (\ `8000098 `_\ )
+
+`0.3.6 `_ (2019-02-22)
+----------------------------------------------------------------------------------------------------------
+
+Continuous Integration
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **travis:** include ``commitlint`` stage (\ `6659a69 `_\ )
+* **travis:** remove obsolete check based on ``$TRAVIS_TEST_RESULT`` (\ `6df9c95 `_\ )
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **contributing:** update with sub-headings and ``commitlint`` details (\ `ea2c9a4 `_\ )
+
+`0.3.5 `_ (2019-02-21)
+----------------------------------------------------------------------------------------------------------
+
+Code Refactoring
+^^^^^^^^^^^^^^^^
+
+
+* **kitchen:** prefer ``kitchen.yml`` to ``.kitchen.yml`` (\ `3860bf9 `_\ )
+
+`0.3.4 `_ (2019-02-21)
+----------------------------------------------------------------------------------------------------------
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **contributing:** add commit message formatting instructions (\ `fb3d173 `_\ )
+
+`0.3.3 `_ (2019-02-20)
+----------------------------------------------------------------------------------------------------------
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **changelog:** add missing entry under ``v0.3.2`` (\ `50352b5 `_\ )
+
+`0.3.2 `_ (2019-02-20)
+----------------------------------------------------------------------------------------------------------
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **README:** remove gitchangelog (\ `2fc85fc `_\ )
+* **contributing:** create blank template (\ `3633e8f `_\ )
+
+`0.3.1 `_ (2019-02-20)
+----------------------------------------------------------------------------------------------------------
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* **changelog:** merge previous ``rst`` into new ``md`` format (\ `2b4e485 `_\ )
+
+`0.3.0 `_ (2019-02-20)
+----------------------------------------------------------------------------------------------------------
+
+Features
+^^^^^^^^
+
+
+* **semantic-release:** configure for this formula (\ `cbcfd75 `_\ )
+
+`0.2.0 `_ (2019-02-17)
+----------------------------------------------------------------------------------------------------------
+
+
+* Added a working testing scaffold and travis support. [Javier Bértoli]
+
+`0.1.7 `_ (2019-02-16)
+----------------------------------------------------------------------------------------------------------
+
+Fix
+^^^
+
+
+* Typo in the installation instructions. [Niels Abspoel]
+
+Other
+^^^^^
+
+
+* Update the changelog. [Niels Abspoel]
+* Update README with link to install gitchangelog [Imran Iqbal]
+
+`0.1.6 `_ (2019-02-16)
+----------------------------------------------------------------------------------------------------------
+
+
+* Add changelog generator. [Niels Abspoel]
+
+`0.1.5 `_ (2019-02-15)
+----------------------------------------------------------------------------------------------------------
+
+
+* Prepare v0.1.5 [Imran Iqbal]
+* Fix missing ')' [gmazrael]
+
+`0.1.4 `_ (2019-02-15)
+----------------------------------------------------------------------------------------------------------
+
+
+* Replace obsolete VERSION file and replace with FORMULA file. [Imran Iqbal]
+
+`0.1.3 `_ (2019-02-12)
+----------------------------------------------------------------------------------------------------------
+
+
+* Updated changelog and version. [Alexander Weidinger]
+*
+ Map.jinja: use grains.filter_by instead of defaults.merge. [Alexander Weidinger]
+
+ because defaults.merge does not work with salt-ssh. https://github.com/saltstack/salt/issues/51605
+
+ Added osfingermap.yaml.
+
+`0.1.2 `_ (2019-02-12)
+----------------------------------------------------------------------------------------------------------
+
+
+* Improve comments and examples in osfamilymap & osmap [Imran Iqbal]
+* Fix map.jinja and add more OSes. [Imran Iqbal]
+
+`0.1.1 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Update. [Niels Abspoel]
+* Update formula with map.jinja and style guide references, improve README and VERSION. [Niels Abspoel]
+
+`0.1.0 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+*
+ Examples must be consistent. [Daniel Dehennin]
+
+ The “template” is kept during rendering.
+
+
+ * TOFS_pattern.md: add “template” to rendered state.
+ * template/macros.jinja: ditoo.
+
+*
+ Remove double slash in generated salt URL. [Daniel Dehennin]
+
+ When the files are “full path” with leading slash “/”, the generated URL contain a double slash because of the join.
+
+
+ * template/macros.jinja: remove leading slash before joining parts.
+ * TOFS_pattern.md: mirror changes of “macros.jinja”.
+
+*
+ Add an example for “ntp” of the use of “files_switch” [Daniel Dehennin]
+
+*
+ Accept pillar separator in “files_switch” prefix. [Daniel Dehennin]
+
+ The prefix was used for 2 purposes:
+
+
+ * define the pillar prefix where to lookup “:files_switch”. It supports the colon “:” separator to lookup in pillar subtree like “foo:bar”
+ *
+ define the path prefix where to look for “files/”, It did not support separator to lookup inside directory tree.
+
+ This patch only replace any colon “:” with “/” when looking up “files/” directory, with the “foo:bar” prefix:
+
+ *
+ lookup “foo:bar:files_switch” pillar to get list of grains to match
+
+ * lookup files under “salt://foo/bar/files/”
+ * TOFS_pattern.md: document the new use of “prefix” supporting colon “:”.
+ * template/macros.jinja: transform any colon “:” in “prefix” by slash
+ “/” to lookup files.
+
+*
+ Make TOFS pattern example usable. [Daniel Dehennin]
+
+ The example could not be used as-is. This commit improve conformity to formula conventions.
+
+
+ * TOFS_pattern.md: add missing commas “,” in “map.jinja” and extra one
+ to ease the addition of new entries. Import “map.jinja” in “init.sls” and “conf.sls”. Declare descriptive state IDs. Use the “module.function” notation. Use the “name” parameter.
+
+*
+ Cosmetics modification of TOFS pattern documentation. [Daniel Dehennin]
+
+
+ * TOFS_pattern.md: add myself as modifier.
+ Trim trailing whitespaces. Separate titles from first paragraph.
+
+*
+ Switch template.config to TOFS pattern. [Daniel Dehennin]
+
+* Import TOFS pattern from Zabbix formula. [Daniel Dehennin]
+
+`0.0.9 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Add VERSION file. [Karim Hamza]
+* Add note about formula versioning. [Karim Hamza]
+
+`0.0.8 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Align with SaltStack official formulas doc page. [Denys Havrysh]
+* Use https in the link to SaltStack documentation. [Denys Havrysh]
+
+`0.0.7 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Map.ninja: fix typos and leftover comments. [Marco Molteni]
+* Remove whitespace in map.jinja comment. [Andrew Gabbitas]
+
+`0.0.6 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Improve style and jinja too match salt-formula. [Niels Abspoel]
+* Propose new-ish formula style - defaults live in defaults.yml - map jinja overrides by grain + merges pillar:lookup - split all contextually similar states in their own files. [puneet kandhari]
+
+`0.0.5 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Change states to use short-dec style. [Seth House]
+* Update CHANGELOG.rst. [Nitin Madhok]
+*
+ Update README.rst. [Nitin Madhok]
+
+ Fix broken link
+
+*
+ Fixing pillar to match the map file. [Forrest]
+
+ Map file and pillar didn't match.
+
+`0.0.4 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Add change log. [Antti Jokipii]
+
+`0.0.3 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Updated the license and readme to match our standards. [Forrest Alvarez]
+* Use map.jinja content in init.sls. [Eugene Vereschagin]
+* Add map.jinja. [Eugene Vereschagin]
+
+`0.0.2 `_ (2019-02-10)
+----------------------------------------------------------------------------------------------------------
+
+
+* Add link to Salt Formula documentation. [Eugene Vereschagin]
+* Change extension from .md to .rst. [Eugene Vereschagin]
+
+`0.0.1 `_ (2019-02-10)
+------------------------------------------------------------------------------------------------------
+
+
+* Initial commit. [Lukas Erlacher]
diff --git a/docs/CONTRIBUTING.rst b/docs/CONTRIBUTING.rst
new file mode 100644
index 00000000..7e971ab9
--- /dev/null
+++ b/docs/CONTRIBUTING.rst
@@ -0,0 +1,143 @@
+
+How to contribute
+=================
+
+This document will eventually outline all aspects of guidance to make your contributing experience a fruitful and enjoyable one.
+What it already contains is information about *commit message formatting* and how that directly affects the numerous automated processes that are used for this repo.
+
+Commit message formatting
+-------------------------
+
+Automation of multiple processes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This formula uses `semantic-release `_ for automating numerous processes such as bumping the version number appropriately, creating new tags/releases and updating the changelog.
+The entire process relies on the structure of commit messages to determine the version bump, which is then used for the rest of the automation.
+
+Full details are available in the upstream docs regarding the `Angular Commit Message Conventions `_.
+The key factor is that the first line of the commit message must follow this format:
+
+.. code-block::
+
+ type(scope): subject
+
+
+* E.g. ``docs(contributing): add commit message formatting instructions``.
+
+Besides the version bump, the changelog and release notes are formatted accordingly.
+So based on the example above:
+
+..
+
+ .. raw:: html
+
+ Documentation
+
+
+
+ * **contributing:** add commit message formatting instructions
+
+
+
+* The ``type`` translates into a ``Documentation`` sub-heading.
+* The ``(scope):`` will be shown in bold text without the brackets.
+* The ``subject`` follows the ``scope`` as standard text.
+
+Linting commit messages in Travis CI
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This formula uses `commitlint `_ for checking commit messages during CI testing.
+This ensures that they are in accordance with the ``semantic-release`` settings.
+
+For more details about the default settings, refer back to the ``commitlint`` `reference rules `_.
+
+Relationship between commit type and version bump
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This formula applies some customisations to the defaults, as outlined in the table below,
+based upon the `type `_ of the commit:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Type
+ - Heading
+ - Description
+ - Bump (default)
+ - Bump (custom)
+ * - ``build``
+ - Build System
+ - Changes related to the build system
+ - –
+ -
+ * - ``chore``
+ - –
+ - Changes to the build process or auxiliary tools and libraries such as documentation generation
+ - –
+ -
+ * - ``ci``
+ - Continuous Integration
+ - Changes to the continuous integration configuration
+ - –
+ -
+ * - ``docs``
+ - Documentation
+ - Documentation only changes
+ - –
+ - 0.0.1
+ * - ``feat``
+ - Features
+ - A new feature
+ - 0.1.0
+ -
+ * - ``fix``
+ - Bug Fixes
+ - A bug fix
+ - 0.0.1
+ -
+ * - ``perf``
+ - Performance Improvements
+ - A code change that improves performance
+ - 0.0.1
+ -
+ * - ``refactor``
+ - Code Refactoring
+ - A code change that neither fixes a bug nor adds a feature
+ - –
+ - 0.0.1
+ * - ``revert``
+ - Reverts
+ - A commit used to revert a previous commit
+ - –
+ - 0.0.1
+ * - ``style``
+ - Styles
+ - Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
+ - –
+ - 0.0.1
+ * - ``test``
+ - Tests
+ - Adding missing or correcting existing tests
+ - –
+ - 0.0.1
+
+
+Use ``BREAKING CHANGE`` to trigger a ``major`` version change
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Adding ``BREAKING CHANGE`` to the footer of the extended description of the commit message will **always** trigger a ``major`` version change, no matter which type has been used.
+This will be appended to the changelog and release notes as well.
+To preserve good formatting of these notes, the following format is prescribed:
+
+
+* ``BREAKING CHANGE: .``
+
+An example of that:
+
+.. code-block:: git
+
+ ...
+
+ BREAKING CHANGE: With the removal of all of the `.sls` files under
+ `template/package/`, this formula no longer supports the installation of
+ packages.
diff --git a/docs/TOFS_pattern.rst b/docs/TOFS_pattern.rst
new file mode 100644
index 00000000..d766acd7
--- /dev/null
+++ b/docs/TOFS_pattern.rst
@@ -0,0 +1,510 @@
+
+TOFS: A pattern for using SaltStack
+===================================
+
+Roberto Moreda moreda@allenta.com
+29/12/2014
+
+Modified by Daniel Dehennin daniel.dehennin@baby-gnu.org
+
+Updated by Imran Iqbal https://github.com/myii
+
+All that follows is a proposal based on my experience with `SaltStack `_. The good thing of a piece of software like this is that you can "bend it" to suit your needs in many possible ways, and this is one of them. All the recommendations and thoughts are given "as it is" with no warranty of any type.
+
+Usage of values in pillar vs templates in ``file_roots``
+------------------------------------------------------------
+
+Among other functions, the *master* (or *salt-master*\ ) serves files to the *minions* (or *salt-minions*\ ). The `file_roots `_ is the list of directories used in sequence to find a file when a minion requires it: the first match is served to the minion. Those files could be `state files `_ or configuration templates, among others.
+
+Using SaltStack is a simple and effective way to implement configuration management, but even in a `non-multitenant `_ scenario, it is not a good idea to generally access some data (e.g. the database password in our `Zabbix `_ server configuration file or the private key of our `Nginx `_ TLS certificate).
+
+To avoid this situation we can use the `pillar mechanism `_\ , which is designed to provide controlled access to data from the minions based on some selection rules. As pillar data could be easily integrated in the `Jinja `_ templates, it is a good mechanism to store values to be used in the final rendering of state files and templates.
+
+There are a variety of approaches on the usage of pillar and templates as seen in the `saltstack-formulas `_\ ' repositories. `Some `_ `developments `_ stress the initial purpose of pillar data into a storage for most of the possible variables for a determined system configuration. This, in my opinion, is shifting too much load from the original template files approach. Adding up some `non-trivial Jinja `_ code as essential part of composing the state file definitely makes SaltStack state files (hence formulas) more difficult to read. The extreme of this approach is that we could end up with a new render mechanism, implemented in Jinja, storing everything needed in pillar data to compose configurations. Additionally, we are establishing a strong dependency with the Jinja renderer.
+
+In opposition to the _put the code in file_roots and the data in pillars_ approach, there is the *pillar as a store for a set of key-values* approach. A full-blown configuration file abstracted in pillar and jinja is complicated to develop, understand and maintain. I think a better and simpler approach is to keep a configuration file templated using just a basic (non-extensive but extensible) set of pillar values.
+
+On the reusability of SaltStack state files
+-------------------------------------------
+
+There is a brilliant initiative of the SaltStack community called `salt-formulas `_. Their goal is to provide state files, pillar examples and configuration templates ready to be used for provisioning. I am a contributor for two small ones: `zabbix-formula `_ and `varnish-formula `_.
+
+The `design guidelines `_ for formulas are clear in many aspects and it is a recommended reading for anyone willing to write state files, even non-formulaic ones.
+
+In the next section, I am going to describe my proposal to extend further the reusability of formulas, suggesting some patterns of usage.
+
+The Template Override and Files Switch (TOFS) pattern
+-----------------------------------------------------
+
+I understand a formula as a **complete, independent set of SaltStack state and configuration template files sufficient to configure a system**. A system could be something as simple as an NTP server or some other much more complex service that requires many state and configuration template files.
+
+The customization of a formula should be done mainly by providing pillar data used later to render either the state or the configuration template files.
+
+Example: NTP before applying TOFS
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Let's work with the NTP example. A basic formula that follows the `design guidelines `_ has the following files and directories tree:
+
+.. code-block::
+
+ /srv/saltstack/salt-formulas/ntp-saltstack-formula/
+ ntp/
+ map.jinja
+ init.sls
+ conf.sls
+ files/
+ default/
+ etc/
+ ntp.conf.jinja
+
+In order to use it, let's assume a `masterless configuration `_ and this relevant section of ``/etc/salt/minion``\ :
+
+.. code-block:: yaml
+
+ pillar_roots:
+ base:
+ - /srv/saltstack/pillar
+ file_client: local
+ file_roots:
+ base:
+ - /srv/saltstack/salt
+ - /srv/saltstack/salt-formulas/ntp-saltstack-formula
+
+.. code-block:: jinja
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/map.jinja
+ {%- set ntp = salt['grains.filter_by']({
+ 'default': {
+ 'pkg': 'ntp',
+ 'service': 'ntp',
+ 'config': '/etc/ntp.conf',
+ },
+ }, merge=salt['pillar.get']('ntp:lookup')) %}
+
+In ``init.sls`` we have the minimal states required to have NTP configured. In many cases ``init.sls`` is almost equivalent to an ``apt-get install`` or a ``yum install`` of the package.
+
+.. code-block:: sls
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/init.sls
+ {%- from 'ntp/map.jinja' import ntp with context %}
+
+ Install NTP:
+ pkg.installed:
+ - name: {{ ntp.pkg }}
+
+ Enable and start NTP:
+ service.running:
+ - name: {{ ntp.service }}
+ - enabled: True
+ - require:
+ - pkg: Install NTP package
+
+In ``conf.sls`` we have the configuration states. In most cases, that is just managing configuration file templates and making them to be watched by the service.
+
+.. code-block:: sls
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
+ include:
+ - ntp
+
+ {%- from 'ntp/map.jinja' import ntp with context %}
+
+ Configure NTP:
+ file.managed:
+ - name: {{ ntp.config }}
+ - template: jinja
+ - source: salt://ntp/files/default/etc/ntp.conf.jinja
+ - watch_in:
+ - service: Enable and start NTP service
+ - require:
+ - pkg: Install NTP package
+
+Under ``files/default``\ , there is a structure that mimics the one in the minion in order to avoid clashes and confusion on where to put the needed templates. There you can find a mostly standard template for the configuration file.
+
+.. code-block:: jinja
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/files/default/etc/ntp.conf.jinja
+ # Managed by saltstack
+ # Edit pillars or override this template in saltstack if you need customization
+ {%- set settings = salt['pillar.get']('ntp', {}) %}
+ {%- set default_servers = ['0.ubuntu.pool.ntp.org',
+ '1.ubuntu.pool.ntp.org',
+ '2.ubuntu.pool.ntp.org',
+ '3.ubuntu.pool.ntp.org'] %}
+
+ driftfile /var/lib/ntp/ntp.drift
+ statistics loopstats peerstats clockstats
+ filegen loopstats file loopstats type day enable
+ filegen peerstats file peerstats type day enable
+ filegen clockstats file clockstats type day enable
+
+ {%- for server in settings.get('servers', default_servers) %}
+ server {{ server }}
+ {%- endfor %}
+
+ restrict -4 default kod notrap nomodify nopeer noquery
+ restrict -6 default kod notrap nomodify nopeer noquery
+
+ restrict 127.0.0.1
+ restrict ::1
+
+With all this, it is easy to install and configure a simple NTP server by just running ``salt-call state.sls ntp.conf``\ : the package will be installed, the service will be running and the configuration should be correct for most of cases, even without pillar data.
+
+Alternatively, you can define a highstate in ``/srv/saltstack/salt/top.sls`` and run ``salt-call state.highstate``.
+
+.. code-block:: sls
+
+ ## /srv/saltstack/salt/top.sls
+ base:
+ '*':
+ - ntp.conf
+
+**Customizing the formula just with pillar data**\ , we have the option to define the NTP servers.
+
+.. code-block:: sls
+
+ ## /srv/saltstack/pillar/top.sls
+ base:
+ '*':
+ - ntp
+
+.. code-block:: sls
+
+ ## /srv/saltstack/pillar/ntp.sls
+ ntp:
+ servers:
+ - 0.ch.pool.ntp.org
+ - 1.ch.pool.ntp.org
+ - 2.ch.pool.ntp.org
+ - 3.ch.pool.ntp.org
+
+Template Override
+^^^^^^^^^^^^^^^^^
+
+If the customization based on pillar data is not enough, we can override the template by creating a new one in ``/srv/saltstack/salt/ntp/files/default/etc/ntp.conf.jinja``
+
+.. code-block:: jinja
+
+ ## /srv/saltstack/salt/ntp/files/default/etc/ntp.conf.jinja
+ # Managed by saltstack
+ # Edit pillars or override this template in saltstack if you need customization
+
+ # Some bizarre configurations here
+ # ...
+
+ {%- for server in settings.get('servers', default_servers) %}
+ server {{ server }}
+ {%- endfor %}
+
+This way we are locally **overriding the template files** offered by the formula in order to make a more complex adaptation. Of course, this could be applied as well to any of the files, including the state files.
+
+Files Switch
+^^^^^^^^^^^^
+
+To bring some order into the set of template files included in a formula, as we commented, we suggest having a similar structure to a normal final file system under ``files/default``.
+
+We can make different templates coexist for different minions, classified by any `grain `_ value, by simply creating new directories under ``files``. This mechanism is based on **using values of some grains as a switch for the directories under ``files/``\ **.
+
+If we decide that we want ``os_family`` as switch, then we could provide the formula template variants for both the ``RedHat`` and ``Debian`` families.
+
+.. code-block::
+
+ /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/files/
+ default/
+ etc/
+ ntp.conf.jinja
+ RedHat/
+ etc/
+ ntp.conf.jinja
+ Debian/
+ etc/
+ ntp.conf.jinja
+
+To make this work we need a ``conf.sls`` state file that takes a list of possible files as the configuration template.
+
+.. code-block:: sls
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
+ include:
+ - ntp
+
+ {%- from 'ntp/map.jinja' import ntp with context %}
+
+ Configure NTP:
+ file.managed:
+ - name: {{ ntp.config }}
+ - template: jinja
+ - source:
+ - salt://ntp/files/{{ grains.get('os_family', 'default') }}/etc/ntp.conf.jinja
+ - salt://ntp/files/default/etc/ntp.conf.jinja
+ - watch_in:
+ - service: Enable and start NTP service
+ - require:
+ - pkg: Install NTP package
+
+If we want to cover the possibility of a special template for a minion identified by ``node01`` then we could have a specific template in ``/srv/saltstack/salt/ntp/files/node01/etc/ntp.conf.jinja``.
+
+.. code-block:: jinja
+
+ ## /srv/saltstack/salt/ntp/files/node01/etc/ntp.conf.jinja
+ # Managed by saltstack
+ # Edit pillars or override this template in saltstack if you need customization
+
+ # Some crazy configurations here for node01
+ # ...
+
+To make this work we could write a specially crafted ``conf.sls``.
+
+.. code-block:: sls
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
+ include:
+ - ntp
+
+ {%- from 'ntp/map.jinja' import ntp with context %}
+
+ Configure NTP:
+ file.managed:
+ - name: {{ ntp.config }}
+ - template: jinja
+ - source:
+ - salt://ntp/files/{{ grains.get('id') }}/etc/ntp.conf.jinja
+ - salt://ntp/files/{{ grains.get('os_family') }}/etc/ntp.conf.jinja
+ - salt://ntp/files/default/etc/ntp.conf.jinja
+ - watch_in:
+ - service: Enable and start NTP service
+ - require:
+ - pkg: Install NTP package
+
+Using the ``files_switch`` macro
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We can simplify the ``conf.sls`` with the new ``files_switch`` macro to use in the ``source`` parameter for the ``file.managed`` state.
+
+.. code-block:: sls
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
+ include:
+ - ntp
+
+ {%- set tplroot = tpldir.split('/')[0] %}
+ {%- from 'ntp/map.jinja' import ntp with context %}
+ {%- from 'ntp/macros.jinja' import files_switch %}
+
+ Configure NTP:
+ file.managed:
+ - name: {{ ntp.config }}
+ - template: jinja
+ - source: {{ files_switch(
+ salt['config.get'](
+ tplroot ~ ':tofs:files:Configure NTP',
+ ['/etc/ntp.conf.jinja']
+ )
+ ) }}
+ - watch_in:
+ - service: Enable and start NTP service
+ - require:
+ - pkg: Install NTP package
+
+
+* This uses ``config.get``\ , searching for ``nfs:tofs:files:Configure NTP`` to determine the list of template files to use.
+* If this does not yield any results, the default of ``['/etc/ntp.conf.jinja']`` will be used.
+
+In ``macros.jinja``\ , we define this new macro ``files_switch``.
+
+.. code-block:: jinja
+
+ ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/macros.jinja
+ {%- macro files_switch(files,
+ default_files_switch=['id', 'os_family'],
+ indent_width=6) %}
+ {#-
+ Returns a valid value for the "source" parameter of a "file.managed"
+ state function. This makes easier the usage of the Template Override and
+ Files Switch (TOFS) pattern.
+
+ Params:
+ * files: ordered list of files to look for
+ * default_files_switch: if there's no pillar
+ ':tofs:files_switch' this is the ordered list of grains to
+ use as selector switch of the directories under
+ "/files"
+ * indent_witdh: indentation of the result value to conform to YAML
+
+ Example (based on a `tplroot` of `xxx`):
+
+ If we have a state:
+
+ Deploy configuration:
+ file.managed:
+ - name: /etc/yyy/zzz.conf
+ - source: {{ files_switch(
+ salt['config.get'](
+ tplroot ~ ':tofs:files:Deploy configuration',
+ ['/etc/yyy/zzz.conf', '/etc/yyy/zzz.conf.jinja']
+ )
+ ) }}
+ - template: jinja
+
+ In a minion with id=theminion and os_family=RedHat, it's going to be
+ rendered as:
+
+ Deploy configuration:
+ file.managed:
+ - name: /etc/yyy/zzz.conf
+ - source:
+ - salt://xxx/files/theminion/etc/yyy/zzz.conf
+ - salt://xxx/files/theminion/etc/yyy/zzz.conf.jinja
+ - salt://xxx/files/RedHat/etc/yyy/zzz.conf
+ - salt://xxx/files/RedHat/etc/yyy/zzz.conf.jinja
+ - salt://xxx/files/default/etc/yyy/zzz.conf
+ - salt://xxx/files/default/etc/yyy/zzz.conf.jinja
+ - template: jinja
+ #}
+ {#- Get the `tplroot` from `tpldir` #}
+ {%- set tplroot = tpldir.split('/')[0] %}
+ {%- set path_prefix = salt['config.get'](tplroot ~ ':tofs:path_prefix', tplroot) %}
+ {%- set files_dir = salt['config.get'](tplroot ~ ':tofs:dirs:files', 'files') %}
+ {%- set files_switch_list = salt['config.get'](
+ tplroot ~ ':tofs:files_switch',
+ default_files_switch
+ ) %}
+ {#- Only add to [''] when supporting older TOFS implementations #}
+ {%- for path_prefix_ext in [''] %}
+ {%- set path_prefix_inc_ext = path_prefix ~ path_prefix_ext %}
+ {#- For older TOFS implementation, use `files_switch` from the pillar #}
+ {#- Use the default, new method otherwise #}
+ {%- set fsl = salt['pillar.get'](
+ tplroot ~ path_prefix_ext|replace('/', ':') ~ ':files_switch',
+ files_switch_list
+ ) %}
+ {#- Append an empty value to evaluate as `default` in the loop below #}
+ {%- if '' not in fsl %}
+ {%- do fsl.append('') %}
+ {%- endif %}
+ {%- for fs in fsl %}
+ {%- for file in files %}
+ {%- if fs %}
+ {%- set fs_dir = salt['config.get'](fs, fs) %}
+ {%- else %}
+ {%- set fs_dir = salt['config.get'](tplroot ~ ':tofs:dirs:default', 'default') %}
+ {%- endif %}
+ {%- set url = '- salt://' ~ '/'.join([
+ path_prefix_inc_ext,
+ files_dir,
+ fs_dir,
+ file.lstrip('/')
+ ]) %}
+ {{ url | indent(indent_width, true) }}
+ {%- endfor %}
+ {%- endfor %}
+ {%- endfor %}
+ {%- endmacro %}
+
+How to customise the ``source`` further
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The examples below are based on an ``Ubuntu`` minion called ``theminion`` being configured via. pillar.
+
+Using the default settings of the ``files_switch`` macro above,
+the ``source`` will be:
+
+.. code-block:: sls
+
+ - source:
+ - salt://ntp/files/theminion/etc/ntp.conf.jinja
+ - salt://ntp/files/Debian/etc/ntp.conf.jinja
+ - salt://ntp/files/default/etc/ntp.conf.jinja
+
+Customise ``files``
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``files`` portion can be customised:
+
+.. code-block:: sls
+
+ ntp:
+ tofs:
+ dirs:
+ files: files_alt
+
+Resulting in:
+
+.. code-block:: sls
+
+ - source:
+ - salt://ntp/files_alt/theminion/etc/ntp.conf.jinja
+ - salt://ntp/files_alt/Debian/etc/ntp.conf.jinja
+ - salt://ntp/files_alt/default/etc/ntp.conf.jinja
+
+Customise the use of grains
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Grains can be customised and even arbitrary paths can be supplied:
+
+.. code-block:: sls
+
+ ntp:
+ tofs:
+ files_switch:
+ - any/path/can/be/used/here
+ - id
+ - os
+ - os_family
+
+Resulting in:
+
+.. code-block:: sls
+
+ - source:
+ - salt://ntp/files/any/path/can/be/used/here/etc/ntp.conf.jinja
+ - salt://ntp/files/theminion/etc/ntp.conf.jinja
+ - salt://ntp/files/Ubuntu/etc/ntp.conf.jinja
+ - salt://ntp/files/Debian/etc/ntp.conf.jinja
+ - salt://ntp/files/default/etc/ntp.conf.jinja
+
+Customise the ``default`` path
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``default`` portion of the path can be customised:
+
+.. code-block:: sls
+
+ ntp:
+ tofs:
+ dirs:
+ default: default_alt
+
+Resulting in:
+
+.. code-block:: sls
+
+ - source:
+ ...
+ - salt://ntp/files/default_alt/etc/ntp.conf.jinja
+
+Customise the list of template ``files``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The list of template ``files`` can be given:
+
+.. code-block:: sls
+
+ ntp:
+ tofs:
+ files:
+ Configure NTP:
+ - '/etc/ntp.conf.jinja'
+ - '/etc/ntp.conf_alt.jinja'
+
+Resulting in:
+
+.. code-block:: sls
+
+ - source:
+ - salt://ntp/files/theminion/etc/ntp.conf.jinja
+ - salt://ntp/files/theminion/etc/ntp.conf_alt.jinja
+ - salt://ntp/files/Debian/etc/ntp.conf.jinja
+ - salt://ntp/files/Debian/etc/ntp.conf_alt.jinja
+ - salt://ntp/files/default/etc/ntp.conf.jinja
+ - salt://ntp/files/default/etc/ntp.conf_alt.jinja
diff --git a/docs/index.rst b/docs/index.rst
index bebbaf03..e13a2b91 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -2,24 +2,17 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
-Welcome to ``template-formula``'s documentation!
-================================================
+Welcome to template-formula's documentation!
+============================================
.. toctree::
- :maxdepth: 2
+ :maxdepth: 3
:caption: Contents
+ :numbered:
+ :glob:
README
-
-
-
-.. :numbered:
-.. :glob:
-.. :titlesonly:
-.. :hidden:
-.. Indices and tables
-.. ==================
-..
-.. * :ref:`genindex`
-.. .. * :ref:`modindex`
-.. * :ref:`search`
+ AUTHORS
+ CONTRIBUTING
+ TOFS_pattern
+ CHANGELOG
diff --git a/release.config.js b/release.config.js
index 36e2a8ec..5281e971 100644
--- a/release.config.js
+++ b/release.config.js
@@ -14,7 +14,7 @@ module.exports = {
prepareCmd: 'sh ./update_FORMULA.sh ${nextRelease.version}',
}],
['@semantic-release/git', {
- assets: ['*.md', 'FORMULA'],
+ assets: ['*.md', '*.rst', 'FORMULA'],
}],
'@semantic-release/github',
],