Any known documentation as tests tooling around bats-core?

[djmcgreal-cc]

Hi,
We've been very happily using bats as our go-to tool for all things testing. We'd like to start aligning our documentation with the tests themselves, so that, when presented to users, we've a fair chance of it working and being consistent with the rest of the application.

Is anyone aware of any frameworks for such that synergise with bats, etc?

Specifically we're doing a lot of Kubernetes API testing.

Thanks! Dan.

[martin-schulze-vireso]

Do you have an example of what you mean from another language/test framework?

[djmcgreal-cc]

I'm just getting into researching the tools, so thought I'd ask in this community (as ideally what we use would have the consistent reporting control and consistency) for anything I should look at (I'm finding Google fairly noisy). I found https://github.com/anko/txm.

When I think about what I'd like ideally, I think I'd like some script to parse a markdown document's front matter for something like

bats:
  setup_file:
    heading: Setup
  tests:
    - heading: Create a Pod that authenticates against the GitLab API
      asserts: kubectl wait --for condition=...
    - heading: Create a Pod that performs git operations
  teardown_file:
    script: kubectl delete -l documentation-tests

Then use remark or mdast or so to generate a bats test script from the code blocks found in the headings configured.

Ideally the test could be generated from bats. I don't know what happens if bats generates a bats test file in e.g. setup_suite or so and then includes it?

[djmcgreal-cc]

I ended up writing something myself, which sprawled out from something I thought was quite simple. In case it's useful for anyone:

With some markdown like:

# Setup <!-- setup_file() -->
`commands to initialise the tests`
# Here's how to do a thing <!-- @test -->
`commands to do a thing`
# Or another <!-- @test -->
`more commands doing something else`
# Cleanup <!-- teardown_file() -->
`clear it up`

The following generates a bats file with a setup/teardown/test per commented header, extracting the code via remark --tree-only.

#!/usr/bin/env sh

echo 'set -o pipefail'
echo 'cd $BATS_FILE_TMPDIR'; echo

output_function(){
    echo "$1 {"
    echo "$2" | sed 's/^/  /g;$d'
    echo "}"
    echo
}

next_function="" should_print=0 code=""
headings_and_codes=$(remark exemplars/internal-gitlab/README.md --tree-out 2>/dev/null \
    | jq -c '.. | select(.type? == "heading" or .type? == "code" or .type? == "inlineCode")') #or .type? == "inlineCode" #trouble with inlineCode is how to exclude?

IFS=$'\n'
for heading_or_code in $headings_and_codes; do
    comment=$(jq -n --argjson heading "$heading_or_code" -rn \
        '$heading | .children[]? | select(.type == "html") | .value')
    case "$comment" in
        "<!-- setup_file() -->") 
            [ "$next_function" != "" ] && output_function "$next_function" "$code"
            next_function="setup_file()" code=""
            continue
            ;;
        "<!-- @test -->")
            [ "$next_function" != "" ] && output_function "$next_function" "$code"
            next_function="@test "$(jq -n --argjson heading "$heading_or_code" -rn \
                '$heading | .children[0].value') code=""
            continue
            ;;
        "<!-- teardown_file() -->") 
            [ "$next_function" != "" ] && output_function "$next_function" "$code"
            next_function="teardown_file()" code=""
            continue
            ;;
    esac

    [ "$next_function" == "" ] && continue

    next_code=$(jq -n --argjson code "$heading_or_code" -r '$code | select(.lang? == "sh" or .type? == "inlineCode") | .value') # or .type == "inlineCode"
    code="${code}${next_code}\n"
done
[ "$next_function" != "" ] && output_function "$next_function" "$code"

I do this offline, and have another @test somewhere else that checks that the .bats file on disk is in-sync with the output from the generator. It doesn't do it in CI so a) can be sanity checked and b) can be run easily locally just by bats from the existing file.