Host-based test coverage does not exclude resources outside of module

[JohnLyman]

In http://rspec-puppet.com/documentation/coverage/#excluded-resources, it says:

Resources declared outside of the module being tested (i.e. resources added by module dependencies) are automatically excluded from the coverage report.

When testing a host, it reports untouched resources that are in a dependent module. Is this intended behavior when testing hosts (i.e. does the documentation not apply for host-based tests)? If so, is there a work-around? I can filter the external resources with add_filter, but this is tedious with numerous resources and I can see it becoming a management nightmare.

Possibly related to #490

Thanks

[rodjek]

Yes, this is intended behaviour as hosts are not associated with any specific module. Can you give me some more details around the host you're testing so that we can try to improve the situation for you?

The first thought that comes to mind is maybe an add_filter_module (probably named something better) method that allows you to filter out all the resources in a module. Or maybe allow host specs to specify a module to focus on which automatically filters out resources outside the named module?

Happy to hear any ideas you have :)

[JohnLyman]

Can you give me some more details around the host you're testing so that we can try to improve the situation for you?

So now that I've been thinking about this, I'm guessing we are not using host-based tests as intended (which is what every developer loves to hear lol.) Often for our internal modules, we add host-based tests to modules to test different commonly-used (at our company) global/environmental hiera settings. This is important for us because hiera is the only interface exposed for our end users to make changes.

We've also been using host-based tests in modules to test that OS-based data-in-module hiera settings are working as expected. In other words, spec/classes covers individual classes & parameters, and spec/hosts covers that we set the pseudo-private module hiera data correctly for each OS. Incidentally, my coworker is the one who requested coverage reports for host-based tests with #466 for the exact use cases I just described.

I'm guessing your intention is that host-based tests are for testing site manifests and/or roles. In that case, it makes sense that the coverage report includes all dependencies, etc. When testing hosts in a module, however, I don't think it ever makes sense. I don't care that I'm not covering some third-party's resources because that's the job of the respective third-party module.

Or maybe allow host specs to specify a module to focus on which automatically filters out resources outside the named module?

I like this option better than the first. Another option I like would be a boolean toggle to include external dependencies in the coverage report or not.

Of course, "only use host-based tests as intended" is also an acceptable answer :)

Thanks for the great work!