ERL-1326: Common Test: empty groups should not fail

[OTP-Maintainer]

Original reporter: JIRAUSER15901
Affected version: OTP-23
Component: common_test
Migrated from: https://bugs.erlang.org/browse/ERL-1326

---

*Sample code*

{code:erlang}
-module(example_SUITE).
-include_lib("stdlib/include/assert.hrl").

-compile(export_all).

all() -> [{group, g1}, {group, g2}].

groups() -> [{g1, [t1]}, {g2, []}].

t1(_Config) ->
  ok.
{code}

*Current functionality*

{{ct_run -suite example_SUITE}}

At present, running this command will cause ct to fail with error "Invalid reference to group g2 in example_SUITE:all/0"

*Improvement suggestion*

This example should not generate an error. An empty list of tests is a valid list. It could be silently skipped instead.

*Motivation*

If you evaluate {{[length(X) || X <- [[1, 2], [], [3]]].}}, you do not expect an error.

A group could be temporarily empty, and it's inconvenient when tests fail based on that.

[OTP-Maintainer]

paulo.ferraz.oliveira@gmail.com said:

I also ran into this very issue (which causes a silent {{rebar3}} to exit without warnings) and a potentially related one:
 * {{ct:run_test/1}} will not output \{error, _} in this very same case, but instead {{{Ok, Failed, {UserSkipped, AutoSkipped}} = \{0, 0, {0, 0}}}}.

[OTP-Maintainer]

paulo.ferraz.oliveira@gmail.com said:

Somewhat related:
 I had my group specified as {ok, [sequence], \{handle_for}} (handle_for should be in a list) and again the issue was silent.