Merge pull request #44 from cdentyou/tcase_tags

RFC: Support arbitrary tagging and selection of testcases.

doc/check.texi

@@ -974,6 +974,8 @@ easier for the developer to write, run, and analyze tests.
 * Test Fixtures::               
 * Multiple Suites in one SRunner::  
 * Selective Running of Tests::
+* Selecting Tests by Suite or Test Case::
+* Selecting Tests Based on Arbitrary Tags::
 * Testing Signal Handling and Exit Values::  
 * Looping Tests::               
 * Test Timeouts::               
@@ -1302,19 +1304,105 @@ srunner_add_suite (sr, make_pack_suite ());
 @end example
 
 @node Selective Running of Tests, Testing Signal Handling and Exit Values, Multiple Suites in one SRunner, Advanced Features
+
 @section Selective Running of Tests
 
+After adding a couple of suites and some test cases in each, it is
+sometimes practical to be able to run only one suite, or one specific
+test case, without recompiling the test code.  Check provides two ways
+to accomplish this, either by specifying a suite or test case by name
+or by assigning tags to test cases and specifying one or more tags to
+run.
+
+@menu
+* Selecting Tests by Suite or Test Case::
+* Selecting Tests Based on Arbitrary Tags::
+@end menu
+
+@node Selecting Tests by Suite or Test Case, Selecting Tests Based on Arbitrary Tags, Selective Running of Tests, Selective Running of Tests
+@subsection Selecting Tests by Suite or Test Case
+
 @vindex CK_RUN_SUITE
 @vindex CK_RUN_CASE
-After adding a couple of suites and some test cases in each, it is
-sometimes practical to be able to run only one suite, or one
-specific test case, without recompiling the test code. There are
-two environment variables available that offers this ability,
-@code{CK_RUN_SUITE} and @code{CK_RUN_CASE}. Just set the value to
-the name of the suite and/or test case you want to run. These
-environment variables can also be a good integration tool for
-running specific tests from within another tool, e.g. an IDE.
-
+
+There are two environment variables available that offer this
+ability, @code{CK_RUN_SUITE} and @code{CK_RUN_CASE}. Just set the
+value to the name of the suite and/or test case you want to run. These
+environment variables can also be a good integration tool for running
+specific tests from within another tool, e.g. an IDE.
+
+@node Selecting Tests Based on Arbitrary Tags, ,Selecting Tests by Suite or Test Case, Selective Running of Tests
+@subsection Selecting Tests Based on Arbitrary Tags
+
+@vindex CK_INCLUDE_TAGS
+@vindex CK_EXCLUDE_TAGS
+
+It can be useful to dynamically include or exclude groups of tests to
+be run based on criteria other than the suite or test case name. For
+example, one or more tags can be assigned to test cases. The tags
+could indicate if a test runs for a long time, so such tests could be
+excluded in order to run quicker tests for a sanity
+check. Alternately, tags may be used to indicate which functional
+areas test cover. Tests can then be run that include all test cases
+for a given set of functional areas.
+
+In Check, a tag is a string of characters without white space. One or
+more tags can be assigned to a test case by using the
+@code{tcase_set_tags} function. This function accepts a string, and
+multiple tags can be specified by delimiting them with spaces. For
+example:
+
+@example
+@verbatim
+    Suite *s;
+
+    TCase *red, *blue, *purple, *yellow, *black;
+
+    s = suite_create("Check Tag Filtering");
+
+    red = tcase_create("Red");
+    tcase_set_tags(red, "Red");
+    suite_add_tcase (s, red);
+    tcase_add_test(red, red_test1);
+
+    blue = tcase_create("Blue");
+    tcase_set_tags(blue, "Blue");
+    suite_add_tcase (s, blue);
+    tcase_add_test(blue, blue_test1);
+
+    purple = tcase_create("Purple");
+    tcase_set_tags(purple, "Red Blue");
+    suite_add_tcase (s, purple);
+    tcase_add_test(purple, purple_test1);
+
+@end verbatim
+@end example
+
+Once test cases are tagged they may be selectively run in one of two ways:
+
+a) Using Environment Variables
+
+There are two environment variables available for selecting test cases
+based on tags: @code{CK_INCLUDE_TAGS} and
+@code{CK_EXCLUDE_TAGS}. These can be set to a space separated list of
+tag names. If @code{CK_INCLUDE_TAGS} is set then test cases which
+include at least one tag in common with @code{CK_INCLUDE_TAGS} will be
+run. If @code{CK_EXCLUDE_TAGS} is set then test cases with one tag in
+common with @code{CK_EXCLUDE_TAGS} will not be run. In cases where
+both @code{CK_INCLUDE_TAGS} and @code{CK_EXCLUDE_TAGS} match a tag for
+a test case the test will be excluded.
+
+Both @code{CK_INCLUDE_TAGS} and @code{CK_EXCLUDE_TAGS} can be
+specified in conjunction with @code{CK_RUN_SUITE} or even
+@code{CK_RUN_CASE} in which case they will have the effect of further
+narrowing the selection.
+
+b) Programmatically
+
+The @code{srunner_run_tagged} function allows one to specify which
+tags to run or exclude from a suite runner. This can be used to
+programmatically control which test cases may run.
+
 @node Testing Signal Handling and Exit Values, Looping Tests, Selective Running of Tests, Advanced Features
 @section Testing Signal Handling and Exit Values
 
@@ -1975,6 +2063,10 @@ CK_RUN_CASE: Name of a test case, runs only that test. See section @ref{Selectiv
 
 CK_RUN_SUITE: Name of a test suite, runs only that suite. See section @ref{Selective Running of Tests}.
 
+CK_INCLUDE_TAGS: String of space separated tags, runs only test cases associated with at least one of the tags, See section @ref{Selecting Tests Based on Arbitrary Tags}.
+
+CK_EXCLUDE_TAGS: String of space separated tags, runs only test cases not associated with any of the tags, See section @ref{Selecting Tests Based on Arbitrary Tags}.
+
 CK_VERBOSITY: How much output to emit, accepts: ``silent'', ``minimal'', ``normal'', ``subunit'', or ``verbose''.  See section @ref{SRunner Output}.
 
 CK_FORK: Set to ``no'' to disable using fork() to run unit tests in their own process. This is useful for debugging segmentation faults.  See section @ref{No Fork Mode}.

src/check.c

@@ -103,6 +103,7 @@ static void suite_free(Suite * s)
     free(s);
 }
 
+
 TCase *tcase_create(const char *name)
 {
     char *env;
@@ -149,10 +150,49 @@ TCase *tcase_create(const char *name)
     tc->ch_sflst = check_list_create();
     tc->unch_tflst = check_list_create();
     tc->ch_tflst = check_list_create();
+    tc->tags = check_list_create();
 
     return tc;
 }
 
+/*
+ * Helper function to create a list of tags from
+ * a space separated string.
+ */
+List *tag_string_to_list(const char *tags_string)
+{
+    List *list;
+    char *tags;
+    char *tag;
+
+    list = check_list_create();
+
+    if (NULL == tags_string)
+    {
+	return list;
+    }
+
+    tags = strdup(tags_string);
+    tag = strtok(tags, " ");
+    while (tag)
+    {
+	check_list_add_end(list, strdup(tag));
+	tag = strtok(NULL, " ");
+    }
+    free(tags);
+    return list;
+}
+
+void tcase_set_tags(TCase * tc, const char *tags_orig)
+{
+    /* replace any pre-existing list */
+    if (tc->tags)
+    {
+	check_list_apply(tc->tags, free);
+	check_list_free(tc->tags);
+    }
+    tc->tags = tag_string_to_list(tags_orig);
+}
 
 static void tcase_free(TCase * tc)
 {
@@ -161,15 +201,40 @@ static void tcase_free(TCase * tc)
     check_list_apply(tc->ch_sflst, free);
     check_list_apply(tc->unch_tflst, free);
     check_list_apply(tc->ch_tflst, free);
+    check_list_apply(tc->tags, free);
     check_list_free(tc->tflst);
     check_list_free(tc->unch_sflst);
     check_list_free(tc->ch_sflst);
     check_list_free(tc->unch_tflst);
     check_list_free(tc->ch_tflst);
-
+    check_list_free(tc->tags);
     free(tc);
 }
 
+unsigned int tcase_matching_tag(TCase *tc, List *check_for)
+{
+
+    if (NULL == check_for)
+    {
+	return 0;
+    }
+
+    for(check_list_front(check_for); !check_list_at_end(check_for);
+        check_list_advance(check_for))
+    {
+	for(check_list_front(tc->tags); !check_list_at_end(tc->tags);
+	    check_list_advance(tc->tags))
+	{
+	    if (0 == strcmp((const char *)check_list_val(tc->tags),
+			    (const char *)check_list_val(check_for)))
+	    {
+		return 1;
+	    }
+	}
+    }
+    return 0;
+}
+
 void suite_add_tcase(Suite * s, TCase * tc)
 {
     if(s == NULL || tc == NULL || check_list_contains(s->tclst, tc))

src/check.h.in

@@ -172,6 +172,19 @@ CK_DLL_EXP void CK_EXPORT suite_add_tcase(Suite * s, TCase * tc);
  * */
 CK_DLL_EXP TCase *CK_EXPORT tcase_create(const char *name);
 
+/**
+ * Associate a test case with certain tags.
+ * Replaces any existing tags with the new set.
+ *
+ * @param tc the test case
+ *
+ * @param tags string containing arbitrary tags separated by spaces.
+ *        This will be copied. Passing NULL clears all tags.
+ *
+ * @since 0.11.0
+ * */
+CK_DLL_EXP void CK_EXPORT tcase_set_tags(TCase * tc,
+					 const char *tags);
 /**
  * Add a test function to a test case
  *
@@ -955,8 +968,9 @@ CK_DLL_EXP void CK_EXPORT srunner_free(SRunner * sr);
  * In addition to running all suites, if the suite runner has been
  * configured to output to a log, that is also performed.
  *
- * Note that if the CK_RUN_CASE and/or CK_RUN_SUITE environment variables
- * are defined, then only the named suite and/or test case is run.
+ * Note that if the CK_RUN_CASE, CK_RUN_SUITE, CK_INCLUDE_TAGS and/or
+ * CK_EXCLUDE_TAGS environment variables are defined, then only the
+ * named suites or test cases will run.
  *
  * @param sr suite runner to run all suites from
  * @param print_mode the verbosity in which to report results to stdout
@@ -974,9 +988,22 @@ CK_DLL_EXP void CK_EXPORT srunner_run_all(SRunner * sr,
  * suite runner has been configured to output to a log, that is also
  * performed.
  *
+ * Note that if the sname and tcname parameters are passed as null
+ * then the function will fallback to using the environment variables
+ * CK_RUN_SUITE and CK_RUN_CASE respectively in order to select the
+ * suite/cases.
+ * 
+ * Similarly if the CK_INCLUDE_TAGS and/or CK_EXCLUDE_TAGS environment
+ * variables are defined then these will further filter the test cases
+ * (see srunner_run_tagged, below).
+ *
  * @param sr suite runner where the given suite or test case must be
- * @param sname suite name to run. A NULL means "any suite".
- * @param tcname test case name to run. A NULL means "any test case"
+ * @param sname suite name to run. A NULL means use the value of the
+ * environment variable CK_RUN_SUITE if set, otherwise run "any/every
+ * suite".
+ * @param tcname test case name to run. A NULL means use the value of
+ * the environment variable CK_RUN_CASE if set, otherwise run
+ * "any/every case".
  * @param print_mode the verbosity in which to report results to stdout
  *
  * @since 0.9.9
@@ -986,6 +1013,46 @@ CK_DLL_EXP void CK_EXPORT srunner_run(SRunner * sr, const char *sname,
                                       enum print_output print_mode);
 
 
+/**
+ * Run a specific suite or test case or testcases with specific tags
+ * from a suite runner, printing results to stdout as specified by the
+ * print_mode.
+ *
+ * In addition to running any applicable suites or test cases, if the
+ * suite runner has been configured to output to a log, that is also
+ * performed.
+ *
+ * Note that if sname, tcname, include_tags, exclude_tags parameters
+ * are passed as NULL then if the environment variables CK_RUN_SUITE,
+ * CK_RUN_CASE, CK_INCLUDE_TAGS, CK_EXCLUDE_TAGS are defined then these
+ * values will be used instead.
+ *
+ * @param sr suite runner where the given suite or test case must be
+ * @param sname suite name to run. A NULL means use the value of the
+ * environment variable CK_RUN_SUITE if set, otherwise run "any/every
+ * suite".
+ * @param tcname test case name to run. A NULL means use the value of
+ * the environment variable CK_RUN_CASE if set, otherwise run
+ * "any/every case".
+ * @param include_tags space separate list of tags. Only run test
+ * cases that share one of these tags. A NULL means use the value of
+ * the environment variable CK_INCLUDE_TAGS if set, otherwise run
+ * "any/every test case".
+ * @param exclude_tags space separate list of tags. Only run test
+ * cases that do not share one of these tags even if they are selected
+ * by an included tag. A NULL means use the value of the environment
+ * variable CK_EXCLUDE_TAGS if set, otherwise run "any/every test
+ * case".
+ * @param print_mode the verbosity in which to report results to stdout
+ *
+ * @since 0.11.0
+ */
+CK_DLL_EXP void CK_EXPORT srunner_run_tagged(SRunner * sr, const char *sname,
+					     const char *tcname,
+					     const char *include_tags,
+					     const char *exclude_tags,
+					     enum print_output print_mode);
+
 /**
  * Retrieve the number of failed tests executed by a suite runner.
  *

src/check_impl.h

@@ -65,6 +65,7 @@ struct TCase
     List *unch_tflst;
     List *ch_sflst;
     List *ch_tflst;
+    List *tags;
 };
 
 typedef struct TestStats
@@ -134,4 +135,7 @@ enum fork_status cur_fork_status(void);
 
 clockid_t check_get_clockid(void);
 
+unsigned int tcase_matching_tag(TCase *tc, List *check_for);
+List *tag_string_to_list(const char *tags_string);
+
 #endif /* CHECK_IMPL_H */