Is-13 add basic API tests #842

pkeroulas · 2024-01-17T20:58:49Z

Successfully tested against:
garethsb/nmos-cpp@04ac4fd
AMWA-TV/is-13@8655e0c

Successfully tested against: garethsb/nmos-cpp@04ac4fd AMWA-TV/is-13@8655e0c

When possible, TestResult.detail should be displayed in stderr since the raised exception prevents from the report creation.

peterbrightwell

Pat demonstrated this on a resource labelling call and it looks good

For each resource&objects: - Read initial value and store - Reset default value, check timestamp and store - Write max-length and check value+timestamp - Write >max-length and check value+timestamp - Reset default value and compare - Restore initial value Apply to objects: label, description, tags TODO: check IS04

This seems to impact the version(timestamp) test.

pkeroulas · 2024-04-23T18:32:18Z

@garethsb @jonathan-r-thorpe please, can you have a 1st look at this?

jonathan-r-thorpe · 2024-04-29T09:24:40Z

nmostesting/suites/IS1301Test.py

+RESOURCES = ["self", "devices", "senders", "receivers"]
+OBJECTS = ["label", "description", "tags"]
+
+# const for label&description-related tests


Suggested change

# const for label&description-related tests

# Constants for label- and description-related tests

jonathan-r-thorpe · 2024-04-29T09:24:58Z

nmostesting/suites/IS1301Test.py

+STRING_OVER_MAX_VALUE = ''.join(['X' for i in range(100)])
+STRING_MAX_VALUE = STRING_OVER_MAX_VALUE[:64]  # this is the max length tolerated
+
+# const for tags-related tests


Suggested change

# const for tags-related tests

# Constants for tags-related tests

jonathan-r-thorpe · 2024-04-29T09:29:06Z

nmostesting/suites/IS1301Test.py

+STRING_OVER_MAX_VALUE = ''.join(['X' for i in range(100)])
+STRING_MAX_VALUE = STRING_OVER_MAX_VALUE[:64]  # this is the max length tolerated


The parameter naming here gives the first impression these are number values (cf. INT_MAX).

Perhaps add LENGTH in as well STRING_MAX_LENGTH_VALUE? A little verbose but less confusing.

Similarly STRING_OVER_MAX_VALUE could be STRING_LENGTH_OVER_MAX_VALUE or STRING_OVER_MAX_LENGTH_VALUE

jonathan-r-thorpe · 2024-04-29T09:30:00Z

nmostesting/suites/IS1301Test.py

+TAGS_OVER_MAX_VALUE = {'location': ['underground'], 'studio': ['42'], 'tech': ['John', 'Mike']}
+TAGS_MAX_VALUE = TAGS_OVER_MAX_VALUE.copy()


The comment for STRING_OVER_MAX_VALUE applies here also.

jonathan-r-thorpe · 2024-04-29T09:33:44Z

nmostesting/suites/IS1301Test.py

+        """
+        FAKE_ORIG = {
+            'description': 'fake_orig_desc',
+            'label': 'fake_orig_label',
+            'tags': {
+                'location': ['fake_location']
+            }
+        }
+        for resource in RESOURCES:
+            url = "{}{}{}".format(self.annotation_url, 'node/', resource)
+            TestHelper.do_request("PATCH", url, json=FAKE_ORIG)
+        """


Redundant code needs removing.

jonathan-r-thorpe · 2024-04-29T09:57:43Z

nmostesting/suites/IS1301Test.py

+    def log(self, msg):
+        print(msg)


Generally the test suites don't print to the console as the console reporting is handled by the testing framework, so the "logging" is best removed.

jonathan-r-thorpe · 2024-04-29T10:00:30Z

nmostesting/suites/IS1301Test.py

+def strip_tags(tags):
+    if TAGS_TO_BE_SKIPPED in list(tags.keys()):


Skipped or stripped? TAGS_TO_BE_STRIPPED perhaps?

jonathan-r-thorpe · 2024-04-29T10:07:32Z

nmostesting/suites/IS1301Test.py

+        else:
+            return False, "GET  Resquest FAIL"
+
+    def compare_resource(self, object, new, resp):


I guess resp here is response, which doesn't make much sense in the context of a comparison function.

Would prefer this to be consistent with other compare functions in the code base (e.g. NMOSUtils.compare_urls, NMOSUtils.compare_api_version etc.) that, (rightly or wrongly) use (url1, url2) and (ver1, ver2), so (res1, res2) might be preferable

Also object both an overly broad and overloaded term; is there something more descriptive that could be used here? tag_type?

Yes, I agree "object" (and OBJECTS for the list of them) etc. seems confusing terminology. I think we're talking about what IS-13 calls the "annotation properties"?

Also agree new and resp are confusing, but this function doesn't seem symmetric, which means it's not a usual comparison function... 😕 @jonathan-r-thorpe, let's discuss while we're face-to-face this week!

jonathan-r-thorpe · 2024-04-29T10:13:35Z

nmostesting/suites/IS1301Test.py

+        """ Patch a resource with one ore several object values """
+
+        object = list(new.keys())[0]
+        valid, resp = TestHelper.do_request("PATCH", url, json=new)


Suggested change

valid, resp = TestHelper.do_request("PATCH", url, json=new)

valid, resp = self.do_request("PATCH", url, json=new)

jonathan-r-thorpe · 2024-04-29T10:20:00Z

nmostesting/suites/IS1301Test.py

+        if not valid:
+            return False, "PATCH Resquest FAIL"


Rather than pass back a flag and message, you could raise NMOSTestExceptions in this function. (See do_receiver_put in IS0401Test.py for some examples of wrapping the test failure in this exception). You could then remove the flag handling code in do_test making it more compact and readable.

Agree, I'd like to see this done consistently with the rest of the test suites, and functions named in a way which indicates whether they return TestResults, raise NMOSTestExceptions, or a boolean and (optional) strings.

jonathan-r-thorpe · 2024-04-29T10:25:19Z

nmostesting/suites/IS1301Test.py

+    def get_url(self, base_url, resource):
+        """
+        Build the url for both annotation and node APIs which behaves differently.


Getting or building? looks more a build to me; idiomatically "create" is used a lot in the other test suites for this sort of thing create_url perhaps?

jonathan-r-thorpe · 2024-04-29T10:41:39Z

nmostesting/suites/IS1301Test.py

+The script implements the IS-13 test suite as specified by the nmos-resource-labelling workgroup.
+At the end of the test, the initial state of the tested unit is supposed to be restored but this
+cannot be garanteed.


I'm guessing the spec is the true "source of truth" here. Where possible it would be great to tie test cases back to normative language in the specifications (you can supply a link= to the test failure case that will embed a url link in the test results seen by the user).

Suggested change

The script implements the IS-13 test suite as specified by the nmos-resource-labelling workgroup.

At the end of the test, the initial state of the tested unit is supposed to be restored but this

cannot be garanteed.

The script implements the IS-13 test suite according to the AMWA IS-13 NMOS Annotation Specification (https://specs.amwa.tv/is-13/).

At the end of the test, the initial state of the tested unit is supposed to be restored but this

cannot be guaranteed.

jonathan-r-thorpe · 2024-04-29T10:46:22Z

A general question - looking at the specification there are a few normative references not covered by this test suite. Will these be covered before the activity closes?
e.g. "The Annotation API SHOULD be advertised as a ‘service’ endpoint under an IS-04 NMOS Node in the services array" https://specs.amwa.tv/is-13/branches/v1.0-dev/docs/Interoperability_-_IS-04.html#discovery,
"the UUIDs of Node Resources in the Annotation API MUST match those in the corresponding IS-04 Node API"
https://specs.amwa.tv/is-13/branches/v1.0-dev/docs/Interoperability_-_IS-04.html#consistent-resources

pkeroulas · 2024-06-05T14:50:21Z

The first spec is not covered indeed, but the second is. I still need to understand the translation of normative SHOULD to code. Thanks @jonathan-r-thorpe

garethsb

Just a few comments, which I'll review with @jonathan-r-thorpe since we're face-to-face today!

garethsb · 2024-07-01T12:08:28Z

nmostesting/suites/IS1301Test.py

+        else:
+            return False, "GET  Resquest FAIL"
+
+    def compare_resource(self, object, new, resp):


Yes, I agree "object" (and OBJECTS for the list of them) etc. seems confusing terminology. I think we're talking about what IS-13 calls the "annotation properties"?

garethsb · 2024-07-01T12:11:27Z

nmostesting/suites/IS1301Test.py

+            except Exception as e:
+                return False, e.msg
+        else:
+            return False, "GET  Resquest FAIL"


At least a typo, but also want to check why this suite needs a function like this if other suites don't, or whether there is already a common approach to this...

Suggested change

return False, "GET Resquest FAIL"

return False, "GET Request FAIL"

garethsb · 2024-07-01T12:15:36Z

nmostesting/suites/IS1301Test.py

+        else:
+            return False, "GET  Resquest FAIL"
+
+    def compare_resource(self, object, new, resp):


Also agree new and resp are confusing, but this function doesn't seem symmetric, which means it's not a usual comparison function... 😕 @jonathan-r-thorpe, let's discuss while we're face-to-face this week!

garethsb · 2024-07-01T12:19:34Z

nmostesting/suites/IS1301Test.py

+        if not valid:
+            return False, "PATCH Resquest FAIL"


Agree, I'd like to see this done consistently with the rest of the test suites, and functions named in a way which indicates whether they return TestResults, raise NMOSTestExceptions, or a boolean and (optional) strings.

garethsb · 2024-07-01T12:35:17Z

nmostesting/suites/IS1301Test.py

+        object = list(new.keys())[0]
+        valid, resp = TestHelper.do_request("PATCH", url, json=new)
+        if not valid:
+            return False, "PATCH Resquest FAIL"


Suggested change

return False, "PATCH Resquest FAIL"

return False, "PATCH Resuest FAIL"

garethsb · 2024-07-01T12:43:21Z

nmostesting/suites/IS1301Test.py

+
+    def do_test(self, test, resource, object):
+        """
+        Perform the test sequence as documented in the file header


Seems more natural to see the explanation of this function here to me?

garethsb · 2024-07-01T12:49:33Z

nmostesting/suites/IS1301Test.py

+        # re-GET
+        valid, resp = self.get_resource(url)
+        if not valid:
+            return False, "Get Resquest FAIL"


Suggested change

return False, "Get Resquest FAIL"

return False, "Get Request FAIL"

garethsb · 2024-07-01T13:58:20Z

nmostesting/suites/IS1301Test.py

+        return self.do_test(test, "receivers", "description")
+
+    def test_04_03(self, test):
+        """Annotation test: receivers/sevices/../tags (reset to default, set 5 tags, set >5 tags, check IS04+version)"""


garethsb · 2024-07-01T13:59:01Z

nmostesting/suites/IS1301Test.py

+        return self.do_test(test, "senders", "description")
+
+    def test_03_03(self, test):
+        """Annotation test: sender/sevices/../tags (reset to default, set 5 tags, set >5 tags, check IS04+version)"""


garethsb · 2024-07-01T14:00:05Z

nmostesting/suites/IS1301Test.py

+        return test.PASS()
+
+    def test_01_01(self, test):
+        """ Annotation test: self/label (reset to default, set 64-byte value, set >64-byte, check IS04+version)"""


These strings appear in the UI to describe the test, so let's wordsmith somewhat, @jonathan-r-thorpe?

pkeroulas · 2024-10-22T19:43:23Z

@jonathan-r-thorpe @garethsb all your change requests were addressed.

Raise exceptions instead of returning tuples. Add reference to the specification when possible.

Default value (1s) increases the duration of the test suite to approx 1min.

pkeroulas added 2 commits January 17, 2024 15:52

IS-13: add basic API tests

1902876

Successfully tested against: garethsb/nmos-cpp@04ac4fd AMWA-TV/is-13@8655e0c

Output more explicit error messages

8a4c1bf

When possible, TestResult.detail should be displayed in stderr since the raised exception prevents from the report creation.

peterbrightwell reviewed Jan 18, 2024

View reviewed changes

peterbrightwell assigned garethsb and jonathan-r-thorpe Jan 18, 2024

pkeroulas added 3 commits January 23, 2024 16:45

IS-13: test consistency with IS-04

9c3f078

is-13: cover senders and receivers

35a2c78

pkeroulas force-pushed the is-13 branch from 423bad2 to 35a2c78 Compare April 5, 2024 15:49

pkeroulas added 4 commits April 15, 2024 17:11

IS-13: strip grouphint tag which pollutes the test

1f3b948

IS-13: improve error msg

3c8bbf4

IS-13: add a pause to accomodate the update propagation

6664276

This seems to impact the version(timestamp) test.

is-13: add test suite description

5fafebb

jonathan-r-thorpe reviewed Apr 29, 2024

View reviewed changes

garethsb reviewed Jul 1, 2024

View reviewed changes

IS-13: rename functions, variables and constants

e5d6cbe

IS-13: remove logger

6080619

pkeroulas force-pushed the is-13 branch from b8a84e0 to 10d668d Compare October 22, 2024 19:11

pkeroulas added 4 commits October 22, 2024 16:31

IS-13: refactor results and errors

1886026

Raise exceptions instead of returning tuples. Add reference to the specification when possible.

IS-13: add service announcement test

0a2f6fd

IS-13: list uncovered tests

24a9817

IS-13: make the propagation delay configurable

8269c65

Default value (1s) increases the duration of the test suite to approx 1min.

pkeroulas force-pushed the is-13 branch from 5b80b58 to 8269c65 Compare October 22, 2024 20:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is-13 add basic API tests #842

Is-13 add basic API tests #842

pkeroulas commented Jan 17, 2024

peterbrightwell left a comment

pkeroulas commented Apr 23, 2024

jonathan-r-thorpe Apr 29, 2024 •

edited by garethsb

Loading

jonathan-r-thorpe Apr 29, 2024 •

edited by garethsb

Loading

jonathan-r-thorpe Apr 29, 2024

jonathan-r-thorpe Apr 29, 2024

jonathan-r-thorpe Apr 29, 2024

jonathan-r-thorpe Apr 29, 2024

jonathan-r-thorpe Apr 29, 2024

jonathan-r-thorpe Apr 29, 2024 •

edited

Loading

jonathan-r-thorpe Apr 29, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

jonathan-r-thorpe Apr 29, 2024

jonathan-r-thorpe Apr 29, 2024

garethsb Jul 1, 2024

jonathan-r-thorpe Apr 29, 2024

jonathan-r-thorpe Apr 29, 2024 •

edited

Loading

jonathan-r-thorpe commented Apr 29, 2024

pkeroulas commented Jun 5, 2024

garethsb left a comment

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

garethsb Jul 1, 2024

pkeroulas commented Oct 22, 2024 •

edited

Loading

	# const for label&description-related tests
	# Constants for label- and description-related tests

	# const for tags-related tests
	# Constants for tags-related tests

		STRING_OVER_MAX_VALUE = ''.join(['X' for i in range(100)])
		STRING_MAX_VALUE = STRING_OVER_MAX_VALUE[:64] # this is the max length tolerated

		TAGS_OVER_MAX_VALUE = {'location': ['underground'], 'studio': ['42'], 'tech': ['John', 'Mike']}
		TAGS_MAX_VALUE = TAGS_OVER_MAX_VALUE.copy()

		def strip_tags(tags):
		if TAGS_TO_BE_SKIPPED in list(tags.keys()):

	valid, resp = TestHelper.do_request("PATCH", url, json=new)
	valid, resp = self.do_request("PATCH", url, json=new)

	return False, "GET Resquest FAIL"
	return False, "GET Request FAIL"

	return False, "PATCH Resquest FAIL"
	return False, "PATCH Resuest FAIL"

Is-13 add basic API tests #842

Are you sure you want to change the base?

Is-13 add basic API tests #842

Conversation

pkeroulas commented Jan 17, 2024

peterbrightwell left a comment

Choose a reason for hiding this comment

pkeroulas commented Apr 23, 2024

jonathan-r-thorpe Apr 29, 2024 • edited by garethsb Loading

Choose a reason for hiding this comment

jonathan-r-thorpe Apr 29, 2024 • edited by garethsb Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-r-thorpe Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-r-thorpe Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

jonathan-r-thorpe commented Apr 29, 2024

pkeroulas commented Jun 5, 2024

garethsb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pkeroulas commented Oct 22, 2024 • edited Loading

jonathan-r-thorpe Apr 29, 2024 •

edited by garethsb

Loading

jonathan-r-thorpe Apr 29, 2024 •

edited by garethsb

Loading

jonathan-r-thorpe Apr 29, 2024 •

edited

Loading

jonathan-r-thorpe Apr 29, 2024 •

edited

Loading

pkeroulas commented Oct 22, 2024 •

edited

Loading