Revamped signature tests #110

honno · 2022-04-12T15:54:23Z

As attempted in #109, this PR revamps the signature tests so that they try using inspect before passing actual arguments. This wasn't done initially due to compiled funcs not being easily inspectable #109 (comment), but I think it's best if we try using them as they give us full coverage very easily when they do work... which is most of the time!

The fallback option of passing args looks a bit different now. All arguments are inferred, and when none can be inferred we just skip the test. It turns out generating arrays somewhat naively works in a slim majority of functions, using our dtype_helpers.py utils to get valid dtypes. Inferred defaults from the stub signatures work most of the time too.

In cases we can't infer arguments, we skip. It seems right now only ~2/3rds of functions can get tested via the fallback option (i.e. passing args, not using inspect). IMO for a solution which has no hard-coded examples, that's a pretty promising start. I had played around heuristics like custom shape and element strategies for certain functions... but it kinda got messy so I thought I'd leave it to later, as it doesn't seem high priority.

Hypothesis is not strictly necessary for the fallback option, but I like writing generalised data specifications rather than hard-coded arguments—this does end up working in our favour when it comes to specifying dtypes, and shapes/elements in the future. I set max_examples=1when used as we don't actually care about testing edge cases here (also worth noting data draws are much faster like this compared to using <strat>.example()).

test_signatures.py is a nasty diff heh as I basically start from scratch, you'll prob just want to look at it as-is.

asmeurer · 2022-04-12T20:18:26Z

array_api_tests/test_signatures.py

+    stub_params = list(stub_sig.parameters.values())
+    # We're not interested if the array module has additional arguments, so we
+    # only iterate through the arguments listed in the spec.
+    for i, stub_param in enumerate(stub_params):


Would this catch functions that have too many positional-only arguments?

Yep! We're iterating over the parameters from the signatures/ stubs and checking them one-by-one, which covers this (added a test for this too).

asmeurer · 2022-04-12T20:19:16Z

Might be useful to add some simple meta-tests for the signature tests.

asmeurer · 2022-04-12T20:22:09Z

I tried running this against numpy (not numpy.array_api), and in both master and this branch I get

array_api_tests/test_signatures.py:32: in <module>
    from . import dtype_helpers as dh
array_api_tests/dtype_helpers.py:137: in <module>
    if dtype_nbits[default_int] == 32:
E   KeyError: dtype('int64')

We should try to figure out what is going on here. I'd like to be able to compare this vs. master for numpy so if you come up with a fix please do it as a separate PR.

asmeurer · 2022-04-12T20:29:05Z

Oh I see the problem. In NumPy, array.dtype isn't the same object as np.int64 (dtype classes vs. objects).

It's not obvious to me that NumPy is even out of compliance with the spec here. The spec only says that dtype objects have to compare equal, which in NumPy they do

>>> import numpy as np
>>> np.int64
<class 'numpy.int64'>
>>> np.array(0).dtype
dtype('int64')
>>> np.array(0).dtype == np.int64
True

The spec also doesn't say dtype objects have to be hashable (they are in NumPy, but this could fail if they weren't in another library).

Perhaps we should create a custom mapping that compares with equality instead of hashing and use that whereever we use dtypes as keys.

asmeurer · 2022-04-12T20:32:36Z

Hypothesis is not strictly necessary for the fallback option, but I like writing generalised data specifications rather than hard-coded arguments—this does end up working in our favour when it comes to specifying dtypes, and shapes/elements in the future. I set max_examples=1when used as we don't actually care about testing edge cases here (also worth noting data draws are much faster like this compared to using .example()).

What's the difference between max_examples=1 and a hard-coded example, though? I remember when writing the hard-coded examples that there were quite a few functions that didn't work unless the examples matched up somehow (e.g., aligned shapes, specific dtypes). I kind of made it a little harder than it needed to be by using the same examples for all functions.

It could be interesting to reuse the strategies for the function tests themselves, but that might require a significant refactoring. The only effective way I can see to do that would be to move the signature tests to the tests for the functions themselves, and then have some simple way to filter it so it only runs the signature part of the tests.

honno · 2022-04-13T08:49:18Z

It could be interesting to reuse the strategies for the function tests themselves, but that might require a significant refactoring. The only effective way I can see to do that would be to move the signature tests to the tests for the functions themselves, and then have some simple way to filter it so it only runs the signature part of the tests.

Ah so to clarify, the use of Hypothesis isn't for testing interesting arrangements of args, but just the args themselves (so that's what the "primary" tests already do.

What's the difference between max_examples=1 and a hard-coded example, though? I remember when writing the hard-coded examples that there were quite a few functions that didn't work unless the examples matched up somehow (e.g., aligned shapes, specific dtypes). I kind of made it a little harder than it needed to be by using the same examples for all functions.

Yep, so the examples are "naive" in that they don't match up with any requirements except for valid dtypes, for now at least. The only real difference is allowing use of dtype/shape/element strategies, although right now we only use dtype strategies (i.e. pass a strategy that generates the valid dtypes, as opposed to having to "pick" a valid dtype). The downside to this is, yaknow, adding the cognitive load of Hypothesis... but any solution is messy IMO and I found writing strategies-as-data-specifications wasn't as messy.

asmeurer · 2022-04-13T20:18:29Z

So thanks to your branch, we can compare before and after.

Before

FAILED array_api_tests/test_signatures.py::test_has_names[unique_all] - AssertionError: numpy is missing the set function unique_all()
FAILED array_api_tests/test_signatures.py::test_has_names[unique_counts] - AssertionError: numpy is missing the set function unique_counts()
FAILED array_api_tests/test_signatures.py::test_has_names[unique_inverse] - AssertionError: numpy is missing the set function unique_inverse()
FAILED array_api_tests/test_signatures.py::test_has_names[unique_values] - AssertionError: numpy is missing the set function unique_values()
FAILED array_api_tests/test_signatures.py::test_has_names[from_dlpack] - AssertionError: numpy is missing the creation function from_dlpack()
FAILED array_api_tests/test_signatures.py::test_has_names[concat] - AssertionError: numpy is missing the manipulation function concat()
FAILED array_api_tests/test_signatures.py::test_has_names[permute_dims] - AssertionError: numpy is missing the manipulation function permute_dims()
FAILED array_api_tests/test_signatures.py::test_has_names[astype] - AssertionError: numpy is missing the data_type function astype()
FAILED array_api_tests/test_signatures.py::test_has_names[acos] - AssertionError: numpy is missing the elementwise function acos()
FAILED array_api_tests/test_signatures.py::test_has_names[acosh] - AssertionError: numpy is missing the elementwise function acosh()
FAILED array_api_tests/test_signatures.py::test_has_names[asin] - AssertionError: numpy is missing the elementwise function asin()
FAILED array_api_tests/test_signatures.py::test_has_names[asinh] - AssertionError: numpy is missing the elementwise function asinh()
FAILED array_api_tests/test_signatures.py::test_has_names[atan] - AssertionError: numpy is missing the elementwise function atan()
FAILED array_api_tests/test_signatures.py::test_has_names[atan2] - AssertionError: numpy is missing the elementwise function atan2()
FAILED array_api_tests/test_signatures.py::test_has_names[atanh] - AssertionError: numpy is missing the elementwise function atanh()
FAILED array_api_tests/test_signatures.py::test_has_names[bitwise_left_shift] - AssertionError: numpy is missing the elementwise function bitwise_left_shift()
FAILED array_api_tests/test_signatures.py::test_has_names[bitwise_invert] - AssertionError: numpy is missing the elementwise function bitwise_invert()
FAILED array_api_tests/test_signatures.py::test_has_names[bitwise_right_shift] - AssertionError: numpy is missing the elementwise function bitwise_right_shift()
FAILED array_api_tests/test_signatures.py::test_has_names[pow] - AssertionError: numpy is missing the elementwise function pow()
FAILED array_api_tests/test_signatures.py::test_has_names[matmul0] - AssertionError: numpy is missing the linear_algebra extension function matmul()
FAILED array_api_tests/test_signatures.py::test_has_names[matrix_transpose0] - AssertionError: numpy is missing the linear_algebra extension function matrix_transpose()
FAILED array_api_tests/test_signatures.py::test_has_names[tensordot0] - AssertionError: numpy is missing the linear_algebra extension function tensordot()
FAILED array_api_tests/test_signatures.py::test_has_names[vecdot0] - AssertionError: numpy is missing the linear_algebra extension function vecdot()
FAILED array_api_tests/test_signatures.py::test_has_names[cross] - AssertionError: numpy is missing the linalg extension function cross()
FAILED array_api_tests/test_signatures.py::test_has_names[diagonal] - AssertionError: numpy is missing the linalg extension function diagonal()
FAILED array_api_tests/test_signatures.py::test_has_names[matmul1] - AssertionError: numpy is missing the linear_algebra extension function matmul()
FAILED array_api_tests/test_signatures.py::test_has_names[matrix_norm] - AssertionError: numpy is missing the linalg extension function matrix_norm()
FAILED array_api_tests/test_signatures.py::test_has_names[matrix_transpose1] - AssertionError: numpy is missing the linear_algebra extension function matrix_transpose()
FAILED array_api_tests/test_signatures.py::test_has_names[outer] - AssertionError: numpy is missing the linalg extension function outer()
FAILED array_api_tests/test_signatures.py::test_has_names[svdvals] - AssertionError: numpy is missing the linalg extension function svdvals()
FAILED array_api_tests/test_signatures.py::test_has_names[tensordot1] - AssertionError: numpy is missing the linear_algebra extension function tensordot()
FAILED array_api_tests/test_signatures.py::test_has_names[trace] - AssertionError: numpy is missing the linalg extension function trace()
FAILED array_api_tests/test_signatures.py::test_has_names[vecdot1] - AssertionError: numpy is missing the linear_algebra extension function vecdot()
FAILED array_api_tests/test_signatures.py::test_has_names[vector_norm] - AssertionError: numpy is missing the linalg extension function vector_norm()
FAILED array_api_tests/test_signatures.py::test_function_positional_args[eye] - AssertionError: eye() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[linspace] - AssertionError: linspace() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[roll] - AssertionError: roll() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[squeeze] - AssertionError: squeeze() should not accept 1 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[broadcast_to] - AssertionError: broadcast_to() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[abs] - AssertionError: abs() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[add] - AssertionError: add() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[bitwise_and] - AssertionError: bitwise_and() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[bitwise_or] - AssertionError: bitwise_or() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[bitwise_xor] - AssertionError: bitwise_xor() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[cos] - AssertionError: cos() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[cosh] - AssertionError: cosh() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[divide] - AssertionError: divide() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[equal] - AssertionError: equal() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[exp] - AssertionError: exp() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[expm1] - AssertionError: expm1() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[floor_divide] - AssertionError: floor_divide() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[greater] - AssertionError: greater() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[greater_equal] - AssertionError: greater_equal() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[isfinite] - AssertionError: isfinite() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[isinf] - AssertionError: isinf() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[isnan] - AssertionError: isnan() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[less] - AssertionError: less() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[less_equal] - AssertionError: less_equal() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[log] - AssertionError: log() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[log1p] - AssertionError: log1p() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[log2] - AssertionError: log2() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[log10] - AssertionError: log10() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[logaddexp] - AssertionError: logaddexp() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[logical_and] - AssertionError: logical_and() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[logical_not] - AssertionError: logical_not() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[logical_or] - AssertionError: logical_or() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[logical_xor] - AssertionError: logical_xor() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[multiply] - AssertionError: multiply() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[negative] - AssertionError: negative() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[not_equal] - AssertionError: not_equal() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[positive] - AssertionError: positive() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[remainder] - AssertionError: remainder() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[sign] - AssertionError: sign() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[sin] - AssertionError: sin() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[sinh] - AssertionError: sinh() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[square] - AssertionError: square() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[sqrt] - AssertionError: sqrt() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[subtract] - AssertionError: subtract() should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[tan] - AssertionError: tan() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_positional_args[tanh] - AssertionError: tanh() should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[std] - AssertionError: Unexpected exception TypeError("_std_dispatcher() got an unexpected ...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[var] - AssertionError: Unexpected exception TypeError("_var_dispatcher() got an unexpected ...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[arange] - AssertionError: Unexpected exception TypeError("arange() got an unexpected keywor...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[asarray] - AssertionError: Unexpected exception TypeError("asarray() got an unexpected keyw...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[empty] - AssertionError: Unexpected exception TypeError("empty() got an unexpected keyword ...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[empty_like] - AssertionError: Unexpected exception TypeError("empty_like() got an unexpecte...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[eye] - AssertionError: Unexpected exception TypeError("eye() got an unexpected keyword argu...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[full] - AssertionError: Unexpected exception TypeError("full() got an unexpected keyword ar...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[full_like] - AssertionError: Unexpected exception TypeError("_full_like_dispatcher() got an...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[linspace] - AssertionError: Unexpected exception TypeError("_linspace_dispatcher() got an u...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[ones] - AssertionError: Unexpected exception TypeError("ones() got an unexpected keyword ar...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[ones_like] - AssertionError: Unexpected exception TypeError("_ones_like_dispatcher() got an...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[zeros] - AssertionError: Unexpected exception TypeError("zeros() got an unexpected keyword ...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[zeros_like] - AssertionError: Unexpected exception TypeError("_zeros_like_dispatcher() got ...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[argsort] - AssertionError: Unexpected exception TypeError("_argsort_dispatcher() got an une...
FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[sort] - AssertionError: Unexpected exception TypeError("_sort_dispatcher() got an unexpecte...

After

FAILED array_api_tests/test_signatures.py::test_func_signature[max] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[mean] - AssertionError: Expected argument 'dtype' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[min] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[prod] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[std] - AssertionError: Expected argument 'dtype' to be named 'correction'
FAILED array_api_tests/test_signatures.py::test_func_signature[sum] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[var] - AssertionError: Expected argument 'dtype' to be named 'correction'
FAILED array_api_tests/test_signatures.py::test_func_signature[unique_all] - AssertionError: unique_all not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[unique_counts] - AssertionError: unique_counts not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[unique_inverse] - AssertionError: unique_inverse not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[unique_values] - AssertionError: unique_values not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[argmax] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[argmin] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[empty_like] - TypeError: empty_like() got an unexpected keyword argument 'device'
FAILED array_api_tests/test_signatures.py::test_func_signature[eye] - AssertionError: Expected argument 'order' to be named 'device'
FAILED array_api_tests/test_signatures.py::test_func_signature[from_dlpack] - AssertionError: from_dlpack not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[full] - AssertionError: Expected argument 'order' to be named 'device'
FAILED array_api_tests/test_signatures.py::test_func_signature[full_like] - AssertionError: Expected argument 'order' to be named 'device'
FAILED array_api_tests/test_signatures.py::test_func_signature[linspace] - AssertionError: Expected argument 'endpoint' to be named 'dtype'
FAILED array_api_tests/test_signatures.py::test_func_signature[meshgrid] - AssertionError: Expected argument 'copy' to be named 'indexing'
FAILED array_api_tests/test_signatures.py::test_func_signature[ones] - AssertionError: Expected argument 'order' to be named 'device'
FAILED array_api_tests/test_signatures.py::test_func_signature[ones_like] - AssertionError: Expected argument 'order' to be named 'device'
FAILED array_api_tests/test_signatures.py::test_func_signature[zeros_like] - AssertionError: Expected argument 'order' to be named 'device'
FAILED array_api_tests/test_signatures.py::test_func_signature[all] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[any] - AssertionError: Expected argument 'out' to be named 'keepdims'
FAILED array_api_tests/test_signatures.py::test_func_signature[concat] - AssertionError: concat not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[permute_dims] - AssertionError: permute_dims not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[reshape] - AssertionError: Expected argument 'newshape' to be named 'shape'
FAILED array_api_tests/test_signatures.py::test_func_signature[argsort] - AssertionError: Expected argument 'kind' to be named 'descending'
FAILED array_api_tests/test_signatures.py::test_func_signature[sort] - AssertionError: Expected argument 'kind' to be named 'descending'
FAILED array_api_tests/test_signatures.py::test_func_signature[astype] - AssertionError: astype not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[acos] - AssertionError: acos not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[acosh] - AssertionError: acosh not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[asin] - AssertionError: asin not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[asinh] - AssertionError: asinh not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[atan] - AssertionError: atan not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[atan2] - AssertionError: atan2 not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[atanh] - AssertionError: atanh not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[bitwise_left_shift] - AssertionError: bitwise_left_shift not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[bitwise_invert] - AssertionError: bitwise_invert not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[bitwise_right_shift] - AssertionError: bitwise_right_shift not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[pow] - AssertionError: pow not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[matrix_transpose] - AssertionError: matrix_transpose not found in array module
FAILED array_api_tests/test_signatures.py::test_func_signature[vecdot] - AssertionError: vecdot not found in array module
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.cholesky] - AssertionError: Argument 'upper' missing from signature
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.cross] - AssertionError: cross not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.diagonal] - AssertionError: diagonal not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.matmul] - AssertionError: matmul not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.matrix_norm] - AssertionError: matrix_norm not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.matrix_rank] - AssertionError: Expected argument 'tol' to be named 'rtol'
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.matrix_transpose] - AssertionError: matrix_transpose not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.outer] - AssertionError: outer not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.pinv] - AssertionError: Expected argument 'rcond' to be named 'rtol'
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.svdvals] - AssertionError: svdvals not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.tensordot] - AssertionError: tensordot not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.trace] - AssertionError: trace not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.vecdot] - AssertionError: vecdot not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_extension_func_signature[linalg.vector_norm] - AssertionError: vector_norm not found in linalg extension
FAILED array_api_tests/test_signatures.py::test_array_method_signature[__array_namespace__] - AssertionError: __array_namespace__ not found in array object array(False)
FAILED array_api_tests/test_signatures.py::test_array_method_signature[to_device] - AssertionError: to_device not found in array object array(False)

asmeurer · 2022-04-13T20:29:05Z

It looks like some of the error messages here are misleading. For example

FAILED array_api_tests/test_signatures.py::test_func_signature[linspace] - AssertionError: Expected argument 'endpoint' to be named 'dtype'

The signature of linspace in NumPy is

linspace(start, stop, num=50, endpoint=True, retstep=False, dtype=None, axis=0)

and in the spec

linspace(start, stop, /, num: int, *, dtype=None, device=None, endpoint=True)

So the problem isn't really about the endpoint or dtype arguments. It has to do with num being a required argument in the spec. linspace also is missing the device keyword argument, which I think the

FAILED array_api_tests/test_signatures.py::test_function_keyword_only_args[linspace] - AssertionError: Unexpected exception TypeError("_linspace_dispatcher() got an u...

from the "before" is supposed to be saying, but unfortunately that test isn't very good at converting errors into useful error messages.

There are some that are missing in the "after". One example is roll (it fails because axis should be keyword-only). This must be a bug in your test code however, because roll is pure Python and has a perfectly inspectable signature.

honno · 2022-04-14T12:30:50Z

First of, I'll be rebasing this to #112 as it's useful to try this on NumPy proper, and so have reverted this PR to a draft until that gets merged.

I've pushed a fix that allows for out-of-order keyword-only args, which also helps out the error message situation—I think none of the errors I see running against NumPy-proper are misleading now. The error you'll see for linspace is

AssertionError: Argument 'device' missing from signature

So the problem isn't really about the endpoint or dtype arguments. It has to do with num being a required argument in the spec.

Actually in this case, I think num in np.linspace() should be passing (which it is in this PR), as this is a strictness question that doesn't effect interop. For example, an array-consuming library supporting the Array API should always pass num, either positionally or with a keyword

xp.linspace(0, 1, 5)
xp.linspace(0, 1, num=5)

which np.linspace() supports in both instances.

There are some that are missing in the "after". One example is roll (it fails because axis should be keyword-only). This must be a bug in your test code however, because roll is pure Python and has a perfectly inspectable signature.

Yep see above, I think np.roll() should be passing, e.g. assuming xp.roll(x, /, shift, *, axis=None) you can always use np.roll(a, shift, axis=None).

honno · 2022-04-25T08:45:07Z

Will like to merge this in a few days

More performative, even for `max_examples=1`

honno requested a review from asmeurer April 12, 2022 15:54

asmeurer reviewed Apr 12, 2022

View reviewed changes

honno force-pushed the try-inspect-sig branch from 715838a to 707eed9 Compare April 13, 2022 08:28

This was referenced Apr 13, 2022

Use equality instead of hasing for dtype helpers #111

Closed

Implement EqualityMapping and use for relevant dtype helpers #112

Merged

honno marked this pull request as draft April 14, 2022 09:19

honno mentioned this pull request Apr 14, 2022

Add an option to not fail on value-based casting errors #113

Closed

honno force-pushed the try-inspect-sig branch 3 times, most recently from 06616e7 to bb0bc60 Compare April 21, 2022 09:27

honno marked this pull request as ready for review April 21, 2022 09:27

honno requested a review from asmeurer April 21, 2022 09:27

honno added 9 commits April 28, 2022 10:50

Reintroduce inspect use for sig tests

c3798d2

Create dh.func_in_dtypes from parsing the spec

8f3bfdc

Special case expm1 input dtypes for now

6071f44

Hard-code array scalar casting input dtypes for dh.func_in_dtypes

ce2e2c0

Skip alias stubs and fallback on source stubs, e.g. for matmul

828627b

Rudimentary support for uninspectable signatures

6206e2f

Test different arg/kwarg arrangements for uninspectable normal args

5f5d5fd

Streamline uninspectable testing, just skipping awkward cases

d6a56b4

Remove support for testing uninspectable normal args

e78e3d5

honno added 7 commits April 28, 2022 10:50

Use data draws over examples

971cd95

More performative, even for `max_examples=1`

Nicer matrixy_names definition

17a56ba

Test uninspectable array methods

15114b7

Some docs and syntax niceties

377c89f

Add failing test for testing out-of-order kw-only args

2f9fb30

More meta tests for test_signatures.py

7e7bd2c

Allow out-of-order kwonly args in signatures

b172d84

honno force-pushed the try-inspect-sig branch from bb0bc60 to b172d84 Compare April 28, 2022 09:51

honno merged commit 03ec5cf into data-apis:master Apr 28, 2022

asmeurer mentioned this pull request Mar 17, 2023

Signature tests not catching missing keyword arguments #175

Closed

honno mentioned this pull request Apr 11, 2023

Specify/infer arguments for testing uninspectable signatures #177

Merged

honno deleted the try-inspect-sig branch February 28, 2024 13:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revamped signature tests #110

Revamped signature tests #110

honno commented Apr 12, 2022 •

edited

Loading

asmeurer Apr 12, 2022

honno Apr 13, 2022

asmeurer commented Apr 12, 2022

asmeurer commented Apr 12, 2022

asmeurer commented Apr 12, 2022

asmeurer commented Apr 12, 2022

honno commented Apr 13, 2022

asmeurer commented Apr 13, 2022

asmeurer commented Apr 13, 2022

honno commented Apr 14, 2022 •

edited

Loading

honno commented Apr 25, 2022

Revamped signature tests #110

Revamped signature tests #110

Conversation

honno commented Apr 12, 2022 • edited Loading

asmeurer Apr 12, 2022

Choose a reason for hiding this comment

honno Apr 13, 2022

Choose a reason for hiding this comment

asmeurer commented Apr 12, 2022

asmeurer commented Apr 12, 2022

asmeurer commented Apr 12, 2022

asmeurer commented Apr 12, 2022

honno commented Apr 13, 2022

asmeurer commented Apr 13, 2022

asmeurer commented Apr 13, 2022

honno commented Apr 14, 2022 • edited Loading

honno commented Apr 25, 2022

honno commented Apr 12, 2022 •

edited

Loading

honno commented Apr 14, 2022 •

edited

Loading