Intention regarding types in tablecloth (based on tablecloth.api.utils)

[ezmiller]

@genmeblog trying out this new discussion context for some of our conversation. I started looking into the type helpers in tablecloth.api.utils, and had a few questions about what's intended.

First, I see that type? there is designed to answer questions about the hierarchy of types specified in type-sets. I.e. (type? :integer :int64). I wondered what this implies about users will interact with types in tablecloth. I'm assuming the intention is that users will usually think about types in terms of the parent categories : integer, :datetime, etc, and not in terms of the more granular types :int32, :int64, etc. Is that the intention?

[ezmiller]

Took some time to look a bit more closely at how Python/Numpy handles this. Will look at R stuff later.

Numpy has detailed typed, e.g. np.integeter but also np.int32, np.int64, etc. It maintains a hierarchy something like what we have in tablecloth. So

np.issubdtype(np.int32, np.integer)
# True

When you check the datatype of the array, you get the granular type:

a = np.arange(3, dtype=np.int32)
a.dtype
# dtype('int32')

[genmeblog]

The main idea was mainly supporting the selection columns by a type. And yes, the assumption is that user wants to select numerical column(s) rather than a concrete type. tech.ml.dataset creates minimal type by default and user doesn't control if it's int16 or int32.