Pandas-style queries

[alimanfoo]

Hi folks, I'm just wondering if there's been any discussion of supporting a pandas-style query method for datasets? It would only have to support 1d variables.

[alimanfoo]

I think something like this is basically what I'm looking for:

https://nbviewer.jupyter.org/gist/alimanfoo/6d2918efb3510a157f9eb736d9eb6442

Would there be any interesting in adding something like this to the Dataset API?

[max-sixty]

Thanks for the clear example.

I'd certainly be open to it. I haven't thought through the semantic differences between a query running on a table vs multi-d array.

What do others think?

[shoyer]

I agree, I think this could be interesting. If the implementation can really be this simple, I would say go for it!

[alimanfoo]

Cool, thanks @shoyer.

Re the implementation, I think the main question would be how to handle the case where data variables are backed by dask arrays. I don't think the pandas eval() function is specifically set up to handle that at the moment, although it might still work anyway when using the "python" engine, need to check that.

Apart from that, hopefully implementation really would be that simple. Possibly need to add some sanity checking to make sure result of evaluating the expression can be used as an indexer.

[raybellwaves]

Looks good. It's probably an overkill for this first pass but @nils-braun has done some good work on querying dask in https://dask-sql.readthedocs.io/en/latest/

[alimanfoo]

Initial PR here: #4984

[andersy005]

Is it ok to mark this as "answered" and point folks to the new .query() method documentation?