Splitting the verbose JSON dump for results from earthaccess.search_datasets from the __repr__ method?

[scottstanie]

While searching and playing with some of the results from search_datasets on the command line/jupyter, I noticed that the default for seeing the earthaccess.results.DataCollection is to see the entire json.dumps of the object:

In [7]: import earthaccess
In [8]: results = earthaccess.search_datasets(daac="ASF", keyword="OPERA_L2_CSLC-S1_V1")
In [9]: results.__repr__()
Out[9]: '[{\n  "meta": {\n    "revision-id": 9,\n    "deleted": false,\n    "format": "application/vnd.nasa.cmr.umm+json",\n    "provider-id": "ASF",\n    "has-combine": false,\n    "user-id": "mharbin",\n
.... 

In [10]: len(str(results[0]).splitlines())
Out[10]: 568

I usually expect the repr to be just the information you might use to recreate it (ideally in the format where you can just copy/paste it back in).
So just to put out the idea: it would be nice to have the __repr__ be console/developer friendly, and to have the json dumps come either from __str__, or from a separate method so that you don't accidentally dump > 500 lines to the console.

Examples of what I mean:

In [1]: from numpy import array

In [2]: arr = array([1, 2, 3])

In [3]: repr(arr)
Out[3]: 'array([1, 2, 3])'

which you can copy/paste to recreate that object.
To make it prettier, the str method is the one that applies formatting:

In [4]: str(arr)
Out[4]: '[1 2 3]'

In [5]: print(arr)
[1 2 3]

Another good example is from pydantic:

In [9]: import datetime
   ...: from typing import List, Optional
   ...: from pydantic import BaseModel
   ...:
   ...: class User(BaseModel):
   ...:     id: int
   ...:     name: str = 'John Doe'
   ...:     signup_ts: Optional[datetime.datetime] = None
   ...:     friends: List[int] = []
   ...:
   ...: external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22', 'friends': [1, '2', b'3']}
   ...: user = User(**external_data)

In [10]: user
Out[10]: User(id=123, name='John Doe', signup_ts=datetime.datetime(2017, 6, 1, 12, 22), friends=[1, 2, 3])

In [11]: User(id=123, name='John Doe', signup_ts=datetime.datetime(2017, 6, 1, 12, 22), friends=[1, 2, 3])
Out[11]: User(id=123, name='John Doe', signup_ts=datetime.datetime(2017, 6, 1, 12, 22), friends=[1, 2, 3])

to pretty-format (or dump to json), they have a model_dump method:

In [13]: user.model_dump()
Out[13]:
{'id': 123,
 'name': 'John Doe',
 'signup_ts': datetime.datetime(2017, 6, 1, 12, 22),
 'friends': [1, 2, 3]}

[jhkennedy]

@scottstanie I think this is a good idea and there's a bit we can improve here.

We'll have to figure out exactly what we mean by:

a string that would yield an object with the same value when passed to eval()

https://docs.python.org/3/library/functions.html#repr

and I think in general we could go a bit farther; we return just plain lists of objects, but a results class could then summarize the results better and allow you to combine multiple search results, etc.

[jhkennedy]

and I think in general we could go a bit farther; we return just plain lists of objects, but a results class could then summarize the results better and allow you to combine multiple search results, etc.

Is this ballpark what you're thinking?

results = earthaccess.search_datasets(...)  # results is a list
print(earthaccess.Results(results).summary_text())

@MattF-NSIDC I'm suggesting this:

>>> results = earthacces.search_datasets(...)
>>> type(results)
Earthaccess.Results
>>> results
# some string that allows you to recreate the search
>>> print(results)
# some string describing the search results; likely something like "432 granules found from ..."

which would also open up space to do things like:

>>> results.to_json()
# JSON dump like we're used to seeing
>>> results.to_parquet('my_results.parquet')
# save the results to a GeoParquet

And the download and open methods could (also?) become a class method:

>>> paths = results.download('.')
>>> ds = xr.open_mfdataset(results.open())

[jhkennedy]

This is of course a much bigger lift; we should start with improving just the earthaccess.results.DataGranule methods.

[mfisher87]

That makes more sense, thanks for clarifying. It would be cheap for us to slap some dunder methods on there to make said class into a sequence.

[jhkennedy]

yeah, we'd definitely want to keep it as a sequence -- both from a usability standpoint and also that'd mean we could expect the change to have a pretty small impact on users

[scottstanie]

🤔 @scottstanie should we consider using custom repr methods for ipython/jupyter? https://ipython.readthedocs.io/en/stable/config/integrating.html#rich-display

oh those are cool! I haven't seen all those methods, I've only occasionally added __rich_repr__ for pretty printing