ENH: create Timedelta scalar / TimedeltaIndex

[jreback]

closes #3009
closes #8187
closes #8209
closes #4533
closes #8190
closes #7869
closes #8214

This PR creates a new timedelta sub-class, the scalar Timedelta, very similar to how Timestamp sub-classes datetime. Further it creates a TimedeltaIndex modelled on DatetimeIndex that holds the underlying data (boxing is only on iteration).

• ToDo
  • Document API changes:
    • to_timedelta return TimedeltaIndex rather than a Series
    • to_timedelta with scalar returns Timedelta (rather than np.timedelta64)
    • Boxing for list(Series(...)) is now Timedelta
    • possible repr changes
    • timedelta_range
    • usage / examples
    • API
  • Timedelta
    • conversions to_timedelta/to_pytimedelta
  • TimedeltaIndex
    • conversions to_pytimedelta
  • what else needs boxing in Series?
  • fix formatters (Timedelta64Formatter; remove repr_timedelta64 (no longer necessary)
  • create timedelta_range & fix freq / inferred_freq
  • fix remaining copied methods from DatetimeIndex
  • more tests (of TimedeltaIndex / scalar)
    • scalar comparisons
    • index comparisons
    • getitem / partial string access
    • sql expecting np.timedelta64 ? (currently skipping these 2 tests!!!!), see: jreback@2743ef1
    • validate series / td ops
  • move methods of TimedeltaIndex/DatetimeIndex
    • move DatetimeOpsMixin to tseries/base.py?
    • shift/take/order
    • slice_indexer
    • append/union/intersection
    • comparbless?
    • .dt accessor are ok now
    • move .td to .tdi

[hughesadam87]

Hi Jeff,

I was attached on the last thread and wanted to add some comments. First, let me say that this is a really nice addition and seeing the output helps clarify some of the points you mentioned to me in an email.

One question I have is can the TimeDeltaIndex be constructed directly from a DatetimeIndex? Something like:

@classmethod
def from_datetimeindex(cls, datetimeindex):
    ....

For experimental research, data is often timestamped, but users would like to have access to the interval representation as well as the original timestamps. Maintaining both representations is what motivated me to create a hybrid Index that I spoke of. I'd imagine that even with this new index, converting between DatetimeIndex and TimeDelta would mainly be out of the scope of these classes per-se, and left to any application that needs to implement such behavior. Or am I mistaken?

I also wanted to share a notebook with you that shows the TimeIndex hybrid that I wrote, just in case seeing it clarifies some of the questions I had asked previously. As you see, the convert() method transfers between various representations of the data. Now that you've built the TimeDelta index, I think I will move all of my intervals into this index, and maybe add some methods like TimeDelta.as_seconds() which would return an array of total seconds etc... or is the behavior already implemented? Part of the motivation for this is for plotting; if the user is working in a system of total seconds, then when they plot, they'd probably want seconds on the axis not timestamps. Again, this is custom behavior that I've implemented in a hybrid index, but want to refactor to utilize the new index in 0.15.

Here's the notebook if you have time to glance at it, maybe it will clarify what I'm trying to say. The TimeIndex section is the important one. Also, the SpecIndex does indeed subclass Float64 as you suggested:

http://nbviewer.ipython.org/github/hugadams/pyuvvis/blob/master/examples/Notebooks/units.ipynb
Thanks

[shoyer]

@hugadams I think the way you spell from_datetimeindex is datetimeindex - datetimeindex[0] (or insert your preferred reference time).

I agree that conversion to floating point is useful, and maybe we want something like a to_seconds() method, but again you can already do very succinctly with arithmetic: timedeltaindex / pd.to_timedelta('1s').

[jreback]

who knew this would work!

In [4]: pd.timedelta_range('1 days, 00:00:02',periods=5,freq='2h')
Out[4]: 
<class 'pandas.tseries.td.TimedeltaIndex'>
[1 days, 00:00:02, ..., 1 days, 08:00:02]
Length: 5, Freq: <2 * Hours>

In [5]: pd.timedelta_range('1 days, 00:00:02',periods=5,freq='2h').tolist()
Out[5]: 
[Timedelta('1 days, 00:00:02'),
 Timedelta('1 days, 02:00:02'),
 Timedelta('1 days, 04:00:02'),
 Timedelta('1 days, 06:00:02'),
 Timedelta('1 days, 08:00:02')]

[hughesadam87]

Thanks Stephan. That's a good point. I didn't realize you could pass those types of string arguments into to_timedelta(), which I believe Jeff was trying to explain to me and my dense self.

[jreback]

@jorisvandenbossche

https://travis-ci.org/jreback/pandas/jobs/34546538

I am having some sql failures when passing Timedelta (which is a sub-class of timedelta which is what was passed before i think).

Is their some sort of conversion done on Timestamp (which is the same thing I would need to do for Timedelta)

?

[jorisvandenbossche]

No, now timedelta64 columns were actually int64's, and they were written to the database as such. Now it get's a Timedelta object instead of an int, and it chokes.

But SQLAlchemy has a Interval type (http://docs.sqlalchemy.org/en/rel_0_9/core/types.html#sqlalchemy.types.Interval) that can handle datetime.timedelta. So I think there are two options:

1. keep it as is (store timedelta as int) and therefore convert Timedelta columns to timedelta64/int64 on input
2. or use the Interval type and make sqlalchemy aware of to treat Timedelta the same as datetime.timedelta (this will store it for postgresql in a native timedelta format, see http://www.postgresql.org/docs/9.1/static/datatype-datetime.html, for others like MySQL it will store it as datetime from 1970)

See here for some discussion on it #6921 and here for the subsequent PR #7076

For Timestamp, there should be some conversion (on some drivers it fails), but this is not yet done. See #7103 for that

[jorisvandenbossche]

a small not-code related comment: I think td.py is not a very clear file name, why not just timedelta.py?

[jreback]

I have this method of .components on an individual Timedelta.
I could expose it on a TimedeltaIndex (and actually just return a frame)
though its prob not super duper performant (its in cython on the indivual timedeltas).

Normally people just use the Timedeltas as the bottom (e.g. express as say number of seconds).
But the breakdown is useful no? (its actually how I do the repr)

In [33]: s = Series(timedelta_range('1 day 23:59:59',periods=4,freq='s'))

In [34]: s
Out[34]: 
0   1 days, 23:59:59
1   2 days, 00:00:00
2   2 days, 00:00:01
3   2 days, 00:00:02
dtype: timedelta64[ns]

In [35]: s.dt.components()
Out[35]: 
   sign  days  hours  minutes  seconds  milli  micro  nano
0     1     1     23       59       59      0      0     0
1     1     2      0        0        0      0      0     0
2     1     2      0        0        1      0      0     0
3     1     2      0        0        2      0      0     0

In [36]: s.dt.minutes
Out[36]: 
0    2880
1    2880
2    2880
3    2880
dtype: int64

[hughesadam87]

I think the components breakdown is quite useful personally.

[shoyer]

@jreback OK, point well taken.

Are you sure you are doing the normalization right? I would like to see a test that looks like this:

>>> to_timestamp(-1, 'us').components
(-1, 23, 59, 59, 999, 999)

[jreback]

@shoyer

That example is super ambiguous! Maybe we should require giving a sign in the string parser if there are any negatives. Or if not, let's just document very carefully. But yes, I would lean that way (which is more compatible with timedelta).

The sign IS parsed in the construction (has to be leading element)

hmm the sign is being ignored in that type of construction....let me see

[shoyer]

I think '-1 days -2 hours -3 min' or '-1 days +2 hours +3 min' is much more explicit (and should be preferred).

[jreback]

I think these look better! (was buggy)

In [1]: pd.Timedelta('-1 days 1us').components_as_tuple
Out[1]: (-2, 23, 59, 59, 999, 999, 0)

In [2]: pd.Timedelta('-2 days 1us').components_as_tuple
Out[2]: (-3, 23, 59, 59, 999, 999, 0)

In [3]: pd.Timedelta('-2 days 1 hours 1us').components_as_tuple
Out[3]: (-3, 22, 59, 59, 999, 999, 0)

In [4]: pd.Timedelta('-2 days 2 hours 1us').components_as_tuple
Out[4]: (-3, 21, 59, 59, 999, 999, 0)

In [5]: pd.Timedelta('-2 days 2 hours 2 min 1us').components_as_tuple
Out[5]: (-3, 21, 57, 59, 999, 999, 0)

In [6]: pd.Timedelta('2 days 2 hours 2 min 1us').components_as_tuple
Out[6]: (2, 2, 2, 0, 0, 1, 0)

[jreback]

Ok only issue is then that the negative values easily have a 'different' look that was input (its the simplied form). Is this a problem?

@jorisvandenbossche @shoyer

In [4]: abs(pd.Timedelta('-1 days, 10:11:12'))
Out[4]: Timedelta('1 days, 10:11:12')

In [5]: pd.Timedelta('-1 days, 10:11:12')
Out[5]: Timedelta('-2 days, 13:48:48')

In [6]: -pd.Timedelta('-1 days, 10:11:12')
Out[6]: Timedelta('1 days, 10:11:12')

In [7]: -pd.Timedelta('-1 days, 10:11:12').value
Out[7]: 123072000000000

In [8]: pd.Timedelta('-1 days, 10:11:12')
Out[8]: Timedelta('-2 days, 13:48:48')

In [9]: pd.Timedelta('-1 days, 10:11:12').value
Out[9]: -123072000000000

[hayd]

I think the behavior of datetime.timedelta is completely dumb.

+1. The string '-2 days, 13:48:48' reads, at least IMO, like '-2 days, -13:48:48' not '-2 days, +13:48:48' (which is what it must parse to). It sucks that's what timedelta uses, it kinda doubly sucks is that they don't offer a method to parse the string to a timedelta (I couldn't find one) BUT that means we can print with the + as we don't need to comply with their parse their function.

That is, I think this should print as '-2 days, +13:48:48'.

---

One small improvement is that components_as_tuple could be a namedtuple, but we can tweak that later!

[jreback]

easy enough on the namedtuple.

In [2]: pd.Timedelta('-1us').components
Out[2]: Components(days=-1, hours=23, minutes=59, seconds=59, milliseconds=999, microseconds=999, nanoseconds=0)

In [3]: pd.Timedelta('1us').components
Out[3]: Components(days=0, hours=0, minutes=0, seconds=0, milliseconds=0, microseconds=1, nanoseconds=0)

[shoyer]

+1 on @hayd's suggestion to always print the sign for each component if the timedelta is negative, e.g., '-2 days, +13:48:48'

[jreback]

Finally fixed the negative simplfying format. This is actually complicated because of float precision issues (you can't convert all at once but have to do seconds then the fractional part after its converted to an integer). jreback@ee6f91f

note - this is the 'all' format (show it can show the ns, normally we don't).

Have full tests on round-tripping thru the repr. whoosh!

def f():
    for v in ['1s', '-1s', '1us', '-1us', '1 day', '-1 day',
          '-23:59:59.999999', '-1 days +23:59:59.999999',
          '-1ns', '1ns', '-23:59:59.999999999']:
    print "%30.30s -> %30.30s" % (v,pd.Timedelta(v)._repr_base(format='all'))
In [13]: f()
                            1s ->      0 days 00:00:01.000000000
                           -1s ->    -1 days +23:59:59.000000000
                           1us ->      0 days 00:00:00.000001000
                          -1us ->    -1 days +23:59:59.999999000
                         1 day ->      1 days 00:00:00.000000000
                        -1 day ->    -1 days +00:00:00.000000000
              -23:59:59.999999 ->    -1 days +00:00:00.000001000
      -1 days +23:59:59.999999 ->    -1 days +23:59:59.999999000
                          -1ns ->    -1 days +23:59:59.999999999
                           1ns ->      0 days 00:00:00.000000001
           -23:59:59.999999999 ->    -1 days +00:00:00.000000001

[jreback]

any final final comments...?

[jorisvandenbossche]

I merged the sql PR, so you can remove the test skip for that.

For the rest, I think I ran out of comments :-)

[shoyer]

Looks pretty solid to me, now. I guess we can save parsing strings like '-1 days -1 hour' until someone actually asks for it.

[TomAugspurger]

Go for it. I've got some tasks I can try it out on in the next couple weeks.

On Sep 13, 2014, at 4:07 PM, jreback notifications@github.com wrote:

any final final comments...?

—
Reply to this email directly or view it on GitHub.

[jreback]

@shoyer yep, I think -1 days -1 hours can wait (will raise ValueError now).

One possible enhancement is to add a repr format like this:

+/-dddd.hh:mm:ss:mmmuuunnn

e.g. -1.23:59:59.999999123

which I have seen (though to me a bit TOO much)

merging shortly

[jreback]

bombs away!