ENH: Restore original convert_objects and add _convert

[bashtage]

Restores the v0.16 behavior of convert_objects and moves the new
version of _convert
Adds to_numeric for directly converting numeric data

closes #11116
closes #11133

[bashtage]

Still a few things to do:

• Update release notes to remove new docs and add deprecation note
• Make sure copy is passed to _possibly_convert_objects
• Add note about bugs fixed in convert_objects
• Restore docs for convert_objects
• Add tests for to_numeric
• Add docs for to_numeric
• Move converters to a new file, convert.py

[jreback]

How about we put _possibly_convert_objects and _soft_convert_objects in pandas/tools/convert.py (new file)
common.py is just so huge. We can eventually move more stuff their as well. It will have to import common for now.

[jreback]

I can do this after as well.

[shoyer]

I'll just note that there is some value in keeping this function intact in core._common. Xray was using this private function internally, with a broken guard against it changing (see pydata/xarray#569 for details). Obviously, that was our fault and we've fixed it now.

[jreback]

@shoyer I don't think you should be using this internal function (I saw you removed it). and instead use the public .convert_objects

[shoyer]

Fair enough -- but there's not any other way to hook directly into pandas' object array of datetimes -> datetime64 conversion. Might be worth thinking about making such an API at some point.

On Wed, Sep 23, 2015 at 5:58 PM, Jeff Reback notifications@github.com
wrote:

@@ -1857,23 +1857,84 @@ def _maybe_box_datetimelike(value):

_values_from_object = lib.values_from_object

-def _possibly_convert_objects(values,

•                          datetime=True,
  

•                          numeric=True,
  

•                          timedelta=True,
  

•                          coerce=False,
  

  +# TODO: Remove in 0.18 or 2017, which ever is sooner
  +def _possibly_convert_objects(values, convert_dates=True,
  @shoyer I don't think you should be using this internal function (I saw you removed it). and instead use the public .convert_objects

  Reply to this email directly or view it on GitHub:
  https://github.com/pydata/pandas/pull/11173/files#r40276398

[jreback]

you mean like this?

In [20]: pd.to_datetime([datetime.datetime(2012,1,1,0,0),datetime.datetime(2013,1,1,1,1)],box=False)       
Out[20]: 
array(['2011-12-31T19:00:00.000000000-0500',
       '2012-12-31T20:01:00.000000000-0500'], dtype='datetime64[ns]')

[shoyer]

Not quite -- I only want soft conversion (like the pd.Series constructor). I don't want to convert text that matches a regex, only an array of all datetime instances (to datetime64) or all timedelta instances (to timedelta64).

[bashtage]

The important changes are:

convert in ObjectManager: https://github.com/pydata/pandas/pull/11173/files#diff-e705e723b2d6e7c0e2a0443f80916abfR1521

This change requires this to 'route' to either the old or new style converter.

to_numeric: https://github.com/pydata/pandas/pull/11173/files#diff-40d1f19921aa9f2932951124fad07cc6R53

New function.

[jorisvandenbossche]

@bashtage Really thanks a lot for this!

(I am still not fully convinced the deprecation is needed, but I will try to summarize my reasons tomorrow)

[jreback]

@bashtage

• pls move the 2 converting functions either to a new module, core/convert.py.
• let's finish up the documentation changes.

pls let me know asap.

[bashtage]

Still no tests for to_numeric or docs for this method.

[jreback]

looks good! ping when tests for pd.to_numeric

[jreback]

@jorisvandenbossche ok with this

[jorisvandenbossche]

to_numeric needs some fixes (and tests): lib is not yet imported, lib.maybe_convert_numeric expects an array so it bugs on a series as arg, and the final result needs to be boxed into a series again if the arg was a series.

[bashtage]

Yes - it definitely needs work. It expects something with the buffer
interface, so I need to ensure that this works as expected.

On Sun, Sep 27, 2015 at 8:12 PM Joris Van den Bossche <
notifications@github.com> wrote:

to_numeric needs some fixes (and tests): lib is not yet imported,
lib.maybe_convert_numeric expects an array so it bugs on a series as arg,
and the final result needs to be boxed into a series again if the arg was a
series.

—
Reply to this email directly or view it on GitHub
#11173 (comment).

[jreback]

@bashtage did you have a chance to add some tests here?

[bashtage]

Should be ready when green I think

[bashtage]

Can squash commits - wanted to make it easier to see changes at each stage since close to release

[bashtage]

@jreback Green - let me know if I missed anything. Only tick left is to add some docs although this can probably wait since convert_objects is still available.

[jorisvandenbossche]

'coerce' shouldn't be here?