BLD: pin cython language level to '2'

[tacaswell]

Not explicitly pinning the language level has been producing future
warnings from cython. The next release of cython is going to change
the default level to '3str' under which the pandas cython extensions
do not compile.

The long term solution is to update the cython files to the next
language level, but this is a stop-gap to keep pandas building.

• closes #xxxx
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[tacaswell]

Alternative to #24538

The next version of cython is going to change the default to '3str', may be worth applying the pinning to bug-fix branches?

[codecov]

Codecov Report

Merging #25145 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25145      +/-   ##
==========================================
- Coverage   92.37%   92.37%   -0.01%     
==========================================
  Files         166      166              
  Lines       52408    52408              
==========================================
- Hits        48412    48411       -1     
- Misses       3996     3997       +1

Flag	Coverage Δ
#multiple	90.79% <ø> (ø)	⬆️
#single	42.86% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/util/testing.py	88.04% <0%> (-0.1%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e38d55...0daffc1. Read the comment docs.

[codecov]

Codecov Report

Merging #25145 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25145      +/-   ##
==========================================
- Coverage   92.37%   92.37%   -0.01%     
==========================================
  Files         166      166              
  Lines       52408    52408              
==========================================
- Hits        48412    48411       -1     
- Misses       3996     3997       +1

Flag	Coverage Δ
#multiple	90.79% <ø> (ø)	⬆️
#single	42.86% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/util/testing.py	88.04% <0%> (-0.1%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e38d55...08fc674. Read the comment docs.

[WillAyd]

Thanks for the PR! Unfortunately we are dropping Python2 support in the 0.25 release so I don't think pinning to that syntax here makes sense. Should be going forward with the PR you've linked to in the near future

[TomAugspurger]

IIUC, this would allow people to compile pandas 0.24.x from source with future versions of Cython? Any reason not to do that for now?

[WillAyd]

@TomAugspurger makes sense. I was thinking originally it would be nice to not have to pin anything and implicitly get everything onto Py3 with next release, but suppose it doesn't hurt to have this pinned for 0.24.x

[WillAyd]

There is a dict in this file which contains directives sent to cythonize. Think it would be more consistent if you set this therein (see PR you linked for sample of how)

[tacaswell]

sure, moved and force-pushed.

[TomAugspurger]

Just to be clear, I'm don't *know* that that's a benefit of pinning it for now, before your proper fix is ready. That was more of a question :)

…

On Mon, Feb 4, 2019 at 3:54 PM William Ayd ***@***.***> wrote: ***@***.**** requested changes on this pull request. ------------------------------ In setup.py <#25145 (comment)>: > @@ -472,6 +472,7 @@ def maybe_cythonize(extensions, *args, **kwargs): """ Render tempita templates before calling cythonize """ + kwargs.setdefault('language_level', 2) There is a dict in this file which contains directives sent to cythonize. Think it would be more consistent if you set this therein (see PR you linked for sample of how) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25145 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHInSagdfPuQAoEu3BlzbZ2SJo260Uks5vKKwrgaJpZM4ahrg3> .

[WillAyd]

@tacaswell where do you see the note about 3str becoming the default on next release? Also, does that actually impact compilation? IIRC there was only one spot in the code base that was really problematic for 2-to-3 comparability and it was the result of unicode usage, which I would think 3str would still respect?

[tacaswell]

https://cython.readthedocs.io/en/latest/src/changes.html?highlight=3str#id7 for "3str" becoming the default in the future.

Empirically, pandas does not cythonize/compile properly without this patch with cython master.

This is setting the dialect of cython (not the version of python) so you can use any of the language levels (2, 3, 3str) with any version of python (as it generates c-extensions that then have the correct bindings to the calling python) so I think this is un-related to deciding to drop py2 support. I think you should back port this to any version of pandas you want people to be able to compile with future versions of cython that you do not want to backport #24538 to.

[jorisvandenbossche]

But Python 2.7 is not yet dropped from 0.24.x. So as @tacaswell said it might make sense to apply this patch for that reason?

[WillAyd]

Yea understood about how it's technically different from our Python support level, though we theoretically want to be consistent in how we write code.

IIRC the original Cython warning would mention a push to 3 syntax in the future but if that is now 3str I'm under the impression our code on 0.24 is already compatible with that syntax, hence why I don't think we need to pin anything here. Are you seeing errors building with 3str?

[jorisvandenbossche]

Sorry, ignore my previous comment, my github notifications were lagging seriously behind. @TomAugspurger already noticed the same

[tacaswell]

pandas master + cython master:

(bleeding) ✘  ~/source/other_source/pandas [master {pandas/master}|✔ ]
jupiter@17:48  ➤  pip install -ve .
Created temporary directory: /tmp/pip-ephem-wheel-cache-xszq9dms
Created temporary directory: /tmp/pip-req-tracker-ecjmim3m
Created requirements tracker '/tmp/pip-req-tracker-ecjmim3m'
Created temporary directory: /tmp/pip-install-f_hm_b2w
Obtaining file:///home/tcaswell/source/other_source/pandas
  Added file:///home/tcaswell/source/other_source/pandas to build tracker '/tmp/pip-req-tracker-ecjmim3m'
    Running setup.py (path:/home/tcaswell/source/other_source/pandas/setup.py) egg_info for package from file:///home/tcaswell/source/other_source/pandas
    Running command python setup.py egg_info
    /home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Compiler/Main.py:366: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/tcaswell/source/other_source/pandas/pandas/_libs/algos.pxd
      tree = Parsing.p_module(s, pxd, full_module_name)
    /home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Compiler/Main.py:366: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/tcaswell/source/other_source/pandas/pandas/_libs/groupby.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)

    Error compiling Cython file:
    ------------------------------------------------------------
    ...

            a = tmp
            n -= na_count

        if n % 2:
            result = kth_smallest_c( a, n / 2, n)
                                         ^
    ------------------------------------------------------------

    pandas/_libs/groupby.pyx:59:38: Cannot assign type 'double' to 'Py_ssize_t'

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
            n -= na_count

        if n % 2:
            result = kth_smallest_c( a, n / 2, n)
        else:
            result = (kth_smallest_c(a, n / 2, n) +
                                         ^
    ------------------------------------------------------------

    pandas/_libs/groupby.pyx:61:38: Cannot assign type 'double' to 'Py_ssize_t'

    Error compiling Cython file:
    ------------------------------------------------------------
    ...

        if n % 2:
            result = kth_smallest_c( a, n / 2, n)
        else:
            result = (kth_smallest_c(a, n / 2, n) +
                      kth_smallest_c(a, n / 2 - 1, n)) / 2
                                             ^
    ------------------------------------------------------------

    pandas/_libs/groupby.pyx:62:42: Cannot assign type 'double' to 'Py_ssize_t'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/tcaswell/source/other_source/pandas/setup.py", line 737, in <module>
        ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
      File "/home/tcaswell/source/other_source/pandas/setup.py", line 489, in maybe_cythonize
        return cythonize(extensions, *args, **kwargs)
      File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1091, in cythonize
        cythonize_one(*args)
      File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1214, in cythonize_one
        raise CompileError(None, pyx_file)
    Cython.Compiler.Errors.CompileError: pandas/_libs/groupby.pyx
    Compiling pandas/_libs/algos.pyx because it changed.
    Compiling pandas/_libs/groupby.pyx because it changed.
    Compiling pandas/_libs/hashing.pyx because it changed.
    Compiling pandas/_libs/hashtable.pyx because it changed.
    Compiling pandas/_libs/index.pyx because it changed.
    Compiling pandas/_libs/indexing.pyx because it changed.
    Compiling pandas/_libs/internals.pyx because it changed.
    Compiling pandas/_libs/interval.pyx because it changed.
    Compiling pandas/_libs/join.pyx because it changed.
    Compiling pandas/_libs/lib.pyx because it changed.
    Compiling pandas/_libs/missing.pyx because it changed.
    Compiling pandas/_libs/parsers.pyx because it changed.
    Compiling pandas/_libs/reduction.pyx because it changed.
    Compiling pandas/_libs/ops.pyx because it changed.
    Compiling pandas/_libs/properties.pyx because it changed.
    Compiling pandas/_libs/reshape.pyx because it changed.
    Compiling pandas/_libs/skiplist.pyx because it changed.
    Compiling pandas/_libs/sparse.pyx because it changed.
    Compiling pandas/_libs/tslib.pyx because it changed.
    Compiling pandas/_libs/tslibs/ccalendar.pyx because it changed.
    Compiling pandas/_libs/tslibs/conversion.pyx because it changed.
    Compiling pandas/_libs/tslibs/fields.pyx because it changed.
    Compiling pandas/_libs/tslibs/frequencies.pyx because it changed.
    Compiling pandas/_libs/tslibs/nattype.pyx because it changed.
    Compiling pandas/_libs/tslibs/np_datetime.pyx because it changed.
    Compiling pandas/_libs/tslibs/offsets.pyx because it changed.
    Compiling pandas/_libs/tslibs/parsing.pyx because it changed.
    Compiling pandas/_libs/tslibs/period.pyx because it changed.
    Compiling pandas/_libs/tslibs/resolution.pyx because it changed.
    Compiling pandas/_libs/tslibs/strptime.pyx because it changed.
    Compiling pandas/_libs/tslibs/timedeltas.pyx because it changed.
    Compiling pandas/_libs/tslibs/timestamps.pyx because it changed.
    Compiling pandas/_libs/tslibs/timezones.pyx because it changed.
    Compiling pandas/_libs/testing.pyx because it changed.
    Compiling pandas/_libs/window.pyx because it changed.
    Compiling pandas/_libs/writers.pyx because it changed.
    Compiling pandas/io/sas/sas.pyx because it changed.
    Compiling pandas/io/msgpack/_packer.pyx because it changed.
    Compiling pandas/io/msgpack/_unpacker.pyx because it changed.
    [ 1/39] Cythonizing pandas/_libs/algos.pyx
    [ 2/39] Cythonizing pandas/_libs/groupby.pyx
Cleaning up...
Removed file:///home/tcaswell/source/other_source/pandas from build tracker '/tmp/pip-req-tracker-ecjmim3m'
Removed build tracker '/tmp/pip-req-tracker-ecjmim3m'
Command "python setup.py egg_info" failed with error code 1 in /home/tcaswell/source/other_source/pandas/
Exception information:
Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/resolve.py", line 225, in _get_abstract_dist_for
    return self.preparer.prepare_editable_requirement(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 158, in prep_for_dist
    self.req.prepare_metadata()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 536, in prepare_metadata
    self.run_egg_info()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 611, in run_egg_info
    call_subprocess(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 759, in call_subprocess
    raise InstallationError(
pip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /home/tcaswell/source/other_source/pandas/

[WillAyd]

Ah OK division is where this gets tripped up. Thanks for confirming I’d be ok with the pin here then

…

Sent from my iPhone

On Feb 4, 2019, at 2:50 PM, Thomas A Caswell ***@***.***> wrote: pandas master + cython master: (bleeding) ✘ ~/source/other_source/pandas [master {pandas/master}|✔ ] ***@***.***:48 ➤ pip install -ve . Created temporary directory: /tmp/pip-ephem-wheel-cache-xszq9dms Created temporary directory: /tmp/pip-req-tracker-ecjmim3m Created requirements tracker '/tmp/pip-req-tracker-ecjmim3m' Created temporary directory: /tmp/pip-install-f_hm_b2w Obtaining file:///home/tcaswell/source/other_source/pandas Added file:///home/tcaswell/source/other_source/pandas to build tracker '/tmp/pip-req-tracker-ecjmim3m' Running setup.py (path:/home/tcaswell/source/other_source/pandas/setup.py) egg_info for package from file:///home/tcaswell/source/other_source/pandas Running command python setup.py egg_info /home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Compiler/Main.py:366: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/tcaswell/source/other_source/pandas/pandas/_libs/algos.pxd tree = Parsing.p_module(s, pxd, full_module_name) /home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Compiler/Main.py:366: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/tcaswell/source/other_source/pandas/pandas/_libs/groupby.pyx tree = Parsing.p_module(s, pxd, full_module_name) Error compiling Cython file: ------------------------------------------------------------ ... a = tmp n -= na_count if n % 2: result = kth_smallest_c( a, n / 2, n) ^ ------------------------------------------------------------ pandas/_libs/groupby.pyx:59:38: Cannot assign type 'double' to 'Py_ssize_t' Error compiling Cython file: ------------------------------------------------------------ ... n -= na_count if n % 2: result = kth_smallest_c( a, n / 2, n) else: result = (kth_smallest_c(a, n / 2, n) + ^ ------------------------------------------------------------ pandas/_libs/groupby.pyx:61:38: Cannot assign type 'double' to 'Py_ssize_t' Error compiling Cython file: ------------------------------------------------------------ ... if n % 2: result = kth_smallest_c( a, n / 2, n) else: result = (kth_smallest_c(a, n / 2, n) + kth_smallest_c(a, n / 2 - 1, n)) / 2 ^ ------------------------------------------------------------ pandas/_libs/groupby.pyx:62:42: Cannot assign type 'double' to 'Py_ssize_t' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/tcaswell/source/other_source/pandas/setup.py", line 737, in <module> ext_modules=maybe_cythonize(extensions, compiler_directives=directives), File "/home/tcaswell/source/other_source/pandas/setup.py", line 489, in maybe_cythonize return cythonize(extensions, *args, **kwargs) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1091, in cythonize cythonize_one(*args) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1214, in cythonize_one raise CompileError(None, pyx_file) Cython.Compiler.Errors.CompileError: pandas/_libs/groupby.pyx Compiling pandas/_libs/algos.pyx because it changed. Compiling pandas/_libs/groupby.pyx because it changed. Compiling pandas/_libs/hashing.pyx because it changed. Compiling pandas/_libs/hashtable.pyx because it changed. Compiling pandas/_libs/index.pyx because it changed. Compiling pandas/_libs/indexing.pyx because it changed. Compiling pandas/_libs/internals.pyx because it changed. Compiling pandas/_libs/interval.pyx because it changed. Compiling pandas/_libs/join.pyx because it changed. Compiling pandas/_libs/lib.pyx because it changed. Compiling pandas/_libs/missing.pyx because it changed. Compiling pandas/_libs/parsers.pyx because it changed. Compiling pandas/_libs/reduction.pyx because it changed. Compiling pandas/_libs/ops.pyx because it changed. Compiling pandas/_libs/properties.pyx because it changed. Compiling pandas/_libs/reshape.pyx because it changed. Compiling pandas/_libs/skiplist.pyx because it changed. Compiling pandas/_libs/sparse.pyx because it changed. Compiling pandas/_libs/tslib.pyx because it changed. Compiling pandas/_libs/tslibs/ccalendar.pyx because it changed. Compiling pandas/_libs/tslibs/conversion.pyx because it changed. Compiling pandas/_libs/tslibs/fields.pyx because it changed. Compiling pandas/_libs/tslibs/frequencies.pyx because it changed. Compiling pandas/_libs/tslibs/nattype.pyx because it changed. Compiling pandas/_libs/tslibs/np_datetime.pyx because it changed. Compiling pandas/_libs/tslibs/offsets.pyx because it changed. Compiling pandas/_libs/tslibs/parsing.pyx because it changed. Compiling pandas/_libs/tslibs/period.pyx because it changed. Compiling pandas/_libs/tslibs/resolution.pyx because it changed. Compiling pandas/_libs/tslibs/strptime.pyx because it changed. Compiling pandas/_libs/tslibs/timedeltas.pyx because it changed. Compiling pandas/_libs/tslibs/timestamps.pyx because it changed. Compiling pandas/_libs/tslibs/timezones.pyx because it changed. Compiling pandas/_libs/testing.pyx because it changed. Compiling pandas/_libs/window.pyx because it changed. Compiling pandas/_libs/writers.pyx because it changed. Compiling pandas/io/sas/sas.pyx because it changed. Compiling pandas/io/msgpack/_packer.pyx because it changed. Compiling pandas/io/msgpack/_unpacker.pyx because it changed. [ 1/39] Cythonizing pandas/_libs/algos.pyx [ 2/39] Cythonizing pandas/_libs/groupby.pyx Cleaning up... Removed file:///home/tcaswell/source/other_source/pandas from build tracker '/tmp/pip-req-tracker-ecjmim3m' Removed build tracker '/tmp/pip-req-tracker-ecjmim3m' Command "python setup.py egg_info" failed with error code 1 in /home/tcaswell/source/other_source/pandas/ Exception information: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 176, in main status = self.run(options, args) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 315, in run resolver.resolve(requirement_set) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/resolve.py", line 131, in resolve self._resolve_one(requirement_set, req) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/resolve.py", line 225, in _get_abstract_dist_for return self.preparer.prepare_editable_requirement( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement abstract_dist.prep_for_dist(finder, self.build_isolation) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 158, in prep_for_dist self.req.prepare_metadata() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 536, in prepare_metadata self.run_egg_info() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 611, in run_egg_info call_subprocess( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 759, in call_subprocess raise InstallationError( pip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /home/tcaswell/source/other_source/pandas/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jreback]

not sure I understand why we need this change? to prevent source building on later cythons? we could just as easily (maybe better) put a max on cython if PY2 in setup.py (like we have a min now).

[TomAugspurger]

Its probably easier for us and users to set this value than to put a max version on Cython. Either should work though.

…

On Feb 4, 2019, at 17:46, Jeff Reback ***@***.***> wrote: not sure I understand why we need this change? to prevent source building on later cythons? we could just as easily (maybe better) put a max on cython if PY2 in setup.py (like we have a min now). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[WillAyd]

I slightly prefer this to code in setup because:

1. I think it's less verbose AND
2. I don't see an indication that 2 syntax gets dropped in the next Cython, only that the default changes

So given point 2 above I don't see a reason to necessarily cap the Cython dependency.

Not a huge sticking point for me either way though

[jreback]

thanks @tacaswell