Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4. #174

jun66j5 · 2015-07-30T11:25:03Z

Python 3.4.3 (default, Apr 13 2015, 22:30:47)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import babel
>>> babel.__version__
'2.0'
>>> from babel.core import Locale
>>> Locale.parse('zh_CN')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/py34/lib/python3.4/site-packages/babel/core.py", line 286, in parse
    language = get_global('language_aliases').get(language, language)
  File "/venv/py34/lib/python3.4/site-packages/babel/core.py", line 58, in get_global
    _global_data = pickle.load(fileobj)
TypeError: an integer is required (got type str)
>>>

Locale.parse('zh_CN') should work and load zh_CN localedata.

On Python 2.7, it works fine.

Python 2.7.3 (default, Jun 22 2015, 19:33:41)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import babel
>>> babel.__version__
'2.0'
>>> from babel.core import Locale
>>> Locale.parse('zh_CN')
Locale('zh', territory='CN', script='Hans')
>>>

The text was updated successfully, but these errors were encountered:

sils · 2015-07-30T11:47:02Z

@jun66j5 thanks for your report!

Babel was barely maintained for some time and we're some new people trying to get it up to speed again. Because of this it might take some time until we're able to adress this bug.

erickwilder · 2015-07-30T16:25:06Z

@jun66j5 is this working in the previous (1.x) release?

jun66j5 · 2015-07-30T18:44:35Z

Babel 1.3 works fine with Python 3.3 and 3.4.

Python 3.3.6 (default, Jan 28 2015, 16:27:12)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import babel
>>> babel.__version__
'1.3'
>>> from babel.core import Locale
>>> Locale.parse('zh_CN')
Locale('zh', territory='CN', script='Hans')

blisc · 2015-07-30T21:27:39Z

I got the a similar error trying to compile Python documentation with Sphinx.
Downgrading to babel 1.3 seems to have resolved the issue on 3.4.3

sidloki · 2015-07-31T00:03:13Z

Same error here. If I clone and checkout branch 2.0 and run:

/path/to/babel$ python3.4 setup.py import_cldr release sdist
/path/to/babel$ pip install dist/Babel-2.0.tar.gz

everything works fine (both python3.4 and python2.7). A package created with python2.7 is not working with python3.4.

erickwilder · 2015-07-31T00:41:07Z

I can't reproduce the error. Both running directly at the Python interpreter and writing a new test case and running against tox/pytest results on the Locale object being created (and tests passing)
I'm on a MacOSX and Python 3.x versions installed with pyenv. Anyone with a Linux environment can test the issue174/locale-parse-py3x branch?

jun66j5 · 2015-07-31T03:38:08Z

babel/global.dat file in Babel-2.0.tar.gz on PyPI seems to be broken.
The global.dat cannot be loaded with Python 3.4 but it can be loaded with Python 2.6.

$ wget -q -P /tmp https://pypi.python.org/packages/source/B/Babel/Babel-2.0.tar.gz#md5=62917719897a81e22dcaa3b17eeb11d8
$ md5sum /tmp/Babel-2.0.tar.gz
62917719897a81e22dcaa3b17eeb11d8  /tmp/Babel-2.0.tar.gz
$ tar tzvf /tmp/Babel-2.0.tar.gz | grep /global.dat
-rw-r--r-- mitsuhiko/staff 116412 2015-07-27 19:35 Babel-2.0/babel/global.dat
$ tar xzf /tmp/Babel-2.0.tar.gz -C /tmp Babel-2.0/babel/global.dat
$ ls -l /tmp/Babel-2.0/babel/global.dat
-rw-r--r-- 1 jun66j5 jun66j5 116412 Jul 27 19:35 /tmp/Babel-2.0/babel/global.dat
$ python3.4
Python 3.4.3 (default, Apr 13 2015, 22:30:47)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('/tmp/Babel-2.0/babel/global.dat', 'rb') as f: data = pickle.load(f)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: an integer is required (got type str)
>>>
$ python2.6
Python 2.6.9 (default, Oct 22 2014, 19:47:46)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('/tmp/Babel-2.0/babel/global.dat', 'rb') as f: data = pickle.load(f)
...
>>> data.keys()
['territory_aliases', 'territory_zones', 'zone_aliases', 'territory_currencies', 'windows_zone_mapping', 'script_aliases', '_version', 'variant_aliases', 'language_aliases', 'meta_zones', 'likely_subtags', 'zone_territories']

sidloki · 2015-07-31T10:12:12Z

To reproduce the error:

# Python 2.7 environment
$ make clean
$ make import-cldr
$ py.test
# all tests passed

# python3.4 environment
$ make clean-pyc
$ py.test
# some tests failed with TypeError: an integer is required (got type str)

The problem is, that the global.dat created in a python2 environment cannot be used in a python3 environment.

If I create the global.dat in a python3 environment and use it under python2, I get the follwoing failures:

[doctest] babel.core.Locale.parse 
215 
216         >>> Locale.parse(l)
217         Locale('de', territory='DE')
218 
219         This also can perform resolving of likely subtags which it does
220         by default.  This is for instance useful to figure out the most
221         likely locale for a territory you can use ``'und'`` as the
222         language tag:
223 
224         >>> Locale.parse('und_AT')
Expected:
    Locale('de', territory='AT')
Got:
    Locale(u'de', territory=u'AT')

[doctest] babel.numbers.get_territory_currencies
076     of when the currency became active.  The longer the currency is being in
077     use the more to the left of the list it will be.
078 
079     The start date defaults to today.  If no end date is given it will be the
080     same as the start date.  Otherwise a range can be defined.  For instance
081     this can be used to find the currencies in use in Austria between 1995 and
082     2011:
083 
084     >>> from datetime import date
085     >>> get_territory_currencies('AT', date(1995, 1, 1), date(2011, 1, 1))
Expected:
    ['ATS', 'EUR']
Got:
    [u'ATS', u'EUR']

shimizukawa · 2015-08-01T15:41:33Z

To reproduce the error:

C:\> python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32
>>> from babel.messages.pofile import read_po
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\py34\lib\site-packages\babel\messages\__init__.py", line 12, in <module>
    from babel.messages.catalog import *
  File "D:\py34\lib\site-packages\babel\messages\catalog.py", line 23, in <module>
    from babel.dates import format_datetime
  File "D:\py34\lib\site-packages\babel\dates.py", line 28, in <module>
    from babel.util import UTC, LOCALTZ
  File "D:\py34\lib\site-packages\babel\util.py", line 278, in <module>
    from babel import localtime
  File "D:\py34\lib\site-packages\babel\localtime\__init__.py", line 21, in <module>
    from babel.localtime._win32 import _get_localzone
  File "D:\py34\lib\site-packages\babel\localtime\_win32.py", line 18, in <module>
    tz_names = get_global('windows_zone_mapping')
  File "D:\py34\lib\site-packages\babel\core.py", line 58, in get_global
    _global_data = pickle.load(fileobj)
TypeError: an integer is required (got type str)

mitsuhiko · 2015-08-01T17:01:19Z

This is a really annoying issue. Ideally we would stop using pickle and switch to something we can efficiently load from disk (msgpack as a start). Long term it would be great if we could mmap something, but unfortunately that might be pretty slow with the constant boxing :(

icoxfog417 · 2015-08-02T01:48:59Z

As @jun66j5 says, global.dat on PyPI may be broken.
I installed from GitHub 2.0 branch as below instead of pip install babel, then the error didn't occur.

pip install git+https://github.com/mitsuhiko/babel.git@2.0

My environments is Windows 8.1 and Python 3.4.3.

thuvh · 2015-08-04T09:48:27Z

I think this error happens in window environments

…ncompatible pickle format between python 2 and 3 (python-babel#174)

…d incompatible pickle format between python 2 and 3 (python-babel#174)

erickwilder · 2015-08-04T14:08:36Z

@jun66j5
I saw that you're working on some changes to solve this issue. Any progress? Do you need some help?

jun66j5 · 2015-08-05T02:36:38Z

Thanks, @erickwilder. Updated https://github.com/jun66j5/babel/commits/issue174/workaround.

Root cause is incompatible pickle data for datetime.date classes between Python 2 and 3. In that branch, the uses of the classes are removed and added unit tests for that.

Should I create a pull request with that branch?

erickwilder · 2015-08-05T11:44:00Z

@jun66j5
Yes please. Submit a pull request so we can review.
We'll need to backport changes of that to the 2.x branch, so your call to use master or the maintenance branch as a target when opening.

To avoid incompatible *.dat files between Python 2 and 3.

ericholscher · 2015-08-11T03:29:19Z

Ran into this one as well on Windows & 3.4: https://ci.appveyor.com/project/ericholscher/sphinx-autoapi/build/1.0.5

=================================== ERRORS ==================================== 
_________________ ERROR collecting tests/test_integration.py __________________ 
tests\test_integration.py:9: in <module> 
    from sphinx.application import Sphinx 
.tox\py34\lib\site-packages\sphinx\application.py:37: in <module> 
    from sphinx.builders import BUILTIN_BUILDERS 
.tox\py34\lib\site-packages\sphinx\builders\__init__.py:23: in <module> 
    from sphinx.util import i18n, path_stabilize 
.tox\py34\lib\site-packages\sphinx\util\i18n.py:15: in <module> 
    from babel.messages.pofile import read_po 
.tox\py34\lib\site-packages\babel\messages\__init__.py:12: in <module> 
    from babel.messages.catalog import * 
.tox\py34\lib\site-packages\babel\messages\catalog.py:23: in <module> 
    from babel.dates import format_datetime 
.tox\py34\lib\site-packages\babel\dates.py:28: in <module>
    from babel.util import UTC, LOCALTZ
.tox\py34\lib\site-packages\babel\util.py:278: in <module>
    from babel import localtime 
.tox\py34\lib\site-packages\babel\localtime\__init__.py:21: in <module> 
    from babel.localtime._win32 import _get_localzone 
.tox\py34\lib\site-packages\babel\localtime\_win32.py:18: in <module> 
    tz_names = get_global('windows_zone_mapping') 
.tox\py34\lib\site-packages\babel\core.py:58: in get_global 
    _global_data = pickle.load(fileobj) 
E   TypeError: an integer is required (got type str)

To avoid incompatible *.dat files between Python 2 and 3. Added unit tests for that *.dat files have only babel classes Author: Jun Omae <jun66j5@gmail.com>

vstinner · 2015-08-28T13:31:13Z

Hi, I got this issue (TypeError in babel.core.get_global()) when working on porting the OpenStack Horizon project to Python 3. I confirm that the problem is that the file babel/global.dat of Babel-2.0.tar.gz (the file hosted at PyPI) cannot be loaded by Python 3.

I also confirm that the root issue is that Babel uses datetime.date objects in global.dat. Python 2 serializes a datetime.date using a native string which is serialized to the pickle opcode SHORT_BINSTRING. On Python 3, this opcode produces a Unicode string, whereas datetime.date constructor only accepts a bytes string.

Python 3 serializes a datetime.date differently: as a Unicode string encoded to latin1 (it calls the method .encode('latin1') on the string). The Python 3 output works on Python 2 and Python 3. See the Python bug 13505: Bytes objects pickled in 3.x with protocol <=2 are unpickled incorrectly in 2.x.

So a workaround is to produce babel/global.dat using Python 3.

A better fix is to avoid the datetime.date type: #188 implements this, +1 for this fix ;-)

Backporting #174 bugfix to 2.x version

vstinner · 2015-09-10T08:41:25Z

Cool! The fix (#188) was merged! Would it be possible to get a bugfix release? The latest Babel release on PyPI is incompatible with Python 3 :-(

sils · 2015-09-10T08:49:39Z

@Haypo we're still not having access to pypi to update that, it's a bit hard to reach the old maintainer. You can install Babel from git in the meantime: https://pip.pypa.io/en/latest/reference/pip_install.html#git

@erickwilder closing as it was fixed

…ironment. Closes #1976. see also: * python-babel/babel#174 * python-babel/babel#188 Version spec syntax "babel>=1.3,!=2.0" is following PEP440: https://www.python.org/dev/peps/pep-0440/#version-exclusion and it works with pip 6.0 or later (I didn't check before pip 6.0).

refs: * 6bce0a1 * python-babel/babel#174

erickwilder added the bug label Jul 30, 2015

erickwilder added a commit that referenced this issue Jul 31, 2015

Add Locale.parse testcase (related to issue #174)

0bd889c

erickwilder self-assigned this Jul 31, 2015

sils mentioned this issue Jul 31, 2015

Exception occurs at unpickling global.dat #175

Closed

shimizukawa mentioned this issue Aug 1, 2015

Babel 2.0 issue with Python v3.4.3 on Windows sphinx-doc/sphinx#1976

Closed

jun66j5 added a commit to jun66j5/babel that referenced this issue Aug 4, 2015

Removed uses date instances from global.dat for workaround to avoid i…

9b725d5

…ncompatible pickle format between python 2 and 3 (python-babel#174)

jun66j5 added a commit to jun66j5/babel that referenced this issue Aug 4, 2015

Removed uses of date instances from global.dat for workaround to avoi…

6860ffa

…d incompatible pickle format between python 2 and 3 (python-babel#174)

jun66j5 mentioned this issue Aug 5, 2015

Avoid incompatible *.dat files between Python 2 and 3 #188

Merged

jun66j5 added a commit to jun66j5/babel that referenced this issue Aug 5, 2015

Removed uses of datetime.date class from *.dat files (python-babel#174)

94f6830

To avoid incompatible *.dat files between Python 2 and 3.

erickwilder pushed a commit to erickwilder/babel that referenced this issue Aug 5, 2015

Removed uses of datetime.date class from *.dat files (python-babel#174)

42832ec

To avoid incompatible *.dat files between Python 2 and 3.

erickwilder mentioned this issue Aug 5, 2015

Backporting #174 bugfix to 2.x version #193

Merged

mdickinson mentioned this issue Aug 12, 2015

Attempt to fix Appveyor failures on Python 3 enthought/traits#261

Merged

jun66j5 mentioned this issue Aug 22, 2015

Traceback in babel, when using it with Python 3.4 & Windows 10 #205

Closed

This was referenced Sep 7, 2015

Fabmanager and appbuilder both throw type error on Python 3.4.3 / windows 7 dpgaspar/Flask-AppBuilder#220

Closed

babel throws a type error on imports python 3.3/3.4 on windows 7 #207

Closed

erickwilder added a commit that referenced this issue Sep 9, 2015

Merge pull request #193 from erickwilder/issue174/2.x-backport

17d77d0

Backporting #174 bugfix to 2.x version

sils closed this as completed Sep 10, 2015

gschizas mentioned this issue Sep 16, 2015

global.dat cannot be unpickled with Python 3.5 #221

Closed

msarahan mentioned this issue Sep 16, 2015

sphinx / babel dependencies lead to downgrade ContinuumIO/anaconda-issues#402

Closed

vstinner mentioned this issue Sep 21, 2015

Need a new release to fix Python 3 compatibility #228

Closed

jun66j5 unassigned erickwilder Sep 21, 2015

pyup-bot mentioned this issue Apr 27, 2017

Pin babel to latest version 2.4.0 osteele/assignment-dashboard#24

Open

pyup-bot mentioned this issue Jul 4, 2017

Initial Update pelson/pyggybank#2

Merged

This was referenced Aug 17, 2017

Pin babel to latest version 2.4.0 springermac/pyinstaller#67

Merged

Pin babel to latest version 2.5.0 osteele/assignment-dashboard#51

Open

This was referenced Aug 26, 2017

Pin babel to latest version 2.5.0 nicfit/nicfit.py#129

Merged

Initial Update oddbird/oddsite#127

Closed

pyup-bot mentioned this issue Sep 4, 2017

Scheduled weekly dependency update for week 36 oddbird/oddsite#128

Merged

pyup-bot mentioned this issue Sep 14, 2017

Pin babel to latest version 2.5.1 osteele/assignment-dashboard#56

Open

This was referenced Jan 15, 2018

Pin babel to latest version 2.5.2 osteele/assignment-dashboard#76

Open

Pin babel to latest version 2.5.3 osteele/assignment-dashboard#78

Open

tk0miya added a commit to tk0miya/sphinx that referenced this issue Feb 7, 2020

Close sphinx-doc#6623: babel-2.0 or above is available (Unpinned)

853e71b

refs: * 6bce0a1 * python-babel/babel#174

tk0miya mentioned this issue Feb 7, 2020

Close #6623: babel-2.0 or above is available (Unpinned) sphinx-doc/sphinx#7110

Merged

vorpal-buildbot mentioned this issue Jun 13, 2020

Initial Update PennyDreadfulMTG/Penny-Dreadful-Tools#7430

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4. #174

Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4. #174

jun66j5 commented Jul 30, 2015

sils commented Jul 30, 2015

erickwilder commented Jul 30, 2015

jun66j5 commented Jul 30, 2015

blisc commented Jul 30, 2015

sidloki commented Jul 31, 2015

erickwilder commented Jul 31, 2015

jun66j5 commented Jul 31, 2015

sidloki commented Jul 31, 2015

shimizukawa commented Aug 1, 2015

mitsuhiko commented Aug 1, 2015

icoxfog417 commented Aug 2, 2015

thuvh commented Aug 4, 2015

erickwilder commented Aug 4, 2015

jun66j5 commented Aug 5, 2015

erickwilder commented Aug 5, 2015

ericholscher commented Aug 11, 2015

vstinner commented Aug 28, 2015

vstinner commented Sep 10, 2015

sils commented Sep 10, 2015

Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4. #174

Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4. #174

Comments

jun66j5 commented Jul 30, 2015

sils commented Jul 30, 2015

erickwilder commented Jul 30, 2015

jun66j5 commented Jul 30, 2015

blisc commented Jul 30, 2015

sidloki commented Jul 31, 2015

erickwilder commented Jul 31, 2015

jun66j5 commented Jul 31, 2015

sidloki commented Jul 31, 2015

shimizukawa commented Aug 1, 2015

mitsuhiko commented Aug 1, 2015

icoxfog417 commented Aug 2, 2015

thuvh commented Aug 4, 2015

erickwilder commented Aug 4, 2015

jun66j5 commented Aug 5, 2015

erickwilder commented Aug 5, 2015

ericholscher commented Aug 11, 2015

vstinner commented Aug 28, 2015

vstinner commented Sep 10, 2015

sils commented Sep 10, 2015