Fix compatibility with Jython #378
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch was taken from #369 (comment), authored by Pekka Klärck <peke@iki.fi>. In short, Jython doesn't support lone surrogates, so importing yaml (and in particular, loading `reader.py`) caused a UnicodeDecodeError. This patch works around this through a clever use of `eval` to defer evaluation of the string containing the lone surrogates, only doing it on non-Jython platforms. This is only done in `lib/yaml/reader.py` and not `lib3/yaml/reader.py` because Jython does not support Python 3. With this patch, Jython's behavior with respect to Unicode code points over 0xFFFF becomes as it was before 0716ae2. It still does not pass all the unit tests on Jython (passes 1275, fails 3, errors on 1); all the failing tests are related to unicode. Still, this is better than simply crashing upon `import yaml`. With this patch, all tests continue to pass on Python 2 / Python 3.
|
Looks great to me! The reasoning is very well explained in the description above. Thanks for creating the PR @anishathalye and hopefully someone can merge it! |
perlpunk
approved these changes
Apr 4, 2020
|
@ingydotnet any news on release planning? |
nitzmahone
approved these changes
Jan 13, 2021
asherf
added a commit
to asherf/pants
that referenced
this pull request
Jan 21, 2021
5.4.1 (2021-01-20) * yaml/pyyaml#480 -- Fix stub compat with older pyyaml versions that may unwittingly load it 5.4 (2021-01-19) * yaml/pyyaml#407 -- Build modernization, remove distutils, fix metadata, build wheels, CI to GHA * yaml/pyyaml#472 -- Fix for CVE-2020-14343, moves arbitrary python tags to UnsafeLoader * yaml/pyyaml#441 -- Fix memory leak in implicit resolver setup * yaml/pyyaml#392 -- Fix py2 copy support for timezone objects * yaml/pyyaml#378 -- Fix compatibility with Jython https://github.com/yaml/pyyaml/blob/master/CHANGES
stuhood
pushed a commit
to pantsbuild/pants
that referenced
this pull request
Jan 21, 2021
5.4.1 (2021-01-20) * yaml/pyyaml#480 -- Fix stub compat with older pyyaml versions that may unwittingly load it 5.4 (2021-01-19) * yaml/pyyaml#407 -- Build modernization, remove distutils, fix metadata, build wheels, CI to GHA * yaml/pyyaml#472 -- Fix for CVE-2020-14343, moves arbitrary python tags to UnsafeLoader * yaml/pyyaml#441 -- Fix memory leak in implicit resolver setup * yaml/pyyaml#392 -- Fix py2 copy support for timezone objects * yaml/pyyaml#378 -- Fix compatibility with Jython https://github.com/yaml/pyyaml/blob/master/CHANGES
This was referenced Mar 6, 2021
This was referenced Mar 14, 2021
otc-zuul bot
pushed a commit
to opentelekomcloud-infra/customer-service-monitoring
that referenced
this pull request
Mar 29, 2021
Bump pyyaml from 5.3.1 to 5.4 in /roles/prepare_build/files Bumps pyyaml from 5.3.1 to 5.4. Changelog Sourced from pyyaml's changelog. 5.4 (2021-01-19) yaml/pyyaml#407 -- Build modernization, remove distutils, fix metadata, build wheels, CI to GHA yaml/pyyaml#472 -- Fix for CVE-2020-14343, moves arbitrary python tags to UnsafeLoader yaml/pyyaml#441 -- Fix memory leak in implicit resolver setup yaml/pyyaml#392 -- Fix py2 copy support for timezone objects yaml/pyyaml#378 -- Fix compatibility with Jython Commits 58d0cb7 5.4 release a60f7a1 Fix compatibility with Jython ee98abd Run CI on PR base branch changes ddf2033 constructor.timezone: _copy & deepcopy fc914d5 Avoid repeatedly appending to yaml_implicit_resolvers a001f27 Fix for CVE-2020-14343 fe15062 Add 3.9 to appveyor file for completeness sake 1e1c7fb Add a newline character to end of pyproject.toml 0b6b7d6 Start sentences and phrases for capital letters c976915 Shell code improvements Additional commits viewable in compare view Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase. Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: @dependabot rebase will rebase this PR @dependabot recreate will recreate this PR, overwriting any edits that have been made to it @dependabot merge will merge this PR after your CI passes on it @dependabot squash and merge will squash and merge this PR after your CI passes on it @dependabot cancel merge will cancel a previously requested merge and block automerging @dependabot reopen will reopen this PR if it is closed @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) @dependabot use these labels will set the current labels as the default for future PRs for this repo and language @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the Security Alerts page. Reviewed-by: None <None> Reviewed-by: Anton Kachurin <katchuring@gmail.com> Reviewed-by: Anton Sidelnikov <None>
mtremer
pushed a commit
to ipfire/ipfire-2.x
that referenced
this pull request
Feb 14, 2022
- Update from 3.13 to 6.0 - Update of rootfile - Changelog 6.0 (2021-10-13) * yaml/pyyaml#327 -- Change README format to Markdown * yaml/pyyaml#483 -- Add a test for YAML 1.1 types * yaml/pyyaml#497 -- fix float resolver to ignore `.` and `._` * yaml/pyyaml#550 -- drop Python 2.7 * yaml/pyyaml#553 -- Fix spelling of “hexadecimal” * yaml/pyyaml#556 -- fix representation of Enum subclasses * yaml/pyyaml#557 -- fix libyaml extension compiler warnings * yaml/pyyaml#560 -- fix ResourceWarning on leaked file descriptors * yaml/pyyaml#561 -- always require `Loader` arg to `yaml.load()` * yaml/pyyaml#564 -- remove remaining direct distutils usage 5.4.1 (2021-01-20) * yaml/pyyaml#480 -- Fix stub compat with older pyyaml versions that may unwittingly load it 5.4 (2021-01-19) * yaml/pyyaml#407 -- Build modernization, remove distutils, fix metadata, build wheels, CI to GHA * yaml/pyyaml#472 -- Fix for CVE-2020-14343, moves arbitrary python tags to UnsafeLoader * yaml/pyyaml#441 -- Fix memory leak in implicit resolver setup * yaml/pyyaml#392 -- Fix py2 copy support for timezone objects * yaml/pyyaml#378 -- Fix compatibility with Jython 5.3.1 (2020-03-18) * yaml/pyyaml#386 -- Prevents arbitrary code execution during python/object/new constructor 5.3 (2020-01-06) * yaml/pyyaml#290 -- Use `is` instead of equality for comparing with `None` * yaml/pyyaml#270 -- Fix typos and stylistic nit * yaml/pyyaml#309 -- Fix up small typo * yaml/pyyaml#161 -- Fix handling of __slots__ * yaml/pyyaml#358 -- Allow calling add_multi_constructor with None * yaml/pyyaml#285 -- Add use of safe_load() function in README * yaml/pyyaml#351 -- Fix reader for Unicode code points over 0xFFFF * yaml/pyyaml#360 -- Enable certain unicode tests when maxunicode not > 0xffff * yaml/pyyaml#359 -- Use full_load in yaml-highlight example * yaml/pyyaml#244 -- Document that PyYAML is implemented with Cython * yaml/pyyaml#329 -- Fix for Python 3.10 * yaml/pyyaml#310 -- Increase size of index, line, and column fields * yaml/pyyaml#260 -- Remove some unused imports * yaml/pyyaml#163 -- Create timezone-aware datetimes when parsed as such * yaml/pyyaml#363 -- Add tests for timezone 5.2 (2019-12-02) * Repair incompatibilities introduced with 5.1. The default Loader was changed, but several methods like add_constructor still used the old default yaml/pyyaml#279 -- A more flexible fix for custom tag constructors yaml/pyyaml#287 -- Change default loader for yaml.add_constructor yaml/pyyaml#305 -- Change default loader for add_implicit_resolver, add_path_resolver * Make FullLoader safer by removing python/object/apply from the default FullLoader yaml/pyyaml#347 -- Move constructor for object/apply to UnsafeConstructor * Fix bug introduced in 5.1 where quoting went wrong on systems with sys.maxunicode <= 0xffff yaml/pyyaml#276 -- Fix logic for quoting special characters * Other PRs: yaml/pyyaml#280 -- Update CHANGES for 5.1 5.1.2 (2019-07-30) * Re-release of 5.1 with regenerated Cython sources to build properly for Python 3.8b2+ 5.1.1 (2019-06-05) * Re-release of 5.1 with regenerated Cython sources to build properly for Python 3.8b1 5.1 (2019-03-13) * yaml/pyyaml#35 -- Some modernization of the test running * yaml/pyyaml#42 -- Install tox in a virtualenv * yaml/pyyaml#45 -- Allow colon in a plain scalar in a flow context * yaml/pyyaml#48 -- Fix typos * yaml/pyyaml#55 -- Improve RepresenterError creation * yaml/pyyaml#59 -- Resolves #57, update readme issues link * yaml/pyyaml#60 -- Document and test Python 3.6 support * yaml/pyyaml#61 -- Use Travis CI built in pip cache support * yaml/pyyaml#62 -- Remove tox workaround for Travis CI * yaml/pyyaml#63 -- Adding support to Unicode characters over codepoint 0xffff * yaml/pyyaml#75 -- add 3.12 changelog * yaml/pyyaml#76 -- Fallback to Pure Python if Compilation fails * yaml/pyyaml#84 -- Drop unsupported Python 3.3 * yaml/pyyaml#102 -- Include license file in the generated wheel package * yaml/pyyaml#105 -- Removed Python 2.6 & 3.3 support * yaml/pyyaml#111 -- Remove commented out Psyco code * yaml/pyyaml#129 -- Remove call to `ord` in lib3 emitter code * yaml/pyyaml#149 -- Test on Python 3.7-dev * yaml/pyyaml#158 -- Support escaped slash in double quotes "\/" * yaml/pyyaml#175 -- Updated link to pypi in release announcement * yaml/pyyaml#181 -- Import Hashable from collections.abc * yaml/pyyaml#194 -- Reverting yaml/pyyaml#74 * yaml/pyyaml#195 -- Build libyaml on travis * yaml/pyyaml#196 -- Force cython when building sdist * yaml/pyyaml#254 -- Allow to turn off sorting keys in Dumper (2) * yaml/pyyaml#256 -- Make default_flow_style=False * yaml/pyyaml#257 -- Deprecate yaml.load and add FullLoader and UnsafeLoader classes * yaml/pyyaml#261 -- Skip certain unicode tests when maxunicode not > 0xffff * yaml/pyyaml#263 -- Windows Appveyor build Signed-off-by: Adolf Belka <adolf.belka@ipfire.org> --git a/config/rootfiles/packages/python3-yaml b/config/rootfiles/packages/python3-yaml x 0870a2346..bd4009a08 100644 * yaml/pyyaml#195 -- Build libyaml on travis * yaml/pyyaml#196 -- Force cython when building sdist * yaml/pyyaml#254 -- Allow to turn off sorting keys in Dumper (2) * yaml/pyyaml#256 -- Make default_flow_style=False * yaml/pyyaml#257 -- Deprecate yaml.load and add FullLoader and Uns oader classes * yaml/pyyaml#261 -- Skip certain unicode tests when maxunicode not xffff * yaml/pyyaml#263 -- Windows Appveyor build Signed-off-by: Adolf Belka <adolf.belka@ipfire.org> Reviewed-by: Peter Müller <peter.mueller@ipfire.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch was taken from #369 (comment), authored by Pekka Klärck peke@iki.fi.
In short, Jython doesn't support lone surrogates, so importing yaml (and in particular, loading
reader.py) caused a UnicodeDecodeError. This patch works around this through a clever use ofevalto defer evaluation of the string containing the lone surrogates, only doing it on non-Jython platforms.This is only done in
lib/yaml/reader.pyand notlib3/yaml/reader.pybecause Jython does not support Python 3.With this patch, Jython's behavior with respect to Unicode code points over 0xFFFF becomes as it was before 0716ae2. It still does not pass all the unit tests on Jython (passes 1275, fails 3, errors on 1); all the failing tests are related to unicode. Still, this is better than simply crashing upon
import yaml.With this patch, all tests continue to pass on Python 2 / Python 3.
CC @pekkaklarck
Fixes #369