tools: from io import StringIO in ninja.py #29371

cclauss · 2019-08-29T20:27:38Z

cStringIO was removed in Python 3 so this PR advocates that we use the io module instead. As reported #25789 (comment), #29196 (comment), and #29236 (comment) this issue is being seen by Windows users who are experimenting with Python 3.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2019-08-29T20:48:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25297/

bnoordhuis · 2019-08-29T21:45:15Z

Aside: if you push fix-ups, it's easier for people to recognize them as such when you prefix them with fixup! or squash!. They're not arbitrary prefixes, git rebase -i recognizes them.

Someone might come along now and be unsure whether the two commits in this PR need to be squashed first or merged separately.

nodejs-github-bot · 2019-08-30T04:00:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25298/

gengjiawen · 2019-08-30T04:05:02Z

This happened on macOS platform too.There are other issues on macOS too. We may need to test more platform on JenkinsCI or travis.

richardlau · 2019-08-30T04:10:48Z

This happened on macOS platform too.There are other issues on macOS too. We may need to test more platform on JenkinsCI or travis.

This file is only used when using Ninja, which I don't think any of our CI (Travis or Jenkins) uses.

cclauss · 2019-08-30T04:44:34Z

@gengjiawen If you are interested to spend some time on adding macOS and Windows to .travis.yml, you might take some inspiration from: https://docs.travis-ci.com/user/languages/python/#running-python-tests-on-multiple-operating-systems

nodejs-github-bot · 2019-08-30T04:45:27Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25299/

gengjiawen · 2019-08-30T04:49:15Z

This happened on macOS platform too.There are other issues on macOS too. We may need to test more platform on JenkinsCI or travis.

This file is only used when using Ninja, which I don't think any of our CI (Travis or Jenkins) uses.

I mean we need to test Python3 compatibility in other Platform like macOS.

Also the crash happened in config phase, python3 ./configure.py

Traceback (most recent call last):
  File "./configure.py", line 1716, in <module>
    run_gyp(gyp_args)
  File "tools/gyp_node.py", line 54, in run_gyp
    rc = gyp.main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 523, in gyp_main
    options.duplicate_basename_check)
  File "tools/gyp/pylib/gyp/__init__.py", line 107, in Load
    generator.CalculateVariables(default_variables, params)
  File "tools/gyp/pylib/gyp/generator/make.py", line 81, in CalculateVariables
    import gyp.generator.xcode as xcode_generator
  File "tools/gyp/pylib/gyp/generator/xcode.py", line 10, in <module>
    import gyp.xcode_ninja
  File "tools/gyp/pylib/gyp/xcode_ninja.py", line 16, in <module>
    import gyp.generator.ninja
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 23, in <module>
    from cStringIO import StringIO
ModuleNotFoundError: No module named 'cStringIO'

cclauss · 2019-08-30T07:42:36Z

That is the line that this PR fixes!!

gengjiawen · 2019-08-30T07:48:52Z

That is the line that this PR fixes!!

yeap, but on macOS will still failed with this patch, maybe add macOS to travisCI:

Traceback (most recent call last):
  File "./configure.py", line 1716, in <module>
    run_gyp(gyp_args)
  File "tools/gyp_node.py", line 54, in run_gyp
    rc = gyp.main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 523, in gyp_main
    options.duplicate_basename_check)
  File "tools/gyp/pylib/gyp/__init__.py", line 139, in Load
    params['parallel'], params['root_targets'])
  File "tools/gyp/pylib/gyp/input.py", line 2779, in Load
    variables, includes, depth, check, True)
  File "tools/gyp/pylib/gyp/input.py", line 459, in LoadTargetBuildFile
    includes, depth, check, load_dependencies)
  File "tools/gyp/pylib/gyp/input.py", line 408, in LoadTargetBuildFile
    build_file_data, PHASE_EARLY, variables, build_file_path)
  File "tools/gyp/pylib/gyp/input.py", line 1290, in ProcessVariablesAndConditionsInDict
    build_file)
  File "tools/gyp/pylib/gyp/input.py", line 1305, in ProcessVariablesAndConditionsInList
    ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
  File "tools/gyp/pylib/gyp/input.py", line 1264, in ProcessVariablesAndConditionsInDict
    ProcessConditionsInDict(the_dict, phase, variables, build_file)
  File "tools/gyp/pylib/gyp/input.py", line 1137, in ProcessConditionsInDict
    build_file)
  File "tools/gyp/pylib/gyp/input.py", line 1061, in EvalCondition
    cond_expr, true_dict, false_dict, phase, variables, build_file)
  File "tools/gyp/pylib/gyp/input.py", line 1087, in EvalSingleCondition
    if eval(ast_code, {'__builtins__': None}, variables):
  File "<string>", line 1, in <module>
TypeError: '>=' not supported between instances of 'int' and 'str' while loading dependencies of /Users/daniel/code/node/node.gyp while trying to load /Users/daniel/code/node/node.gyp

Trott · 2019-09-03T04:01:44Z

Landed in 020c2ea

PR-URL: #29371 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>

targos · 2019-09-03T07:02:01Z

This change broke ./configure --ninja:

$ ./configure --ninja 
INFO: Using floating patch "tools/icu/patches/64/source/common/putil.cpp" from "tools/icu"
INFO: Using floating patch "tools/icu/patches/64/source/i18n/dtptngen.cpp" from "tools/icu"
Traceback (most recent call last):
  File "./configure", line 28, in <module>
    import configure
  File "/home/mzasso/git/nodejs/node/configure.py", line 1716, in <module>
    run_gyp(gyp_args)
  File "tools/gyp_node.py", line 54, in run_gyp
    rc = gyp.main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 2490, in GenerateOutput
    config_name)
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 2395, in GenerateOutputForConfig
    target = writer.WriteSpec(spec, config_name, generator_flags)
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 483, in WriteSpec
    spec)
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 921, in WriteSources
    precompiled_header, spec)
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 980, in WriteSourcesForArch
    [Define(d, self.flavor) for d in defines])
  File "tools/gyp/pylib/gyp/generator/ninja.py", line 1585, in WriteVariableList
    ninja_file.variable(var, ' '.join(values))
  File "tools/gyp/pylib/gyp/ninja_syntax.py", line 35, in variable
    self._line('%s = %s' % (key, value), indent)
  File "tools/gyp/pylib/gyp/ninja_syntax.py", line 139, in _line
    self.output.write(leading_space + text[0:space] + ' $\n')
TypeError: unicode argument expected, got 'str'

Allow both Python 2 and 3 to access StringIO. This fixes `./configure --ninja`, which was broken by #29371. See: #29371 (comment) PR-URL: #29414 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Sam Roberts <vieuxtech@gmail.com>

PR-URL: #29371 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>

Allow both Python 2 and 3 to access StringIO. This fixes `./configure --ninja`, which was broken by #29371. See: #29371 (comment) PR-URL: #29414 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Sam Roberts <vieuxtech@gmail.com>

nodejs-github-bot added build Issues and PRs related to build files or the CI. tools Issues and PRs related to the tools directory. labels Aug 29, 2019

cclauss requested a review from joaocgreis August 29, 2019 20:28

cclauss added the python PRs and issues that require attention from people who are familiar with Python. label Aug 29, 2019

cclauss requested review from sam-github and gengjiawen August 29, 2019 20:28

cclauss changed the title ~~Py3 ninja.py~~ tools: from io import StringIO in ninja.py Aug 29, 2019

This was referenced Aug 29, 2019

Accept Python 3 on Windows #29236

Closed

build: ongoing list of actions for Python 3 compatibility #25789

Closed

bnoordhuis approved these changes Aug 29, 2019

View reviewed changes

gengjiawen approved these changes Aug 30, 2019

View reviewed changes

tools: from io import StringIO in ninja.py

64fa594

cclauss force-pushed the Py3-ninja.py branch from 331b268 to 64fa594 Compare August 30, 2019 03:53

richardlau approved these changes Aug 30, 2019

View reviewed changes

lpinca approved these changes Aug 30, 2019

View reviewed changes

targos added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Aug 31, 2019

Trott closed this Sep 3, 2019

This was referenced Sep 3, 2019

gyp: make StringIO work in ninja.py #29414

Closed

Tests do not test ./configure --ninja #29415

Closed

cclauss deleted the Py3-ninja.py branch September 3, 2019 20:34

BridgeAR mentioned this pull request Sep 3, 2019

v12.10.0 proposal #29429

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: from io import StringIO in ninja.py #29371

tools: from io import StringIO in ninja.py #29371

cclauss commented Aug 29, 2019 •

edited

Loading

nodejs-github-bot commented Aug 29, 2019

bnoordhuis commented Aug 29, 2019

nodejs-github-bot commented Aug 30, 2019

gengjiawen commented Aug 30, 2019

richardlau commented Aug 30, 2019

cclauss commented Aug 30, 2019

nodejs-github-bot commented Aug 30, 2019

gengjiawen commented Aug 30, 2019

cclauss commented Aug 30, 2019

gengjiawen commented Aug 30, 2019

Trott commented Sep 3, 2019

targos commented Sep 3, 2019

tools: from io import StringIO in ninja.py #29371

tools: from io import StringIO in ninja.py #29371

Conversation

cclauss commented Aug 29, 2019 • edited Loading

Checklist

nodejs-github-bot commented Aug 29, 2019

bnoordhuis commented Aug 29, 2019

nodejs-github-bot commented Aug 30, 2019

gengjiawen commented Aug 30, 2019

richardlau commented Aug 30, 2019

cclauss commented Aug 30, 2019

nodejs-github-bot commented Aug 30, 2019

gengjiawen commented Aug 30, 2019

cclauss commented Aug 30, 2019

gengjiawen commented Aug 30, 2019

Trott commented Sep 3, 2019

targos commented Sep 3, 2019

cclauss commented Aug 29, 2019 •

edited

Loading