build,tools: refactor build config into config.json #25798

refack · 2019-01-29T19:06:03Z

First step in a process to normalize and formalize process.config and allow more generic consumption of the build config by other tools as config.json.

Ref: https://mobile.twitter.com/hashseed/status/1090284584116371462

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

nodejs-github-bot · 2019-01-29T19:06:05Z

@refack build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/2439/pipeline

https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/2440/

refack · 2019-01-29T19:06:29Z

/CC @nodejs/build-files @nodejs/python @hashseed

joyeecheung

This would need to be added to make clean?

joyeecheung · 2019-01-29T19:34:54Z

tools/js2c.py

@@ -272,8 +276,13 @@ def AddModule(module, source):
    if name.endswith('.gypi'):
      # Currently only config.gypi is allowed
      assert name == 'config.gypi'
-      lines = re.sub(r'\'true\'', 'true', lines)
-      lines = re.sub(r'\'false\'', 'false', lines)
+      config_filename = str(name).replace('gypi', 'json')


Why not just pass config.json to js2c in node.gyp?

For the time being I'm assuming 1:1 correspondence, and I want to keep the embedded config_raw data exactly the same using:
https://github.com/nodejs/node/blob/c663277042ed56a221bfbb25b5acf18a1ce2fdbd/tools/js2c.py#L285
So the tight coupling is there anyway...

joyeecheung · 2019-01-29T19:39:35Z

On a second thought...@bnoordhuis suggested replacing config.gypi with something closer to JavaScript (in stead of closer to Python) in #24816 , can't we just replace config.gypi with config.json, or does that break anybody?

refack · 2019-01-29T19:46:28Z

can't we just replace config.gypi with config.json, or does that break anybody?

Since process.config is documented API, we need to deprecate it in it's current format.
I'm setting things up for an alternative property (maybe call it process.buildConfiguration) that should be exactly that.
https://github.com/nodejs/node/blob/c663277042ed56a221bfbb25b5acf18a1ce2fdbd/tools/js2c.py#L196-L198
that will leave config.gypi as something we can deprecate and later eliminate.

joyeecheung · 2019-01-29T19:49:33Z

@refack If it's just for preserving the format of process.config, we can just restore it back in bootstrap/node.js after parsing it though?

process.buildConfiguration

I am going to be -1 if we are going to add yet another property to process..

joyeecheung · 2019-01-29T19:54:47Z

BTW, I think this breaks tools/install.py which evaluates config.gypi as a Python dict and uses the variables directly?

Umm...wait, we even distribute config.gypi in the releases under include..

refack · 2019-01-29T19:57:25Z

BTW, I think this breaks tools/install.py which evaluates config.gypi as a Python dict
and uses the variables directly?

Yep...
(that why I wanted to do this in small steps, since config.gypi is used in surprising places. For building native addons for example...)

joyeecheung · 2019-01-29T20:06:45Z

@refack I believe if we do this in small steps, this commit simply breaks the releases if landed as-is without a follow-up landed in time?

refack · 2019-01-29T20:20:36Z

@refack I believe if we do this in small steps, this commit simply breaks the releases if landed as-is without a follow-up landed in time?

Yes, sorry. I Do consider those as bugs and I'll fix them until everything works as at was.

To the clear the goal of this PR is to have the "interesting" parts of config.gypi available as a JSON conforming config.json, while keeping the rest of the tree stable.
After that I'd try to:

Add schema validation to config.json and document the properties.
Replace config.gypi with the JSON file in our GYP scaffold.
Deprecate process.config and try to design a better defined alternative.

joyeecheung · 2019-01-29T20:31:55Z

@refack

To the clear the goal of this PR is to have the "interesting" parts of config.gypi available as a JSON conforming config.json, while keeping the rest of the tree stable.

Can't we just rename config.gypi to config.json everywhere and use json.dumps/json.loads in the python scripts? (That is, skip step 1). I don't think it's entirely necessary to remove target_defaults and hoist variables to the top level in the resulting object? That way to maintain the compatibility of install.py I believe we just need to dump it as a json but rename it as config.gypi in the destination.

refack · 2019-01-29T20:40:05Z

P.S. I fixed install.py

I don't think it's entirely necessary to remove target_defaults

IMHO we need to start with getting rid of the cruft... For example what's left now in config.gypi is a purly static non-informative wrapper:

# Do not edit. Generated by the configure script.
{ 'target_defaults': { 'cflags': [],
                       'default_configuration': 'Release',
                       'defines': [],
                       'include_dirs': [],
                       'libraries': []},
  'variables': { 'includes': ['config.json']}}

but the scope of where that cruft is required to keep thing from breaking need to be mapped out...

RE you twitter comment, I agree that process.features is exactly the place we should refactor the information to.

joyeecheung · 2019-01-29T20:53:29Z

tools/install.py

@@ -173,6 +166,7 @@ def ignore_inspector_headers(files, dest):
  action([
    'common.gypi',
    'config.gypi',
+    'config.json',


Does any downstream rely on the variables being in config.gypi instead of being included? (Not sure who to ping but I guess this could make the PR semver-major?)

Can't we just recreate config.gypi in this script? Or does that defeat the purpose of this PR if there is an inconsistency between what's generated in the build and what gets distributed in the tarballs?

I have considered that. Since they are both created by ./configure we could do that to solve this if we find it's a problem.
I wanted to do it like this to see if we can map the scope by looking for things that break.

(P.S. if we decide to make this semver-major, then I guess it would make sense to add the deprecation of process.config here)

joyeecheung · 2019-01-29T20:54:07Z

tools/js2c.py

+      config = ReadFile(config_filename)
+      config = re.sub(r'\'true\'', 'true', config)
+      config = re.sub(r'\'false\'', 'false', config)
+      config_definition = GetDefinition('build_config', lines)


Following the convention this should be build_config_raw

jasnell · 2019-01-29T23:37:21Z

Since process.config is documented API, we need to deprecate it in it's current format.
I'm setting things up for an alternative property (maybe call it process.buildConfiguration) that should be exactly that.

Also keep in mind that there are userland modules that do weird things with process.config, including completely replacing it with their own objects... which is precisely why we don't rely on it much internally.

tools/js2c.py

refack · 2019-01-30T21:41:11Z

CI: https://ci.nodejs.org/job/node-test-pull-request/20462/

refack · 2019-01-30T21:54:18Z

Ohh I just remembered why we have process.config be config.gypi verbatim - we use it in node-gyp:
https://github.com/nodejs/node-gyp/blob/53b607426f9e8780e403a82dc5944456f5823336/lib/configure.js#L108

We could still keep it internal

refack · 2019-01-31T02:29:16Z

~~CI: https://ci.nodejs.org/job/node-test-pull-request/20472/~~
https://ci.nodejs.org/job/node-test-pull-request/20473/

BridgeAR · 2019-03-05T17:13:04Z

Ping @refack this failed across the board and it requires rebasing.

BridgeAR · 2019-04-03T03:02:12Z

Ping @refack

cclauss · 2019-11-01T08:19:11Z

Should we close this one?

jasnell · 2020-06-25T01:20:45Z

This has not been worked on in a while and is out of date. Closing.

nodejs-github-bot added build Issues and PRs related to build files or the CI. meta Issues and PRs related to the general management of the project. labels Jan 29, 2019

refack added python PRs and issues that require attention from people who are familiar with Python. tools Issues and PRs related to the tools directory. and removed meta Issues and PRs related to the general management of the project. labels Jan 29, 2019

joyeecheung reviewed Jan 29, 2019

View reviewed changes

hashseed reviewed Jan 30, 2019

View reviewed changes

tools/js2c.py Outdated Show resolved Hide resolved

build,tools: simply build config as config.json

23b0104

refack force-pushed the normalize-process-config branch from 669c6d9 to 23b0104 Compare January 31, 2019 02:25

squash! weaken assert

b027c5f

refack self-assigned this Jan 31, 2019

fixup! forgot a file

ae3e267

BridgeAR added the stalled Issues and PRs that are stalled. label Apr 3, 2019

refack mentioned this pull request Jun 10, 2019

fix: enable ltcg on windows electron/node#104

Merged

Trott force-pushed the master branch from 1ecc406 to 49cf67e Compare September 17, 2019 16:51

devnexen force-pushed the master branch from e8a4568 to 5289f80 Compare December 26, 2019 19:46

BridgeAR force-pushed the master branch 2 times, most recently from 8ae28ff to 2935f72 Compare May 31, 2020 12:19

jasnell closed this Jun 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build,tools: refactor build config into config.json #25798

build,tools: refactor build config into config.json #25798

refack commented Jan 29, 2019 •

edited

Loading

nodejs-github-bot commented Jan 29, 2019 •

edited by refack

Loading

refack commented Jan 29, 2019

joyeecheung left a comment

joyeecheung Jan 29, 2019

refack Jan 29, 2019

joyeecheung commented Jan 29, 2019 •

edited

Loading

refack commented Jan 29, 2019

joyeecheung commented Jan 29, 2019

joyeecheung commented Jan 29, 2019 •

edited

Loading

refack commented Jan 29, 2019

joyeecheung commented Jan 29, 2019

refack commented Jan 29, 2019

joyeecheung commented Jan 29, 2019

refack commented Jan 29, 2019

joyeecheung Jan 29, 2019

joyeecheung Jan 29, 2019

refack Jan 29, 2019

joyeecheung Jan 29, 2019

jasnell commented Jan 29, 2019

refack commented Jan 30, 2019

refack commented Jan 30, 2019

refack commented Jan 31, 2019 •

edited

Loading

BridgeAR commented Mar 5, 2019

BridgeAR commented Apr 3, 2019

cclauss commented Nov 1, 2019

jasnell commented Jun 25, 2020

build,tools: refactor build config into config.json #25798

build,tools: refactor build config into config.json #25798

Conversation

refack commented Jan 29, 2019 • edited Loading

Checklist

nodejs-github-bot commented Jan 29, 2019 • edited by refack Loading

refack commented Jan 29, 2019

joyeecheung left a comment

Choose a reason for hiding this comment

joyeecheung Jan 29, 2019

Choose a reason for hiding this comment

refack Jan 29, 2019

Choose a reason for hiding this comment

joyeecheung commented Jan 29, 2019 • edited Loading

refack commented Jan 29, 2019

joyeecheung commented Jan 29, 2019

joyeecheung commented Jan 29, 2019 • edited Loading

refack commented Jan 29, 2019

joyeecheung commented Jan 29, 2019

refack commented Jan 29, 2019

joyeecheung commented Jan 29, 2019

refack commented Jan 29, 2019

joyeecheung Jan 29, 2019

Choose a reason for hiding this comment

joyeecheung Jan 29, 2019

Choose a reason for hiding this comment

refack Jan 29, 2019

Choose a reason for hiding this comment

joyeecheung Jan 29, 2019

Choose a reason for hiding this comment

jasnell commented Jan 29, 2019

refack commented Jan 30, 2019

refack commented Jan 30, 2019

refack commented Jan 31, 2019 • edited Loading

BridgeAR commented Mar 5, 2019

BridgeAR commented Apr 3, 2019

cclauss commented Nov 1, 2019

jasnell commented Jun 25, 2020

refack commented Jan 29, 2019 •

edited

Loading

nodejs-github-bot commented Jan 29, 2019 •

edited by refack

Loading

joyeecheung commented Jan 29, 2019 •

edited

Loading

joyeecheung commented Jan 29, 2019 •

edited

Loading

refack commented Jan 31, 2019 •

edited

Loading