-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doc: add guide on maintaining the build files #16975
Conversation
or `.\configure --help` (Windows). Consider updating this file | ||
when you need to update the configuration process. | ||
|
||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure if pasting the help text here is useful..although it does give you an idea about what it does if you don't know what it is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the above paragraph is enough. Including the text here will only drift from the actual help text over time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed with @richardlau , don't think we should duplicate the ./configure --help
output.
To see the help text, run `.\vcbuild help`. Consider updating this file | ||
when you need to update the build and testing process on Windows. | ||
|
||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should document the similarity/differences between vcbuild.bat
and Makefile
, though not necessarily in this PR (I am not familiar enough with vcbuild.bat
to be comfortable to write a guide on it anyway)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that'd be quite a job for someone! one of it's biggest problems is that it lacks most of the smarts that make brings to the table and we only do a partial job of emulating the bits that are needed
Makefile
Outdated
@@ -390,13 +402,17 @@ CI_JS_SUITES ?= default | |||
CI_DOC := doctool | |||
|
|||
# Build and test addons without building anything else | |||
# The CI runs this on ARM(Raspberry Pis) because the addons and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Uh, wait, I don't think this is true..? From what I can tell node-test-binary-arm
only runs test-ci-js
, But then I am not sure who uses this target (test-ci-native
) and how are the addons tested on the pis @nodejs/build
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
node-test-binary-arm:
case $RUN_SUBSET in
addons)
PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE make test-ci-native
;;
*)
export CC=should-not-compile CXX=should-not-compile # Prevent building
PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE TEST_CI_ARGS="--run=${RUN_SUBSET},7" make test-ci-js
;;
esac
i.e. it's used for the addons tests on arm-fanned.
### Options | ||
|
||
- `-j <n>`: number of threads used to build the binary or run the parallel | ||
tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Parallel tests are run in parallel by default (by virtue of -J
specified to the test runner in Makefile), even if the make
command is -j1
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's an awkward catch to this that's not documented anywhere that probably should be since I'm finding Jenkins jobs that don't do this properly: a lot of the -ci targets use this PARALLEL_ARGS
variable to insert -j X
for tools/test.py
(-J
isn't used in any of the -ci targets). Unfortunately this variable is only set if $JOBS
exists, and this isn't automatically set just because you supply -j X
to make
. In fact, it's kind of difficult to determine the -j
argument inside a Makefile in a cross-platform way (but we should probably try, there's an interesting stackoverflow suggestion here). So you have to give JOBS=X
when running the -ci targets.
-J
is done automatically for tests on non-ci targets so it takes up "as many cores as you have" for tests as @TimothyGu said.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@TimothyGu @rvagg Thanks for the correction, I'll add a note in the Makefile about this, and change this part to just "number of threads used to build the binary"
|
||
## Makefile | ||
|
||
This file is not generated and mantained by humans. Note that this is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo maintained
.
Maybe reword this, e.g. and is maintained
to avoid being misread as is not (generated and maintained)
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, I totally misread this 😁
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure about putting CI job names in the Makefile. It would put the onus on @nodejs/build to keep them in sync.
Makefile
Outdated
@@ -811,6 +849,7 @@ $(PKG): release-only | |||
--package-path $(MACOSOUTDIR)/pkgs ./$(PKG) | |||
SIGN="$(PRODUCTSIGN_CERT)" PKG="$(PKG)" bash tools/osx-productsign.sh | |||
|
|||
# Builds the MAC OS installer for releases |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
macOS
Makefile
Outdated
@@ -368,6 +379,7 @@ clear-stalled: | |||
echo $${PS_OUT} | xargs kill; \ | |||
fi | |||
|
|||
# GC test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment doesn't really convey any additional information than can be inferred from the target name.
Makefile
Outdated
@@ -100,6 +101,7 @@ install: all | |||
uninstall: | |||
$(PYTHON) tools/install.py $@ '$(DESTDIR)' '$(PREFIX)' | |||
|
|||
# Clean the artifacts generated by Makefile in most targets |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think clean
is a standard enough makefile target name that this comment shouldn't be necessary. If it is kept, suggest Remove the artifacts...
reword.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is mostly to distinguish clean
from docclean
, distclean
, test-addons-clean
and friends.
|
||
#### Release Targets | ||
|
||
- `binary`: builds binary tarballs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a note that this target will fail unless NODE_VERSION_IS_RELEASE
is #define'd as 1
in src/node_version.h
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@richardlau good catch, though I think this should be documented in the makefile instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would a maintainer need to know that? Seems more relevant for users.
Makefile
Outdated
@@ -1085,15 +1131,29 @@ lint-addon-docs: test/addons/.docbuildstamp | |||
cpplint: lint-cpp | |||
@echo "Please use lint-cpp instead of cpplint" | |||
|
|||
ifneq ("","$(wildcard tools/eslint/)") | |||
# tools/eslint is not avialable in the source tarball |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
available
Makefile
Outdated
@@ -234,6 +244,7 @@ test-only: all | |||
$(CI_NATIVE_SUITES) \ | |||
known_issues | |||
|
|||
# Used by make coverage-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe use backticks for `make coverage-test`. I read this wrong on first parse and thought it was just awkward wording
Makefile
Outdated
@@ -408,6 +424,9 @@ test-ci-js: | clear-stalled | |||
echo $${PS_OUT} | xargs kill; exit 1; \ | |||
fi | |||
|
|||
# Run by most CI, exceptions: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think rpi is the only exception, I've just been grepping through jenkins config files and it's the only one I can find.
Makefile
Outdated
@@ -584,10 +615,17 @@ docopen: $(apidocs_html) | |||
docclean: | |||
$(RM) -r out/doc | |||
|
|||
# Prepare the build for running the tests | |||
# Note that this is not run by the CI jobs on ARM or Linux with FIPS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now that I've seen this and noticed CONFIG_FLAGS
I've gone and edited node-test-commit-linux-fips to not be an exception here, it now does
PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE CONFIG_FLAGS=--openssl-fips=$WORKSPACE/fipscanisterdir $MAKE run-ci -j $JOBS
so it's using run-ci
like the rest. I'll do the same for the new jobs I'm building out in node-test-commit-linux-linked that do a similar ./configure
override thing.
So the only thing I'd suggest about this option is that you note that it's called by run-ci
, we don't actually call it directly anywhere.
_(FYI I also think that node-test-commit-linux-fips wasn't running tests in parallel due to a mis-application of JOBS
and -j
and I think that should be fixed now, that would explain why it's been taking more time than it should in some instances).
Makefile
Outdated
# Run by most testing CIs, exceptions are: | ||
# - ARM (Raspberry Pis): the binaries are cross-compiled elsewhere | ||
# before being used to run tests. See test-ci-native. | ||
# - Linux with FIPS: the build is done differently, the tests are still |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fips isn't an exception here, although it does run test-ci
directly a second time with FIPS enabled
coverage is another exception that doesn't run this
Makefile
Outdated
@@ -916,6 +957,7 @@ ifeq ($(XZ), 0) | |||
endif | |||
$(RM) $(BINARYNAME).tar | |||
|
|||
# Builds the binary tarball for releases | |||
binary: $(BINARYTAR) | |||
|
|||
binary-upload: binary |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you could note that these -upload tasks are strictly for release builds on release machines only, unless you think that's obvious
|
||
## configure | ||
|
||
To see the help text, run `./configure --help`(unix-like) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
space needed before (unix-like)
## configure | ||
|
||
To see the help text, run `./configure --help`(unix-like) | ||
or `.\configure --help` (Windows). Consider updating this file |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should probably be python configure --help
- `bench-all` | ||
- `bench-ci` | ||
|
||
### Environment Variables |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also worth listing:
BUILDTYPE
JOBS
CONFIG_FLAGS
BUILD_DOWNLOAD_FLAGS
BUILD_INTL_FLAGS
are you restricting your comment lines shorter than 80 chars? thanks for doing this btw! I hope we can keep it all in sync as it evolves |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for so many nits)
a subset of the targets avilable in the [Makefile](#makefile), and a few | ||
targets of its own. This script also calls [configure](#configure) before | ||
building the Node.js binary. For a detailed guide on this script, | ||
see [vcbuild.bat](#vcbuild.bat). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#vcbuild.bat
-> #vcbuildbat
?
On unix-like systems: | ||
|
||
- configure: A (mostly) Python 2 script that accepts a set of options, | ||
configure and prepare the build using gyp. For a detailed guide on |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
configures and prepares?
Options: | ||
-h, --help show this help message and exit | ||
--prefix=PREFIX select the install prefix [default: /usr/local] | ||
--coverage Build node with code coverage enabled |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A nit: most of the explanations start with a lower case word, some with an upper case one: Build, Generate, Generate, Enable etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think these come direct from the configure
help, so any changes should be made there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@richardlau Yes.
@vsemozhetbyt Do you want to open a PR to fix the configure script? Or I can do that as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Though these are rather simple fixes, I would rather abstain as I know neither Python nor the format of this file. Sorry(
default compilation environment | ||
--enable-vtune-profiling | ||
Enable profiling support for Intel VTune profiler to | ||
profileJavaScript code executed in nodejs. This |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
profileJavaScript -> profile JavaScript ?
nodejs -> Node.js ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
vcbuild.bat release msi : builds release build and MSI installer package | ||
vcbuild.bat test : builds debug build and runs tests | ||
vcbuild.bat build-release : builds the release distribution as used by nodejs.org | ||
vcbuild.bat enable-vtune : builds nodejs with Intel VTune profiling support to profile JavaScript |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nodejs -> Node.js ?
On Windows: | ||
|
||
- vcbuild.bat: A Windows Batch Script that locates build tools, provides | ||
a subset of the targets avilable in the [Makefile](#makefile), and a few |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
avilable -> available
- Makefile: A makefile that can be run with BSD make or GNU make. It | ||
provides a set of targets that build and test the Node.js | ||
binary, produce releases and documentation, and interact with the | ||
CI to run benchamarks or tests. For a detailed guide on this file, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
benchamarks -> benchmarks
alternative lib name to link to [default: uv] | ||
--shared-libuv-libpath=SHARED_LIBUV_LIBPATH | ||
a directory to search for the shared libuv DLL | ||
--shared-openssl link to a shared OpenSSl DLL instead of static linking |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OpenSSl -> OpenSSL
### Targets | ||
|
||
For the targets listed below that do not have detailed descriptions | ||
in this docuement, please see their comments in the makefile instead. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docuement -> document
For the targets listed below that do not have detailed descriptions | ||
in this docuement, please see their comments in the makefile instead. | ||
|
||
Here is a list of targets that will commonly triggered by its users: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
triggered -> be triggered
Makefile
Outdated
v8: | ||
tools/make-v8.sh | ||
$(MAKE) -C deps/v8 $(V8_ARCH).$(BUILDTYPE_LOWER) $(V8_BUILD_OPTIONS) | ||
|
||
|
||
# This is meant to be run by the developers, which builds Node.js and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove the
and which
If we're going to say this here, is it worth saying something similar for make all
?
Makefile
Outdated
@@ -390,13 +402,17 @@ CI_JS_SUITES ?= default | |||
CI_DOC := doctool | |||
|
|||
# Build and test addons without building anything else | |||
# The CI runs this on ARM(Raspberry Pis) because the addons and | |||
# the binary are cross-compiled elsewhere |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
binary -> binaries
Makefile
Outdated
test-doc: doc-only | ||
$(MAKE) lint | ||
$(PYTHON) tools/test.py $(CI_DOC) | ||
|
||
test-known-issues: all | ||
$(PYTHON) tools/test.py known_issues | ||
|
||
# Used by the node-test-npm CI job |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can be run manually too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm I think it's obvious without saying that all the targets here can be run manually (some needs certain environment variables and some probably shouldn't be run manually though)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would agree, but if you say:
# Note that this is for developers, it is not run on the CI. See run-ci.
elsewhere, then this line suggests that it's used exclusively by the CI job.
or `.\configure --help` (Windows). Consider updating this file | ||
when you need to update the configuration process. | ||
|
||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed with @richardlau , don't think we should duplicate the ./configure --help
output.
|
||
## Makefile | ||
|
||
This file is not generated and mantained by humans. Note that this is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, I totally misread this 😁
--debug-nghttp2 build nghttp2 with DEBUGBUILD (default is false) | ||
``` | ||
|
||
## Makefile |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was compiling Alacritty the other day, and I noticed that their make
actually defaults to showing help (alias for make help
) rather than doing anything. It actually looks really nice, and it'd be really useful if we could do something similar.
I think a help option to make would be less likely to bitrot than separate documentation.
https://github.com/jwilm/alacritty/blob/245a80078180acc2a0a1addc569c15b77991c1c3/Makefile#L18-L21
|
||
## configure | ||
|
||
To see the help text, run `./configure --help`(unix-like) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can just say python configure
for both Unix and Windows.
💯 on this, nothing worse than slightly out-of-date documentation that you can never quite trust. |
77be3b8
to
7d77bfe
Compare
I think I've addressed most comments. @rvagg @TimothyGu @richardlau @vsemozhetbyt @gibfahn PTAL |
7d77bfe
to
e80fb74
Compare
Ping @nodejs/build @rvagg @gibfahn @richardlau for reviews, thanks |
e80fb74
to
ce180ee
Compare
Rebased. New CI: https://ci.nodejs.org/job/node-test-pull-request/11548/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small nit otherwise LGTM.
Makefile
Outdated
lint-ci: lint | ||
else | ||
# Inside the git repository | ||
# This runs linters ond different type of files. It's supposed to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: on
?
2d3c92a
to
fef681b
Compare
Rebased. Planning on landing this on Monday if there are no more outstanding reviews. cc @nodejs/collaborators |
PR-URL: #16975 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
PR-URL: #16975 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Landed in c91bd2f...89d8657, thanks! |
PR-URL: #16975 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
PR-URL: #16975 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
PR-URL: #16975 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
PR-URL: #16975 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Should this land? It will need a manual backport to 6.x and 8.x |
@MylesBorins it's just some doc changes so I would say nope |
This documents how to maintain the build files and some commonly-used targets in the Makefile. It also documents about how to update certain targets to reflect the changes on CI (especially the Raspberry Pis and Linux with FIPS).
This is still a WIP, and only talks about the makefile, configure script and the vcbuild.bat script. There are other build files (e.g. the gyp files) that need to be documented later.
Personally I prefer to leave most of the actual documentation in the code, whiling leave some pointers and necessary explanations in the guide so it's easier to get started with maintaining them.
cc @nodejs/build
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passesAffected core subsystem(s)
doc, build