Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: add guide on maintaining the build files #16975

Closed
wants to merge 2 commits into from

Conversation

joyeecheung
Copy link
Member

This documents how to maintain the build files and some commonly-used targets in the Makefile. It also documents about how to update certain targets to reflect the changes on CI (especially the Raspberry Pis and Linux with FIPS).

This is still a WIP, and only talks about the makefile, configure script and the vcbuild.bat script. There are other build files (e.g. the gyp files) that need to be documented later.

Personally I prefer to leave most of the actual documentation in the code, whiling leave some pointers and necessary explanations in the guide so it's easier to get started with maintaining them.

cc @nodejs/build

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

doc, build

@nodejs-github-bot nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Nov 12, 2017
or `.\configure --help` (Windows). Consider updating this file
when you need to update the configuration process.

```
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if pasting the help text here is useful..although it does give you an idea about what it does if you don't know what it is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the above paragraph is enough. Including the text here will only drift from the actual help text over time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with @richardlau , don't think we should duplicate the ./configure --help output.

To see the help text, run `.\vcbuild help`. Consider updating this file
when you need to update the build and testing process on Windows.

```
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should document the similarity/differences between vcbuild.bat and Makefile, though not necessarily in this PR (I am not familiar enough with vcbuild.bat to be comfortable to write a guide on it anyway)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that'd be quite a job for someone! one of it's biggest problems is that it lacks most of the smarts that make brings to the table and we only do a partial job of emulating the bits that are needed

@joyeecheung joyeecheung added the doc Issues and PRs related to the documentations. label Nov 12, 2017
Makefile Outdated
@@ -390,13 +402,17 @@ CI_JS_SUITES ?= default
CI_DOC := doctool

# Build and test addons without building anything else
# The CI runs this on ARM(Raspberry Pis) because the addons and
Copy link
Member Author

@joyeecheung joyeecheung Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uh, wait, I don't think this is true..? From what I can tell node-test-binary-arm only runs test-ci-js, But then I am not sure who uses this target (test-ci-native) and how are the addons tested on the pis @nodejs/build

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node-test-binary-arm:

case $RUN_SUBSET in
  addons)
    PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE make test-ci-native
    ;;
  *)
    export CC=should-not-compile CXX=should-not-compile # Prevent building
    PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE TEST_CI_ARGS="--run=${RUN_SUBSET},7" make test-ci-js
    ;;
esac

i.e. it's used for the addons tests on arm-fanned.

### Options

- `-j <n>`: number of threads used to build the binary or run the parallel
tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parallel tests are run in parallel by default (by virtue of -J specified to the test runner in Makefile), even if the make command is -j1.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an awkward catch to this that's not documented anywhere that probably should be since I'm finding Jenkins jobs that don't do this properly: a lot of the -ci targets use this PARALLEL_ARGS variable to insert -j X for tools/test.py (-J isn't used in any of the -ci targets). Unfortunately this variable is only set if $JOBS exists, and this isn't automatically set just because you supply -j X to make. In fact, it's kind of difficult to determine the -j argument inside a Makefile in a cross-platform way (but we should probably try, there's an interesting stackoverflow suggestion here). So you have to give JOBS=X when running the -ci targets.

-J is done automatically for tests on non-ci targets so it takes up "as many cores as you have" for tests as @TimothyGu said.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimothyGu @rvagg Thanks for the correction, I'll add a note in the Makefile about this, and change this part to just "number of threads used to build the binary"


## Makefile

This file is not generated and mantained by humans. Note that this is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo maintained.

Maybe reword this, e.g. and is maintained to avoid being misread as is not (generated and maintained).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, I totally misread this 😁

Copy link
Member

@richardlau richardlau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about putting CI job names in the Makefile. It would put the onus on @nodejs/build to keep them in sync.

Makefile Outdated
@@ -811,6 +849,7 @@ $(PKG): release-only
--package-path $(MACOSOUTDIR)/pkgs ./$(PKG)
SIGN="$(PRODUCTSIGN_CERT)" PKG="$(PKG)" bash tools/osx-productsign.sh

# Builds the MAC OS installer for releases
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

macOS

Makefile Outdated
@@ -368,6 +379,7 @@ clear-stalled:
echo $${PS_OUT} | xargs kill; \
fi

# GC test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment doesn't really convey any additional information than can be inferred from the target name.

Makefile Outdated
@@ -100,6 +101,7 @@ install: all
uninstall:
$(PYTHON) tools/install.py $@ '$(DESTDIR)' '$(PREFIX)'

# Clean the artifacts generated by Makefile in most targets
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think clean is a standard enough makefile target name that this comment shouldn't be necessary. If it is kept, suggest Remove the artifacts... reword.

Copy link
Member Author

@joyeecheung joyeecheung Nov 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mostly to distinguish clean from docclean, distclean, test-addons-clean and friends.


#### Release Targets

- `binary`: builds binary tarballs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a note that this target will fail unless NODE_VERSION_IS_RELEASE is #define'd as 1 in src/node_version.h.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@richardlau good catch, though I think this should be documented in the makefile instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would a maintainer need to know that? Seems more relevant for users.

Makefile Outdated
@@ -1085,15 +1131,29 @@ lint-addon-docs: test/addons/.docbuildstamp
cpplint: lint-cpp
@echo "Please use lint-cpp instead of cpplint"

ifneq ("","$(wildcard tools/eslint/)")
# tools/eslint is not avialable in the source tarball
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

available

Makefile Outdated
@@ -234,6 +244,7 @@ test-only: all
$(CI_NATIVE_SUITES) \
known_issues

# Used by make coverage-test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe use backticks for `make coverage-test`. I read this wrong on first parse and thought it was just awkward wording

Makefile Outdated
@@ -408,6 +424,9 @@ test-ci-js: | clear-stalled
echo $${PS_OUT} | xargs kill; exit 1; \
fi

# Run by most CI, exceptions:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think rpi is the only exception, I've just been grepping through jenkins config files and it's the only one I can find.

Makefile Outdated
@@ -584,10 +615,17 @@ docopen: $(apidocs_html)
docclean:
$(RM) -r out/doc

# Prepare the build for running the tests
# Note that this is not run by the CI jobs on ARM or Linux with FIPS
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now that I've seen this and noticed CONFIG_FLAGS I've gone and edited node-test-commit-linux-fips to not be an exception here, it now does

PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE CONFIG_FLAGS=--openssl-fips=$WORKSPACE/fipscanisterdir $MAKE run-ci -j $JOBS

so it's using run-ci like the rest. I'll do the same for the new jobs I'm building out in node-test-commit-linux-linked that do a similar ./configure override thing.

So the only thing I'd suggest about this option is that you note that it's called by run-ci, we don't actually call it directly anywhere.

_(FYI I also think that node-test-commit-linux-fips wasn't running tests in parallel due to a mis-application of JOBS and -j and I think that should be fixed now, that would explain why it's been taking more time than it should in some instances).

Makefile Outdated
# Run by most testing CIs, exceptions are:
# - ARM (Raspberry Pis): the binaries are cross-compiled elsewhere
# before being used to run tests. See test-ci-native.
# - Linux with FIPS: the build is done differently, the tests are still
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fips isn't an exception here, although it does run test-ci directly a second time with FIPS enabled

coverage is another exception that doesn't run this

Makefile Outdated
@@ -916,6 +957,7 @@ ifeq ($(XZ), 0)
endif
$(RM) $(BINARYNAME).tar

# Builds the binary tarball for releases
binary: $(BINARYTAR)

binary-upload: binary
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could note that these -upload tasks are strictly for release builds on release machines only, unless you think that's obvious


## configure

To see the help text, run `./configure --help`(unix-like)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

space needed before (unix-like)

## configure

To see the help text, run `./configure --help`(unix-like)
or `.\configure --help` (Windows). Consider updating this file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should probably be python configure --help

- `bench-all`
- `bench-ci`

### Environment Variables
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also worth listing:

BUILDTYPE
JOBS
CONFIG_FLAGS
BUILD_DOWNLOAD_FLAGS
BUILD_INTL_FLAGS

@rvagg
Copy link
Member

rvagg commented Nov 13, 2017

are you restricting your comment lines shorter than 80 chars?

thanks for doing this btw! I hope we can keep it all in sync as it evolves

Copy link
Contributor

@vsemozhetbyt vsemozhetbyt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for so many nits)

a subset of the targets avilable in the [Makefile](#makefile), and a few
targets of its own. This script also calls [configure](#configure) before
building the Node.js binary. For a detailed guide on this script,
see [vcbuild.bat](#vcbuild.bat).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#vcbuild.bat -> #vcbuildbat ?

On unix-like systems:

- configure: A (mostly) Python 2 script that accepts a set of options,
configure and prepare the build using gyp. For a detailed guide on
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configures and prepares?

Options:
-h, --help show this help message and exit
--prefix=PREFIX select the install prefix [default: /usr/local]
--coverage Build node with code coverage enabled
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A nit: most of the explanations start with a lower case word, some with an upper case one: Build, Generate, Generate, Enable etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these come direct from the configure help, so any changes should be made there.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@richardlau Yes.

@vsemozhetbyt Do you want to open a PR to fix the configure script? Or I can do that as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though these are rather simple fixes, I would rather abstain as I know neither Python nor the format of this file. Sorry(

default compilation environment
--enable-vtune-profiling
Enable profiling support for Intel VTune profiler to
profileJavaScript code executed in nodejs. This
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

profileJavaScript -> profile JavaScript ?
nodejs -> Node.js ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vcbuild.bat release msi : builds release build and MSI installer package
vcbuild.bat test : builds debug build and runs tests
vcbuild.bat build-release : builds the release distribution as used by nodejs.org
vcbuild.bat enable-vtune : builds nodejs with Intel VTune profiling support to profile JavaScript
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nodejs -> Node.js ?

On Windows:

- vcbuild.bat: A Windows Batch Script that locates build tools, provides
a subset of the targets avilable in the [Makefile](#makefile), and a few
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avilable -> available

- Makefile: A makefile that can be run with BSD make or GNU make. It
provides a set of targets that build and test the Node.js
binary, produce releases and documentation, and interact with the
CI to run benchamarks or tests. For a detailed guide on this file,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benchamarks -> benchmarks

alternative lib name to link to [default: uv]
--shared-libuv-libpath=SHARED_LIBUV_LIBPATH
a directory to search for the shared libuv DLL
--shared-openssl link to a shared OpenSSl DLL instead of static linking
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenSSl -> OpenSSL

### Targets

For the targets listed below that do not have detailed descriptions
in this docuement, please see their comments in the makefile instead.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docuement -> document

For the targets listed below that do not have detailed descriptions
in this docuement, please see their comments in the makefile instead.

Here is a list of targets that will commonly triggered by its users:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

triggered -> be triggered

Makefile Outdated
v8:
tools/make-v8.sh
$(MAKE) -C deps/v8 $(V8_ARCH).$(BUILDTYPE_LOWER) $(V8_BUILD_OPTIONS)


# This is meant to be run by the developers, which builds Node.js and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the and which

If we're going to say this here, is it worth saying something similar for make all?

Makefile Outdated
@@ -390,13 +402,17 @@ CI_JS_SUITES ?= default
CI_DOC := doctool

# Build and test addons without building anything else
# The CI runs this on ARM(Raspberry Pis) because the addons and
# the binary are cross-compiled elsewhere
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

binary -> binaries

Makefile Outdated
test-doc: doc-only
$(MAKE) lint
$(PYTHON) tools/test.py $(CI_DOC)

test-known-issues: all
$(PYTHON) tools/test.py known_issues

# Used by the node-test-npm CI job
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be run manually too.

Copy link
Member Author

@joyeecheung joyeecheung Nov 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I think it's obvious without saying that all the targets here can be run manually (some needs certain environment variables and some probably shouldn't be run manually though)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would agree, but if you say:

# Note that this is for developers, it is not run on the CI. See run-ci.

elsewhere, then this line suggests that it's used exclusively by the CI job.

or `.\configure --help` (Windows). Consider updating this file
when you need to update the configuration process.

```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with @richardlau , don't think we should duplicate the ./configure --help output.


## Makefile

This file is not generated and mantained by humans. Note that this is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, I totally misread this 😁

--debug-nghttp2 build nghttp2 with DEBUGBUILD (default is false)
```

## Makefile
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was compiling Alacritty the other day, and I noticed that their make actually defaults to showing help (alias for make help) rather than doing anything. It actually looks really nice, and it'd be really useful if we could do something similar.

I think a help option to make would be less likely to bitrot than separate documentation.

https://github.com/jwilm/alacritty/blob/245a80078180acc2a0a1addc569c15b77991c1c3/Makefile#L18-L21


## configure

To see the help text, run `./configure --help`(unix-like)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can just say python configure for both Unix and Windows.

@gibfahn
Copy link
Member

gibfahn commented Nov 15, 2017

Personally I prefer to leave most of the actual documentation in the code, whiling leave some pointers and necessary explanations in the guide so it's easier to get started with maintaining them.

💯 on this, nothing worse than slightly out-of-date documentation that you can never quite trust.

@joyeecheung joyeecheung changed the title [WIP] doc: add guide on maintaining the build files doc: add guide on maintaining the build files Nov 18, 2017
@joyeecheung
Copy link
Member Author

I think I've addressed most comments. @rvagg @TimothyGu @richardlau @vsemozhetbyt @gibfahn PTAL

@joyeecheung
Copy link
Member Author

@joyeecheung
Copy link
Member Author

joyeecheung commented Nov 20, 2017

Ping @nodejs/build @rvagg @gibfahn @richardlau for reviews, thanks

@joyeecheung
Copy link
Member Author

Copy link
Member

@richardlau richardlau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit otherwise LGTM.

Makefile Outdated
lint-ci: lint
else
# Inside the git repository
# This runs linters ond different type of files. It's supposed to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: on?

@joyeecheung
Copy link
Member Author

Rebased. Planning on landing this on Monday if there are no more outstanding reviews. cc @nodejs/collaborators

CI: https://ci.nodejs.org/job/node-test-pull-request/12435/

joyeecheung added a commit that referenced this pull request Jan 9, 2018
PR-URL: #16975
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
joyeecheung added a commit that referenced this pull request Jan 9, 2018
PR-URL: #16975
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
@joyeecheung
Copy link
Member Author

joyeecheung commented Jan 9, 2018

Landed in c91bd2f...89d8657, thanks!

@joyeecheung joyeecheung closed this Jan 9, 2018
MylesBorins pushed a commit that referenced this pull request Jan 10, 2018
PR-URL: #16975
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
MylesBorins pushed a commit that referenced this pull request Jan 10, 2018
PR-URL: #16975
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
@MylesBorins MylesBorins mentioned this pull request Jan 10, 2018
MylesBorins pushed a commit that referenced this pull request Jan 10, 2018
PR-URL: #16975
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
MylesBorins pushed a commit that referenced this pull request Jan 10, 2018
PR-URL: #16975
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
@MylesBorins
Copy link
Contributor

Should this land? It will need a manual backport to 6.x and 8.x

@joyeecheung
Copy link
Member Author

@MylesBorins it's just some doc changes so I would say nope

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Issues and PRs related to build files or the CI. doc Issues and PRs related to the documentations.
Projects
None yet
Development

Successfully merging this pull request may close these issues.