Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools: add release tool, remove old tools (WIP) #681

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -174,9 +174,6 @@ $(apidoc_dirs):
out/doc/api/assets/%: doc/api_assets/% out/doc/api/assets/
cp $< $@

out/doc/changelog.html: CHANGELOG.md doc/changelog-head.html doc/changelog-foot.html tools/build-changelog.sh $(NODE_EXE)
bash tools/build-changelog.sh

out/doc/%: doc/%
cp -r $< $@

Expand All @@ -186,13 +183,6 @@ out/doc/api/%.json: doc/api/%.markdown $(NODE_EXE)
out/doc/api/%.html: doc/api/%.markdown $(NODE_EXE)
out/Release/$(NODE_EXE) tools/doc/generate.js --format=html --template=doc/template.html $< > $@

email.md: CHANGELOG.md tools/email-footer.md
bash tools/changelog-head.sh | sed 's|^\* #|* \\#|g' > $@
cat tools/email-footer.md | sed -e 's|__VERSION__|'$(VERSION)'|g' >> $@

blog.html: email.md
cat $< | ./$(NODE_EXE) tools/doc/node_modules/.bin/marked > $@

docopen: out/doc/api/all.html
-google-chrome out/doc/api/all.html

Expand Down
63 changes: 60 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
io.js
===
=====

[![Gitter](https://badges.gitter.im/Join Chat.svg)](https://gitter.im/iojs/io.js?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

This repository began as a GitHub fork of
Expand All @@ -8,7 +9,8 @@ This repository began as a GitHub fork of
io.js contributions, releases, and contributorship are under an
[open governance model](./GOVERNANCE.md).
We intend to land, with increasing regularity, releases which are
compatible with the npm ecosystem that has been built to date for Node.js.
compatible with the npm ecosystem that has been built to date for
Node.js.

## Is it io.js or IO.js or iojs or IOjs or iOjS?

Expand All @@ -17,7 +19,61 @@ especially not at the start of a sentence, unless it is being
displayed in a location that is customarily all-caps (such as
the title of man pages).

## To build:
## Download

Binaries, installers and source tarballs are available at
<https://iojs.org>.

**Releases** are available at <https://iojs.org/dist/>, listed under
their version string. The <https://iojs.org/dist/latest/> symlink
will point to the latest release directory.

**Nightly** builds are available at
<https://iojs.org/download/nightly/>, listed under their version
string which includes their date (in UTC time) and the commit SHA at
the HEAD of the release.

**API documentation** is available in each release and nightly
directory under _docs_. <http://iojs.org/api/> points to the the
latest version.

### Verifying Binaries

Release and nightly download directories all contain a *SHASUM256.txt*
file that lists the SHA checksums for each file available for
download. To check that a downloaded file matches the checksum, run
it through `sha256sum` with a command such as:

```
$ grep iojs-vx.y.z.tar.gz SHASUMS256.txt | sha256sum -c -
```

_(Where "iojs-vx.y.z.tar.gz" is the name of the file you have
downloaded)_

Additionally, releases (not nightlies) have GPG signed copies of
SHASUM256.txt files available as SHASUM256.txt.asc. You can use `gpg`
to verify that the file has not been tampered with.

To verify a SHASUM256.txt.asc, you will first need to import all of
the GPG keys of individuals authorized to create releases. They are
listed at the bottom of this README. Use a command such as this to
import the keys:

```
$ gpg --keyserver pool.sks-keyservers.net \
--recv-keys DD8F2338BAE7501E3DD5AC78C273792F7D83545D
```

_(Include each of the key fingerprints at the end of this command.)_

You can then use `gpg --verify SHASUMS256.txt.asc` to verify that the
file has been signed by an authorized member of the io.js team.

Once verified, use the SHASUMS256.txt.asc file to get the checksum for
the binary verification command above.

## Build

### Unix / Macintosh

Expand Down Expand Up @@ -209,6 +265,7 @@ information about the governance of the io.js project, see
* **Colin Ihrig** ([@cjihrig](https://github.com/cjihrig)) &lt;cjihrig@gmail.com&gt; (Technical Committee)
* **Mikeal Rogers** ([@mikeal](https://github.com/mikeal)) &lt;mikeal.rogers@gmail.com&gt;
* **Rod Vagg** ([@rvagg](https://github.com/rvagg)) &lt;rod@vagg.org&gt;
<br>Release GPG key: DD8F2338BAE7501E3DD5AC78C273792F7D83545D
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a naive question but can't we use a single master key with as many sub-keys as there are release managers? It would make all releases be signed by "io.js Release Engineering <releng@iojs.org>" (or whatever we decide to call ourselves) and it makes revocation a little easier. Master key management is an open issue, though; who gets to own it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, to which I don't have a simple answer unfortunately.

My personal bias is towards decentralising this and making individuals responsible for signing their releases with their own keys, simply because I think decentralisation in most aspects of the project are more healthy for the community focus and ethos that we're trying to foster. It also makes that individual properly responsible for that release rather than it just being a mechanical task.

However, I don't have any strong technical objections to a more rigid signing mechanism if others prefer that. Perhaps a question for @iojs/tc.

* **Thorsten Lorenz** ([@thlorenz](https://github.com/thlorenz)) &lt;thlorenz@gmx.de&gt;
* **Stephen Belanger** ([@qard](https://github.com/qard)) &lt;admin@stephenbelanger.com&gt;
* **Jeremiah Senkpiel** ([@fishrock123](https://github.com/fishrock123)) &lt;fishrock123@rocketmail.com&gt;
Expand Down
36 changes: 0 additions & 36 deletions doc/changelog-foot.html

This file was deleted.

38 changes: 0 additions & 38 deletions doc/changelog-head.html

This file was deleted.

209 changes: 209 additions & 0 deletions doc/releases.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
io.js Release Process
=====================

This document describes the technical aspects of the io.js release process. The intended audience is those who have been authorized by the Technical Committee (TC) to create, promote and sign official release builds for io.js, hosted on <https://iojs.org>.

## Who can make a release?

Release authorization is given by the io.js TC. Once authorized, an individual must be have the following:

### 1. Jenkins Release Access

There are three relevant Jenkins jobs that should be used for a release flow:

**a.** **[iojs+any-pr+multi](https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/)** is used for a final full-test run to ensure that the current *HEAD* is stable.

**b.** (optional) **[iojs+release+nightly](https://jenkins-iojs.nodesource.com/job/iojs+release+nightly/)** can be used to create a nightly release for the current *HEAD* if public test releases are required. Builds triggered with this job are published straight to <http://iojs.org/download/nightly/> and are available for public download.

**c.** **[iojs+release](https://jenkins-iojs.nodesource.com/job/iojs+release/)** does all of the work to build all required release assets. Promotion of the release files is a manual step once they are ready (see below).

The [io.js build team](https://github.com/iojs/build) is able to provide this access to individuals authorized by the TC.

### 2. <iojs.org> Access

The _dist_ user on iojs.org controls the assets available in <http://iojs.org/download/> (note that <http://iojs.org/dist/> is an alias for <https://iojs.org/download/release/>).

The Jenkins release build slaves upload their artefacts to the web server as the _staging_ user, the _dist_ user has access to move these assets to public access (the _staging_ user does not, for security purposes).

Nightly builds are promoted automatically on the server by a cron task for the _dist_ user.

Release builds require manual promotion by an individual with SSH access to the server as the _dist_ user. The [io.js build team](https://github.com/iojs/build) is able to provide this access to individuals authorized by the TC.

### 3. A Publicly Listed GPG Key

A SHASUMS256.txt file is produced for every promoted build, nightly and releases. Additionally for releases, this file is signed by the individual responsible for that release. In order to be able to verify downloaded binaries, the public should be able to check that the SHASUMS256.txt file has been signed by someone who has been authorized to create a release.

The GPG keys should be fetchable from a known third-party keyserver, currently the SKS Keyservers at <https://sks-keyservers.net> are recommended. Use the [submission](https://sks-keyservers.net/i/#submit) form to submit a new GPG key. Keys should be fetchable via:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting an insecure connection warning when I try to submit my pubkey to this server. Is this something I should be worried about?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, is there any reason to not use https://pgp.mit.edu/ ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually went ahead with sending my pubkey despite the insecure connection. Was rewarded with a 500 Internal Server Error. ):

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason I chose sks-keyservers.net was because the only precedent I'm aware of for checking signed shasums is in the Docker images for both Node (official Docker ones) and io.js and that's what they've chosen to use. I don't know the reasoning behind this but I went with that as an established convention.

I'm happy to change based on recommendation, @bnoordhuis also had some thoughts about PGP keys so we should chat about this tomorrow at TC.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, sounds like a good topic for @iojs/tc agenda. @rvagg let's land it the way it is for now, we will fix it later.


```
gpg --keyserver pool.sks-keyservers.net --recv-keys <FINGERPRINT>
```

Additionally, full GPG key fingerprints for individuals authorized to release should be listed in the io.js GitHub README.md file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I totally missed this :|


## How to create a release

Notes:

- Dates listed below as _"YYYY-MM-DD"_ should be the date of the release **as UTC**. Use `date -u +'%Y-%m-%d'` to find out what this is.
- Version strings are listed below as _"vx.y.z"_, substitute for the release version.

### 1. Ensure that HEAD Is Stable

Run a **[iojs+any-pr+multi](https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/)** test run to ensure that the build is stable and the HEAD commit is ready for release.

### 2. Produce a Nightly Build _(optional)_

If there is reason to produce a test release for the purpose of having others try out installers or specifics of builds, produce a nightly build using **[iojs+release+nightly](https://jenkins-iojs.nodesource.com/job/iojs+release+nightly/)** and wait for it to drop in <http://iojs.org/download/nightly/>.

This is particularly recommended if there has been recent work relating to the OS X or Windows installers as they are not tested in any way by CI.

### 4. Update the _CHANGELOG.md_

Use the following git command to produce a list of commits since the last release:

```
git log --pretty=format:"* [%h] - %s (%aN)" \
--since="$(git show -s --format=%ad `git rev-list --tags --max-count=1`)"
```

_(You will probably need to omit the first two commits as they relate to the last release.)_

The _CHANGELOG.md_ entry should take the following form:

```
## YYYY-MM-DD, Version x.y.z, @user

### Notable changes

* List interesting changes here
* Particularly changes that are responsible for minor or major version bumps
* Also be sure to look at any changes introduced by dependencies such as npm
* ... and include any notable items from there

### Known issues

* Include this section if there are any known problems with this release
* Scan GitHub for unresolved problems that users may need to be aware of

### Commits

* Include the full list of commits since the last release here
```

### 5. Update _src/node_version.h_

The following macros should already be set for the release since they will have been updated directly following the last release. They shouldn't require changing:

```
#define NODE_MAJOR_VERSION x
#define NODE_MINOR_VERSION y
#define NODE_PATCH_VERSION z
```

However, the `NODE_VERSION_IS_RELEASE` macro needs to be set to `1` for the build to be produced with a version string that does not have a trailing pre-release tag:

```
#define NODE_VERSION_IS_RELEASE 1
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section should explain when to bump NODE_MODULE_VERSION and what it means for the version number (at least a minor bump; I don't think we've decided on a major bump policy yet.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, adding this:

Also consider whether to bump NODE_MODULE_VERSION:

This macro is used to signal an ABI version for native addons. It currently has two common uses in the community:

  • Determining what API to work against for compiling native addons, e.g. NAN uses it to form a compatibility-layer for much of what it wraps.
  • Determining the ABI for downloading pre-built binaries of native addons, e.g. node-pre-gyp uses this value as exposed via process.versions.modules
    to help determine the appropriate binary to download at install-time.

The general rule is to bump this version when there are breaking ABI changes and also if there are non-trivial API changes. The rules are not yet strictly defined, so if in doubt, please confer with someone that will have a more informed perspective, such as a member of the NAN team.


**Also consider whether to bump `NODE_MODULE_VERSION`**:

This macro is used to signal an ABI version for native addons. It currently has two common uses in the community:

* Determining what API to work against for compiling native addons, e.g. [NAN](https://github.com/rvagg/nan) uses it to form a compatibility-layer for much of what it wraps.
* Determining the ABI for downloading pre-built binaries of native addons, e.g. [node-pre-gyp](https://github.com/mapbox/node-pre-gyp) uses this value as exposed via `process.versions.modules` to help determine the appropriate binary to download at install-time.

The general rule is to bump this version when there are _breaking ABI_ changes and also if there are non-trivial API changes. The rules are not yet strictly defined, so if in doubt, please confer with someone that will have a more informed perspective, such as a member of the NAN team.

### 6. Create Release Commit

The _CHANGELOG.md_ and _src/node_version.h_ changes should be the final commit that will be tagged for the release.

When committing these to git, use the following message format:

```
YYYY-MM-DD io.js vx.y.z Release

Notable changes:

* Copy the notable changes list here
```

### 7. Tag and Sign the Release Commit

Tag the release as <b><code>vx.y.z</code></b> and sign **using the same GPG key that will be used to sign SHASUMS256.txt**.

```
git tag -sm 'YYYY-MM-DD io.js vz.y.x Release' vx.y.z
```

### 8. Set Up For the Next Release

Edit _src/node_version.h_ again and:

* Increment `NODE_PATCH_VERSION` by one
* Change `NODE_VERSION_IS_RELEASE` back to `0`

Commit this change with:

```
git commit -am 'Working on vx.y.z' # where 'z' is the incremented patch number
```

This sets up the branch so that nightly builds are produced with the next version number _and_ a pre-release tag.

### 9. Push to GitHub

Push the changes along with the tag you created:

```
git push origin branch vx.y.z
# where "branch" is the working branch and "vx.y.z" the the release version
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(we may later add e.g., v1.x here)

```

### 9. Produce Release Builds

Use **[iojs+release](https://jenkins-iojs.nodesource.com/job/iojs+release/)** to produce release artefacts. Enter the "vx.y.z" version string for this release and it will fetch your tagged commit.

Artefacts from each slave are uploaded to Jenkins and are available if further testing is required. Use this opportunity particularly to test OS X and Windows installers if there are any concerns. Click through to the individual slaves for a run to find the artefacts. For example, the Windows 64-bit .msi file for v1.0.4 can be found [here](https://jenkins-iojs.nodesource.com/job/iojs+release/20/nodes=iojs-win2008r2-release-x64/).

All release slaves should achieve "SUCCESS" (and be blue, not red). A release with failures should not be promoted, there are likely problems to be investigated.

Note that you do not have to wait for the ARM builds if they are take longer than the others. It is only necessary to have the main Linux (x64 and x86), OS X .pkg and .tar.gz, Windows (x64 and x86) .msi and .exe, source and docs (both produced currently by an OS X slave). i.e. the slaves with "arm" in their name don't need to have finished to progress to the next step. However, **if you promote builds _before_ ARM builds have finished, you must repeat the promotion step for the ARM builds when they are ready**.

### 10. Promote and Sign the Release Builds

**It is important that the same individual who signed the release tag be the one to promote the builds as the SHASUMS256.txt file needs to be signed with the same GPG key!**

When you are confident that the build slaves have properly produced usable artefacts and uploaded them to the web server you can promote them to release status. This is done by interacting with the web server via the _dist_ user.

The _tools/release.sh_ script may be used to promote and sign the build. When run, it will perform the following actions:

**a.** Select a GPG key from your private keys, it will use a command similar to: `gpg --list-secret-keys` to list your keys. If you don't have any keys, it will bail (why are you releasing? Your tag should be signed!). If you have only one key, it will use that. If you have more than one key it will ask you to select one from the list. Be sure to use the same key that you signed your git tag with.

**b.** Log in to the server via SSH and check for releases that can be promoted, along with the list of artefacts. It will use the `dist-promotable` command on the server to find these. You will be asked, for each promotable release, whether you want to proceed. If there is more than one release to promote (there shouldn't be), be sure to only promote the release you are responsible for.

**c.** Log in to the server via SSH and run the promote script for the given release. The command on the server will be similar to: `dist-promote vx.y.z`. After this step, the release artefacts will be available for download and a SHASUMS256.txt file will be present. The release will still be unsigned, however.

**d.** Download SHASUMS256.txt to your computer using SCP into a temporary directory.

**e.** Sign the SHASUMS256.txt file using a command similar to: `gpg --default-key YOURKEY --clearsign /path/to/SHASUMS256.txt`. You will be prompted by GPG for your password for this to work. The signed file will be named SHASUMS256.txt.asc.

**f.** Output an ASCII armored version of your public GPG key, using a command similar to: `gpg --default-key YOURKEY --armor --export --output /path/to/SHASUMS256.txt.gpg`. This does not require your password and is mainly a convenience for users although not the recommended way to get a copy of your key.

**g.** Upload the SHASUMS256.txt\* files back to the server into the release directory.

If you didn't wait for ARM builds in the previous step before promoting the release, you should re-run _tools/release.sh_ after the ARM builds have finished and it will move the ARM artefacts into the correct location and you will be prompted to re-sign SHASUMS256.txt.

### 11. Check the Release

Your release should be available at <https://iojs.org/dist/vx.y.z/> and also <https://iojs.org/dist/latest/>. Check that the appropriate files are in place, you may also want to check that the binaries are working as appropriate and have the right internal version strings. Check that the API docs are available at <https://iojs.org/api/>. Check that the release catalog files are correct at <https://iojs.org/dist/index.tab> and <https://iojs.org/dist/index.json>.

### 12. Announce

_TODO: Update doc with announce procedure when we figure this out._

### 13. Celebrate

_In whatever form you do this..._
Loading