User stories for Jenkins and Mariposa #3

tgross · 2017-01-20T14:45:57Z

misterbisson · 2017-01-20T22:24:53Z

rfd/0079/jenkins-mariposa.md

@@ -0,0 +1 @@
+c1bcd137fadd26ca2998eec192d04c08f62beb1f


@tgross this should be content, not a sha, right?

misterbisson · 2017-01-23T16:44:59Z

rfd/0079/jenkins-mariposa.md

@@ -0,0 +1,76 @@
+# jupiter.example.com, Jenkins, and Mariposa


Can you put this in 00036/user-stories/jupiter-example-com-jenkins.md and add it to the user stories in the main readme?

Sure thing.

misterbisson · 2017-01-23T17:08:38Z

rfd/0036/user-stories/jupiter-example-com-jenkins.md

+
+- Jenkins must be provisioned with an SSH key for Mariposa
+- Jenkins must be provisioned with an SSH key for GitHub deploy keys
+- Jenkins will effectively own the entire account until we have RBAC v2.


This is a good callout, but it doesn't help us specify what permissions would be expected in an RBACv2 world

Brainstorming for RBACv2 world, Jenkins would need permissions to:

create/copy project manifests

update project manifests

trigger deploys of projects

Jenkins would not need permissions to:

launch/destroy containers

change account information

(not yet a complete list)

misterbisson · 2017-01-23T17:09:16Z

rfd/0036/user-stories/jupiter-example-com-jenkins.md

+
+Jenkins would generate secrets as part of the job for the dev environment, add those to the manifest’s metadata (this isn’t in the manifest file itself!). This allows the dev credentials be ephemeral for each deployment.
+
+For production metadata, the operators can add those secrets to Mariposa out-of-band from the CI server. This way Jenkins never gets the production secrets. It only has the manifest not the metadata for production. (Although in practice in has the keys to the universe until RBAC v2.)


Although in practice in has the keys to the universe until RBAC v2

We need to get more particular about what permissions are required to support this process

misterbisson · 2017-01-23T17:10:36Z

rfd/0036/user-stories/jupiter-example-com-jenkins.md

+
+This Jenkins job is configured to receive GitHub webhook pushes. The Jenkins job should run build and deployment tasks defined in the code repository so that the process for deployment is owned by the team that owns the code.
+
+Each GitHub branch has an associated Mariposa project. The Jenkins job clones a template manifest (from the repo) for each new branch. Each changeset received on a given branch will be deployed as a new version of that manifest.


Each GitHub branch has an associated Mariposa project

Does this assume the project is pre-created, or does Jenkins create it?

Not sure... is creating a new manifest for a project a separate action from creating the project? If so, then we could split the two behaviors for Jenkins. We don't have to though.

I imagine there's manifest per project (though we track changes in that manifest and can restore old versions).

I think my real point is that the passive voice there suggests that Jenkins isn't creating projects, so I wanted to be clear.

misterbisson · 2017-01-23T17:26:29Z

rfd/0036/user-stories/jupiter-example-com-jenkins.md

+
+```
+git tag v1.2.3
+git tag production


This is focused on the git side of things, but that may not fully capture what we staged and tested, no?

More discussion in #3 (comment)

It is, but we need a way to identify to the production-deploying job which branch or tag it's supposed to be promoting to production. Using git tag is one way, but in the workflow in https://github.com/joyent/joyent.com-kirby/pull/181 we're using the master branch (or whatever branch we've parameterized otherwise).

Either works, but by having the git tag be the marker we end up with a record of what source version corresponds to what image version. (i.e. although you can move a tag to a different commit a branch is always pointing to the last commit of its branch, even after you deploy).

Please discuss in #3 (comment)

(copying this thread there)

misterbisson · 2017-01-23T17:29:54Z

rfd/0036/user-stories/jupiter-example-com-jenkins.md

+git push --tags
+```
+
+The operator uses the Jenkins UI (or CLI) to start the deployment job with the particular tag needed (with the default being part of the Jenkins job definition). The job tags the correct version as “production” and renders the production manifest with the new version and pushes it to Mariposa for deployment. Note that Mariposa isn’t watching registry server; you need to explicitly update images in IMGAPI to get new images, so updating the Docker image tag in the registry is not enough to trigger a deployment.


you need to explicitly update images in IMGAPI to get new images

There are a number of intersecting "versions" here.

The git repo with the docker file

The other repos sources that we incorporate in the build, and which are not as strongly versioned as the primary git repo (1)

The image tag, which we hope will be strongly versioned with the sum of all its components from the above (2), but independently from the primary git repo (1)

The project manifest, which might specify an image tag more specific than "latest" or "prod" (3)

Tagging the release in git addresses (1), but seems to leave a lot of gaps elsewhere. I think we need to work on how to close those gaps.

Agreed that tagging suitably addresses (1).

Can we address (2) by incorporating the git SHA into the build? Similar to what we do in ContainerPilot or Joyent.com. This makes the git SHA indelibly marked in the image layers.

Resolving (3) by making that same git SHA show up in the project manifest is a tiny bit harder because we probably can't enforce any of this workflow (it's all owned by the end user's Jenkins job). But we could easily expose the Labels from the Docker images associated with a manifest version though. Maybe we provide a conventional label to the end users? (like triton.mariposa.version or something)

I'm probably not explaining my question well. Let me try a different approach.

Let's say we build an image on day one. The base image is pinned to php:5.6-apache, it stages well, and everything works perfectly.

However, on day 2 a developer does a manual build of the image using the exactly same code/git sha from day one. The difference is that on this day the content of the base image is different and now it has a bug.

The git sha represents one aspect of the "version," but it clearly doesn't represent everything.

Updated this following our conversion this afternoon. How's this look?

misterbisson reviewed Jan 20, 2017

View reviewed changes

User stories for Jenkins and Mariposa

f763068

misterbisson reviewed Jan 23, 2017

View reviewed changes

Move Jenkins user story

8be3336

misterbisson reviewed Jan 23, 2017

View reviewed changes

misterbisson and others added 5 commits January 24, 2017 16:35

Copy and grammar edits

c021bb8

Updated TOC

8bc6afe

Hopefully clarify the first step of production

b910652

Clarify required permissions for Jenkins

d26899b

Differentiate image versioning, project versioning, source versioning

c18a849

misterbisson merged commit 9678d9e into misterbisson:master Jan 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User stories for Jenkins and Mariposa #3

User stories for Jenkins and Mariposa #3

tgross commented Jan 20, 2017

misterbisson Jan 20, 2017

tgross Jan 23, 2017

misterbisson Jan 23, 2017 •

edited

Loading

tgross Jan 23, 2017

tgross Jan 23, 2017

misterbisson Jan 23, 2017

tgross Jan 23, 2017

misterbisson Jan 23, 2017

misterbisson Jan 23, 2017

tgross Jan 23, 2017

misterbisson Jan 23, 2017

misterbisson Jan 23, 2017 •

edited

Loading

tgross Jan 23, 2017

misterbisson Jan 23, 2017

tgross Jan 24, 2017

misterbisson Jan 23, 2017 •

edited

Loading

tgross Jan 24, 2017

misterbisson Jan 25, 2017

tgross Jan 25, 2017

		@@ -0,0 +1,76 @@
		# jupiter.example.com, Jenkins, and Mariposa


		Jenkins would generate secrets as part of the job for the dev environment, add those to the manifest’s metadata (this isn’t in the manifest file itself!). This allows the dev credentials be ephemeral for each deployment.

		For production metadata, the operators can add those secrets to Mariposa out-of-band from the CI server. This way Jenkins never gets the production secrets. It only has the manifest not the metadata for production. (Although in practice in has the keys to the universe until RBAC v2.)


		This Jenkins job is configured to receive GitHub webhook pushes. The Jenkins job should run build and deployment tasks defined in the code repository so that the process for deployment is owned by the team that owns the code.

		Each GitHub branch has an associated Mariposa project. The Jenkins job clones a template manifest (from the repo) for each new branch. Each changeset received on a given branch will be deployed as a new version of that manifest.

		@@ -0,0 +1 @@
		c1bcd137fadd26ca2998eec192d04c08f62beb1f

User stories for Jenkins and Mariposa #3

User stories for Jenkins and Mariposa #3

Conversation

tgross commented Jan 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterbisson Jan 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterbisson Jan 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterbisson Jan 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterbisson Jan 23, 2017 •

edited

Loading

misterbisson Jan 23, 2017 •

edited

Loading

misterbisson Jan 23, 2017 •

edited

Loading