From 320c4b3f3a35e7265bf52826f5698f60913da982 Mon Sep 17 00:00:00 2001
From: Alex Suraci <asuraci@pivotal.io>
Date: Tue, 3 Apr 2018 17:31:34 -0400
Subject: [PATCH 01/19] add wip resources v2 proposal

---
 00-resources-v2/proposal.md | 158 ++++++++++++++++++++++++++++++++++++
 1 file changed, 158 insertions(+)
 create mode 100644 00-resources-v2/proposal.md

diff --git a/00-resources-v2/proposal.md b/00-resources-v2/proposal.md
new file mode 100644
index 00000000..fd55b1e7
--- /dev/null
+++ b/00-resources-v2/proposal.md
@@ -0,0 +1,158 @@
+# Summary
+
+1. Tighten up the existing resource interface to better represent its intended
+   use case: versioned objects, changing over time, with an external source of
+   truth.
+
+1. Introduce versioning to the resource interface, so that we can maintain
+   backwards-compatibility.
+
+1. Introduce new interfaces to support today's use cases that are currently
+   being shoehorned into the resource interface.
+
+1. Provide an answer for how spaces are discovered and used, in light of
+   concourse/concourse#1707.
+
+
+# Motivation
+
+Resources today are used for many things that they should not be used for. This
+leads to surprising behavior, difficult workarounds, and a lack of consistency
+in what you can expect from any given resource type.
+
+Many of them only implement a subset of the interface, or only provide stubs
+for part of it. This makes them not a true resource.
+
+We have a need in concourse/concourse#1707 to discover change over *space*, not
+*time*, which requires additions or changes to the resource interface. We also
+have
+
+
+
+* External un-versioned resources that have a "current state" (i.e. a deployment, a set of git branches, a set of git PRs)
+* Push-only things with no state, like notifications
+* Un-versioned blobs that can be passed around and deleted
+
+# Proposal
+
+```elm
+-- arbitrary configuration
+type alias Config = Dict String Json.Value
+
+-- data on disk (it's not in memory, so blank return value)
+type alias Bits = ()
+
+-- arbitrary metadata to show to the user (this may become more structured later)
+type alias Metadata = List (String, String)
+
+-- Versioned Dependencies (Git repo)
+type alias Version = Dict String String
+
+check  : Config -> List Version
+get    : Config -> Version -> Bits
+put    : Config -> Bits -> Set Version
+delete : Config -> Bits -> Set Version
+
+-- Spaces (BOSH deployments, Git branches, Git PRs, Semver trees)
+type alias Space = Dict String String
+
+check  : Config -> Set Space
+get    : Config -> Space -> Bits
+put    : Config -> Bits -> Set Space
+delete : Config -> Bits -> Set Space
+
+-- Notifications (slack, email)
+type alias BuildMetadata =
+  { buildStatus : String
+  , buildNumber : Int
+  , jobName : String
+  , pipelineName : String
+  , teamName : String
+  }
+
+notify : Config -> BuildMetadata -> ()
+```
+
+
+# Examples
+
+- Pull Requests
+- BOSH deploys
+- Feature branches
+- Arbitrary branches
+- IaaSes
+
+
+## Pull Requests
+
+```yaml
+space_types:
+- name: github-pr
+  type: docker-image
+  source: {repository: concourse/github-pr-space-type}
+
+spaces:
+- name: atc-prs
+  type: github-pr
+  source: {repository: concourse/atc}
+
+resources:
+- name: atc-pr
+  type: git
+  source: {uri: "https://github.com/concourse/atc"}
+  spaces: atc-prs
+
+jobs:
+- name: atc-pr-unit
+  plan:
+  - get: atc-pr
+    trigger: true
+    spaces: all
+  - task: unit
+    file: atc/ci/pr.yml
+```
+
+!! stateful spaces? hooks on build start/finish with state available
+!! this could be used to reflect PR status, send slack alerts, track pool entries
+
+
+
+### Scheduling
+
+There should be a single "space combination" of `atc-pr-unit` for every active
+space returned by `atc-prs`. So, the `atc-prs` space must be periodically
+`check`ed to determine the set of PRs available, and the `atc-pr` resource in
+turn must run a `check` for every space.
+
+When any of the space combinations of `atc-pr-unit` has a new version
+available, I expect a build of `atc-pr-unit` to automatically kick off.
+
+1. The `github-pr` space type is implemented like so:
+  * `check` returns the set of PRs, e.g. `[{"remote": "pull/23/head"}]`
+  * `put` is used to update the PR status
+  * `get` returns metadata pertaining to the PR
+  * `delete` is unimplemented (maybe it closes the pr? not important here.)
+
+1. The `atc-prs` space is configured to detect the spaces of the
+  `concourse/atc` repo.
+
+1. The `atc-pr` resource is configured as a regular old `git` resource,
+  performing `checks` across all spaces returned by `atc-prs`.
+
+  * This works by having one *resource* `check` for each space returned by
+    `atc-prs`, by merging the space object `{"remote":"pull/23/head"}` with the
+    rest of the resource config.
+
+    The `git` resource would have to understand this config param to fetch the
+    given remote and emit the version.
+
+  * Because this resource is configured `across: atc-prs`, any use of the
+    resource must be space-aware.
+
+1. The `atc-pr-unit` job configures a `get` step of `atc-pr`. Because the
+  `atc-pr` resource is spatial, each space satisfying the `spaces: all` filter
+  results in a "space combination" for the job, each with independent
+  scheduling.
+
+
+### Runtime

From b4b63ec8a8aaf289b2d0b91dcfc2385a55c9a707 Mon Sep 17 00:00:00 2001
From: Alex Suraci <asuraci@pivotal.io>
Date: Thu, 5 Apr 2018 17:14:19 -0400
Subject: [PATCH 02/19] more wip; currently stuck pondering notification api

---
 00-resources-v2/bosh.yml          |  57 +++++++++++++++
 00-resources-v2/branch-gen.yml    |  41 +++++++++++
 00-resources-v2/commit-status.yml |  36 ++++++++++
 00-resources-v2/notifications.yml |  37 ++++++++++
 00-resources-v2/proposal.md       | 114 +++++++++++++++---------------
 00-resources-v2/prs.yml           |  37 ++++++++++
 6 files changed, 265 insertions(+), 57 deletions(-)
 create mode 100644 00-resources-v2/bosh.yml
 create mode 100644 00-resources-v2/branch-gen.yml
 create mode 100644 00-resources-v2/commit-status.yml
 create mode 100644 00-resources-v2/notifications.yml
 create mode 100644 00-resources-v2/prs.yml

diff --git a/00-resources-v2/bosh.yml b/00-resources-v2/bosh.yml
new file mode 100644
index 00000000..597ef566
--- /dev/null
+++ b/00-resources-v2/bosh.yml
@@ -0,0 +1,57 @@
+resource_types:
+- name: boshdir
+  type: docker-image
+  source: {repository: concourse/github-pr-space-type}
+
+resources:
+- name: director
+  type: boshdir
+  source:
+    target: bosh.concourse-ci.org
+    client_id: abc
+    client_secret: def
+
+- name: release
+  type: git
+  source:
+    uri: https://github.com/concourse/concourse
+    branch: master
+
+- name: deployment
+  type: git
+  source:
+    uri: https://github.com/concourse/concourse-bosh-deployment
+    branch: master
+
+jobs:
+- name: unit
+  plan:
+  - get: release
+    trigger: true
+  - task: unit
+    file: release/ci/unit.yml
+
+- name: deploy
+  plan:
+  - get: release
+    passed: [unit]
+    trigger: true
+  - get: deployment
+    trigger: true
+  - task: create-rc
+    file: release/ci/create.yml
+  - put: director
+    params:
+      manifest: deployment/cluster/concourse.yml
+      releases: [release-rc/*.tgz]
+
+- name: acceptance
+  plan:
+  - get: release
+    passed: [deploy]
+    trigger: true
+  - get: deployment
+    passed: [deploy]
+    trigger: true
+  - get: director
+    passed: [deploy]
diff --git a/00-resources-v2/branch-gen.yml b/00-resources-v2/branch-gen.yml
new file mode 100644
index 00000000..9cfc0281
--- /dev/null
+++ b/00-resources-v2/branch-gen.yml
@@ -0,0 +1,41 @@
+resources:
+- name: atc
+  type: git
+  source: {uri: https://github.com/concourse/atc}
+  space: {branch: master}
+
+- name: atc-gen
+  type: git
+  source: {uri: "https://github.com/concourse/atc"}
+
+jobs:
+- name: gen
+  plan:
+  - get: atc
+    trigger: true
+  - task: gen
+    file: atc/ci/gen.yml
+    config:
+      platform: ...
+      image_resource: ...
+      outputs:
+      - name: generated-repo
+        # has the generated code committed to the repo
+      - name: branch-space
+        # has a 'space' file with `{branch: gen-(some deterministic hash)}`
+  - put: atc-gen
+    # dynamically determines the space; resource idempotently creates it
+    space: {load: gen-branch/space}
+    params: {repository: gen-repo}
+
+- name: test
+  plan:
+  - get: atc
+    passed: [gen]
+    trigger: true
+  - get: atc-gen
+    passed: [gen]
+    spaces: [{branch: gen-*}]
+    trigger: true
+  - task: test
+    file: atc/ci/test.yml
diff --git a/00-resources-v2/commit-status.yml b/00-resources-v2/commit-status.yml
new file mode 100644
index 00000000..f13f13d7
--- /dev/null
+++ b/00-resources-v2/commit-status.yml
@@ -0,0 +1,36 @@
+resource_types:
+- name: github-status
+  type: docker-image
+  source: {repository: concourse/github-status-resource}
+
+resources:
+- name: atc
+  type: git
+  source:
+    uri: https://github.com/concourse/atc
+
+- name: atc-status
+  type: github-status
+  notify:
+  - on: used_in_build
+    follow: atc
+  source:
+    repository: concourse/atc
+    access_token: ((token))
+
+jobs:
+- name: atc-pr-unit
+  plan:
+  - get: atc-pr
+    trigger: true
+    spaces: all
+  - task: unit
+    file: atc/ci/pr.yml
+
+# github-status /info:
+{
+  "version": "2.0",
+  "notifier": {
+    "used_in_build": "/opt/resource/notify"
+  }
+}
diff --git a/00-resources-v2/notifications.yml b/00-resources-v2/notifications.yml
new file mode 100644
index 00000000..6f6af50e
--- /dev/null
+++ b/00-resources-v2/notifications.yml
@@ -0,0 +1,37 @@
+resource_types:
+- name: slack-notifier
+  type: docker-image
+  source: {repository: concourse/slack-notifier-resource}
+
+resources:
+- name: atc
+  type: git
+  source:
+    uri: https://github.com/concourse/atc
+
+- name: slack-alert
+  type: slack-notifier
+  source:
+    url: https://hooks.slack.com/services/XXXX
+  notify:
+  - on: build_failed
+    params:
+      template: |
+        Oh no! <{{.ExternalURL}}/builds/{{.BuildID}}|{{.BuildName}}> failed. :(
+
+jobs:
+- name: atc-pr-unit
+  plan:
+  - get: atc-pr
+    trigger: true
+    spaces: all
+  - task: unit
+    file: atc/ci/pr.yml
+
+# github-status /info:
+{
+  "version": "2.0",
+  "notifier": {
+    "build_failed": "/opt/resource/notify"
+  }
+}
diff --git a/00-resources-v2/proposal.md b/00-resources-v2/proposal.md
index fd55b1e7..80bcca58 100644
--- a/00-resources-v2/proposal.md
+++ b/00-resources-v2/proposal.md
@@ -1,17 +1,19 @@
 # Summary
 
-1. Tighten up the existing resource interface to better represent its intended
-   use case: versioned objects, changing over time, with an external source of
-   truth.
+Introduces a new resource interface with the following goals:
+
+1. Support for spaces (concourse/concourse#1707).
+
+1. Marking versions as deleted.
+
+1. Tightening up loopholes in the API to ensure that resources are pointing at
+   an external source of truth and cannot be "partially implemented".
 
 1. Introduce versioning to the resource interface, so that we can maintain
    backwards-compatibility.
 
-1. Introduce new interfaces to support today's use cases that are currently
-   being shoehorned into the resource interface.
-
-1. Provide an answer for how spaces are discovered and used, in light of
-   concourse/concourse#1707.
+1. Establish a pattern for notification-style resources like Slack, GitHub
+   commit/PR status, etc.
 
 
 # Motivation
@@ -28,10 +30,35 @@ We have a need in concourse/concourse#1707 to discover change over *space*, not
 have
 
 
+## New Implications
+
+Here are a few use cases that resources were sometimes used for inappropriately:
+
+1. Resources that really only have a "current state", such as deployments. This
+  is still "change over time", but the difference is that old versions become
+  invalid as soon as there's a new one.
+
+1. Pushing un-versioned artifacts through a pipeline, such that the version
+  history for the resource becomes nonlinear (the "latest" version has nothing
+  to do with the versions prior). This is a cardinal sin.
+
+The new interface improves the story around these two use cases.
+
+The first case is resolved by having versions be deletable; we can now
+represent that previous states of an external dependency are no longer
+available.
+
+The second use case can be resolved by using a new space to represent nonlinear
+versions.
+
 
-* External un-versioned resources that have a "current state" (i.e. a deployment, a set of git branches, a set of git PRs)
-* Push-only things with no state, like notifications
-* Un-versioned blobs that can be passed around and deleted
+## xx hooks use case
+
+- git: commit status
+- github pr: pr status
+
+- pool?: validate lock still available on start? configurable to release on sad path? (maybe from get/put params?)
+- slack notification: `put` to start msg, hooks reply to it with result? (kind of an abuse tho)
 
 # Proposal
 
@@ -48,29 +75,27 @@ type alias Metadata = List (String, String)
 -- Versioned Dependencies (Git repo)
 type alias Version = Dict String String
 
-check  : Config -> List Version
-get    : Config -> Version -> Bits
-put    : Config -> Bits -> Set Version
-delete : Config -> Bits -> Set Version
-
--- Spaces (BOSH deployments, Git branches, Git PRs, Semver trees)
 type alias Space = Dict String String
 
-check  : Config -> Set Space
-get    : Config -> Space -> Bits
-put    : Config -> Bits -> Set Space
-delete : Config -> Bits -> Set Space
+discover : Config -> Set Space
+check    : Config -> Space -> Maybe Version -> List Version
+get      : Config -> Space -> Version -> Bits
+put      : Config -> Space -> Bits -> Set Version
+destroy  : Config -> Space -> Bits -> Set Version
+
+notify : Config -> Notification -> ()
+
+-- Optional: runs whenever build status changes while using resource as input
+-- notify : Config -> Space -> Version -> Status -> ()
 
--- Notifications (slack, email)
 type alias BuildMetadata =
-  { buildStatus : String
-  , buildNumber : Int
-  , jobName : String
+  { teamName : String
   , pipelineName : String
-  , teamName : String
+  , jobName : String
+  , buildName : String
+  , buildID : Int
+  , status : String
   }
-
-notify : Config -> BuildMetadata -> ()
 ```
 
 
@@ -79,42 +104,17 @@ notify : Config -> BuildMetadata -> ()
 - Pull Requests
 - BOSH deploys
 - Feature branches
-- Arbitrary branches
+- Generated branches
 - IaaSes
+- Pool resource?
 
 
 ## Pull Requests
 
-```yaml
-space_types:
-- name: github-pr
-  type: docker-image
-  source: {repository: concourse/github-pr-space-type}
-
-spaces:
-- name: atc-prs
-  type: github-pr
-  source: {repository: concourse/atc}
-
-resources:
-- name: atc-pr
-  type: git
-  source: {uri: "https://github.com/concourse/atc"}
-  spaces: atc-prs
-
-jobs:
-- name: atc-pr-unit
-  plan:
-  - get: atc-pr
-    trigger: true
-    spaces: all
-  - task: unit
-    file: atc/ci/pr.yml
-```
-
 !! stateful spaces? hooks on build start/finish with state available
 !! this could be used to reflect PR status, send slack alerts, track pool entries
-
+!! these hooks might not just be notifications - it could be important (ie release lock on abort)
+!! need a way for `put` to create a new space
 
 
 ### Scheduling
diff --git a/00-resources-v2/prs.yml b/00-resources-v2/prs.yml
new file mode 100644
index 00000000..6a91818f
--- /dev/null
+++ b/00-resources-v2/prs.yml
@@ -0,0 +1,37 @@
+resource_types:
+- name: github-pr
+  type: docker-image
+  source: {repository: concourse/github-pr-resource}
+
+resources:
+- name: atc-pr
+  type: github-pr
+  source:
+    repository: concourse/atc
+    access_token: ((token))
+
+notifiers:
+- resource: atc-pr
+  follow: atc-pr
+
+jobs:
+- name: atc-pr-unit
+  plan:
+  - get: atc-pr
+    trigger: true
+    spaces: all
+  - task: unit
+    file: atc/ci/pr.yml
+
+# github-pr /info:
+{
+  "version": "2.0",
+  "artifacts": {
+    "check": "/opt/resource/check",
+    "in": "/opt/resource/in",
+    "out": "/opt/resource/out"
+  },
+  "notifier": {
+    "used_in_build": "/opt/resource/notify"
+  }
+}

From c0b36832ebbe28ffd1e5c1d1c1527a6e15865689 Mon Sep 17 00:00:00 2001
From: Alex Suraci <asuraci@pivotal.io>
Date: Tue, 10 Apr 2018 16:34:52 -0400
Subject: [PATCH 03/19] clean up proposal, add summary of changes

---
 00-resources-v2/bosh.yml                      |  57 ----
 00-resources-v2/proposal.md                   | 158 ----------
 .../branch-gen.yml                            |   8 +-
 .../commit-status.yml                         |  16 +-
 .../notifications.yml                         |  17 +-
 01-resources-v2/proposal.md                   | 285 ++++++++++++++++++
 {00-resources-v2 => 01-resources-v2}/prs.yml  |  20 +-
 7 files changed, 302 insertions(+), 259 deletions(-)
 delete mode 100644 00-resources-v2/bosh.yml
 delete mode 100644 00-resources-v2/proposal.md
 rename {00-resources-v2 => 01-resources-v2}/branch-gen.yml (72%)
 rename {00-resources-v2 => 01-resources-v2}/commit-status.yml (67%)
 rename {00-resources-v2 => 01-resources-v2}/notifications.yml (65%)
 create mode 100644 01-resources-v2/proposal.md
 rename {00-resources-v2 => 01-resources-v2}/prs.yml (55%)

diff --git a/00-resources-v2/bosh.yml b/00-resources-v2/bosh.yml
deleted file mode 100644
index 597ef566..00000000
--- a/00-resources-v2/bosh.yml
+++ /dev/null
@@ -1,57 +0,0 @@
-resource_types:
-- name: boshdir
-  type: docker-image
-  source: {repository: concourse/github-pr-space-type}
-
-resources:
-- name: director
-  type: boshdir
-  source:
-    target: bosh.concourse-ci.org
-    client_id: abc
-    client_secret: def
-
-- name: release
-  type: git
-  source:
-    uri: https://github.com/concourse/concourse
-    branch: master
-
-- name: deployment
-  type: git
-  source:
-    uri: https://github.com/concourse/concourse-bosh-deployment
-    branch: master
-
-jobs:
-- name: unit
-  plan:
-  - get: release
-    trigger: true
-  - task: unit
-    file: release/ci/unit.yml
-
-- name: deploy
-  plan:
-  - get: release
-    passed: [unit]
-    trigger: true
-  - get: deployment
-    trigger: true
-  - task: create-rc
-    file: release/ci/create.yml
-  - put: director
-    params:
-      manifest: deployment/cluster/concourse.yml
-      releases: [release-rc/*.tgz]
-
-- name: acceptance
-  plan:
-  - get: release
-    passed: [deploy]
-    trigger: true
-  - get: deployment
-    passed: [deploy]
-    trigger: true
-  - get: director
-    passed: [deploy]
diff --git a/00-resources-v2/proposal.md b/00-resources-v2/proposal.md
deleted file mode 100644
index 80bcca58..00000000
--- a/00-resources-v2/proposal.md
+++ /dev/null
@@ -1,158 +0,0 @@
-# Summary
-
-Introduces a new resource interface with the following goals:
-
-1. Support for spaces (concourse/concourse#1707).
-
-1. Marking versions as deleted.
-
-1. Tightening up loopholes in the API to ensure that resources are pointing at
-   an external source of truth and cannot be "partially implemented".
-
-1. Introduce versioning to the resource interface, so that we can maintain
-   backwards-compatibility.
-
-1. Establish a pattern for notification-style resources like Slack, GitHub
-   commit/PR status, etc.
-
-
-# Motivation
-
-Resources today are used for many things that they should not be used for. This
-leads to surprising behavior, difficult workarounds, and a lack of consistency
-in what you can expect from any given resource type.
-
-Many of them only implement a subset of the interface, or only provide stubs
-for part of it. This makes them not a true resource.
-
-We have a need in concourse/concourse#1707 to discover change over *space*, not
-*time*, which requires additions or changes to the resource interface. We also
-have
-
-
-## New Implications
-
-Here are a few use cases that resources were sometimes used for inappropriately:
-
-1. Resources that really only have a "current state", such as deployments. This
-  is still "change over time", but the difference is that old versions become
-  invalid as soon as there's a new one.
-
-1. Pushing un-versioned artifacts through a pipeline, such that the version
-  history for the resource becomes nonlinear (the "latest" version has nothing
-  to do with the versions prior). This is a cardinal sin.
-
-The new interface improves the story around these two use cases.
-
-The first case is resolved by having versions be deletable; we can now
-represent that previous states of an external dependency are no longer
-available.
-
-The second use case can be resolved by using a new space to represent nonlinear
-versions.
-
-
-## xx hooks use case
-
-- git: commit status
-- github pr: pr status
-
-- pool?: validate lock still available on start? configurable to release on sad path? (maybe from get/put params?)
-- slack notification: `put` to start msg, hooks reply to it with result? (kind of an abuse tho)
-
-# Proposal
-
-```elm
--- arbitrary configuration
-type alias Config = Dict String Json.Value
-
--- data on disk (it's not in memory, so blank return value)
-type alias Bits = ()
-
--- arbitrary metadata to show to the user (this may become more structured later)
-type alias Metadata = List (String, String)
-
--- Versioned Dependencies (Git repo)
-type alias Version = Dict String String
-
-type alias Space = Dict String String
-
-discover : Config -> Set Space
-check    : Config -> Space -> Maybe Version -> List Version
-get      : Config -> Space -> Version -> Bits
-put      : Config -> Space -> Bits -> Set Version
-destroy  : Config -> Space -> Bits -> Set Version
-
-notify : Config -> Notification -> ()
-
--- Optional: runs whenever build status changes while using resource as input
--- notify : Config -> Space -> Version -> Status -> ()
-
-type alias BuildMetadata =
-  { teamName : String
-  , pipelineName : String
-  , jobName : String
-  , buildName : String
-  , buildID : Int
-  , status : String
-  }
-```
-
-
-# Examples
-
-- Pull Requests
-- BOSH deploys
-- Feature branches
-- Generated branches
-- IaaSes
-- Pool resource?
-
-
-## Pull Requests
-
-!! stateful spaces? hooks on build start/finish with state available
-!! this could be used to reflect PR status, send slack alerts, track pool entries
-!! these hooks might not just be notifications - it could be important (ie release lock on abort)
-!! need a way for `put` to create a new space
-
-
-### Scheduling
-
-There should be a single "space combination" of `atc-pr-unit` for every active
-space returned by `atc-prs`. So, the `atc-prs` space must be periodically
-`check`ed to determine the set of PRs available, and the `atc-pr` resource in
-turn must run a `check` for every space.
-
-When any of the space combinations of `atc-pr-unit` has a new version
-available, I expect a build of `atc-pr-unit` to automatically kick off.
-
-1. The `github-pr` space type is implemented like so:
-  * `check` returns the set of PRs, e.g. `[{"remote": "pull/23/head"}]`
-  * `put` is used to update the PR status
-  * `get` returns metadata pertaining to the PR
-  * `delete` is unimplemented (maybe it closes the pr? not important here.)
-
-1. The `atc-prs` space is configured to detect the spaces of the
-  `concourse/atc` repo.
-
-1. The `atc-pr` resource is configured as a regular old `git` resource,
-  performing `checks` across all spaces returned by `atc-prs`.
-
-  * This works by having one *resource* `check` for each space returned by
-    `atc-prs`, by merging the space object `{"remote":"pull/23/head"}` with the
-    rest of the resource config.
-
-    The `git` resource would have to understand this config param to fetch the
-    given remote and emit the version.
-
-  * Because this resource is configured `across: atc-prs`, any use of the
-    resource must be space-aware.
-
-1. The `atc-pr-unit` job configures a `get` step of `atc-pr`. Because the
-  `atc-pr` resource is spatial, each space satisfying the `spaces: all` filter
-  results in a "space combination" for the job, each with independent
-  scheduling.
-
-
-### Runtime
diff --git a/00-resources-v2/branch-gen.yml b/01-resources-v2/branch-gen.yml
similarity index 72%
rename from 00-resources-v2/branch-gen.yml
rename to 01-resources-v2/branch-gen.yml
index 9cfc0281..214e1bd4 100644
--- a/00-resources-v2/branch-gen.yml
+++ b/01-resources-v2/branch-gen.yml
@@ -21,12 +21,10 @@ jobs:
       outputs:
       - name: generated-repo
         # has the generated code committed to the repo
-      - name: branch-space
-        # has a 'space' file with `{branch: gen-(some deterministic hash)}`
+      - name: branch-name
+        # has a 'name' file with `gen-(some deterministic hash)`
   - put: atc-gen
-    # dynamically determines the space; resource idempotently creates it
-    space: {load: gen-branch/space}
-    params: {repository: gen-repo}
+    params: {branch_name: branch-name/name, repository: gen-repo}
 
 - name: test
   plan:
diff --git a/00-resources-v2/commit-status.yml b/01-resources-v2/commit-status.yml
similarity index 67%
rename from 00-resources-v2/commit-status.yml
rename to 01-resources-v2/commit-status.yml
index f13f13d7..566cd958 100644
--- a/00-resources-v2/commit-status.yml
+++ b/01-resources-v2/commit-status.yml
@@ -8,12 +8,14 @@ resources:
   type: git
   source:
     uri: https://github.com/concourse/atc
+  notify:
+  - on: build_started
+    using: atc-status # invoke atc-status notifier with 'self' set to this resource
+  - on: build_finished
+    using: atc-status # invoke atc-status notifier with 'self' set to this resource
 
 - name: atc-status
   type: github-status
-  notify:
-  - on: used_in_build
-    follow: atc
   source:
     repository: concourse/atc
     access_token: ((token))
@@ -26,11 +28,3 @@ jobs:
     spaces: all
   - task: unit
     file: atc/ci/pr.yml
-
-# github-status /info:
-{
-  "version": "2.0",
-  "notifier": {
-    "used_in_build": "/opt/resource/notify"
-  }
-}
diff --git a/00-resources-v2/notifications.yml b/01-resources-v2/notifications.yml
similarity index 65%
rename from 00-resources-v2/notifications.yml
rename to 01-resources-v2/notifications.yml
index 6f6af50e..1305434a 100644
--- a/00-resources-v2/notifications.yml
+++ b/01-resources-v2/notifications.yml
@@ -14,10 +14,13 @@ resources:
   source:
     url: https://hooks.slack.com/services/XXXX
   notify:
-  - on: build_failed
+  - on: build_finished
     params:
-      template: |
-        Oh no! <{{.ExternalURL}}/builds/{{.BuildID}}|{{.BuildName}}> failed. :(
+      template:
+        succeeded: |
+          Yay! <{{.ExternalURL}}/builds/{{.BuildID}}|{{.BuildName}}> succeeded. :)
+        failed: |
+          Oh no! <{{.ExternalURL}}/builds/{{.BuildID}}|{{.BuildName}}> failed. :(
 
 jobs:
 - name: atc-pr-unit
@@ -27,11 +30,3 @@ jobs:
     spaces: all
   - task: unit
     file: atc/ci/pr.yml
-
-# github-status /info:
-{
-  "version": "2.0",
-  "notifier": {
-    "build_failed": "/opt/resource/notify"
-  }
-}
diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
new file mode 100644
index 00000000..adc15ad2
--- /dev/null
+++ b/01-resources-v2/proposal.md
@@ -0,0 +1,285 @@
+# Summary
+
+Introduces a new resource interface with the following goals:
+
+* Introduce versioning to the resource interface, so that we can maintain
+  backwards-compatibility.
+
+* Support for spaces (concourse/concourse#1707).
+
+* Introduce a more airtight "versioned artifacts" interface, tightening up
+  loopholes in today's resource API to ensure that resources are pointing at an
+  external source of truth and cannot be partially implemented or hacky.
+
+* Introduce a "notifier" interface. This is to replace a whole class of resource
+  types that will not be able to fit in to an "artifact" interface.
+
+* Extend the versioned artifacts interface to support deletion of versions,
+  either by an explicit `delete` call or by somehow noticing that versions have
+  disappeared.
+
+
+# Proposal
+
+At this early stage of the RFC, it's easiest for me to just use Elm syntax and
+pretend this is all in a type system.
+
+In reality, each of these functions would be scripts with JSON requests passed
+to them on stdin. I'm going to use the `Bits` type just to represent which calls
+have state on disk to either access (like `put`) or return (like `get`). This
+would normally just be the working directory of the script.
+
+## General Types
+
+```elm
+-- arbitrary configuration
+type alias Config = Dict String Json.Value
+
+-- identifier for space, i.e. {"branch":"foo"}
+type alias Space = Dict String String
+
+-- identifier for version, i.e. {"version":"1.2"}
+type alias Version = Dict String String
+
+-- arbitrary ordered metadata (we may make this fancier in the future)
+type alias Metadata = List (String, String)
+
+-- data on disk
+type alias Bits = ()
+```
+
+## Versioned Artifacts interface
+
+```elm
+spaces  : Config -> Set Space
+check   : Config -> Space -> Maybe Version -> List (Version, Metadata)
+get     : Config -> Space -> Version -> Bits
+put     : Config -> Bits -> (Space, Set Version)
+destroy : Config -> Bits -> Dict Space (Set Version)
+```
+
+## Notifications interface
+
+```elm
+notify : Config -> Notification -> ()
+
+type Notification
+  = BuildStarted
+      { build : BuildMetadata
+      , self : Maybe BuildInput
+      }
+  | BuildFinished
+      { build : BuildMetadata
+      , status : Status
+      , self : Maybe BuildInput
+      }
+
+type alias BuildMetadata =
+  { teamName : String
+  , pipelineName : String
+  , jobName : String
+  , buildName : String
+  , buildID : Int
+  , status : String
+  }
+
+type alias BuildInput =
+  { space : Space
+  , version : Version
+  }
+```
+
+
+# Examples
+
+TODO:
+
+- Pull Requests
+- Feature branches
+- Build matrixes
+- Generating branches (and propagating them downstream)
+- Semver artifacts
+- Fanning out against multiple IaaSes
+- Pool resource?
+- BOSH deploys
+
+
+# Summary of Changes
+
+## Overarching Changes
+
+* Add a `info` script which prints the resource's API version, e.g.
+  `{"version":"2.0"}`. This will start at `2.0`. If `/info` does not exist we'll
+  execute today's resource interface behavior.
+
+* Rather than running `/opt/resource/X`, discover the supported resource APIs by
+  invoking `info`. This allows us to be more flexible in what kinds of resources
+  we can support (versioned artifacts, notifications, ???), and where the
+  scripts may live (`/opt/resource` is very Linux specific).
+
+* Today's resource interface (`/in`, `/out`, `/check`) becomes more specifically
+  a "versioned artifacts" or just "artifacts" resource interface.
+
+* Introduction of some sort of schema validation for resource configuration.
+
+* Remove the distinction between `source` and `params`; resources will receive a
+  single `config`. The distinction will remain in the pipeline. This makes it
+  easier to implement a resource without planning ahead for interesting dynamic
+  vs. static usage patterns, and will get more powerful with #684.
+
+
+## Changes to Versioned Artifact resources
+
+* Add a `spaces` actopm, which is used to discover spaces.
+
+  It's not a verb, partly because most verbs would be ambiguous with `check`,
+  and partly because the API is practically stateless; it just returns whatever
+  spaces there are.
+
+* Add a `destroy` action, which looks like `put` (can be given files and params
+  to determine what to destroy) but returns the set of versions that it
+  deleted.
+
+  This is to support bulk deletes, e.g. to garbage collect intermediate
+  artifacts after a final build is shipped.
+
+* Change `check` and `get` to always run against a particular space, given by
+  the request payload.
+
+* Change `check` to include metadata for each version. Change `get` and `put` to
+  no longer return it.
+
+  This way metadata is always immediately available, and only comes from one
+  place.
+
+  The original thought was that metadata collection may be expensive, but so far
+  we haven't seen that to be the case.
+
+* Change `get` script to no longer return a version, since it's always given one
+  now. As a result, `get` no longer has a response; it just succeeds or fails.
+
+* Change `get` and `put` to run with the bits as their working directory, rather
+  than taking the path as an argument. This was something people would trip up
+  on when implementing a resource.
+
+* Change `put` to emit a list of versions, rather than just one.
+
+  TODO: it's undecided whether `put` should be given a single space, return a
+  single space, or return many spaces + versions.
+
+  Technically the `git` resource may push many commits, so this is necessary to
+  track them all as outputs of a build. This could also support batch creation.
+
+  To ensure `check` is the source of truth for ordering, the versions returned
+  by `put` are not order dependent. A `check` will be performed to discover
+  them in the correct order, and then each version will be saved as an output
+  of the build. The latest version of the set will then be fetched.
+
+* Change `put` to write its JSON response to a specified file, rather than
+  `stdout`, so that we don't have to be attached to process its response.
+
+  This is one of the few ways a build can error after the ATC reattaches
+  (`unexpected end of JSON`). With it written to a file, we can just try to read
+  the file when we re-attach after seeing that the process exited. This also
+  frees up stdout/stderr for normal logging, which has been an occasional
+  pitfall during resource development/debugging.
+
+
+## Introduction of "notifier" resource interface
+
+A resource may now implement a "notifier" interface. Notifications must be
+explicitly enabled in the pipeline. This is so that a resource type can't
+suddenly decide to include a notifier and start doing surprising things.
+
+A notifier has a single hook, `notify`, and receives some sort of JSON payload
+describing the event. The event is purely for side-effects and has no
+implications for pipeline semantics.
+
+If a notification fails to send, the build shall error. To be honest, I haven't
+thought about this much yet, but I think it's better to be conservative.
+
+A resource type may implement both "artifacts" and "notifier" interfaces. If a
+resource implements both, it will be given information about its occurrence in
+the build (space and version). This will be crucial for e.g. reporting the
+status of a pull request or commit back to GitHub.
+
+Add support for a "notifier" resource type. They have one hook, `notify`,
+which is invoked with various types of notifications
+(`build_started`, `build_finished`, as the poster children).
+
+
+# Caveats
+
+Terminology is now even more confusing. "Resource type" could mean either the
+"git" vs. "s3" or "notifier" vs. "artifact".
+
+
+# Open Questions
+
+## Can we reduce the `check` overhead?
+
+With spaces there will be more `check`s than ever. Right now, there's one
+container per recurring `check`. Can we reduce the container overhead here by
+requiring that resource `check`s be side-effect free and able to run in parallel?
+
+There may be substantial security implications for this.
+
+
+## Is `destroy` general enough to be a part of the interface?
+
+It may be the case that most resources cannot easily support `destroy`. One
+example is the `git` resource. It doesn't really make sense to `destroy` a
+commit. Even if it did (`push -f`?), it's a kind of weird workflow to support
+out of the box.
+
+Could we instead just have `put` and ensure that we `check` in such a way that
+deleted versions are automatically noticed? What would the overhead of this be?
+
+
+## Should `put` be given a space or return the space?
+
+The verb `PUT` in HTTP implies an idempotent action against a given resource. So
+it's intuitive that the `put` verb here would do the same.
+
+However, many of today's usage of `put` would be against a dynamically
+determined space. For example, most semver workflows involve `put`ing with the
+version determined by a file (often coming from the `semver` resource). So the
+space isn't known statically at pipeline configuration time.
+
+What's more, the resulting space for a semver push would only be `MAJOR.MINOR`,
+excluding the final patch segment. This is annoying to have to explicitly
+configure in your build.
+
+If we instead have `put` return both the space and the versions, this would be a
+lot simpler.
+
+
+## Should a notifier resource be able to "observe" another resource?
+
+This would allow e.g. a generic `git` resource and specific
+`github-commit-status` resource type rather than having to bake them all
+together.
+
+The downside here is that it would be introducing cross-resource-type interface
+contracts. Resource A would have to understand resource B's versions/spaces.
+This gets more possible with schema verification but still feels risky.
+
+
+# New Implications
+
+Here are a few use cases that resources were sometimes used for inappropriately:
+
+## Single-state resources
+
+Resources that really only have a "current state", such as deployments. This is
+still "change over time", but the difference is that old versions become invalid
+as soon as there's a new one.
+
+These can now be done by always marking the old versions as "deleted".
+
+
+## Non-linearly versioned artifact storage
+
+This can be done by representing each non-linear version in a separate space.
+For example, generated code could be pushed to a generated (but deterministic)
+branch name, and that space could then be passed along.
diff --git a/00-resources-v2/prs.yml b/01-resources-v2/prs.yml
similarity index 55%
rename from 00-resources-v2/prs.yml
rename to 01-resources-v2/prs.yml
index 6a91818f..7cc47579 100644
--- a/00-resources-v2/prs.yml
+++ b/01-resources-v2/prs.yml
@@ -9,10 +9,9 @@ resources:
   source:
     repository: concourse/atc
     access_token: ((token))
-
-notifiers:
-- resource: atc-pr
-  follow: atc-pr
+  notify:
+  - on: build_started
+  - on: build_finished
 
 jobs:
 - name: atc-pr-unit
@@ -22,16 +21,3 @@ jobs:
     spaces: all
   - task: unit
     file: atc/ci/pr.yml
-
-# github-pr /info:
-{
-  "version": "2.0",
-  "artifacts": {
-    "check": "/opt/resource/check",
-    "in": "/opt/resource/in",
-    "out": "/opt/resource/out"
-  },
-  "notifier": {
-    "used_in_build": "/opt/resource/notify"
-  }
-}

From 8e853a51e5ffb13b3cbc07b6cbe858c76bab8e4b Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Tue, 29 May 2018 20:20:38 +0000
Subject: [PATCH 04/19] fix typos

---
 01-resources-v2/branch-gen.yml | 2 +-
 01-resources-v2/proposal.md    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/01-resources-v2/branch-gen.yml b/01-resources-v2/branch-gen.yml
index 214e1bd4..d4368d44 100644
--- a/01-resources-v2/branch-gen.yml
+++ b/01-resources-v2/branch-gen.yml
@@ -24,7 +24,7 @@ jobs:
       - name: branch-name
         # has a 'name' file with `gen-(some deterministic hash)`
   - put: atc-gen
-    params: {branch_name: branch-name/name, repository: gen-repo}
+    params: {branch_name: branch-name/name, repository: generated-repo}
 
 - name: test
   plan:
diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index adc15ad2..07b27e25 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -130,7 +130,7 @@ TODO:
 
 ## Changes to Versioned Artifact resources
 
-* Add a `spaces` actopm, which is used to discover spaces.
+* Add a `spaces` action, which is used to discover spaces.
 
   It's not a verb, partly because most verbs would be ambiguous with `check`,
   and partly because the API is practically stateless; it just returns whatever

From 217f5754433de8a702e46c1fb8a90621bb6c7816 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Mon, 4 Jun 2018 19:25:42 +0000
Subject: [PATCH 05/19] next iteration on v2 interface, add git example

* removed the `destroy` action. instead, `put` returns a set of versions
  that were deleted, alongside the set of versions that were created.
* removed the `spaces` action. instead, `check` now runs in batch across
  all spaces.
* `check` now returns *all versions*, both on the initial call and if
  the given versions are no longer present (push -f). this will be a
  problem for resources with lots of history - maybe either streaming or
  pagination would help in the future?
* space identifiers are back to normal strings, rather than JSON
  objects. i'm not 100% convinced of this but it definitely makes it
  easier to use them as keys in maps, and should be cleaner in the UI
  and in YAML.
---
 01-resources-v2/git-example/.gitignore   |   2 +
 01-resources-v2/git-example/Gemfile      |   4 +
 01-resources-v2/git-example/Gemfile.lock |  19 +++
 01-resources-v2/git-example/README.md    |  13 ++
 01-resources-v2/git-example/artifact     | 104 ++++++++++++
 01-resources-v2/git-example/info         |  13 ++
 01-resources-v2/proposal.md              | 203 ++++++++++++++---------
 7 files changed, 279 insertions(+), 79 deletions(-)
 create mode 100644 01-resources-v2/git-example/.gitignore
 create mode 100644 01-resources-v2/git-example/Gemfile
 create mode 100644 01-resources-v2/git-example/Gemfile.lock
 create mode 100644 01-resources-v2/git-example/README.md
 create mode 100755 01-resources-v2/git-example/artifact
 create mode 100755 01-resources-v2/git-example/info

diff --git a/01-resources-v2/git-example/.gitignore b/01-resources-v2/git-example/.gitignore
new file mode 100644
index 00000000..bbb00daa
--- /dev/null
+++ b/01-resources-v2/git-example/.gitignore
@@ -0,0 +1,2 @@
+check-repo
+dot
diff --git a/01-resources-v2/git-example/Gemfile b/01-resources-v2/git-example/Gemfile
new file mode 100644
index 00000000..8e376927
--- /dev/null
+++ b/01-resources-v2/git-example/Gemfile
@@ -0,0 +1,4 @@
+source :rubygems
+
+gem 'git'
+gem 'pry'
diff --git a/01-resources-v2/git-example/Gemfile.lock b/01-resources-v2/git-example/Gemfile.lock
new file mode 100644
index 00000000..cc170e92
--- /dev/null
+++ b/01-resources-v2/git-example/Gemfile.lock
@@ -0,0 +1,19 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    coderay (1.1.2)
+    git (1.4.0)
+    method_source (0.9.0)
+    pry (0.11.3)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  git
+  pry
+
+BUNDLED WITH
+   1.16.2
diff --git a/01-resources-v2/git-example/README.md b/01-resources-v2/git-example/README.md
new file mode 100644
index 00000000..cc5b51f8
--- /dev/null
+++ b/01-resources-v2/git-example/README.md
@@ -0,0 +1,13 @@
+# Git Resource v2
+
+This implementation is done in Ruby using the `git` gem. I chose Ruby
+over Bash because having a real language with more accessible data
+structures is probably going to be more important with this new
+interface, and Ruby feels pretty well suited (really just need a bit more
+than Bash).
+
+Please leave comments on parts you like/don't like! But bear in mind the
+goal here isn't necessarily the prettiness of the code, it's to see what
+kinds of things the resource has to do. I'll be using Ruby purely as a
+scripting language, hacking things together where needed in the interest
+of brevity.
diff --git a/01-resources-v2/git-example/artifact b/01-resources-v2/git-example/artifact
new file mode 100755
index 00000000..94056188
--- /dev/null
+++ b/01-resources-v2/git-example/artifact
@@ -0,0 +1,104 @@
+#!/usr/bin/env ruby
+
+require "json"
+require "git"
+
+# $request = JSON.parse(STDIN.read, symbolize_names: true)
+
+def commit_versions(log)
+  log.collect do |c|
+    {
+      version: {ref: c.sha},
+      metadata: [
+        {name: "author", value: c.author.name},
+        {name: "author_date", value: c.author_date},
+        {name: "commit", value: c.sha},
+        {name: "committer", value: c.committer.name},
+        {name: "committer_date", value: c.committer_date},
+        {name: "message", value: c.message}
+      ]
+    }
+  end
+end
+
+case ARGV[0]
+when "check"
+  $request = {
+    config: {uri: "https://github.com/vito/booklit"},
+    from: {master: {ref: "40bc6986197e411471d306bb8eb3a21c5b5f9d26"}}
+  }
+
+  git =
+    if Dir.exists?("check-repo")
+      Git.open("check-repo").tap(&:fetch)
+    else
+      Git.clone($request[:config][:uri], "check-repo")
+    end
+
+  spaces = []
+  git.branches.local.each do |b|
+    # skip "default branch" entry
+    next if b.name =~ /HEAD ->/
+
+    b.checkout
+
+    paths = $request[:config][:paths] || ["."]
+    paths += ($request[:config][:ignore_paths] || []).collect { |p| ":!" + p }
+
+    log = git.log(nil).path(paths)
+
+    # TODO: this will get *all* commits and load it into memory, which is
+    # probably a bad idea. the linux repo for example has 710k+ commits.
+    #
+    # should this be paginated? or should it stream each version back to the
+    # caller somehow so everything doesn't have to be sucked into memory?
+    #
+    # TODO: when checking from a given version, should the given version be
+    # returned? this was done in the original API so that you could run
+    # `check-resource -f <version>`, but in this new world where we just always
+    # collect all versions, that won't be necessary. in fact, `check-resource
+    # -f <version>` would need to be given a space that it's checking against.
+    commits =
+      if version = $request[:from][b.name]
+        begin
+          commit_versions(log.between(version[:ref]))
+        rescue
+          # bad ref; emit all versions
+          commit_versions(log)
+        end
+      else
+        commit_versions(log)
+      end
+
+    $stderr.puts "#{b.name}: #{commits.length} commits"
+
+    spaces << {
+      space: {branch: b.name},
+      versions: commits
+    }
+  end
+
+  response = JSON.dump(spaces)
+  puts response
+  puts response.size
+when "get"
+  $request = {
+    config: {uri: "https://github.com/vito/booklit"},
+    space: {branch: "master"},
+    version: {ref: "f828f2758256b0e93dc3c101f75604efe92ca07e"}
+  }
+
+  git =
+    Git.clone($request[:config][:uri], "dot", # TODO: irl this would be '.'
+              branch: $request[:space][:branch],
+              recursive: true)
+
+  git.checkout($request[:version][:ref])
+
+  # TODO: draw the rest of the owl
+  #
+  # most of this is uninteresting.
+
+when "put"
+  puts "putting"
+end
diff --git a/01-resources-v2/git-example/info b/01-resources-v2/git-example/info
new file mode 100755
index 00000000..2b4a580b
--- /dev/null
+++ b/01-resources-v2/git-example/info
@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+
+require "json"
+
+puts JSON.dump({
+  version: "2.0"
+
+  artifacts: {
+    check: "artifact check",
+    get: "artifact get",
+    put: "artifact put"
+  }
+})
diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 07b27e25..0e92f7ed 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -35,8 +35,8 @@ would normally just be the working directory of the script.
 -- arbitrary configuration
 type alias Config = Dict String Json.Value
 
--- identifier for space, i.e. {"branch":"foo"}
-type alias Space = Dict String String
+-- identifier for space, i.e. 'foo' or '1.2'
+type alias Space = String
 
 -- identifier for version, i.e. {"version":"1.2"}
 type alias Version = Dict String String
@@ -51,11 +51,9 @@ type alias Bits = ()
 ## Versioned Artifacts interface
 
 ```elm
-spaces  : Config -> Set Space
-check   : Config -> Space -> Maybe Version -> List (Version, Metadata)
+check   : Config -> Dict Space Version -> Dict Space (List (Version, Metadata))
 get     : Config -> Space -> Version -> Bits
-put     : Config -> Bits -> (Space, Set Version)
-destroy : Config -> Bits -> Dict Space (Set Version)
+put     : Config -> Bits -> Dict Space { created : Set Version, deleted : Set Version }
 ```
 
 ## Notifications interface
@@ -92,6 +90,16 @@ type alias BuildInput =
 
 # Examples
 
+## Resource Implementations
+
+I've started implementing a new `git` resource alongside this
+document. See
+[`git-example/`](https://github.com/vito/rfcs/tree/resources-v2/01-resources-v2/git-example).
+I've left `TODO`s for parts that need more thinking or discussion. Please
+leave comments!
+
+## Pipeline Usage
+
 TODO:
 
 - Pull Requests
@@ -130,58 +138,60 @@ TODO:
 
 ## Changes to Versioned Artifact resources
 
-* Add a `spaces` action, which is used to discover spaces.
+* Change `check` to run against all spaces. It will be given a mapping of each
+  space to its current latest version, and return the set of all spaces, along
+  with any new versions in each space.
 
-  It's not a verb, partly because most verbs would be ambiguous with `check`,
-  and partly because the API is practically stateless; it just returns whatever
-  spaces there are.
+  This is all done as one batch call so that resources can decide how to
+  efficiently perform the check. It also keeps the container overhead down to
+  one per resource, rather than one per space.
 
-* Add a `destroy` action, which looks like `put` (can be given files and params
-  to determine what to destroy) but returns the set of versions that it
-  deleted.
+* Change `put` to emit a set of created versions for each space, rather than
+  just one.
 
-  This is to support bulk deletes, e.g. to garbage collect intermediate
-  artifacts after a final build is shipped.
+  Technically the `git` resource may push many commits, so returning more than
+  one version is necessary to track them all as outputs of a build. This could
+  also support batch creation.
 
-* Change `check` and `get` to always run against a particular space, given by
-  the request payload.
-
-* Change `check` to include metadata for each version. Change `get` and `put` to
-  no longer return it.
+  To ensure `check` is the source of truth for ordering, the versions returned
+  by `put` are not order dependent. A `check` will be performed to discover
+  them in the correct order, and then each version will be saved as an output
+  of the build. The latest version of the set will then be fetched.
 
-  This way metadata is always immediately available, and only comes from one
-  place.
+* Change `put` to additionally return a set of *deleted* versions.
 
-  The original thought was that metadata collection may be expensive, but so far
-  we haven't seen that to be the case.
+  There has long been a call for a batch `delete` or `destroy` action. Adding
+  this to `put` alongside the set of created versions allows `put` to become a
+  general idempotent side-effect performer, rather than implying that each
+  resource must support a separate `delete` action.
 
-* Change `get` script to no longer return a version, since it's always given one
-  now. As a result, `get` no longer has a response; it just succeeds or fails.
+* Change `get` to always run against a particular space, given by
+  the request payload.
 
-* Change `get` and `put` to run with the bits as their working directory, rather
-  than taking the path as an argument. This was something people would trip up
-  on when implementing a resource.
+* Change `check` to include metadata for each version. Change `get` and `put`
+  to no longer return it.
 
-* Change `put` to emit a list of versions, rather than just one.
+  This way metadata is always immediately available, and only comes from one
+  place.
 
-  TODO: it's undecided whether `put` should be given a single space, return a
-  single space, or return many spaces + versions.
+  The original thought was that metadata collection may be expensive, but so
+  far we haven't seen that to be the case.
 
-  Technically the `git` resource may push many commits, so this is necessary to
-  track them all as outputs of a build. This could also support batch creation.
+* Change `get` script to no longer return a version, since it's always given
+  one now. As a result, `get` no longer has a response; it just succeeds or
+  fails.
 
-  To ensure `check` is the source of truth for ordering, the versions returned
-  by `put` are not order dependent. A `check` will be performed to discover
-  them in the correct order, and then each version will be saved as an output
-  of the build. The latest version of the set will then be fetched.
+* Change `get` and `put` to run with the bits as their working directory,
+  rather than taking the path as an argument. This was something people would
+  trip up on when implementing a resource.
 
 * Change `put` to write its JSON response to a specified file, rather than
   `stdout`, so that we don't have to be attached to process its response.
 
   This is one of the few ways a build can error after the ATC reattaches
-  (`unexpected end of JSON`). With it written to a file, we can just try to read
-  the file when we re-attach after seeing that the process exited. This also
-  frees up stdout/stderr for normal logging, which has been an occasional
+  (`unexpected end of JSON`). With it written to a file, we can just try to
+  read the file when we re-attach after seeing that the process exited. This
+  also frees up stdout/stderr for normal logging, which has been an occasional
   pitfall during resource development/debugging.
 
 
@@ -216,44 +226,6 @@ Terminology is now even more confusing. "Resource type" could mean either the
 
 # Open Questions
 
-## Can we reduce the `check` overhead?
-
-With spaces there will be more `check`s than ever. Right now, there's one
-container per recurring `check`. Can we reduce the container overhead here by
-requiring that resource `check`s be side-effect free and able to run in parallel?
-
-There may be substantial security implications for this.
-
-
-## Is `destroy` general enough to be a part of the interface?
-
-It may be the case that most resources cannot easily support `destroy`. One
-example is the `git` resource. It doesn't really make sense to `destroy` a
-commit. Even if it did (`push -f`?), it's a kind of weird workflow to support
-out of the box.
-
-Could we instead just have `put` and ensure that we `check` in such a way that
-deleted versions are automatically noticed? What would the overhead of this be?
-
-
-## Should `put` be given a space or return the space?
-
-The verb `PUT` in HTTP implies an idempotent action against a given resource. So
-it's intuitive that the `put` verb here would do the same.
-
-However, many of today's usage of `put` would be against a dynamically
-determined space. For example, most semver workflows involve `put`ing with the
-version determined by a file (often coming from the `semver` resource). So the
-space isn't known statically at pipeline configuration time.
-
-What's more, the resulting space for a semver push would only be `MAJOR.MINOR`,
-excluding the final patch segment. This is annoying to have to explicitly
-configure in your build.
-
-If we instead have `put` return both the space and the versions, this would be a
-lot simpler.
-
-
 ## Should a notifier resource be able to "observe" another resource?
 
 This would allow e.g. a generic `git` resource and specific
@@ -265,6 +237,79 @@ contracts. Resource A would have to understand resource B's versions/spaces.
 This gets more possible with schema verification but still feels risky.
 
 
+# Answered(?) Questions
+
+<details><summary>Can we reduce the `check` overhead?</summary>
+
+<p>
+~~With spaces there will be more `check`s than ever. Right now, there's one
+container per recurring `check`. Can we reduce the container overhead here by
+requiring that resource `check`s be side-effect free and able to run in
+parallel?~~
+</p>
+
+<p>
+~~There may be substantial security implications for this.~~
+</p>
+
+<p>
+This is now done as one big `check` across all spaces, run in a single
+container. Resources can choose how to perform this efficiently and safely.
+This may mean GraphQL requests or just iterating over local shared state in
+series. Even in the worst-case, where no parallelism is involved, it will at
+least consume only one container.
+</p>
+</details>
+
+<details><summary>Is `destroy` general enough to be a part of the interface?</summary>
+
+<p>
+~~It may be the case that most resources cannot easily support `destroy`. One
+example is the `git` resource. It doesn't really make sense to `destroy` a
+commit. Even if it did (`push -f`?), it's a kind of weird workflow to support
+out of the box.~~
+</p>
+
+<p>
+~~Could we instead just have `put` and ensure that we `check` in such a way that
+deleted versions are automatically noticed? What would the overhead of this
+be?~~ This only works if the versions are "chained", as with the `git` case.
+</p>
+
+<p>
+Decided against introducing `destroy` in favor of having `put` return two sets
+for each space: versions created and versions deleted. This generalizes `put`
+into an idempotent versioned artifact side effect performer.
+</p>
+</details>
+
+<details><summary>Should `put` be given a space or return the space?</summary>
+
+<p>
+~~The verb `PUT` in HTTP implies an idempotent action against a given resource. So
+it's intuitive that the `put` verb here would do the same.~~
+</p>
+<p>
+~~However, many of today's usage of `put` would be against a dynamically
+determined space. For example, most semver workflows involve `put`ing with the
+version determined by a file (often coming from the `semver` resource). So the
+space isn't known statically at pipeline configuration time.~~
+</p>
+<p>
+~~What's more, the resulting space for a semver push would only be `MAJOR.MINOR`,
+excluding the final patch segment. This is annoying to have to explicitly
+configure in your build.~~
+</p>
+<p>
+~~If we instead have `put` return both the space and the versions, this would be a
+lot simpler.~~
+</p>
+<p>
+Answered this at the same time as having `put` return a set of deleted
+versions. It'll return multiple spaces and versions created/deleted for them.
+</p>
+</details>
+
 # New Implications
 
 Here are a few use cases that resources were sometimes used for inappropriately:

From 5551f0398d5ccb3b14a4cee5a831395cc70e66c4 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Mon, 4 Jun 2018 20:26:44 +0000
Subject: [PATCH 06/19] fix inconsistency with 'get' action in example

space is a string, not a JSON object
---
 01-resources-v2/git-example/artifact | 4 ++--
 01-resources-v2/proposal.md          | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/01-resources-v2/git-example/artifact b/01-resources-v2/git-example/artifact
index 94056188..c689bc8a 100755
--- a/01-resources-v2/git-example/artifact
+++ b/01-resources-v2/git-example/artifact
@@ -84,13 +84,13 @@ when "check"
 when "get"
   $request = {
     config: {uri: "https://github.com/vito/booklit"},
-    space: {branch: "master"},
+    space: "master",
     version: {ref: "f828f2758256b0e93dc3c101f75604efe92ca07e"}
   }
 
   git =
     Git.clone($request[:config][:uri], "dot", # TODO: irl this would be '.'
-              branch: $request[:space][:branch],
+              branch: $request[:space],
               recursive: true)
 
   git.checkout($request[:version][:ref])
diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 0e92f7ed..3543c66a 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -310,6 +310,7 @@ versions. It'll return multiple spaces and versions created/deleted for them.
 </p>
 </details>
 
+
 # New Implications
 
 Here are a few use cases that resources were sometimes used for inappropriately:

From 251c645432fc6534fc7f8dbd466a38c51bff1931 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Tue, 12 Jun 2018 16:49:17 -0400
Subject: [PATCH 07/19] remove notification stuff from proposal

also update examples to use simple string space identifiers, and clarify
some of the change summaries
---
 01-resources-v2/branch-gen.yml       |   4 +-
 01-resources-v2/commit-status.yml    |  30 ------
 01-resources-v2/git-example/artifact |  10 +-
 01-resources-v2/git-example/info     |   3 +-
 01-resources-v2/proposal.md          | 134 ++++++++-------------------
 01-resources-v2/prs.yml              |   5 +-
 6 files changed, 51 insertions(+), 135 deletions(-)
 delete mode 100644 01-resources-v2/commit-status.yml

diff --git a/01-resources-v2/branch-gen.yml b/01-resources-v2/branch-gen.yml
index d4368d44..c0ea55c4 100644
--- a/01-resources-v2/branch-gen.yml
+++ b/01-resources-v2/branch-gen.yml
@@ -2,7 +2,7 @@ resources:
 - name: atc
   type: git
   source: {uri: https://github.com/concourse/atc}
-  space: {branch: master}
+  space: master
 
 - name: atc-gen
   type: git
@@ -33,7 +33,7 @@ jobs:
     trigger: true
   - get: atc-gen
     passed: [gen]
-    spaces: [{branch: gen-*}]
+    spaces: [gen-*]
     trigger: true
   - task: test
     file: atc/ci/test.yml
diff --git a/01-resources-v2/commit-status.yml b/01-resources-v2/commit-status.yml
deleted file mode 100644
index 566cd958..00000000
--- a/01-resources-v2/commit-status.yml
+++ /dev/null
@@ -1,30 +0,0 @@
-resource_types:
-- name: github-status
-  type: docker-image
-  source: {repository: concourse/github-status-resource}
-
-resources:
-- name: atc
-  type: git
-  source:
-    uri: https://github.com/concourse/atc
-  notify:
-  - on: build_started
-    using: atc-status # invoke atc-status notifier with 'self' set to this resource
-  - on: build_finished
-    using: atc-status # invoke atc-status notifier with 'self' set to this resource
-
-- name: atc-status
-  type: github-status
-  source:
-    repository: concourse/atc
-    access_token: ((token))
-
-jobs:
-- name: atc-pr-unit
-  plan:
-  - get: atc-pr
-    trigger: true
-    spaces: all
-  - task: unit
-    file: atc/ci/pr.yml
diff --git a/01-resources-v2/git-example/artifact b/01-resources-v2/git-example/artifact
index c689bc8a..7532f33e 100755
--- a/01-resources-v2/git-example/artifact
+++ b/01-resources-v2/git-example/artifact
@@ -25,7 +25,10 @@ case ARGV[0]
 when "check"
   $request = {
     config: {uri: "https://github.com/vito/booklit"},
-    from: {master: {ref: "40bc6986197e411471d306bb8eb3a21c5b5f9d26"}}
+    from: {
+      master: {ref: "40bc6986197e411471d306bb8eb3a21c5b5f9d26"},
+      travis: {ref: "8b0a526e233b1599f8d5fcc1179e0ec7642acd90"}
+    }
   }
 
   git =
@@ -58,6 +61,9 @@ when "check"
     # `check-resource -f <version>`, but in this new world where we just always
     # collect all versions, that won't be necessary. in fact, `check-resource
     # -f <version>` would need to be given a space that it's checking against.
+    #
+    # TODO: how does the caller know if the given version no longer exists, and
+    # that the version history should be reset?
     commits =
       if version = $request[:from][b.name]
         begin
@@ -73,7 +79,7 @@ when "check"
     $stderr.puts "#{b.name}: #{commits.length} commits"
 
     spaces << {
-      space: {branch: b.name},
+      space: b.name,
       versions: commits
     }
   end
diff --git a/01-resources-v2/git-example/info b/01-resources-v2/git-example/info
index 2b4a580b..668a79fd 100755
--- a/01-resources-v2/git-example/info
+++ b/01-resources-v2/git-example/info
@@ -3,9 +3,8 @@
 require "json"
 
 puts JSON.dump({
-  version: "2.0"
-
   artifacts: {
+    version: "2.0",
     check: "artifact check",
     get: "artifact get",
     put: "artifact put"
diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 3543c66a..d8dde7cb 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -11,9 +11,6 @@ Introduces a new resource interface with the following goals:
   loopholes in today's resource API to ensure that resources are pointing at an
   external source of truth and cannot be partially implemented or hacky.
 
-* Introduce a "notifier" interface. This is to replace a whole class of resource
-  types that will not be able to fit in to an "artifact" interface.
-
 * Extend the versioned artifacts interface to support deletion of versions,
   either by an explicit `delete` call or by somehow noticing that versions have
   disappeared.
@@ -56,38 +53,6 @@ get     : Config -> Space -> Version -> Bits
 put     : Config -> Bits -> Dict Space { created : Set Version, deleted : Set Version }
 ```
 
-## Notifications interface
-
-```elm
-notify : Config -> Notification -> ()
-
-type Notification
-  = BuildStarted
-      { build : BuildMetadata
-      , self : Maybe BuildInput
-      }
-  | BuildFinished
-      { build : BuildMetadata
-      , status : Status
-      , self : Maybe BuildInput
-      }
-
-type alias BuildMetadata =
-  { teamName : String
-  , pipelineName : String
-  , jobName : String
-  , buildName : String
-  , buildID : Int
-  , status : String
-  }
-
-type alias BuildInput =
-  { space : Space
-  , version : Version
-  }
-```
-
-
 # Examples
 
 ## Resource Implementations
@@ -116,28 +81,30 @@ TODO:
 
 ## Overarching Changes
 
-* Add a `info` script which prints the resource's API version, e.g.
-  `{"version":"2.0"}`. This will start at `2.0`. If `/info` does not exist we'll
-  execute today's resource interface behavior.
+* Add an `info` script which returns a JSON object indicating the supported
+  interfaces, their protocol versions, and any other interface-specific
+  meta-configuration (for example, which commands to execute for the
+  interface's hooks).
 
-* Rather than running `/opt/resource/X`, discover the supported resource APIs by
-  invoking `info`. This allows us to be more flexible in what kinds of resources
-  we can support (versioned artifacts, notifications, ???), and where the
-  scripts may live (`/opt/resource` is very Linux specific).
+* The first supported interface will be called `artifacts`, and its version
+  will start at `2.0` as it's really the next iteration of the existing
+  "resources" concept, but with a more specific name.
 
-* Today's resource interface (`/in`, `/out`, `/check`) becomes more specifically
-  a "versioned artifacts" or just "artifacts" resource interface.
+* There are no more hardcoded paths (`/opt/resource/X`) - instead there's the
+  single `info` entrypoint, which is run in the container's working directory.
+  This is more platform-agnostic.
 
 * Introduction of some sort of schema validation for resource configuration.
 
-* Remove the distinction between `source` and `params`; resources will receive a
-  single `config`. The distinction will remain in the pipeline. This makes it
-  easier to implement a resource without planning ahead for interesting dynamic
-  vs. static usage patterns, and will get more powerful with #684.
-
 
 ## Changes to Versioned Artifact resources
 
+* Remove the distinction between `source` and `params`; resources will receive
+  a single `config`. The distinction will remain in the pipeline. This makes it
+  easier to implement a resource without planning ahead for interesting dynamic
+  vs. static usage patterns, and this will become more powerful with
+  concourse/concourse#684.
+
 * Change `check` to run against all spaces. It will be given a mapping of each
   space to its current latest version, and return the set of all spaces, along
   with any new versions in each space.
@@ -195,48 +162,6 @@ TODO:
   pitfall during resource development/debugging.
 
 
-## Introduction of "notifier" resource interface
-
-A resource may now implement a "notifier" interface. Notifications must be
-explicitly enabled in the pipeline. This is so that a resource type can't
-suddenly decide to include a notifier and start doing surprising things.
-
-A notifier has a single hook, `notify`, and receives some sort of JSON payload
-describing the event. The event is purely for side-effects and has no
-implications for pipeline semantics.
-
-If a notification fails to send, the build shall error. To be honest, I haven't
-thought about this much yet, but I think it's better to be conservative.
-
-A resource type may implement both "artifacts" and "notifier" interfaces. If a
-resource implements both, it will be given information about its occurrence in
-the build (space and version). This will be crucial for e.g. reporting the
-status of a pull request or commit back to GitHub.
-
-Add support for a "notifier" resource type. They have one hook, `notify`,
-which is invoked with various types of notifications
-(`build_started`, `build_finished`, as the poster children).
-
-
-# Caveats
-
-Terminology is now even more confusing. "Resource type" could mean either the
-"git" vs. "s3" or "notifier" vs. "artifact".
-
-
-# Open Questions
-
-## Should a notifier resource be able to "observe" another resource?
-
-This would allow e.g. a generic `git` resource and specific
-`github-commit-status` resource type rather than having to bake them all
-together.
-
-The downside here is that it would be introducing cross-resource-type interface
-contracts. Resource A would have to understand resource B's versions/spaces.
-This gets more possible with schema verification but still feels risky.
-
-
 # Answered(?) Questions
 
 <details><summary>Can we reduce the `check` overhead?</summary>
@@ -318,14 +243,31 @@ Here are a few use cases that resources were sometimes used for inappropriately:
 ## Single-state resources
 
 Resources that really only have a "current state", such as deployments. This is
-still "change over time", but the difference is that old versions become invalid
-as soon as there's a new one.
-
-These can now be done by always marking the old versions as "deleted".
-
+still "change over time", but the difference is that old versions become
+invalid as soon as there's a new one. This can now be made more clear by
+marking the old versions as "deleted", either proactively via `put` or by
+`check` discovering the new version.
 
 ## Non-linearly versioned artifact storage
 
 This can be done by representing each non-linear version in a separate space.
 For example, generated code could be pushed to a generated (but deterministic)
 branch name, and that space could then be passed along.
+
+
+# Implementation Notes
+
+## Performance Implications
+
+Now that we're going to be collecting all versions of every resource, we should
+be careful not to be scanning the entire table all the time, and even make an
+effort to share data when possible. For example, we may want to associate
+collected versions to a global resource config object, rather than saving them
+all per-pipeline-resource.
+
+Here are some optimizations we probably want to make:
+
+* `(db.Pipeline).GetLatestVersionedResource` is called every minute and scans
+  the table to find the latest version of a given resource. We should reduce
+  this to a simple join column from the resource to the latest version,
+  maintained every time we save new versions.
diff --git a/01-resources-v2/prs.yml b/01-resources-v2/prs.yml
index 7cc47579..64e8aa83 100644
--- a/01-resources-v2/prs.yml
+++ b/01-resources-v2/prs.yml
@@ -9,9 +9,6 @@ resources:
   source:
     repository: concourse/atc
     access_token: ((token))
-  notify:
-  - on: build_started
-  - on: build_finished
 
 jobs:
 - name: atc-pr-unit
@@ -21,3 +18,5 @@ jobs:
     spaces: all
   - task: unit
     file: atc/ci/pr.yml
+
+  # TODO: this currently doesn't do any commit status indication

From a5895f9bb0382898092b33153e9dd9bb6b4e8849 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Tue, 12 Jun 2018 16:50:42 -0400
Subject: [PATCH 08/19] remove mention of schema validation

didn't get time to map this out - let's do it as a separate RFC
---
 01-resources-v2/proposal.md | 2 --
 1 file changed, 2 deletions(-)

diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index d8dde7cb..11f674eb 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -94,8 +94,6 @@ TODO:
   single `info` entrypoint, which is run in the container's working directory.
   This is more platform-agnostic.
 
-* Introduction of some sort of schema validation for resource configuration.
-
 
 ## Changes to Versioned Artifact resources
 

From 60b015c167a1efddf714d782eeb570b4989a9b53 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Fri, 22 Jun 2018 11:13:53 -0400
Subject: [PATCH 09/19] add has_latest, flesh out json api via Go structs

also switch git example to rugged, for much more efficient checking
across branches (no need to checkout tree, no need to shell out to git)
---
 01-resources-v2/git-example/Gemfile           |   2 +-
 01-resources-v2/git-example/Gemfile.lock      |   4 +-
 01-resources-v2/git-example/artifact          | 179 +++++++------
 .../git-example/collect-all-versions          |  74 ++++++
 01-resources-v2/git-example/info              |   2 +-
 01-resources-v2/proposal.md                   | 239 ++++++++++++++----
 6 files changed, 373 insertions(+), 127 deletions(-)
 create mode 100755 01-resources-v2/git-example/collect-all-versions

diff --git a/01-resources-v2/git-example/Gemfile b/01-resources-v2/git-example/Gemfile
index 8e376927..10d1a910 100644
--- a/01-resources-v2/git-example/Gemfile
+++ b/01-resources-v2/git-example/Gemfile
@@ -1,4 +1,4 @@
 source :rubygems
 
-gem 'git'
+gem 'rugged'
 gem 'pry'
diff --git a/01-resources-v2/git-example/Gemfile.lock b/01-resources-v2/git-example/Gemfile.lock
index cc170e92..0d15428b 100644
--- a/01-resources-v2/git-example/Gemfile.lock
+++ b/01-resources-v2/git-example/Gemfile.lock
@@ -2,18 +2,18 @@ GEM
   remote: http://rubygems.org/
   specs:
     coderay (1.1.2)
-    git (1.4.0)
     method_source (0.9.0)
     pry (0.11.3)
       coderay (~> 1.1.0)
       method_source (~> 0.9.0)
+    rugged (0.27.2)
 
 PLATFORMS
   ruby
 
 DEPENDENCIES
-  git
   pry
+  rugged
 
 BUNDLED WITH
    1.16.2
diff --git a/01-resources-v2/git-example/artifact b/01-resources-v2/git-example/artifact
index 7532f33e..ff9eee4d 100755
--- a/01-resources-v2/git-example/artifact
+++ b/01-resources-v2/git-example/artifact
@@ -1,92 +1,120 @@
 #!/usr/bin/env ruby
 
 require "json"
-require "git"
-
-# $request = JSON.parse(STDIN.read, symbolize_names: true)
-
-def commit_versions(log)
-  log.collect do |c|
-    {
-      version: {ref: c.sha},
-      metadata: [
-        {name: "author", value: c.author.name},
-        {name: "author_date", value: c.author_date},
-        {name: "commit", value: c.sha},
-        {name: "committer", value: c.committer.name},
-        {name: "committer_date", value: c.committer_date},
-        {name: "message", value: c.message}
-      ]
-    }
-  end
+require "rugged"
+require "pry"
+require "benchmark"
+
+$request = JSON.parse(STDIN.read, symbolize_names: true)
+
+# arbitrary number to limit memory usage; total commits to return at a time
+# (across all branches)
+CHECK_BATCH_LIMIT = 10000
+
+def commit_version(c)
+  {
+    version: {ref: c.oid},
+    metadata: [
+      {name: "author", value: enc(c, c.author[:name])},
+      {name: "author_date", value: c.author[:time].to_s},
+      {name: "committer", value: enc(c, c.committer[:name])},
+      {name: "committer_date", value: c.committer[:time].to_s},
+      {name: "message", value: enc(c, c.message)}
+    ]
+  }
+end
+
+def bench(label, &blk)
+  time = Benchmark.realtime(&blk)
+  $stderr.puts "#{label}: #{time}s"
+end
+
+def enc(c, str)
+  str = str.force_encoding("ISO-8859-1") unless c.header_field("Encoding")
+  str.encode("UTF-8")
 end
 
 case ARGV[0]
 when "check"
-  $request = {
-    config: {uri: "https://github.com/vito/booklit"},
-    from: {
-      master: {ref: "40bc6986197e411471d306bb8eb3a21c5b5f9d26"},
-      travis: {ref: "8b0a526e233b1599f8d5fcc1179e0ec7642acd90"}
-    }
-  }
+  repo_dir = File.basename($request[:config][:uri])
 
-  git =
-    if Dir.exists?("check-repo")
-      Git.open("check-repo").tap(&:fetch)
+  repo =
+    if Dir.exists?(repo_dir)
+      Rugged::Repository.new(repo_dir).tap do |r|
+        r.fetch("origin")
+      end
     else
-      Git.clone($request[:config][:uri], "check-repo")
+      Rugged::Repository.clone_at(
+        $request[:config][:uri],
+        repo_dir,
+        bare: true,
+        progress: lambda { |t| print t })
     end
 
   spaces = []
-  git.branches.local.each do |b|
-    # skip "default branch" entry
-    next if b.name =~ /HEAD ->/
-
-    b.checkout
-
-    paths = $request[:config][:paths] || ["."]
-    paths += ($request[:config][:ignore_paths] || []).collect { |p| ":!" + p }
-
-    log = git.log(nil).path(paths)
-
-    # TODO: this will get *all* commits and load it into memory, which is
-    # probably a bad idea. the linux repo for example has 710k+ commits.
-    #
-    # should this be paginated? or should it stream each version back to the
-    # caller somehow so everything doesn't have to be sucked into memory?
-    #
-    # TODO: when checking from a given version, should the given version be
-    # returned? this was done in the original API so that you could run
-    # `check-resource -f <version>`, but in this new world where we just always
-    # collect all versions, that won't be necessary. in fact, `check-resource
-    # -f <version>` would need to be given a space that it's checking against.
-    #
-    # TODO: how does the caller know if the given version no longer exists, and
-    # that the version history should be reset?
-    commits =
-      if version = $request[:from][b.name]
-        begin
-          commit_versions(log.between(version[:ref]))
-        rescue
-          # bad ref; emit all versions
-          commit_versions(log)
+  default_branch = nil
+
+  total_commits = 0
+
+  repo.branches.each do |b|
+    unless b.remote?
+      # assume the only local branch is the default one
+      default_branch = b.name
+      next
+    end
+
+    space_name = b.name.sub("#{b.remote_name}/", "")
+
+    has_latest = false
+    commits = []
+
+    walker = Rugged::Walker.new(repo)
+    walker.sorting(Rugged::SORT_TOPO|Rugged::SORT_REVERSE)
+    walker.simplify_first_parent
+    walker.push(b.target)
+
+    from = $request[:from][space_name.to_sym]
+
+    if from && repo.include?(from[:ref])
+      commit = repo.lookup(from[:ref])
+      walker.hide(commit)
+
+      commits << commit_version(commit)
+      total_commits += 1
+      has_latest = commit.oid == b.target.oid
+    end
+
+    unless has_latest || total_commits >= CHECK_BATCH_LIMIT
+      bench("#{space_name} walk") do
+        walker.walk do |c|
+          # TODO: test if commit satisfies paths/ignore_paths
+          commits << commit_version(c)
+          total_commits += 1
+          has_latest = c.oid == b.target.oid
+
+          break if total_commits >= CHECK_BATCH_LIMIT
         end
-      else
-        commit_versions(log)
       end
+    end
+
+    next if commits.empty?
 
-    $stderr.puts "#{b.name}: #{commits.length} commits"
+    $stderr.puts "#{space_name} commits: #{commits.size} (latest: #{has_latest})"
 
     spaces << {
-      space: b.name,
-      versions: commits
+      space: space_name,
+      versions: commits,
+      has_latest: has_latest
     }
   end
 
-  response = JSON.dump(spaces)
-  puts response
-  puts response.size
+  bench("dump") do
+    puts JSON.dump({
+      spaces: spaces,
+      default_space: default_branch
+    })
+  end
+
 when "get"
   $request = {
     config: {uri: "https://github.com/vito/booklit"},
@@ -94,12 +122,15 @@ when "get"
     version: {ref: "f828f2758256b0e93dc3c101f75604efe92ca07e"}
   }
 
-  git =
-    Git.clone($request[:config][:uri], "dot", # TODO: irl this would be '.'
-              branch: $request[:space],
-              recursive: true)
+  repo =
+    Rugged::Repository.clone_at(
+      $request[:config][:uri],
+      "dot", # TODO: irl this would be '.'
+      checkout_branch: $request[:space])
+
+  repo.checkout($request[:version][:ref])
 
-  git.checkout($request[:version][:ref])
+  # TODO: update/init submodules recursively
 
   # TODO: draw the rest of the owl
   #
diff --git a/01-resources-v2/git-example/collect-all-versions b/01-resources-v2/git-example/collect-all-versions
new file mode 100755
index 00000000..d3d03608
--- /dev/null
+++ b/01-resources-v2/git-example/collect-all-versions
@@ -0,0 +1,74 @@
+#!/usr/bin/env ruby
+
+require "json"
+require "subprocess"
+require "stringio"
+
+# linux (~2 branches, ~766k commits)
+$request = {
+  config: {uri: "https://github.com/torvalds/linux"},
+  from: {}
+}
+
+# concourse (~15 branches, ~8500 commits)
+$request = {
+  config: {uri: "https://github.com/concourse/concourse"},
+  from: {}
+}
+
+# booklit (~10 branches, ~200 commits)
+$request = {
+  config: {uri: "https://github.com/vito/booklit"},
+  from: {}
+}
+
+# rails (~36 branches, ~70k commits)
+$request = {
+  config: {uri: "https://github.com/rails/rails"},
+  from: {}
+}
+
+def check_all
+  all_start = Time.now
+  check_has_latest = false
+  done_last_check = false
+
+  while true
+    check_start = Time.now
+
+    Subprocess::Process.new(
+      ["bundle", "exec", "./artifact", "check"],
+      stdin: Subprocess::PIPE,
+      stdout: Subprocess::PIPE) do |check|
+      out, _ = check.communicate(JSON.dump($request))
+
+      done_last_check = true if check_has_latest
+
+      res = JSON.parse(out, symbolize_names: true)
+
+      res[:spaces].each do |s|
+        space = s[:space]
+        versions = s[:versions]
+
+        if versions.first[:version] == versions.last[:version]
+          puts "checked #{space}: #{versions.first[:version][:ref]}"
+        else
+          puts "checked #{space}: #{versions.first[:version][:ref]}..#{versions.last[:version][:ref]}"
+        end
+
+        $request[:from][space] = versions.last[:version]
+      end
+
+      check_has_latest = res[:spaces].all? { |s| s[:has_latest] }
+    end
+
+    if done_last_check
+      puts "time for stable check: #{Time.now - check_start}s"
+      break
+    end
+  end
+
+  puts "total time for all checks: #{Time.now - all_start}s"
+end
+
+check_all
diff --git a/01-resources-v2/git-example/info b/01-resources-v2/git-example/info
index 668a79fd..be0e0398 100755
--- a/01-resources-v2/git-example/info
+++ b/01-resources-v2/git-example/info
@@ -4,7 +4,7 @@ require "json"
 
 puts JSON.dump({
   artifacts: {
-    version: "2.0",
+    api_version: "2.0",
     check: "artifact check",
     get: "artifact get",
     put: "artifact put"
diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 11f674eb..02d9147e 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -9,59 +9,201 @@ Introduces a new resource interface with the following goals:
 
 * Introduce a more airtight "versioned artifacts" interface, tightening up
   loopholes in today's resource API to ensure that resources are pointing at an
-  external source of truth and cannot be partially implemented or hacky.
+  external source of truth, so that we can explicitly design for new features
+  and workflows rather than forcing everything into the resource interface.
 
-* Extend the versioned artifacts interface to support deletion of versions,
-  either by an explicit `delete` call or by somehow noticing that versions have
-  disappeared.
+* Extend the versioned artifacts interface to support deletion of versions.
 
 
 # Proposal
 
-At this early stage of the RFC, it's easiest for me to just use Elm syntax and
-pretend this is all in a type system.
+## General Types
 
-In reality, each of these functions would be scripts with JSON requests passed
-to them on stdin. I'm going to use the `Bits` type just to represent which calls
-have state on disk to either access (like `put`) or return (like `get`). This
-would normally just be the working directory of the script.
+```go
+// Space is a name of a space, e.g. "master", "release/3.14", "1.0".
+type Space string
+
+// Config is a black box containing all user-supplied configuration, combining
+// `source` in the resource definition with `params` from the step (in the
+// case of `get` or `put`).
+type Config map[string]interface{}
+
+// Version is a key-value identifier for a version of a resource, e.g.
+// `{"ref":"abcdef"}`, `{"version":"1.2.3"}`.
+type Version map[string]string
+
+// Metadata is an ordered list of metadata fields to display to the user about
+// a resource version. It's ordered so that the resource can decide the best
+// way to show it.
+type Metadata []MetadataField
+
+// MetadataField is an arbitrary key-value to display to the user about a
+// version of a resource.
+type MetadataField struct {
+  Name  string `json:"name"`
+  Value string `json:"value"`
+}
+```
 
-## General Types
+## Versioned Artifacts interface
+
+### `check`: Detect versions across spaces.
+
+The `check` command will be invoked with the following JSON structure on
+`stdin`:
+
+```go
+// CheckRequest contains the resource's configuration and latest version
+// associated to each space.
+type CheckRequest struct {
+  Config Config            `json:"config"`
+  From   map[Space]Version `json:"from"`
+}
+```
 
-```elm
--- arbitrary configuration
-type alias Config = Dict String Json.Value
+The first call will have an empty object as `from`.
+
+Any spaces discovered by the resource but not present in `from` should collect
+versions from the beginning.
+
+For each space in `from`, the resource should collect all versions that appear
+*after* the current version, including the given version if it's still present.
+If the given version is no longer present, the resource should instead collect
+from the beginning, as if the space was not specified.
+
+If any space in `from` is no longer present, the resource should ignore it, and
+not include it in the response.
+
+The resource may limit the number of versions it returns however it likes in
+order to save RAM and incrementally build the history. The `has_latest` field in
+the response (see below) is used to indicate this scenario.
+
+The resource should also determine a "default space", if any. Having a default
+space is useful for things like Git repos which have a default branch, or
+version spaces (e.g. `1.8`, `2.0`) which can point to the latest version line by
+default. If there is no default space, the user must specify it explicitly in
+the pipeline, either by configuring one on the resource (`space: foo`) or on the
+`get` step (`spaces: [foo]`).
+
+The command should then emit the collected versions and default space (if any)
+as following JSON response structure to `stdout`:
+
+```go
+// CheckResponse returns the detected spaces and the default space, if any.
+type CheckResponse struct {
+  Spaces       []DetectedSpace `json:"spaces"`
+  DefaultSpace *Space          `json:"default_space"`
+}
+
+// DetectedSpace is a space and the versions detected therein, listed in
+// chronological order.
+//
+// The HasLatest field indicates whether the list of versions includes the
+// latest version. Until the latest version is collected, Concourse will not
+// schedule the space. This allows resources to limit the number of versions
+// they return at once for things with a ton of history.
+type DetectedSpace struct {
+  Space     Space             `json:"space"`
+  Versions  []DetectedVersion `json:"versions"`
+  HasLatest bool              `json:"has_latest"`
+}
+
+// DetectedVersion is a version within a space with its associated metadata.
+type DetectedVersion struct {
+  Version  Version  `json:"version"`
+  Metadata Metadata `json:"metadata"`
+}
+```
 
--- identifier for space, i.e. 'foo' or '1.2'
-type alias Space = String
 
--- identifier for version, i.e. {"version":"1.2"}
-type alias Version = Dict String String
+### `get`: Fetch a version from the resource's space.
 
--- arbitrary ordered metadata (we may make this fancier in the future)
-type alias Metadata = List (String, String)
+The `get` command will be invoked with the following JSON structure on `stdin`:
 
--- data on disk
-type alias Bits = ()
+```go
+type GetRequest struct {
+  Config  Config  `json:"config"`
+  Space   Space   `json:"space"`
+  Version Version `json:"version"`
+}
 ```
 
-## Versioned Artifacts interface
+The command will be invoked with a completely empty working directory. The
+command should populate this directory with the requested bits. The `git`
+resource, for example, would clone directly into the working directory.
+
+If the requested version is unavailable, the command should exit nonzero.
+
+No response is expected.
+
+Anything printed to `stdout` and `stderr` will propagate to the build logs.
+
+
+### `put`: Idempotently create or destroy resource versions in a space.
+
+The `put` command will be invoked with the following JSON structure on `stdin`:
 
-```elm
-check   : Config -> Dict Space Version -> Dict Space (List (Version, Metadata))
-get     : Config -> Space -> Version -> Bits
-put     : Config -> Bits -> Dict Space { created : Set Version, deleted : Set Version }
+```go
+type PutRequest struct {
+  Config       Config `json:"config"`
+  ResponsePath string `json:"response_path"`
+}
 ```
 
+The command will be invoked with all of the build plan's artifacts present in
+the working directory, each as `./(artifact name)`.
+
+The command should perform any and all side-effects idempotently, and then
+record the following JSON response structure to the file specified by
+`response_path`:
+
+```go
+type PutResponse struct {
+  Space   Space     `json:"space"`
+  Created []Version `json:"created"`
+  Deleted []Version `json:"deleted"`
+}
+```
+
+The `space` field determines the space that has been modified or created. This
+allows new spaces to be generated by a `put` dynamically (based on params
+and/or the bits in its working directory) and propagated to the rest of the
+pipeline.
+
+Versions returned under `created` will be recorded as outputs of the build. A
+`check` will then be performed to fill in the metadata and determine the
+ordering of the versions. Once the ordering is learned, the latest version will
+be fetched by the implicit `get`.
+
+Versions returned under `deleted` will be marked as deleted. They will remain
+in the database for archival purposes, but will no longer be input candidates
+for any builds, and can no longer be fetched.
+
+Anything printed to `stdout` and `stderr` will propagate to the build logs.
+
+
 # Examples
 
 ## Resource Implementations
 
-I've started implementing a new `git` resource alongside this
-document. See
-[`git-example/`](https://github.com/vito/rfcs/tree/resources-v2/01-resources-v2/git-example).
-I've left `TODO`s for parts that need more thinking or discussion. Please
-leave comments!
+I've started cooking up new resources using this interface. I've left `TODO`s
+for parts that need more thinking or discussion. Please leave comments!
+
+### `git`
+
+[Code](https://github.com/vito/rfcs/tree/resources-v2/01-resources-v2/git-example)
+
+This resource models the original `git` resource. It represents each branch as a space.
+
+### `semver-git`
+
+[Code](https://github.com/vito/rfcs/tree/resources-v2/01-resources-v2/semver-example)
+
+This is a whole new semver resource intended to replace the original `semver`
+resource with a better model that supports concurrent version lines (i.e.
+supporting multiple major/minor releases with patches). It does this by managing
+tags in an existing Git repository.
+
 
 ## Pipeline Usage
 
@@ -111,8 +253,7 @@ TODO:
   efficiently perform the check. It also keeps the container overhead down to
   one per resource, rather than one per space.
 
-* Change `put` to emit a set of created versions for each space, rather than
-  just one.
+* Change `put` to emit a set of created versions, rather than just one.
 
   Technically the `git` resource may push many commits, so returning more than
   one version is necessary to track them all as outputs of a build. This could
@@ -165,14 +306,14 @@ TODO:
 <details><summary>Can we reduce the `check` overhead?</summary>
 
 <p>
-~~With spaces there will be more `check`s than ever. Right now, there's one
+<strike>With spaces there will be more `check`s than ever. Right now, there's one
 container per recurring `check`. Can we reduce the container overhead here by
 requiring that resource `check`s be side-effect free and able to run in
-parallel?~~
+parallel?</strike>
 </p>
 
 <p>
-~~There may be substantial security implications for this.~~
+<strike>There may be substantial security implications for this.</strike>
 </p>
 
 <p>
@@ -187,16 +328,16 @@ least consume only one container.
 <details><summary>Is `destroy` general enough to be a part of the interface?</summary>
 
 <p>
-~~It may be the case that most resources cannot easily support `destroy`. One
+<strike>It may be the case that most resources cannot easily support `destroy`. One
 example is the `git` resource. It doesn't really make sense to `destroy` a
 commit. Even if it did (`push -f`?), it's a kind of weird workflow to support
-out of the box.~~
+out of the box.</strike>
 </p>
 
 <p>
-~~Could we instead just have `put` and ensure that we `check` in such a way that
+<strike>Could we instead just have `put` and ensure that we `check` in such a way that
 deleted versions are automatically noticed? What would the overhead of this
-be?~~ This only works if the versions are "chained", as with the `git` case.
+be?</strike> This only works if the versions are "chained", as with the `git` case.
 </p>
 
 <p>
@@ -209,23 +350,23 @@ into an idempotent versioned artifact side effect performer.
 <details><summary>Should `put` be given a space or return the space?</summary>
 
 <p>
-~~The verb `PUT` in HTTP implies an idempotent action against a given resource. So
-it's intuitive that the `put` verb here would do the same.~~
+<strike>The verb `PUT` in HTTP implies an idempotent action against a given resource. So
+it's intuitive that the `put` verb here would do the same.</strike>
 </p>
 <p>
-~~However, many of today's usage of `put` would be against a dynamically
+<strike>However, many of today's usage of `put` would be against a dynamically
 determined space. For example, most semver workflows involve `put`ing with the
 version determined by a file (often coming from the `semver` resource). So the
-space isn't known statically at pipeline configuration time.~~
+space isn't known statically at pipeline configuration time.</strike>
 </p>
 <p>
-~~What's more, the resulting space for a semver push would only be `MAJOR.MINOR`,
+<strike>What's more, the resulting space for a semver push would only be `MAJOR.MINOR`,
 excluding the final patch segment. This is annoying to have to explicitly
-configure in your build.~~
+configure in your build.</strike>
 </p>
 <p>
-~~If we instead have `put` return both the space and the versions, this would be a
-lot simpler.~~
+<strike>If we instead have `put` return both the space and the versions, this would be a
+lot simpler.</strike>
 </p>
 <p>
 Answered this at the same time as having `put` return a set of deleted

From 3bc00098143d7f1d59c7c25b8614ddc545a05d81 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Fri, 22 Jun 2018 11:16:46 -0400
Subject: [PATCH 10/19] add semver example resource type + pipelines

---
 01-resources-v2/main-pipeline.yml           | 121 ++++++++++++++++
 01-resources-v2/release-pipeline.yml        |  58 ++++++++
 01-resources-v2/semver-example/.gitignore   |   2 +
 01-resources-v2/semver-example/Gemfile      |   5 +
 01-resources-v2/semver-example/Gemfile.lock |  21 +++
 01-resources-v2/semver-example/README.md    |  13 ++
 01-resources-v2/semver-example/artifact     | 150 ++++++++++++++++++++
 01-resources-v2/semver-example/info         |  12 ++
 8 files changed, 382 insertions(+)
 create mode 100644 01-resources-v2/main-pipeline.yml
 create mode 100644 01-resources-v2/release-pipeline.yml
 create mode 100644 01-resources-v2/semver-example/.gitignore
 create mode 100644 01-resources-v2/semver-example/Gemfile
 create mode 100644 01-resources-v2/semver-example/Gemfile.lock
 create mode 100644 01-resources-v2/semver-example/README.md
 create mode 100755 01-resources-v2/semver-example/artifact
 create mode 100755 01-resources-v2/semver-example/info

diff --git a/01-resources-v2/main-pipeline.yml b/01-resources-v2/main-pipeline.yml
new file mode 100644
index 00000000..7be40cc3
--- /dev/null
+++ b/01-resources-v2/main-pipeline.yml
@@ -0,0 +1,121 @@
+# In this example, we demonstrate that today's model of an
+# always-rolling-forward pipeline doesn't need to mention spaces at all, so
+# long as the semver-git resource continuously rolls its "default space"
+# forward to the latest major.minor version available.
+#
+# Interesting note: the bin-rc and bosh-rc jobs will dynamically switch the
+# space they run against because they have a `get` of the resource's default
+# space, which will change as the version is bumped. This would result in the
+# box's color changing to 'pending' in the UI until a build in that space runs.
+
+---
+resources:
+- name: booklit
+  type: git
+  source:
+    uri: https://github.com/vito/booklit
+
+- name: version
+  type: semver-git
+  source:
+    uri: https://github.com/vito/booklit
+
+- name: bin-rc
+  type: s3
+
+- name: bosh-rc
+  type: s3
+
+jobs:
+- name: unit
+  plan:
+  - get: booklit
+    trigger: true
+  - task: unit
+    file: booklit/ci/test.yml
+
+- name: major
+  plan:
+  - get: booklit
+  - put: version
+    params: {bump: major, pre: rc, repo: booklit}
+
+- name: minor
+  plan:
+  - get: booklit
+  - put: version
+    params: {bump: minor, pre: rc, repo: booklit}
+
+- name: rc
+  plan:
+  - get: booklit
+    passed: [unit]
+    trigger: true
+  - put: version
+    params: {pre: rc, repo: booklit}
+
+- name: bin-rc
+  plan:
+  - get: booklit
+    passed: [rc]
+    trigger: true
+  - get: version
+    passed: [rc]
+    trigger: true
+  - task: build-rc
+    file: booklit/ci/bin-rc.yml
+  - put: bin-rc
+
+- name: bin-testflight
+  plan:
+  - get: booklit
+    passed: [bin-rc]
+    trigger: true
+  - get: bin-rc
+    passed: [bin-rc]
+    trigger: true
+  - task: integration
+    file: booklit/ci/testflight.yml
+
+- name: bosh-rc
+  plan:
+  - get: booklit
+    passed: [rc]
+    trigger: true
+  - get: version
+    passed: [rc]
+    trigger: true
+  - task: build-rc
+    file: booklit/ci/bosh-rc.yml
+  - put: bosh-rc
+
+- name: bosh-testflight
+  plan:
+  - get: booklit
+    passed: [bosh-rc]
+    trigger: true
+  - get: bosh-rc
+    passed: [bosh-rc]
+    trigger: true
+  - task: integration
+    file: booklit/ci/testflight.yml
+
+- name: ship
+  plan:
+  - get: booklit
+    passed: [bin-testflight, bosh-testflight]
+    trigger: true
+  - get: bin-rc
+    passed: [bin-testflight]
+  - get: bosh-rc
+    passed: [bosh-testflight]
+  - put: version
+    params: {bump: final, repo: booklit}
+
+- name: patch
+  plan:
+  - get: booklit
+    passed: [ship]
+    trigger: true
+  - put: version
+    params: {bump: patch, pre: rc, repo: booklit}
diff --git a/01-resources-v2/release-pipeline.yml b/01-resources-v2/release-pipeline.yml
new file mode 100644
index 00000000..336cf4ba
--- /dev/null
+++ b/01-resources-v2/release-pipeline.yml
@@ -0,0 +1,58 @@
+# This example shows a subset of the main pipeline for supporting release
+# branches. There's quite a bit of duplication, but in principle you may have a
+# different set of checks and balances for patch releases.
+
+---
+resources:
+- name: booklit
+  type: git
+  source:
+    uri: https://github.com/vito/booklit
+
+- name: version
+  type: semver-git
+  source:
+    uri: https://github.com/vito/booklit
+
+jobs:
+- name: unit
+  plan:
+  - get: booklit
+    spaces: [release/*]
+    trigger: true
+  - task: unit
+    file: booklit/ci/test.yml
+
+- name: rc
+  plan:
+  - get: booklit
+    spaces: [release/*]
+    passed: [unit]
+    trigger: true
+  - put: version
+    params: {pre: rc, repo: booklit}
+
+- name: integration
+  plan:
+  - get: booklit
+    spaces: [release/*]
+    passed: [rc]
+    trigger: true
+  - task: integration
+    file: booklit/ci/integration.yml
+
+- name: ship
+  plan:
+  - get: booklit
+    spaces: [release/*]
+    passed: [integration]
+    trigger: true
+  - put: version
+    params: {bump: final, repo: booklit}
+
+- name: patch
+  plan:
+  - get: booklit
+    spaces: [release/*]
+  - put: version
+    params: {bump: patch, pre: rc, repo: booklit}
diff --git a/01-resources-v2/semver-example/.gitignore b/01-resources-v2/semver-example/.gitignore
new file mode 100644
index 00000000..bbb00daa
--- /dev/null
+++ b/01-resources-v2/semver-example/.gitignore
@@ -0,0 +1,2 @@
+check-repo
+dot
diff --git a/01-resources-v2/semver-example/Gemfile b/01-resources-v2/semver-example/Gemfile
new file mode 100644
index 00000000..72115298
--- /dev/null
+++ b/01-resources-v2/semver-example/Gemfile
@@ -0,0 +1,5 @@
+source :rubygems
+
+gem 'git'
+gem 'semantic'
+gem 'pry'
diff --git a/01-resources-v2/semver-example/Gemfile.lock b/01-resources-v2/semver-example/Gemfile.lock
new file mode 100644
index 00000000..44bb5be8
--- /dev/null
+++ b/01-resources-v2/semver-example/Gemfile.lock
@@ -0,0 +1,21 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    coderay (1.1.2)
+    git (1.4.0)
+    method_source (0.9.0)
+    pry (0.11.3)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+    semantic (1.6.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  git
+  pry
+  semantic
+
+BUNDLED WITH
+   1.16.2
diff --git a/01-resources-v2/semver-example/README.md b/01-resources-v2/semver-example/README.md
new file mode 100644
index 00000000..cc5b51f8
--- /dev/null
+++ b/01-resources-v2/semver-example/README.md
@@ -0,0 +1,13 @@
+# Git Resource v2
+
+This implementation is done in Ruby using the `git` gem. I chose Ruby
+over Bash because having a real language with more accessible data
+structures is probably going to be more important with this new
+interface, and Ruby feels pretty well suited (really just need a bit more
+than Bash).
+
+Please leave comments on parts you like/don't like! But bear in mind the
+goal here isn't necessarily the prettiness of the code, it's to see what
+kinds of things the resource has to do. I'll be using Ruby purely as a
+scripting language, hacking things together where needed in the interest
+of brevity.
diff --git a/01-resources-v2/semver-example/artifact b/01-resources-v2/semver-example/artifact
new file mode 100755
index 00000000..35d3d2e0
--- /dev/null
+++ b/01-resources-v2/semver-example/artifact
@@ -0,0 +1,150 @@
+#!/usr/bin/env ruby
+
+require "json"
+require "git"
+require "semantic"
+
+# $request = JSON.parse(STDIN.read, symbolize_names: true)
+
+def commit_versions(log)
+  log.collect do |c|
+    {
+      version: {ref: c.sha},
+      metadata: [
+        {name: "author", value: c.author.name},
+        {name: "author_date", value: c.author_date},
+        {name: "commit", value: c.sha},
+        {name: "committer", value: c.committer.name},
+        {name: "committer_date", value: c.committer_date},
+        {name: "message", value: c.message}
+      ]
+    }
+  end
+end
+
+case ARGV[0]
+when "check"
+  $request = {
+    config: {uri: "https://github.com/concourse/concourse"},
+    from: {
+      "2.7": {version: "2.7.5"},
+      "3.0": {version: "3.0.0"},
+      "3.9": {version: "3.9.1"}
+    }
+  }
+
+  tags = Git.ls_remote($request[:config][:uri])["tags"]
+
+  versions = []
+  tags.each do |tag, _|
+    next unless tag =~ /^v\d/
+
+    # TODO: skip non-matches
+    versions << Semantic::Version.new(tag[1..-1])
+  end
+
+  versions.sort!
+
+  from = {}
+  $request[:from].each do |space, version|
+    from[space] = Semantic::Version.new(version[:version])
+  end
+
+  latest_space = nil
+
+  space_versions = {}
+  versions.each do |v|
+    name = "#{v.major}.#{v.minor}"
+    latest_space = name
+    next if from[name] && v < from[name]
+
+    space_versions[name] ||= []
+    space_versions[name] << {
+      version: {version: v.to_s},
+      metadata: []
+    }
+  end
+
+  spaces = []
+  space_versions.each do |space, vs|
+    spaces << {
+      space: space,
+      versions: vs,
+      has_latest: true
+    }
+  end
+
+  puts JSON.dump({
+    spaces: spaces,
+    default_space: latest_space
+  })
+
+when "get"
+  File.open("version", "w") do |file|
+    file.write($request[:version][:version])
+  end
+
+when "put"
+  $request = {
+    config: {
+      uri: "https://github.com/concourse/concourse",
+      file: "version/version",
+      repo: "concourse"
+    },
+    response: "./response.json"
+  }
+
+  # TODO: set up auth/etc
+  repo = Git.open($request[:config][:repo]).fetch("origin", tags: true)
+
+  if file_path = $config[:file]
+    version = Semantic::Version.new(File.read(file_path))
+  else
+    # get current version from latest tag
+    latest_tag = repo.describe("HEAD", tags: true, abbrev: 0, match: "v*")
+
+    version = Semantic::Version.new(latest_tag)
+
+    case bump = $config[:bump]
+    when "final"
+      version.pre = nil
+    else
+      # this ends with a ! but doesn't mutate it... weird.
+      version = version.increment!(bump)
+    end
+
+    if pre = $config[:pre]
+      num =
+        if version.pre
+          current_pre, current_num = version.identifiers(version.pre)
+          if pre == current_pre
+            current_num + 1
+          else
+            1
+          end
+        else
+          1
+        end
+
+      version.pre = "#{pre}.#{num}"
+    end
+  end
+
+  space = "#{version.major}.#{version.minor}"
+
+  tag = repo.add_tag("v#{version}", force: true)
+
+  # TODO: set up auth/etc
+  repo.push("origin", "refs/tags/#{tag.name}")
+
+  response = JSON.dump({
+    space => {
+      created: [{version: tag.name}],
+      deleted: []
+    }
+  })
+
+  File.open($request[:response], "w") do |f|
+    f.write(response)
+  end
+end
diff --git a/01-resources-v2/semver-example/info b/01-resources-v2/semver-example/info
new file mode 100755
index 00000000..be0e0398
--- /dev/null
+++ b/01-resources-v2/semver-example/info
@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+
+require "json"
+
+puts JSON.dump({
+  artifacts: {
+    api_version: "2.0",
+    check: "artifact check",
+    get: "artifact get",
+    put: "artifact put"
+  }
+})

From 918a6be989255bbe031d9cd5385748f82b4f8c44 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Wed, 18 Jul 2018 14:16:44 -0400
Subject: [PATCH 11/19] git example 'check': print progress to stderr

also print total versions collected per-space in collect-all-versions
---
 01-resources-v2/git-example/artifact          |  2 +-
 .../git-example/collect-all-versions          | 21 +++++++++++++------
 2 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/01-resources-v2/git-example/artifact b/01-resources-v2/git-example/artifact
index ff9eee4d..022f207d 100755
--- a/01-resources-v2/git-example/artifact
+++ b/01-resources-v2/git-example/artifact
@@ -48,7 +48,7 @@ when "check"
         $request[:config][:uri],
         repo_dir,
         bare: true,
-        progress: lambda { |t| print t })
+        progress: lambda { |t| $stderr.print t })
     end
 
   spaces = []
diff --git a/01-resources-v2/git-example/collect-all-versions b/01-resources-v2/git-example/collect-all-versions
index d3d03608..d7579958 100755
--- a/01-resources-v2/git-example/collect-all-versions
+++ b/01-resources-v2/git-example/collect-all-versions
@@ -4,12 +4,6 @@ require "json"
 require "subprocess"
 require "stringio"
 
-# linux (~2 branches, ~766k commits)
-$request = {
-  config: {uri: "https://github.com/torvalds/linux"},
-  from: {}
-}
-
 # concourse (~15 branches, ~8500 commits)
 $request = {
   config: {uri: "https://github.com/concourse/concourse"},
@@ -28,11 +22,19 @@ $request = {
   from: {}
 }
 
+# linux (~2 branches, ~766k commits)
+$request = {
+  config: {uri: "https://github.com/torvalds/linux"},
+  from: {}
+}
+
 def check_all
   all_start = Time.now
   check_has_latest = false
   done_last_check = false
 
+  total_versions = {}
+
   while true
     check_start = Time.now
 
@@ -56,6 +58,12 @@ def check_all
           puts "checked #{space}: #{versions.first[:version][:ref]}..#{versions.last[:version][:ref]}"
         end
 
+        total_versions[space] ||= 0
+        total_versions[space] += versions.size
+        if versions.first[:version] == $request[:from][space]
+          total_versions[space] -= 1
+        end
+
         $request[:from][space] = versions.last[:version]
       end
 
@@ -69,6 +77,7 @@ def check_all
   end
 
   puts "total time for all checks: #{Time.now - all_start}s"
+  puts "total versions: #{total_versions.to_json}"
 end
 
 check_all

From d0d8774ef3b72d4e8df09c1fbe19a244d51a776a Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Tue, 21 Aug 2018 13:47:00 -0400
Subject: [PATCH 12/19] change 'check' to emit its response to a file

stdout is risky - it's not guaranteed to be lossless if the output is
printed quickly enough, which could easily happen when a ton of versions
are being emitted.

this also makes the command consistent with `put`, and frees up stdout
and stderr both for logging.
---
 01-resources-v2/proposal.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 02d9147e..af6431b8 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -56,8 +56,9 @@ The `check` command will be invoked with the following JSON structure on
 // CheckRequest contains the resource's configuration and latest version
 // associated to each space.
 type CheckRequest struct {
-  Config Config            `json:"config"`
-  From   map[Space]Version `json:"from"`
+  Config       Config            `json:"config"`
+  From         map[Space]Version `json:"from"`
+  ResponsePath string            `json:"response_path"`
 }
 ```
 
@@ -86,7 +87,7 @@ the pipeline, either by configuring one on the resource (`space: foo`) or on the
 `get` step (`spaces: [foo]`).
 
 The command should then emit the collected versions and default space (if any)
-as following JSON response structure to `stdout`:
+as following JSON response structure the file specified by `response_path`:
 
 ```go
 // CheckResponse returns the detected spaces and the default space, if any.

From 1404725f2c2c6558449818e240fd92db26359d86 Mon Sep 17 00:00:00 2001
From: Clara Fu <cfu@pivotal.io>
Date: Tue, 23 Oct 2018 10:31:52 -0400
Subject: [PATCH 13/19] update rfc to include new interface decisions

Signed-off-by: Mark Huang <mhuang@pivotal.io>
---
 01-resources-v2/git-example/artifact    |  50 ++++-------
 01-resources-v2/proposal.md             |  95 ++++++++++----------
 01-resources-v2/s3-example/Gemfile      |   4 +
 01-resources-v2/s3-example/Gemfile.lock |  30 +++++++
 01-resources-v2/s3-example/artifact     | 114 ++++++++++++++++++++++++
 01-resources-v2/s3-example/info         |  12 +++
 6 files changed, 220 insertions(+), 85 deletions(-)
 create mode 100644 01-resources-v2/s3-example/Gemfile
 create mode 100644 01-resources-v2/s3-example/Gemfile.lock
 create mode 100755 01-resources-v2/s3-example/artifact
 create mode 100755 01-resources-v2/s3-example/info

diff --git a/01-resources-v2/git-example/artifact b/01-resources-v2/git-example/artifact
index 022f207d..792e47cb 100755
--- a/01-resources-v2/git-example/artifact
+++ b/01-resources-v2/git-example/artifact
@@ -7,12 +7,9 @@ require "benchmark"
 
 $request = JSON.parse(STDIN.read, symbolize_names: true)
 
-# arbitrary number to limit memory usage; total commits to return at a time
-# (across all branches)
-CHECK_BATCH_LIMIT = 10000
-
-def commit_version(c)
-  {
+def commit_version(c, s)
+  JSON.dump({
+    space: s,
     version: {ref: c.oid},
     metadata: [
       {name: "author", value: enc(c, c.author[:name])},
@@ -21,7 +18,7 @@ def commit_version(c)
       {name: "committer_date", value: c.committer[:time].to_s},
       {name: "message", value: enc(c, c.message)}
     ]
-  }
+  })
 end
 
 def bench(label, &blk)
@@ -53,19 +50,21 @@ when "check"
 
   spaces = []
   default_branch = nil
-
-  total_commits = 0
+  file = File.new $request[:response_path], 'w'
 
   repo.branches.each do |b|
     unless b.remote?
       # assume the only local branch is the default one
       default_branch = b.name
+
+      file.puts JSON.dump({
+        default_space: default_branch
+      })
       next
     end
 
     space_name = b.name.sub("#{b.remote_name}/", "")
 
-    has_latest = false
     commits = []
 
     walker = Rugged::Walker.new(repo)
@@ -79,41 +78,22 @@ when "check"
       commit = repo.lookup(from[:ref])
       walker.hide(commit)
 
-      commits << commit_version(commit)
-      total_commits += 1
-      has_latest = commit.oid == b.target.oid
+      file.puts commit_version(commit, space_name)
     end
 
-    unless has_latest || total_commits >= CHECK_BATCH_LIMIT
-      bench("#{space_name} walk") do
-        walker.walk do |c|
-          # TODO: test if commit satisfies paths/ignore_paths
-          commits << commit_version(c)
-          total_commits += 1
-          has_latest = c.oid == b.target.oid
-
-          break if total_commits >= CHECK_BATCH_LIMIT
-        end
+    bench("#{space_name} walk") do
+      walker.walk do |c|
+        # TODO: test if commit satisfies paths/ignore_paths
+        file.puts commit_version(c, space_name)
       end
     end
 
     next if commits.empty?
 
     $stderr.puts "#{space_name} commits: #{commits.size} (latest: #{has_latest})"
-
-    spaces << {
-      space: space_name,
-      versions: commits,
-      has_latest: has_latest
-    }
   end
 
-  bench("dump") do
-    puts JSON.dump({
-      spaces: spaces,
-      default_space: default_branch
-    })
-  end
+  file.close
 
 when "get"
   $request = {
diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index af6431b8..85c7bae8 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -75,48 +75,31 @@ from the beginning, as if the space was not specified.
 If any space in `from` is no longer present, the resource should ignore it, and
 not include it in the response.
 
-The resource may limit the number of versions it returns however it likes in
-order to save RAM and incrementally build the history. The `has_latest` field in
-the response (see below) is used to indicate this scenario.
-
-The resource should also determine a "default space", if any. Having a default
+The resource should determine a "default space", if any. Having a default
 space is useful for things like Git repos which have a default branch, or
 version spaces (e.g. `1.8`, `2.0`) which can point to the latest version line by
 default. If there is no default space, the user must specify it explicitly in
 the pipeline, either by configuring one on the resource (`space: foo`) or on the
 `get` step (`spaces: [foo]`).
 
-The command should then emit the collected versions and default space (if any)
-as following JSON response structure the file specified by `response_path`:
-
-```go
-// CheckResponse returns the detected spaces and the default space, if any.
-type CheckResponse struct {
-  Spaces       []DetectedSpace `json:"spaces"`
-  DefaultSpace *Space          `json:"default_space"`
-}
-
-// DetectedSpace is a space and the versions detected therein, listed in
-// chronological order.
-//
-// The HasLatest field indicates whether the list of versions includes the
-// latest version. Until the latest version is collected, Concourse will not
-// schedule the space. This allows resources to limit the number of versions
-// they return at once for things with a ton of history.
-type DetectedSpace struct {
-  Space     Space             `json:"space"`
-  Versions  []DetectedVersion `json:"versions"`
-  HasLatest bool              `json:"has_latest"`
-}
-
-// DetectedVersion is a version within a space with its associated metadata.
-type DetectedVersion struct {
-  Version  Version  `json:"version"`
-  Metadata Metadata `json:"metadata"`
-}
+The command should first emit the default space (if any) and then stream the
+collected versions for each space. Streaming would enable resource authors to
+write versions as they find them rather than hold them all in memory and do one
+big JSON marshal. Each version will be written as individual JSON objects
+streamed to the response_path file. They will include the version, the space
+associated with that version, and the version's metadata. The response should
+look like the following within the file specified by response_path:
+
+```JSON
+{"space":"a","version":{"v":"1"},"metadata":[{"Name":"status","Value":"pending"}]}
+{"space":"a","version":{"v":"2"},"metadata":[{"Name":"status","Value":"pending"}]}
+{"space":"a","version":{"v":"3"},"metadata":[{"Name":"status","Value":"pending"}]}
+{"space":"b","version":{"v":"1"},"metadata":[{"Name":"status","Value":"pending"}]}
+{"space":"b","version":{"v":"2"},"metadata":[{"Name":"status","Value":"pending"}]}
+{"space":"b","version":{"v":"3"},"metadata":[{"Name":"status","Value":"pending"}]}
+// ...
 ```
 
-
 ### `get`: Fetch a version from the resource's space.
 
 The `get` command will be invoked with the following JSON structure on `stdin`:
@@ -155,30 +138,32 @@ The command will be invoked with all of the build plan's artifacts present in
 the working directory, each as `./(artifact name)`.
 
 The command should perform any and all side-effects idempotently, and then
-record the following JSON response structure to the file specified by
-`response_path`:
-
-```go
-type PutResponse struct {
-  Space   Space     `json:"space"`
-  Created []Version `json:"created"`
-  Deleted []Version `json:"deleted"`
-}
+stream the following response to the file specified by `response_path`:
+
+```JSON
+{"type":"created","space":"a","version":{"v":"1"}}
+{"type":"created","space":"a","version":{"v":"2"}}
+{"type":"created","space":"a","version":{"v":"3"}}
+{"type":"created","space":"b","version":{"v":"1"}}
+{"type":"created","space":"b","version":{"v":"2"}}
+{"type":"deleted","space":"b","version":{"v":"3"}}
+// ...
 ```
 
-The `space` field determines the space that has been modified or created. This
-allows new spaces to be generated by a `put` dynamically (based on params
+`put` allows new spaces to be generated dynamically (based on params
 and/or the bits in its working directory) and propagated to the rest of the
 pipeline.
 
-Versions returned under `created` will be recorded as outputs of the build. A
-`check` will then be performed to fill in the metadata and determine the
+Versions returned with `created` type will be recorded as outputs of the build.
+A `check` will then be performed to fill in the metadata and determine the
 ordering of the versions. Once the ordering is learned, the latest version will
 be fetched by the implicit `get`.
 
-Versions returned under `deleted` will be marked as deleted. They will remain
-in the database for archival purposes, but will no longer be input candidates
-for any builds, and can no longer be fetched.
+Versions returned with `deleted` type will be marked as deleted. They will
+remain in the database for archival purposes, but will no longer be input
+candidates for any builds, and can no longer be fetched. The implicit `get`
+after `deleted` puts should not happen so this will result in changes to that
+behavior.
 
 Anything printed to `stdout` and `stderr` will propagate to the build logs.
 
@@ -205,6 +190,16 @@ resource with a better model that supports concurrent version lines (i.e.
 supporting multiple major/minor releases with patches). It does this by managing
 tags in an existing Git repository.
 
+### `s3`
+
+[Code](https://github.com/vito/rfcs/tree/resources-v2/01-resources-v2/s3-example)
+
+This resource models the original `s3` resource. Only regex versions were
+implemented, each space corresponds to a major.minor version. For example, 1.2.0
+and 1.2.1 is the same space but 1.3.0 is a different space. Single numbers are
+also supported with default minor of 0. The default space is set to the latest
+minor version.
+
 
 ## Pipeline Usage
 
diff --git a/01-resources-v2/s3-example/Gemfile b/01-resources-v2/s3-example/Gemfile
new file mode 100644
index 00000000..b64884b3
--- /dev/null
+++ b/01-resources-v2/s3-example/Gemfile
@@ -0,0 +1,4 @@
+source :rubygems
+
+gem 'aws-sdk-s3'
+gem 'semverly'
diff --git a/01-resources-v2/s3-example/Gemfile.lock b/01-resources-v2/s3-example/Gemfile.lock
new file mode 100644
index 00000000..1858f0ae
--- /dev/null
+++ b/01-resources-v2/s3-example/Gemfile.lock
@@ -0,0 +1,30 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    aws-eventstream (1.0.1)
+    aws-partitions (1.106.0)
+    aws-sdk-core (3.33.0)
+      aws-eventstream (~> 1.0)
+      aws-partitions (~> 1.0)
+      aws-sigv4 (~> 1.0)
+      jmespath (~> 1.0)
+    aws-sdk-kms (1.9.0)
+      aws-sdk-core (~> 3, >= 3.26.0)
+      aws-sigv4 (~> 1.0)
+    aws-sdk-s3 (1.21.0)
+      aws-sdk-core (~> 3, >= 3.26.0)
+      aws-sdk-kms (~> 1)
+      aws-sigv4 (~> 1.0)
+    aws-sigv4 (1.0.3)
+    jmespath (1.4.0)
+    semverly (1.0.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  aws-sdk-s3
+  semverly
+
+BUNDLED WITH
+   1.16.2
diff --git a/01-resources-v2/s3-example/artifact b/01-resources-v2/s3-example/artifact
new file mode 100755
index 00000000..7705fc84
--- /dev/null
+++ b/01-resources-v2/s3-example/artifact
@@ -0,0 +1,114 @@
+#!/usr/bin/env ruby
+
+require 'json'
+require 'aws-sdk-s3'
+require 'semverly'
+
+Aws.config.update(
+  region: 'us-east-1',
+  credentials: Aws::Credentials.new(ENV['ACCESS_KEY_ID'],
+                                    ENV['SECRET_ACCESS_KEY'])
+)
+
+client = Aws::S3::Client.new
+
+case ARGV[0]
+when 'check'
+  request = {
+    config: {
+      bucket: 'concourse-s3-test',
+      regex: 'concourse-(.*).pivotal'
+    },
+    response_path: '/tmp/response'
+  }
+
+  bucket = request[:config][:bucket]
+  prefix = request[:config][:regex].partition('(').first
+  file = File.new request[:response_path], 'w'
+
+  response = client.list_objects_v2(
+    bucket: bucket,
+    prefix: prefix
+  )
+
+  versions = []
+
+  response.contents.each do |object|
+    matches = object.key.match request[:config][:regex]
+    continue unless matches
+
+    semver = SemVer.parse(matches[1])
+    space = "#{semver.major}.#{semver.minor}"
+
+    versions << { space: space, path: object.key, version: matches[1] }
+  end
+
+  versions = versions.sort_by do |version|
+    [version[:space], SemVer.parse(version[:version])]
+  end
+
+  latest_version = versions.last
+  if latest_version
+    file.puts JSON.dump(
+      default_space: latest_version[:space]
+    )
+  else
+    file.puts JSON.dump(
+      default_space: nil
+    )
+  end
+
+  versions.each do |version|
+    file.puts JSON.dump(
+      space: version[:space],
+      version: { path: version[:path] },
+      metadata: []
+    )
+  end
+
+  file.close
+
+when 'get'
+  request = {
+    config: {
+      bucket: 'concourse-s3-test',
+      regex: 'concourse-(.*).pivotal'
+    },
+    space: '1.0',
+    version: { path: 'concourse-1.0.0.pivotal' }
+  }
+
+  client.get_object(
+    response_target: request[:version][:path],
+    bucket: request[:config][:bucket],
+    key: request[:version][:path]
+  )
+
+when 'put'
+  request = {
+    config: {
+      bucket: 'concourse-s3-test',
+      regex: 'concourse-(.*).pivotal',
+      file: 'concourse-3.0.0.pivotal'
+    },
+    response_path: '/tmp/response'
+  }
+
+  matches = request[:config][:file].match request[:config][:regex]
+  raise 'file path does not match regex' unless matches
+
+  semver = SemVer.parse(matches[1])
+  space = "#{semver.major}.#{semver.minor}"
+
+  File.open(request[:config][:file], 'rb') do |f|
+    client.put_object(bucket: request[:config][:bucket],
+                      key: request[:config][:file], body: f)
+  end
+
+  File.open(request[:response_path], 'w') do |f|
+    f.puts JSON.dump(
+      space: space,
+      Created: [{ path: request[:config][:file] }]
+    )
+  end
+end
diff --git a/01-resources-v2/s3-example/info b/01-resources-v2/s3-example/info
new file mode 100755
index 00000000..be0e0398
--- /dev/null
+++ b/01-resources-v2/s3-example/info
@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+
+require "json"
+
+puts JSON.dump({
+  artifacts: {
+    api_version: "2.0",
+    check: "artifact check",
+    get: "artifact get",
+    put: "artifact put"
+  }
+})

From 91a23b0c9224ff4bc4c3574936835921dd1ad23a Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Tue, 30 Oct 2018 11:11:42 -0400
Subject: [PATCH 14/19] add clarifying paragraph regarding 'put' space

---
 01-resources-v2/proposal.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index af6431b8..1a9a374d 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -171,6 +171,11 @@ allows new spaces to be generated by a `put` dynamically (based on params
 and/or the bits in its working directory) and propagated to the rest of the
 pipeline.
 
+Note that a `put` may only affect one space at a time, otherwise it becomes
+difficult to express things like "`get` after `put`" to fetch the version that
+was created. If multiple spaces are returned, it's unclear which space the
+`get` would fetch from.
+
 Versions returned under `created` will be recorded as outputs of the build. A
 `check` will then be performed to fill in the metadata and determine the
 ordering of the versions. Once the ordering is learned, the latest version will

From de3d604cbee158af7264953ac4373e8d4d0b775e Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Tue, 20 Nov 2018 16:07:49 -0500
Subject: [PATCH 15/19] resources v2: updates based on impl feedback

this is mostly from learnings we found while implementing #2386 and the
beginnings of the itnerface.
---
 01-resources-v2/proposal.md | 300 ++++++++++++++++++++++++++----------
 1 file changed, 215 insertions(+), 85 deletions(-)

diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 442e0adb..73def312 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -14,7 +14,6 @@ Introduces a new resource interface with the following goals:
 
 * Extend the versioned artifacts interface to support deletion of versions.
 
-
 # Proposal
 
 ## General Types
@@ -62,44 +61,110 @@ type CheckRequest struct {
 }
 ```
 
-The first call will have an empty object as `from`.
+The `check` script responds by writing JSON objects ("events") to a file
+specified by `response_path`. Each JSON object has an `action` and a different
+set of fields based on the action.
+
+The following event types may be emitted by `check`:
+
+* `default_space`: Emitted when the resource has learned of a space which
+  should be considered the "default", e.g. the default branch of a `git` repo
+  or the latest version available for a semver'd resource.
+
+  Required fields for this event:
+
+  * `space`: The name of the space.
+
+* `discovered`: Emitted when a version is discovered for a given space. These
+  must be emitted in chronological order (relative to other `discovered` events
+  for the given space - other events may be intermixed).
+
+  Required fields for this event:
+
+  * `space`: The space the version is in.
+  * `version`: The version object.
+  * `metadata`: A list of JSON objects with `name` and `value`, shown to the
+    user.
+
+* `reset`: Emitted when a given space's "current version" is no longer present
+  (e.g. someone ran `git push -f`). This has the effect of marking all
+  currently-recorded versions of the space 'deleted', after which the resource
+  will emit any and all versions from the beginning, thus 'un-deleting'
+  anything that's actually still there.
+
+  Required fields for this event:
 
-Any spaces discovered by the resource but not present in `from` should collect
-versions from the beginning.
+  * `space`: The name of the space.
 
-For each space in `from`, the resource should collect all versions that appear
-*after* the current version, including the given version if it's still present.
-If the given version is no longer present, the resource should instead collect
-from the beginning, as if the space was not specified.
+The first request will have an empty object as `from`.
 
-If any space in `from` is no longer present, the resource should ignore it, and
-not include it in the response.
+Any spaces discovered by the resource but not present in `from` should emit
+versions from the very first version.
 
-The resource should determine a "default space", if any. Having a default
-space is useful for things like Git repos which have a default branch, or
-version spaces (e.g. `1.8`, `2.0`) which can point to the latest version line by
+For each space and associated version in `from`, the resource should emit all
+versions that appear *after* the given version (not including the given
+version).
+
+If a space or given version in `from` is no longer present (in the case of `git
+push -f` or branch deletion), the resource should emit a `reset` event for the
+space. If the space is still there, but the verion was gone, it should follow
+the `reset` event with all versions detected from the beginning, as if the
+`from` value was never specified.
+
+The resource should determine a "default space", if any. Having a default space
+is useful for things like Git repos which have a default branch, or version
+spaces (e.g. `1.8`, `2.0`) which can point to the latest version line by
 default. If there is no default space, the user must specify it explicitly in
-the pipeline, either by configuring one on the resource (`space: foo`) or on the
-`get` step (`spaces: [foo]`).
-
-The command should first emit the default space (if any) and then stream the
-collected versions for each space. Streaming would enable resource authors to
-write versions as they find them rather than hold them all in memory and do one
-big JSON marshal. Each version will be written as individual JSON objects
-streamed to the response_path file. They will include the version, the space
-associated with that version, and the version's metadata. The response should
-look like the following within the file specified by response_path:
-
-```JSON
-{"space":"a","version":{"v":"1"},"metadata":[{"Name":"status","Value":"pending"}]}
-{"space":"a","version":{"v":"2"},"metadata":[{"Name":"status","Value":"pending"}]}
-{"space":"a","version":{"v":"3"},"metadata":[{"Name":"status","Value":"pending"}]}
-{"space":"b","version":{"v":"1"},"metadata":[{"Name":"status","Value":"pending"}]}
-{"space":"b","version":{"v":"2"},"metadata":[{"Name":"status","Value":"pending"}]}
-{"space":"b","version":{"v":"3"},"metadata":[{"Name":"status","Value":"pending"}]}
-// ...
+the pipeline, either by configuring one on the resource (`default_space: foo`)
+or on every `get` step using the resource (`spaces: [foo]`).
+
+#### example
+
+Given the following request on `stdin`:
+
+```json
+{
+  "config": {
+    "uri": "https://github.com/concourse/concourse"
+  },
+  "from": {
+    "master": {"ref": "abc123"},
+    "feature/foo": {"ref":"def456"}
+    "feature/bar": {"ref":"987cia"}
+  },
+  "response_path": "/tmp/check-response.json"
+}
+```
+
+If the `feature/foo` branch has new commits, `master` is the default branch and
+has no new commits, and `feature/bar` has been `push -f`ed, you may see
+something like the following in `/tmp/check-response.json`:
+
+```json
+{"action":"discovered","space":"feature/foo","version":{"ref":"abcdf8"},"metadata":[{"name":"message","value":"fix thing"}]}
+{"action":"reset","space":"feature/bar"}
+{"action":"discovered","space":"feature/bar","version":{"ref":"abcde0"},"metadata":[{"name":"message","value":"initial commit"}]}
+{"action":"discovered","space":"feature/bar","version":{"ref":"abcde1"},"metadata":[{"name":"message","value":"add readme"}]}
+{"action":"default_space","space":"master"}
+{"action":"discovered","space":"feature/foo","version":{"ref":"abcdf9"},"metadata":[{"name":"message","value":"fix thing even more"}]}
+{"action":"discovered","space":"feature/bar","version":{"ref":"abcde2"},"metadata":[{"name":"message","value":"finish the feature"}]}
 ```
 
+A few things to note:
+
+* A `reset` event is emitted immediately upon detecting that the given version
+  for `feature/bar` (`987cia`) is no longer available, followed by a
+  `discovered` event for every commit going back to the initial commit on the
+  branch.
+
+* No versions are emitted for `master`, because it's already up to date
+  (`abc123` is the latest commit).
+
+* The versions detected for `feature/foo` may appear between events for
+  `feature/bar`, as they're for unrelated spaces. The order only matters within
+  the space.
+
+
 ### `get`: Fetch a version from the resource's space.
 
 The `get` command will be invoked with the following JSON structure on `stdin`:
@@ -137,40 +202,99 @@ type PutRequest struct {
 The command will be invoked with all of the build plan's artifacts present in
 the working directory, each as `./(artifact name)`.
 
-The command should perform any and all side-effects idempotently, and then
-stream the following response to the file specified by `response_path`:
-
-```JSON
-{"type":"created","space":"a","version":{"v":"1"}}
-{"type":"created","space":"a","version":{"v":"2"}}
-{"type":"created","space":"a","version":{"v":"3"}}
-{"type":"created","space":"b","version":{"v":"1"}}
-{"type":"created","space":"b","version":{"v":"2"}}
-{"type":"deleted","space":"b","version":{"v":"3"}}
-// ...
-```
+The `put` script responds by writing JSON objects ("events") to a file
+specified by `response_path`, just like `check`. Each JSON object has an
+`action` and a different set of fields based on the action.
+
+Anything printed to `stdout` and `stderr` will propagate to the build logs.
+
+The following event types may be emitted by `put`:
+
+* `created`: Emitted when the resource has created (perhaps idempotently) a
+  version. The version will be recorded as an output of the build.
+
+  Versions produced by `put` will *not* be directly inserted into the
+  resource's version history in the pipeline, as they were with v1 resources.
+  This enables one-off versions to be created and fetched within a build
+  without disrupting the normal detection of resource versions across the
+
+  Required fields for this event:
+
+  * `space`: The space the version is in.
+  * `version`: The version object.
+  * `metadata`: A list of JSON objects with `name` and `value`, shown to the
+    user. Note that this is return by both `put` and `check`, because there's a
+    chance that `put` produces a version that wouldn't normally be discovered
+    by `check`.
+
+* `deleted`: Emitted when a version has been deleted. The version record will
+  remain in the database for archival purposes, but it will no longer be a
+  candidate for any builds.
+
+  Required fields for this event:
 
-`put` allows new spaces to be generated dynamically (based on params
-and/or the bits in its working directory) and propagated to the rest of the
-pipeline.
+  * `space`: The space the version is in.
+  * `version`: The version object.
 
-Note that a `put` may only affect one space at a time, otherwise it becomes
+Because the space is included on each event, `put` allows a new space to be
+generated dynamically (based on params and/or the bits in its working
+directory) and propagated to the rest of the pipeline. However it must take
+care to only affect one space at a time. Without this restriction it becomes
 difficult to express things like "`get` after `put`" to fetch the version that
 was created. If multiple spaces are returned, it's unclear which space the
 `get` would fetch from.
 
-Versions returned with `created` type will be recorded as outputs of the build.
-A `check` will then be performed to fill in the metadata and determine the
-ordering of the versions. Once the ordering is learned, the latest version will
-be fetched by the implicit `get`.
+#### the `get` after the `put`
 
-Versions returned with `deleted` type will be marked as deleted. They will
-remain in the database for archival purposes, but will no longer be input
-candidates for any builds, and can no longer be fetched. The implicit `get`
-after `deleted` puts should not happen so this will result in changes to that
-behavior.
+With v1 resources, every `put` implied a `get` of the version that was created.
+With v2 we will change that, so that the `get` is opt-in. This has been a
+long-time ask, and one objective reason to make it opt-in is that Concourse
+can't know ahead of time that there will even be anything to `get` - for
+example, the `put` could emit only `deleted` events.
 
-Anything printed to `stdout` and `stderr` will propagate to the build logs.
+So, to `get` the latest version that was produced by the `put`, you would
+configure something like:
+
+```yaml
+- put: my-resource
+  get: my-created-resource
+- task: use-my-created-resource
+```
+
+The value for the `get` field is the name of the artifact to save. When
+specified, the last version emitted will be fetched.
+
+This added flexibility enables resources to provide explicitly versioned
+'variants' of original versions without doubling up the version history. One
+use case for this is pull-requests: you may want a build to pull in one
+resource for the PR itself, another resource for the base branch of the
+upstream reap, and then `put` to produce a "combined" version of the two,
+representing the PR merged into the upstream repo:
+
+```yaml
+jobs:
+- name: run-pr
+  plan:
+  - get: concourse-pr  # pr: 123, ref: deadbeef
+    trigger: true
+  - get: concourse     # ref: abcdef
+  - put: concourse-pr
+    get: merged-pr
+    params:
+      merge_base: concourse
+      status: pending
+
+    # the `put` will learns base ref from `concourse` input and param, and emit
+    # a 'created' event with the following version:
+    #
+    #   pr: 123, ref: deadbeef, base: abcdef
+    #
+    # the `get` will then run with that version and knows to merge onto the
+    # given base ref
+
+  - task: unit
+    # uses 'merged-pr' as an input
+```
 
 
 # Examples
@@ -242,9 +366,9 @@ TODO:
 
 * Remove the distinction between `source` and `params`; resources will receive
   a single `config`. The distinction will remain in the pipeline. This makes it
-  easier to implement a resource without planning ahead for interesting dynamic
-  vs. static usage patterns, and this will become more powerful with
-  concourse/concourse#684.
+  easier to implement a resource without planning ahead for dynamic vs. static
+  usage patterns. This will become more powerful if concourse/concourse#684 is
+  implemented.
 
 * Change `check` to run against all spaces. It will be given a mapping of each
   space to its current latest version, and return the set of all spaces, along
@@ -254,18 +378,27 @@ TODO:
   efficiently perform the check. It also keeps the container overhead down to
   one per resource, rather than one per space.
 
-* Change `put` to emit a set of created versions, rather than just one.
+* Remove the implicit `get` after every `put`, now requiring the pipeline to
+  explicitly configure a `get` field on the same step. This is necessary now
+  that `put` can potentially perform an operation resulting solely in `deleted`
+  events, in which case there is nothing to fetch.
+
+  This has also been requested by users for quite a while, for the sake of
+  optimizing jobs that have no need for the implicit `get`.
+
+* Change `put` to emit a sequence of created versions, rather than just one.
 
   Technically the `git` resource may push many commits, so returning more than
   one version is necessary to track them all as outputs of a build. This could
   also support batch creation.
 
-  To ensure `check` is the source of truth for ordering, the versions returned
-  by `put` are not order dependent. A `check` will be performed to discover
-  them in the correct order, and then each version will be saved as an output
-  of the build. The latest version of the set will then be fetched.
+  To ensure `check` is the source of truth for ordering, the versions emitted
+  by `put` are not directly inserted into the database. Instead, they are
+  simply recorded as outputs of the build. The order does matter, however - if
+  a user configures a `get` on the `put` step, the last version emitted will be
+  fetched. For this reason they should be emitted in chronological order.
 
-* Change `put` to additionally return a set of *deleted* versions.
+* Change `put` to additionally return a sequence of *deleted* versions.
 
   There has long been a call for a batch `delete` or `destroy` action. Adding
   this to `put` alongside the set of created versions allows `put` to become a
@@ -275,11 +408,11 @@ TODO:
 * Change `get` to always run against a particular space, given by
   the request payload.
 
-* Change `check` to include metadata for each version. Change `get` and `put`
-  to no longer return it.
+* Change `check` to include metadata for each version. Change `get` to no
+  longer return it.
 
-  This way metadata is always immediately available, and only comes from one
-  place.
+  This way metadata is always immediately available, which could enable us to
+  have a richer UI for the version history page.
 
   The original thought was that metadata collection may be expensive, but so
   far we haven't seen that to be the case.
@@ -292,14 +425,19 @@ TODO:
   rather than taking the path as an argument. This was something people would
   trip up on when implementing a resource.
 
-* Change `put` to write its JSON response to a specified file, rather than
-  `stdout`, so that we don't have to be attached to process its response.
+* Change `check` and `put` to write its JSON response to a specified file,
+  rather than `stdout`, so that we don't have to be attached to process its
+  response.
 
   This is one of the few ways a build can error after the ATC reattaches
   (`unexpected end of JSON`). With it written to a file, we can just try to
   read the file when we re-attach after seeing that the process exited. This
-  also frees up stdout/stderr for normal logging, which has been an occasional
-  pitfall during resource development/debugging.
+  also frees up `stdout`/`stderr` for normal logging, which has been an
+  occasional pitfall during resource development/debugging.
+
+  Another motivation for this is safety: with `check` emitting a ton of data,
+  there is danger in Garden losing windows of the output due to a slow
+  consumer. Writing to a file circumvents this issue.
 
 
 # Answered(?) Questions
@@ -401,13 +539,5 @@ branch name, and that space could then be passed along.
 
 Now that we're going to be collecting all versions of every resource, we should
 be careful not to be scanning the entire table all the time, and even make an
-effort to share data when possible. For example, we may want to associate
-collected versions to a global resource config object, rather than saving them
-all per-pipeline-resource.
-
-Here are some optimizations we probably want to make:
-
-* `(db.Pipeline).GetLatestVersionedResource` is called every minute and scans
-  the table to find the latest version of a given resource. We should reduce
-  this to a simple join column from the resource to the latest version,
-  maintained every time we save new versions.
+effort to share data when possible. We have implemented this with
+https://github.com/concourse/concourse/issues/2386.

From 2fa7158edcf612d6d50c7f26e1c14dca26a11a12 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Tue, 20 Nov 2018 16:11:33 -0500
Subject: [PATCH 16/19] add comma, move blurb to 'new implications'

---
 01-resources-v2/proposal.md | 68 +++++++++++++++++++------------------
 1 file changed, 35 insertions(+), 33 deletions(-)

diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 73def312..636aa000 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -129,7 +129,7 @@ Given the following request on `stdin`:
   },
   "from": {
     "master": {"ref": "abc123"},
-    "feature/foo": {"ref":"def456"}
+    "feature/foo": {"ref":"def456"},
     "feature/bar": {"ref":"987cia"}
   },
   "response_path": "/tmp/check-response.json"
@@ -264,38 +264,6 @@ configure something like:
 The value for the `get` field is the name of the artifact to save. When
 specified, the last version emitted will be fetched.
 
-This added flexibility enables resources to provide explicitly versioned
-'variants' of original versions without doubling up the version history. One
-use case for this is pull-requests: you may want a build to pull in one
-resource for the PR itself, another resource for the base branch of the
-upstream reap, and then `put` to produce a "combined" version of the two,
-representing the PR merged into the upstream repo:
-
-```yaml
-jobs:
-- name: run-pr
-  plan:
-  - get: concourse-pr  # pr: 123, ref: deadbeef
-    trigger: true
-  - get: concourse     # ref: abcdef
-  - put: concourse-pr
-    get: merged-pr
-    params:
-      merge_base: concourse
-      status: pending
-
-    # the `put` will learns base ref from `concourse` input and param, and emit
-    # a 'created' event with the following version:
-    #
-    #   pr: 123, ref: deadbeef, base: abcdef
-    #
-    # the `get` will then run with that version and knows to merge onto the
-    # given base ref
-
-  - task: unit
-    # uses 'merged-pr' as an input
-```
-
 
 # Examples
 
@@ -532,6 +500,40 @@ This can be done by representing each non-linear version in a separate space.
 For example, generated code could be pushed to a generated (but deterministic)
 branch name, and that space could then be passed along.
 
+## Build-local Versions
+
+Now that `put` doesn't directly modify the resource's version history, it can
+be used to provide explicitly versioned 'variants' of original versions without
+doubling up the version history. One use case for this is pull-requests: you
+may want a build to pull in one resource for the PR itself, another resource
+for the base branch of the upstream reap, and then `put` to produce a
+"combined" version of the two, representing the PR merged into the upstream
+repo:
+
+```yaml
+jobs:
+- name: run-pr
+  plan:
+  - get: concourse-pr  # pr: 123, ref: deadbeef
+    trigger: true
+  - get: concourse     # ref: abcdef
+  - put: concourse-pr
+    get: merged-pr
+    params:
+      merge_base: concourse
+      status: pending
+
+    # the `put` will learns base ref from `concourse` input and param, and emit
+    # a 'created' event with the following version:
+    #
+    #   pr: 123, ref: deadbeef, base: abcdef
+    #
+    # the `get` will then run with that version and knows to merge onto the
+    # given base ref
+
+  - task: unit
+    # uses 'merged-pr' as an input
+```
 
 # Implementation Notes
 

From 9ce0cabe601cfc3ab67080665188be25d8dd7a5c Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Wed, 5 Dec 2018 16:41:29 +0000
Subject: [PATCH 17/19] revise summary and add motivation section

---
 01-resources-v2/proposal.md | 64 +++++++++++++++++++++++++++++++------
 1 file changed, 55 insertions(+), 9 deletions(-)

diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index 636aa000..f1b7e167 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -1,21 +1,67 @@
 # Summary
 
-Introduces a new resource interface with the following goals:
+This RFC proposes a new resource interface to replace the existing resource
+interface.
 
-* Introduce versioning to the resource interface, so that we can maintain
-  backwards-compatibility.
+As part of this proposal, the interface will now be versioned, starting at 2.0.
+Today's resource interface (documented
+[here](https://github.com/concourse/docs/blob/b9d291e5a821046b8a5de48c50b5ccba5a977493/lit/reference/resource-types/implementing.lit))
+will be called version 1, even though it was never really versioned.
 
-* Support for spaces (concourse/concourse#1707).
+The introduction of this new interface will be gradual, allowing Concourse
+users to use a mix of v1 and v2 resources throughout their pipelines. While the
+new interface is defined in terms of entirely new concepts like
+[spaces](https://github.com/concourse/concourse/issues/1707), v1 resources will
+be silently 'adapted' to v2 automatically.
 
-* Introduce a more airtight "versioned artifacts" interface, tightening up
-  loopholes in today's resource API to ensure that resources are pointing at an
-  external source of truth, so that we can explicitly design for new features
-  and workflows rather than forcing everything into the resource interface.
 
-* Extend the versioned artifacts interface to support deletion of versions.
+# Motivation
+
+* Support for multi-branch workflows and build matrixes:
+    * https://github.com/concourse/concourse/issues/1172
+    * https://github.com/concourse/concourse/issues/1707
+
+* Support for creating new branches dynamically (as spaces):
+    * https://github.com/concourse/git-resource/pull/172
+
+* Support for creating multiple versions at once:
+    * https://github.com/concourse/concourse/issues/535
+    * https://github.com/concourse/concourse/issues/2660
+
+* Support for deleting versions:
+    * https://github.com/concourse/concourse/issues/362
+
+* Having resource metadata immediately available via `check`:
+    * https://github.com/concourse/git-resource/issues/193
+
+* Unifying `source` and `params` as just `config` so that resources don't have
+  to care where configuration is being set in pipelines:
+    * https://github.com/concourse/concourse/issues/310
+
+* Improving stability of reattaching to builds by reading resource responses
+  from files instead of `stdout`:
+    * https://github.com/concourse/concourse/issues/1580
+
+* Ensuring resource version history is always correct and up-to-date, enabling
+  it to be [deduped](https://github.com/concourse/concourse/issues/2386) and
+  removing the need for [purging
+  history](https://github.com/concourse/concourse/issues/145) and
+  [removing/renaming
+  resources](https://github.com/concourse/concourse/issues/372).
+
+* Closing gaps in the resource interface that turned them into a "local maxima"
+  and resulted in their being used in somewhat cumbersome ways (notifications,
+  partially-implemented resources, etc.)
+
 
 # Proposal
 
+* TODO: document 'info'
+* TODO: make metadata more flexible?
+    * https://github.com/concourse/concourse/issues/310
+    * https://github.com/concourse/concourse/issues/2900
+
+
 ## General Types
 
 ```go

From 5a3d3ab4c199e26a70581957acb6b0ac4f8f7bf7 Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Wed, 5 Dec 2018 16:43:59 +0000
Subject: [PATCH 18/19] allow `put` to affect multiple spaces

also move 'new implications' above the questions section, and raise 'get
after put' as an open question related specifically to pipelines
---
 01-resources-v2/proposal.md | 203 +++++++++++++++++++++---------------
 1 file changed, 119 insertions(+), 84 deletions(-)

diff --git a/01-resources-v2/proposal.md b/01-resources-v2/proposal.md
index f1b7e167..c6dc358d 100644
--- a/01-resources-v2/proposal.md
+++ b/01-resources-v2/proposal.md
@@ -234,7 +234,7 @@ No response is expected.
 Anything printed to `stdout` and `stderr` will propagate to the build logs.
 
 
-### `put`: Idempotently create or destroy resource versions in a space.
+### `put`: Idempotently create or destroy resource versions across spaces.
 
 The `put` command will be invoked with the following JSON structure on `stdin`:
 
@@ -282,33 +282,9 @@ The following event types may be emitted by `put`:
   * `space`: The space the version is in.
   * `version`: The version object.
 
-Because the space is included on each event, `put` allows a new space to be
-generated dynamically (based on params and/or the bits in its working
-directory) and propagated to the rest of the pipeline. However it must take
-care to only affect one space at a time. Without this restriction it becomes
-difficult to express things like "`get` after `put`" to fetch the version that
-was created. If multiple spaces are returned, it's unclear which space the
-`get` would fetch from.
-
-#### the `get` after the `put`
-
-With v1 resources, every `put` implied a `get` of the version that was created.
-With v2 we will change that, so that the `get` is opt-in. This has been a
-long-time ask, and one objective reason to make it opt-in is that Concourse
-can't know ahead of time that there will even be anything to `get` - for
-example, the `put` could emit only `deleted` events.
-
-So, to `get` the latest version that was produced by the `put`, you would
-configure something like:
-
-```yaml
-- put: my-resource
-  get: my-created-resource
-- task: use-my-created-resource
-```
-
-The value for the `get` field is the name of the artifact to save. When
-specified, the last version emitted will be fetched.
+Because the space is included on each event, `put` allows new spaces to be
+generated dynamically based on params and/or the bits in its working directory
+and propagated to the rest of the pipeline.
 
 
 # Examples
@@ -450,11 +426,123 @@ TODO:
   occasional pitfall during resource development/debugging.
 
   Another motivation for this is safety: with `check` emitting a ton of data,
-  there is danger in Garden losing windows of the output due to a slow
-  consumer. Writing to a file circumvents this issue.
+  there is danger in Garden losing chunks of the output due to a slow consumer.
+  Writing to a file circumvents this issue.
+
+
+# New Implications
+
+## The `get` after the `put` in Concourse pipelines
+
+With v1 resources, every `put` in a Concourse pipeline implied a `get` of the
+version that was created. With v2, the `get` will be made opt-in. This has been
+a long-time ask, and one objective reason to make it opt-in is that Concourse
+can't know ahead of time that there will even be anything to `get` - for
+example, the `put` could emit only `deleted` events.
+
+So, to `get` the latest version that was produced by the `put`, you would
+configure something like:
+
+```yaml
+- put: my-resource
+  get: my-created-resource
+- task: use-my-created-resource
+```
+
+The value for the `get` field is the name under which the artifact will be
+saved (just like `get` steps). When specified, the last version emitted will be
+fetched (from whichever space it was in).
+
+## Single-state resources
+
+Resources that really only have a "current state", such as deployments, can now
+represent their state more clearly because old versions that are no longer
+there will be marked 'deleted'.
+
+## Non-linearly versioned artifact storage
+
+This can be done by representing each non-linear version in a separate space.
+For example, generated code could be pushed to a generated (but deterministic)
+branch name, and that space could then be passed along.
+
+## Build-local Versions
+
+Now that `put` doesn't directly modify the resource's version history, it can
+be used to provide explicitly versioned 'variants' of original versions without
+doubling up the version history. One use case for this is pull-requests: you
+may want a build to pull in one resource for the PR itself, another resource
+for the base branch of the upstream reap, and then `put` to produce a
+"combined" version of the two, representing the PR merged into the upstream
+repo:
+
+```yaml
+jobs:
+- name: run-pr
+  plan:
+  - get: concourse-pr  # pr: 123, ref: deadbeef
+    trigger: true
+  - get: concourse     # ref: abcdef
+  - put: concourse-pr
+    get: merged-pr
+    params:
+      merge_base: concourse
+      status: pending
+
+    # the `put` will learns base ref from `concourse` input and param, and emit
+    # a 'created' event with the following version:
+    #
+    #   pr: 123, ref: deadbeef, base: abcdef
+    #
+    # the `get` will then run with that version and knows to merge onto the
+    # given base ref
+
+  - task: unit
+    # uses 'merged-pr' as an input
+```
+
+
+# Open Questions
+
+## Are there examples of `put`ing to multiple spaces at once?
+
+Initially there was a limitation that `put` could only emit versions pertaining
+to a single space. This was to prevent ambiguity with "`get` after `put`" -
+which space would the `get` fetch from? We loosened this constraint because it
+felt somewhat arbitrary, as the protocol allows it easily, and recording
+outputs and marking versions as deleted across spaces isn't any harder than
+with a single space.
+
+To loosen the constraint we've instead constrained the `get` to only fetch the
+last version, from whichever space it was in. But are there any good examples
+of this being useful, or have we just moved the arbitrary restriction
+elsewhere? (At least we've moved it out of the resource interface - technically
+this is a pipeline concern, not a resource interface concern.)
+
+Would users want to fetch multiple spaces that were created? Would they want to
+do this statically (at pipeline definition time) or dynamically (at runtime)?
+Static would be relatively easily as the build plan would just result in
+multiple `get` steps, but dynamic would run into the same challenges as with
+[dynamic build plan
+generation](https://github.com/concourse/concourse/issues/684). However users
+could always just separate it into a different job spanning the spaces
+dynamically with wildcards.
+
+Here's a mockup for static configuration:
+
+```yaml
+- put: foo
+  params: bar
+  get: {artifact-name-a: space-a, artifact-name-b: space-b}
+```
+
+...but is that useful?
+
+This is really in need of a use case to define it further, but for now the
+constraint has been lifted from the resource interface, and it's up to the rest
+of Concourse's pipeline mechanics to determine what's possible from there.
 
 
-# Answered(?) Questions
+# Answered Questions
 
 <details><summary>Can we reduce the `check` overhead?</summary>
 
@@ -528,59 +616,6 @@ versions. It'll return multiple spaces and versions created/deleted for them.
 </details>
 
 
-# New Implications
-
-Here are a few use cases that resources were sometimes used for inappropriately:
-
-## Single-state resources
-
-Resources that really only have a "current state", such as deployments. This is
-still "change over time", but the difference is that old versions become
-invalid as soon as there's a new one. This can now be made more clear by
-marking the old versions as "deleted", either proactively via `put` or by
-`check` discovering the new version.
-
-## Non-linearly versioned artifact storage
-
-This can be done by representing each non-linear version in a separate space.
-For example, generated code could be pushed to a generated (but deterministic)
-branch name, and that space could then be passed along.
-
-## Build-local Versions
-
-Now that `put` doesn't directly modify the resource's version history, it can
-be used to provide explicitly versioned 'variants' of original versions without
-doubling up the version history. One use case for this is pull-requests: you
-may want a build to pull in one resource for the PR itself, another resource
-for the base branch of the upstream reap, and then `put` to produce a
-"combined" version of the two, representing the PR merged into the upstream
-repo:
-
-```yaml
-jobs:
-- name: run-pr
-  plan:
-  - get: concourse-pr  # pr: 123, ref: deadbeef
-    trigger: true
-  - get: concourse     # ref: abcdef
-  - put: concourse-pr
-    get: merged-pr
-    params:
-      merge_base: concourse
-      status: pending
-
-    # the `put` will learns base ref from `concourse` input and param, and emit
-    # a 'created' event with the following version:
-    #
-    #   pr: 123, ref: deadbeef, base: abcdef
-    #
-    # the `get` will then run with that version and knows to merge onto the
-    # given base ref
-
-  - task: unit
-    # uses 'merged-pr' as an input
-```
-
 # Implementation Notes
 
 ## Performance Implications

From 7e5471ae805e43d11c5d50a5d1fe44d2272a806f Mon Sep 17 00:00:00 2001
From: Alex Suraci <suraci.alex@gmail.com>
Date: Wed, 5 Dec 2018 16:45:07 +0000
Subject: [PATCH 19/19] add a 0 to the folder name

---
 {01-resources-v2 => 001-resources-v2}/branch-gen.yml              | 0
 {01-resources-v2 => 001-resources-v2}/git-example/.gitignore      | 0
 {01-resources-v2 => 001-resources-v2}/git-example/Gemfile         | 0
 {01-resources-v2 => 001-resources-v2}/git-example/Gemfile.lock    | 0
 {01-resources-v2 => 001-resources-v2}/git-example/README.md       | 0
 {01-resources-v2 => 001-resources-v2}/git-example/artifact        | 0
 .../git-example/collect-all-versions                              | 0
 {01-resources-v2 => 001-resources-v2}/git-example/info            | 0
 {01-resources-v2 => 001-resources-v2}/main-pipeline.yml           | 0
 {01-resources-v2 => 001-resources-v2}/notifications.yml           | 0
 {01-resources-v2 => 001-resources-v2}/proposal.md                 | 0
 {01-resources-v2 => 001-resources-v2}/prs.yml                     | 0
 {01-resources-v2 => 001-resources-v2}/release-pipeline.yml        | 0
 {01-resources-v2 => 001-resources-v2}/s3-example/Gemfile          | 0
 {01-resources-v2 => 001-resources-v2}/s3-example/Gemfile.lock     | 0
 {01-resources-v2 => 001-resources-v2}/s3-example/artifact         | 0
 {01-resources-v2 => 001-resources-v2}/s3-example/info             | 0
 {01-resources-v2 => 001-resources-v2}/semver-example/.gitignore   | 0
 {01-resources-v2 => 001-resources-v2}/semver-example/Gemfile      | 0
 {01-resources-v2 => 001-resources-v2}/semver-example/Gemfile.lock | 0
 {01-resources-v2 => 001-resources-v2}/semver-example/README.md    | 0
 {01-resources-v2 => 001-resources-v2}/semver-example/artifact     | 0
 {01-resources-v2 => 001-resources-v2}/semver-example/info         | 0
 23 files changed, 0 insertions(+), 0 deletions(-)
 rename {01-resources-v2 => 001-resources-v2}/branch-gen.yml (100%)
 rename {01-resources-v2 => 001-resources-v2}/git-example/.gitignore (100%)
 rename {01-resources-v2 => 001-resources-v2}/git-example/Gemfile (100%)
 rename {01-resources-v2 => 001-resources-v2}/git-example/Gemfile.lock (100%)
 rename {01-resources-v2 => 001-resources-v2}/git-example/README.md (100%)
 rename {01-resources-v2 => 001-resources-v2}/git-example/artifact (100%)
 rename {01-resources-v2 => 001-resources-v2}/git-example/collect-all-versions (100%)
 rename {01-resources-v2 => 001-resources-v2}/git-example/info (100%)
 rename {01-resources-v2 => 001-resources-v2}/main-pipeline.yml (100%)
 rename {01-resources-v2 => 001-resources-v2}/notifications.yml (100%)
 rename {01-resources-v2 => 001-resources-v2}/proposal.md (100%)
 rename {01-resources-v2 => 001-resources-v2}/prs.yml (100%)
 rename {01-resources-v2 => 001-resources-v2}/release-pipeline.yml (100%)
 rename {01-resources-v2 => 001-resources-v2}/s3-example/Gemfile (100%)
 rename {01-resources-v2 => 001-resources-v2}/s3-example/Gemfile.lock (100%)
 rename {01-resources-v2 => 001-resources-v2}/s3-example/artifact (100%)
 rename {01-resources-v2 => 001-resources-v2}/s3-example/info (100%)
 rename {01-resources-v2 => 001-resources-v2}/semver-example/.gitignore (100%)
 rename {01-resources-v2 => 001-resources-v2}/semver-example/Gemfile (100%)
 rename {01-resources-v2 => 001-resources-v2}/semver-example/Gemfile.lock (100%)
 rename {01-resources-v2 => 001-resources-v2}/semver-example/README.md (100%)
 rename {01-resources-v2 => 001-resources-v2}/semver-example/artifact (100%)
 rename {01-resources-v2 => 001-resources-v2}/semver-example/info (100%)

diff --git a/01-resources-v2/branch-gen.yml b/001-resources-v2/branch-gen.yml
similarity index 100%
rename from 01-resources-v2/branch-gen.yml
rename to 001-resources-v2/branch-gen.yml
diff --git a/01-resources-v2/git-example/.gitignore b/001-resources-v2/git-example/.gitignore
similarity index 100%
rename from 01-resources-v2/git-example/.gitignore
rename to 001-resources-v2/git-example/.gitignore
diff --git a/01-resources-v2/git-example/Gemfile b/001-resources-v2/git-example/Gemfile
similarity index 100%
rename from 01-resources-v2/git-example/Gemfile
rename to 001-resources-v2/git-example/Gemfile
diff --git a/01-resources-v2/git-example/Gemfile.lock b/001-resources-v2/git-example/Gemfile.lock
similarity index 100%
rename from 01-resources-v2/git-example/Gemfile.lock
rename to 001-resources-v2/git-example/Gemfile.lock
diff --git a/01-resources-v2/git-example/README.md b/001-resources-v2/git-example/README.md
similarity index 100%
rename from 01-resources-v2/git-example/README.md
rename to 001-resources-v2/git-example/README.md
diff --git a/01-resources-v2/git-example/artifact b/001-resources-v2/git-example/artifact
similarity index 100%
rename from 01-resources-v2/git-example/artifact
rename to 001-resources-v2/git-example/artifact
diff --git a/01-resources-v2/git-example/collect-all-versions b/001-resources-v2/git-example/collect-all-versions
similarity index 100%
rename from 01-resources-v2/git-example/collect-all-versions
rename to 001-resources-v2/git-example/collect-all-versions
diff --git a/01-resources-v2/git-example/info b/001-resources-v2/git-example/info
similarity index 100%
rename from 01-resources-v2/git-example/info
rename to 001-resources-v2/git-example/info
diff --git a/01-resources-v2/main-pipeline.yml b/001-resources-v2/main-pipeline.yml
similarity index 100%
rename from 01-resources-v2/main-pipeline.yml
rename to 001-resources-v2/main-pipeline.yml
diff --git a/01-resources-v2/notifications.yml b/001-resources-v2/notifications.yml
similarity index 100%
rename from 01-resources-v2/notifications.yml
rename to 001-resources-v2/notifications.yml
diff --git a/01-resources-v2/proposal.md b/001-resources-v2/proposal.md
similarity index 100%
rename from 01-resources-v2/proposal.md
rename to 001-resources-v2/proposal.md
diff --git a/01-resources-v2/prs.yml b/001-resources-v2/prs.yml
similarity index 100%
rename from 01-resources-v2/prs.yml
rename to 001-resources-v2/prs.yml
diff --git a/01-resources-v2/release-pipeline.yml b/001-resources-v2/release-pipeline.yml
similarity index 100%
rename from 01-resources-v2/release-pipeline.yml
rename to 001-resources-v2/release-pipeline.yml
diff --git a/01-resources-v2/s3-example/Gemfile b/001-resources-v2/s3-example/Gemfile
similarity index 100%
rename from 01-resources-v2/s3-example/Gemfile
rename to 001-resources-v2/s3-example/Gemfile
diff --git a/01-resources-v2/s3-example/Gemfile.lock b/001-resources-v2/s3-example/Gemfile.lock
similarity index 100%
rename from 01-resources-v2/s3-example/Gemfile.lock
rename to 001-resources-v2/s3-example/Gemfile.lock
diff --git a/01-resources-v2/s3-example/artifact b/001-resources-v2/s3-example/artifact
similarity index 100%
rename from 01-resources-v2/s3-example/artifact
rename to 001-resources-v2/s3-example/artifact
diff --git a/01-resources-v2/s3-example/info b/001-resources-v2/s3-example/info
similarity index 100%
rename from 01-resources-v2/s3-example/info
rename to 001-resources-v2/s3-example/info
diff --git a/01-resources-v2/semver-example/.gitignore b/001-resources-v2/semver-example/.gitignore
similarity index 100%
rename from 01-resources-v2/semver-example/.gitignore
rename to 001-resources-v2/semver-example/.gitignore
diff --git a/01-resources-v2/semver-example/Gemfile b/001-resources-v2/semver-example/Gemfile
similarity index 100%
rename from 01-resources-v2/semver-example/Gemfile
rename to 001-resources-v2/semver-example/Gemfile
diff --git a/01-resources-v2/semver-example/Gemfile.lock b/001-resources-v2/semver-example/Gemfile.lock
similarity index 100%
rename from 01-resources-v2/semver-example/Gemfile.lock
rename to 001-resources-v2/semver-example/Gemfile.lock
diff --git a/01-resources-v2/semver-example/README.md b/001-resources-v2/semver-example/README.md
similarity index 100%
rename from 01-resources-v2/semver-example/README.md
rename to 001-resources-v2/semver-example/README.md
diff --git a/01-resources-v2/semver-example/artifact b/001-resources-v2/semver-example/artifact
similarity index 100%
rename from 01-resources-v2/semver-example/artifact
rename to 001-resources-v2/semver-example/artifact
diff --git a/01-resources-v2/semver-example/info b/001-resources-v2/semver-example/info
similarity index 100%
rename from 01-resources-v2/semver-example/info
rename to 001-resources-v2/semver-example/info