Image registry cache #669

modulo11 · 2021-04-20T12:08:52Z

This implements parts of #652 and additionally offers images to be used as caches. The current default behavior is not changed but some parts are renamed for better clarification.

Currently, it's an either volume or registry cache although supporting both (as depicted in #652) can be achieved easily. Using both caching arguments (-cache-dir and -cache-image) with the CNB lifecycle resulted in only utilizing the cache image (during my experiments).

@c0d1ngm0nk3y
@loewenstein

codecov-commenter · 2021-04-20T12:12:57Z

Codecov Report

Merging #669 (e1e91de) into main (33c4d25) will increase coverage by 0.11%.
The diff coverage is 66.10%.

@@            Coverage Diff             @@
##             main     #669      +/-   ##
==========================================
+ Coverage   71.24%   71.35%   +0.11%     
==========================================
  Files         117      118       +1     
  Lines        5421     5439      +18     
==========================================
+ Hits         3862     3881      +19     
+ Misses       1191     1187       -4     
- Partials      368      371       +3

Impacted Files	Coverage Δ
pkg/apis/build/v1alpha2/build.go	`35.71% <ø> (+7.30%)`	⬆️
pkg/apis/build/v1alpha2/image_types.go	`50.00% <ø> (ø)`
pkg/cnb/cnb_metadata.go	`46.29% <0.00%> (-14.68%)`	⬇️
pkg/reconciler/build/build.go	`72.80% <50.00%> (-0.76%)`	⬇️
pkg/apis/build/v1alpha2/image_builds.go	`60.62% <64.28%> (+1.30%)`	⬆️
pkg/apis/build/v1alpha2/build_pod.go	`97.92% <100.00%> (+<0.01%)`	⬆️
pkg/apis/build/v1alpha2/build_types.go	`100.00% <100.00%> (ø)`
pkg/apis/build/v1alpha2/image_validation.go	`97.84% <100.00%> (+0.09%)`	⬆️
pkg/reconciler/image/image.go	`50.99% <100.00%> (ø)`
pkg/reconciler/image/reconcile_build.go	`88.76% <100.00%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33c4d25...e1e91de. Read the comment docs.

pkg/apis/build/v1alpha1/build_pod.go

pkg/apis/build/v1alpha1/build_types.go

matthewmcnew · 2021-04-20T22:26:33Z

pkg/apis/build/v1alpha1/build_types.go

@@ -69,12 +69,17 @@ type BuildSpec struct {
 	Notary    *NotaryConfig               `json:"notary,omitempty"`
 }

+type BuildCacheConfig struct {
+	VolumeName string `json:"volumeName,omitempty"`
+	ImageTag   string `json:"imageTag,omitempty"`


In the spirit of reproducibility should we provide an image with a fully qualified digest here?

We think no: If "someone" pushed an update to the cache we'd like to always use the most current version.

The current image reference that is provided to the analyze step is the fully qualified built image reference from the last build. This allows deterministic builds that will not run into outside interference if something else is writing to the same tag. I think the same thing applies here, another image could write to the provided cache image tag and we would want to ensure we are using the exact cache digest from the last build.

Got it. Should be implemented with 6b3e580 and 177e139. I'll also add some test for that. WDYT?

matthewmcnew · 2021-04-20T22:28:33Z

Thanks for submitting this @modulo11! It will be a breaking change to the image (and build) api so we need to provide it in a new api version. This PR for service bindings introduces the v1alpha2 api. I think it would make sense to include this api update alongside that api update.

modulo11 · 2021-04-21T11:50:38Z

Hi @matthewmcnew thanks for the review. We tried bumping the api from v1alpha1 to v1alpha2 for just Build and Image types analogous to PR #555. We gave up in the middle because we we're overwhelmed by subtle changes which are easily overlooked when mixing both versions in the code. For us it would be more feasible to bump all api objects to v1alpha2 and update all references.

We use a latest api package (referencing the current api object versions) in our own controllers, so bumping all versions can be done in one place. We can supply a (draft) PR if you're interested.

WDYT?

matthewmcnew · 2021-04-21T18:45:07Z

I'm not sure of the best path forward as implementing a new api version will be quite involved. Likely we should merge our PR for service bindings that will introduce a new api version and then allow you to provide your PR on top of that so you don't need to manage adding a new api version.

matthewmcnew · 2021-04-21T18:45:30Z

To help me better understand the desired timelines could you tell me a bit more about your desired usecase for this functionality?

c0d1ngm0nk3y · 2021-04-21T19:20:09Z

I'm not sure of the best path forward as implementing a new api version will be quite involved.

But we have doubts that this is the best way forward. The #555 shows the problems with the imports:

´´´
"github.com/pivotal/kpack/pkg/apis/build/v1alpha1"
"github.com/pivotal/kpack/pkg/apis/build/v1alpha2"
´´´

So you have to be very careful which api object take from which import ( ´Build´ -> v1alpha2, but ´SourceResolver` -> v1alpha1). Those errors can be quite subtle.

When bumping the ´build´ api as a whole, we could do something like

´´´
build "github.com/pivotal/kpack/pkg/apis/build/v1alpha2"
´´´

and then changing the api version is changing only the import in each relevant file.

matthewmcnew · 2021-04-21T20:18:18Z

Yeah, we should introduce an entire new api version for all resources at v1alpha2

modulo11 · 2021-04-22T14:34:08Z

Alright, then we'll put that PR on hold and continue after #555 is merged.

c0d1ngm0nk3y · 2021-04-22T15:06:09Z

Yeah, we should introduce an entire new api version for all resources at v1alpha2

But then, there would be no real benefit waiting for #555, right? v1alpha2 can be changed again until it is released.

matthewmcnew · 2021-04-26T22:02:33Z

But then, there would be no real benefit waiting for #555, right? v1alpha2 can be changed again until it is released.

I'm not quite sure what you mean by this @c0d1ngm0nk3y. Are you suggesting that we create a different PR to bump to v1alpha?

c0d1ngm0nk3y · 2021-04-28T07:34:50Z

I'm not quite sure what you mean by this @c0d1ngm0nk3y. Are you suggesting that we create a different PR to bump to v1alpha?

Sorry, yes. Our idea was to create a different PR bumping to v1alpha2. And then both PRs could rebase on that and focus only on the relevant parts. We have started with some documentation and renaming to make the bumping not so "involved".

c0d1ngm0nk3y · 2021-04-28T07:54:33Z

@matthewmcnew Maybe you can have a look at #675 - we would propose to use this as a first step before actually bumping the api. We could then add another PR bumping the api to v1alpha2, but not changing anything. #555 and #669 could then base on that and only adapt what is really relevant. WDYT?

matthewmcnew · 2021-05-10T22:31:22Z

Hi, @modulo11 & @c0d1ngm0nk3y. Thanks for your patience on this! Our current goal is to provide our planned v0.3 release of kpack in the next few weeks with the underlying git library moved to use git2go libgit2 bindings as a fix to our long running azure devops woes. Migrating/building kpack images to use the libgit2 has been a major hurdle so our plan is to release this before releasing any other major kpack updates.

We would like to target the release of the new api version in a short follow-up v0.4 release. I think we are going to wait on merging this an other changes to the api until after the v0.3 release is released.

tylerphelan · 2021-08-17T20:35:06Z

build-webhook.sh

+
+set -e
+
+pack build europe-west3-docker.pkg.dev/sap-se-gcp-istio-dev/public/kpack-667/webhook --builder gcr.io/cf-build-service-public/ci/tiny-builder -e BP_GO_TARGETS=./cmd/controller --publish --trust-builder


could the the tag argument to pack be parameterized in this script? also the pack args for the controller have changed since this pr to

-B gcr.io/cf-build-service-public/ci/kpack-builder -e BP_GO_TARGETS=./cmd/controller -e BP_BUILD_LIBGIT2=true -e BP_GO_BUILD_FLAGS="-tags=\"static\"" --publish

You could also just remove the scripts if you don't want to update them.

tylerphelan · 2021-08-17T20:49:04Z

docs/build.md

@@ -19,7 +19,8 @@ spec:
    image: gcr.io/paketo-buildpacks/builder:base
    imagePullSecrets: 
    - name: builder-secret  
-  cacheName: persisent-volume-claim-name
+  cache:
+    volumeName: persisent-volume-claim-name


I think pvcName is more accurate here

I wonder if we should match the long name in VolumeSource and use the long name persistentVolumeClaimName?

tylerphelan · 2021-08-17T21:05:25Z

pkg/apis/build/v1alpha2/build_pod.go

@@ -641,14 +652,14 @@ func (b *Build) rebasePod(secrets []corev1.Secret, images BuildPodImages, config
 }

 func (b *Build) cacheVolume(os string) []corev1.Volume {
-	if b.Spec.CacheName == "" || os == "windows" {
+	if b.Spec.Cache.VolumeName == "" || os == "windows" {


do we need to protect from .Cache being nil here?

pkg/apis/build/v1alpha2/build_pod_test.go

tylerphelan · 2021-08-17T21:15:19Z

pkg/apis/build/v1alpha2/image_validation_test.go

@@ -262,9 +279,9 @@ func testImageValidation(t *testing.T, when spec.G, it spec.S) {
 		it("image.cacheSize has not decreased", func() {


might be nice to update the text here

@tylerphelan Which text? 😕 Sorry...

We got it :) Thanks.

I actually meant the test text: it("image.cacheSize has not decreased", func() { -> it("image.cache.volume.size has not decreased", func() { 😀

pkg/cnb/cnb_metadata.go

tylerphelan · 2021-08-17T21:19:54Z

docs/image.md

@@ -9,7 +9,9 @@ The following defines the relevant fields of the `image` resource spec in more d
 - `builder`: Configuration of the `builder` resource the image builds will use. See more info [Builder Configuration](builders.md).
 - `serviceAccount`: The Service Account name that will be used for credential lookup.
 - `source`: The source code that will be monitored/built into images. See the [Source Configuration](#source-config) section below.
- `cacheSize`: The size of the Volume Claim that will be used by the build cache.
+- `cache`: Caching configuration, two variants are available:
+  - `volume.request`: Creates a Volume Claim of the given size


should this be volume.size ? request seems less user-friendly. anyone else have thoughts?

Hmm. I like how request matches the k8s terminology for PVCs.

Should this actually be requests and be a RequestsList?

Latest iteration:

- `cache`: Caching configuration, two variants are available: - `volume.size`: Creates a Volume Claim of the given size - `registry.imageTag`: Creates an image with cached contents

tylerphelan

very nice pr, I had just a few notes

c0d1ngm0nk3y · 2021-08-18T06:13:01Z

very nice pr, I had just a few notes

Thanks, we will have a look. Probably after #670 is merged because they are in conflict anyway.

matthewmcnew · 2021-08-18T18:42:12Z

pkg/apis/build/v1alpha2/build_types.go

-	Image   string `json:"image,omitempty"`
-	StackId string `json:"stackId,omitempty"`
+	Image      string `json:"image,omitempty"`
+	CacheImage string `json:"cacheImage,omitempty"`


To this point we have tried to keep every reference in the api to an image to use the image key. Could we move this to cache.image?

c0d1ngm0nk3y · 2021-08-20T11:48:27Z

build-webhook.sh

@@ -0,0 +1,9 @@
+#!/bin/bash


Should this file really be part of this PR? That is just our local "hack" to not build everything, right? If, it should be done for all components I guess. Maybe even in the Makefile?

c0d1ngm0nk3y · 2021-08-20T11:49:03Z

build.sh

@@ -0,0 +1,9 @@
+#!/bin/bash


This is also some local "hack" and should not be part of this PR I guess.

c0d1ngm0nk3y · 2021-08-20T11:53:37Z

docs/build.md

@@ -19,7 +19,8 @@ spec:
    image: gcr.io/paketo-buildpacks/builder:base
    imagePullSecrets: 
    - name: builder-secret  
-  cacheName: persisent-volume-claim-name
+  cache:


This text above should also reflect the new cache option

v1alpha1 is outdated (line 10)

Is this some kind of example? Should we mention the other cache possibility?

c0d1ngm0nk3y · 2021-08-20T11:56:00Z

pkg/apis/build/v1alpha2/build_pod.go

@@ -164,17 +163,28 @@ func (b *Build) BuildPod(images BuildPodImages, secrets []corev1.Secret, taints
 	}

 	var cacheArgs []string


The cacheArgs are not needed in this scope.

c0d1ngm0nk3y · 2021-08-20T11:56:23Z

pkg/apis/build/v1alpha2/build_pod.go

 	} else {
 		cacheArgs = []string{"-cache-dir=/cache"}
+		exportCacheArgs = cacheArgs


If we fix the scope, this is not needed anymore.

c0d1ngm0nk3y · 2021-08-20T12:12:01Z

pkg/apis/build/v1alpha2/image_validation.go

@@ -45,8 +45,12 @@ func (i *Image) SetDefaults(ctx context.Context) {
 		i.Spec.SuccessBuildHistoryLimit = &defaultSuccessfulBuildHistoryLimit
 	}

-	if i.Spec.CacheSize == nil && ctx.Value(HasDefaultStorageClass) != nil {
-		i.Spec.CacheSize = &defaultCacheSize
+	if i.Spec.Cache == nil && ctx.Value(HasDefaultStorageClass) != nil {


Is this correct? Should we check for i.Spec.Cache.Request != nil as well? What if it uses registryCache?

I think this is right!

c0d1ngm0nk3y · 2021-08-20T12:13:43Z

pkg/apis/build/v1alpha2/image_validation.go

-	if is.CacheSize != nil && ctx.Value(HasDefaultStorageClass) == nil {
-		return apis.ErrGeneric("spec.cacheSize cannot be set with no default StorageClass")
+func (is *ImageSpec) validateVolumeCache(ctx context.Context) *apis.FieldError {
+	if is.Cache.Volume != nil && ctx.Value(HasDefaultStorageClass) == nil {


What is with is.Cache != nil?

with the current implementation it will get defaulted, so I think it's ok

c0d1ngm0nk3y · 2021-08-20T12:15:30Z

pkg/apis/build/v1alpha2/image_validation_test.go

@@ -38,7 +38,11 @@ func testImageValidation(t *testing.T, when spec.G, it spec.S) {
 					Revision: "master",
 				},
 			},
-			CacheSize:                &cacheSize,
+			Cache: &CacheConfig{


What is with an empty volumeCache?

c0d1ngm0nk3y · 2021-08-20T12:16:49Z

pkg/apis/build/v1alpha2/image_validation_test.go

@@ -262,9 +279,9 @@ func testImageValidation(t *testing.T, when spec.G, it spec.S) {
 		it("image.cacheSize has not decreased", func() {


@tylerphelan Which text? 😕 Sorry...

c0d1ngm0nk3y · 2021-08-20T12:21:40Z

pkg/reconciler/image/image_test.go

@@ -713,6 +717,7 @@ func testImageReconciler(t *testing.T, when spec.G, it spec.S) {
 									Image: builder.Status.LatestImage,
 								},
 								ServiceAccount: image.Spec.ServiceAccount,
+								Cache:          &buildapi.BuildCacheConfig{},


Shouldn't nil be valid as well?

c0d1ngm0nk3y · 2021-08-20T12:37:22Z

test/execute_build_test.go


-		imageName := fmt.Sprintf("%s-%s", "test-git-image", "cluster-builder")
+	imageName := fmt.Sprintf("%s-%s", "test-git-image", "cluster-builder")


Maybe it is worth to use different imageNames? Since this generateRebuild is called twice (for each cache).

pkg/apis/build/v1alpha2/image_types.go

pkg/reconciler/build/build.go

pkg/apis/build/v1alpha2/build.go

c0d1ngm0nk3y · 2021-08-23T12:56:12Z

@tylerphelan @matthewmcnew We think we have addressed all feedback. Except the part with "cache image lost on failure" which we didn't understand 100% to be honest. Could you have another look?

tylerphelan · 2021-08-23T15:42:27Z

Nice, all of my comments have been addressed. I had one more thought, what do others think about adding validation ensuring only one of image.spec.cache.volume and image.spec.cache.registry is set? Right now the registry will be used by the lifecycle and in the future multiple cache types may be possible, but it might be nice to validate this for now.

pkg/reconciler/build/build.go

c0d1ngm0nk3y · 2021-08-24T11:11:42Z

I had one more thought, what do others think about adding validation ensuring only one of image.spec.cache.volume and image.spec.cache.registry is set?

Good point. We have added validation to Build and Image that only one type of cache can be used.

c0d1ngm0nk3y · 2021-08-24T11:30:54Z

@matthewmcnew @tylerphelan We have rebased and squashed this pr and from our point of view we are good to go. Please check and let us know if there is still something else to do.

matthewmcnew · 2021-08-25T18:22:22Z

pkg/apis/build/v1alpha2/image_types.go

+}
+
+type RegistryCache struct {
+	ImageTag string `json:"imageTag"`


Could we make this tag to match the other instance of tags in the api?

matthewmcnew · 2021-08-25T18:27:13Z

I think we are essentially ready to Merge! (Thank you so much for your patience on this)

One last request, could we update spec.cache.registry.imageTag to spec.cache.registry.tag?

c0d1ngm0nk3y · 2021-08-25T18:58:47Z

I think we are essentially ready to Merge

Sounds great :)

One last request, could we update spec.cache.registry.imageTag to spec.cache.registry.tag?

Ah. It was called spec.cache.registry.tag in an earlier state of the pr. We have changed it to spec.cache.registry.imageTag because we thought that is a bit clearer. tag sounded like it is only the tag that will be applied to the cache image. But it is the whole image name.

But I have no strong opinion on that. We can change it back to spec.cache.registry.tag.

tomkennedy513

Great work!

matthewmcnew · 2021-08-25T20:40:28Z

Ah. It was called spec.cache.registry.tag in an earlier state of the pr. We have changed it to spec.cache.registry.imageTag because we thought that is a bit clearer. tag sounded like it is only the tag that will be applied to the cache image. But it is the whole image name.

Yeah, the docker terminology is a bit cumbersome, especially with the tag field. I think tag makes sense in this case because it matches spec.tag on images,builders, and builds.

* Split up image cache configuration * Split up build cache configuration * Add registy/image based cache * Remove unused buildCacheName arg * Update documentation * Add basic registry-cache e2e test * Fix duplicate import * Fix unmatched quotes * Add home volume to restore step * Allow build.cache to be nil * Persist cache image in build * Use cache image digest if present * Move changes to buildapi from v1alpha1 to v1alpha2 * Add test cache for Spec.Cache = nil * Add validation that only one type of cache is used for build and image * Use `spec.cache.registry.tag` to be more consistent Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com> Co-authored-by: Johannes Dillmann <j.dillmann@sap.com> Co-authored-by: Sumit Kulhadia <sumit.kulhadia@sap.com> Co-authored-by: Pavel Busko <pavel.busko@sap.com>

c0d1ngm0nk3y · 2021-08-26T07:50:59Z

Yeah, the docker terminology is a bit cumbersome, especially with the tag field. I think tag makes sense in this case because it matches spec.tag on images,builders, and builds.

Done. I think we are good to go...

matthewmcnew reviewed Apr 20, 2021

View reviewed changes

pkg/apis/build/v1alpha1/build_pod.go Outdated Show resolved Hide resolved

matthewmcnew reviewed Apr 20, 2021

View reviewed changes

pkg/apis/build/v1alpha1/build_types.go Outdated Show resolved Hide resolved

matthewmcnew reviewed Apr 20, 2021

View reviewed changes

modulo11 force-pushed the image-registry-cache branch from bfb4704 to 177e139 Compare April 22, 2021 14:31

modulo11 requested a review from matthewmcnew April 22, 2021 14:31

c0d1ngm0nk3y mentioned this pull request Apr 28, 2021

Bump build api from v1alpha1 v1alpha2 #676

Merged

matthewmcnew added the 0.4.0 label May 11, 2021

matthewmcnew mentioned this pull request May 11, 2021

Add support for project descriptor #690

Merged

modulo11 force-pushed the image-registry-cache branch from 177e139 to 5e7df8a Compare June 7, 2021 09:18

c0d1ngm0nk3y force-pushed the image-registry-cache branch from 5e7df8a to a54eebd Compare July 22, 2021 13:19

tylerphelan reviewed Aug 17, 2021

View reviewed changes

pkg/apis/build/v1alpha2/build_pod_test.go Outdated Show resolved Hide resolved

tylerphelan reviewed Aug 17, 2021

View reviewed changes

pkg/cnb/cnb_metadata.go Outdated Show resolved Hide resolved

tylerphelan reviewed Aug 17, 2021

View reviewed changes

tylerphelan suggested changes Aug 17, 2021

View reviewed changes

matthewmcnew reviewed Aug 18, 2021

View reviewed changes

c0d1ngm0nk3y suggested changes Aug 20, 2021

View reviewed changes

c0d1ngm0nk3y reviewed Aug 20, 2021

View reviewed changes

c0d1ngm0nk3y force-pushed the image-registry-cache branch from a54eebd to 590aba5 Compare August 20, 2021 12:38

matthewmcnew reviewed Aug 23, 2021

View reviewed changes

pkg/apis/build/v1alpha2/image_types.go Show resolved Hide resolved

matthewmcnew reviewed Aug 23, 2021

View reviewed changes

pkg/reconciler/build/build.go Show resolved Hide resolved

matthewmcnew reviewed Aug 23, 2021

View reviewed changes

pkg/apis/build/v1alpha2/build.go Show resolved Hide resolved

c0d1ngm0nk3y suggested changes Aug 23, 2021

View reviewed changes

pkg/reconciler/build/build.go Show resolved Hide resolved

c0d1ngm0nk3y force-pushed the image-registry-cache branch from 4f4f13f to 06bf555 Compare August 24, 2021 11:10

c0d1ngm0nk3y force-pushed the image-registry-cache branch from 06bf555 to 5b57689 Compare August 24, 2021 11:26

matthewmcnew reviewed Aug 25, 2021

View reviewed changes

tylerphelan approved these changes Aug 25, 2021

View reviewed changes

tomkennedy513 approved these changes Aug 25, 2021

View reviewed changes

c0d1ngm0nk3y force-pushed the image-registry-cache branch from 5b57689 to 0d971c5 Compare August 26, 2021 07:49

matthewmcnew approved these changes Aug 26, 2021

View reviewed changes

tomkennedy513 merged commit beec5d1 into buildpacks-community:main Aug 26, 2021


		set -e

		pack build europe-west3-docker.pkg.dev/sap-se-gcp-istio-dev/public/kpack-667/webhook --builder gcr.io/cf-build-service-public/ci/tiny-builder -e BP_GO_TARGETS=./cmd/controller --publish --trust-builder

		@@ -262,9 +279,9 @@ func testImageValidation(t *testing.T, when spec.G, it spec.S) {
		it("image.cacheSize has not decreased", func() {

		@@ -164,17 +163,28 @@ func (b *Build) BuildPod(images BuildPodImages, secrets []corev1.Secret, taints
		}

		var cacheArgs []string


		imageName := fmt.Sprintf("%s-%s", "test-git-image", "cluster-builder")
		imageName := fmt.Sprintf("%s-%s", "test-git-image", "cluster-builder")

Image registry cache #669

Image registry cache #669

Conversation

modulo11 commented Apr 20, 2021

codecov-commenter commented Apr 20, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewmcnew commented Apr 20, 2021

modulo11 commented Apr 21, 2021

matthewmcnew commented Apr 21, 2021

matthewmcnew commented Apr 21, 2021

c0d1ngm0nk3y commented Apr 21, 2021

matthewmcnew commented Apr 21, 2021 • edited Loading

modulo11 commented Apr 22, 2021

c0d1ngm0nk3y commented Apr 22, 2021

matthewmcnew commented Apr 26, 2021

c0d1ngm0nk3y commented Apr 28, 2021

c0d1ngm0nk3y commented Apr 28, 2021

matthewmcnew commented May 10, 2021

tylerphelan Aug 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dumez-k Aug 23, 2021 • edited Loading

Choose a reason for hiding this comment

tylerphelan left a comment

Choose a reason for hiding this comment

c0d1ngm0nk3y commented Aug 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c0d1ngm0nk3y commented Aug 23, 2021

tylerphelan commented Aug 23, 2021

c0d1ngm0nk3y commented Aug 24, 2021

c0d1ngm0nk3y commented Aug 24, 2021

Choose a reason for hiding this comment

matthewmcnew commented Aug 25, 2021

c0d1ngm0nk3y commented Aug 25, 2021

tomkennedy513 left a comment

Choose a reason for hiding this comment

matthewmcnew commented Aug 25, 2021

c0d1ngm0nk3y commented Aug 26, 2021

codecov-commenter commented Apr 20, 2021 •

edited

Loading

matthewmcnew commented Apr 21, 2021 •

edited

Loading

tylerphelan Aug 17, 2021 •

edited

Loading

dumez-k Aug 23, 2021 •

edited

Loading