decoupling Function, Trigger and Runtimes #620

murali-reddy · 2018-03-07T09:29:49Z

Issue Ref: [Issue number related to this PR or None]

Basic issues this PR addresses are detailed/or dicussed in this proposal

Description:

This PR is result of collabrative development effort from @andresmgot, @ngtuna and @murali-reddy

What problems does this PR address?

decoupling function and event sources

In the current design of Kubeless, there is just Function abstraction, respresented as CRD. This object is used to specify both function details (like function handler, code, runtime etc) and event source details (like Kafkat topic to which this function to be called, cron job etc).

This PR brings in notion of Triggers as concept in to Kubeless. A Trigger is basically represents the details on the event source and the associated function that needs to be called when event occurs. Existing supported event sources (Kafka, cron job, http) are modelled in to seperate triggers. Corresponding to each trigger there will be seperate CRD controller. This seperation at API layer, provides clean seperation of concerns of functions and event sources.

Also this seperation enables Kubeless to support n:m association between the function and event sources. For e.g, Kafka trigger object uses label selector to express the set of functions that need to be associated with trigger.

decoupling event source listener and runtimes

In the current Kubeless architecture, pod that is deployed to run the function, also has the event source listener code (like Kafka client subsribing to a particualr topic. Problem with this approach is we need to maintain a separate image for each language, language version, event source combination.

With this PR each pod that is deployed to run function exposes a http endpoint, irrespective of event source and language type. Also fuction singature has been altered to carry event data and conext information.

design changes

Some notes on the design to help the reviewer.

Each trigger object has a corresponding CRD controller. Controller watches for add/delete/update events from the k8s api server and process the update to reflect to desired state expressed by the user.
Tigger object CRD controllers, also watches for Function object updates. Controller will take an action only if required. For e.g, when a function is deleted, Kafka trigger controller checks if there is any Kafka trigger object is associated. If there is Kafka trigger object associated then it will stop sending topic message to the function service.
By default CRD objects (like core objects) are deleted by API server when they are deleted by API. It's not possible for controllers to sync/clean-up when the object is gone. Kubernetes provides a declarative patterns of Finalizers which provides a mechansim through which interested controller can request for soft delete of the API object. This PR leverages Finalizers to process the deleted objects. Please see PVC protection controller for a reference.
kubeless-controller has been renamed to kubeless-controller-manager, and includes controllers for http triggers and cronjob triggers.
a new deployment manifest is added for Kafka. When deployed, it creates seperate deployment afka-trigger-controller, which has Kafka trigger controller CRD.

Known Issue

CI is failing for GKE and Kafka build. I will fix it. But the reviews for the PR can get started.
Some inconsistent behaviour in edge cases was seen with API machinery for CRD controllers (for instance Custom resources with finalizers can "deadlock" customresourcecleanup.apiextensions.k8s.io finalizer kubernetes/kubernetes#60538). If any issue found we need to workaround or get it fixed upstream.
No explict CLI support for trigger creation in this PR. Kubeless function deploy implicitly creats the trigger objects as well. You can use API to create trigger object directly. CLI support will be added in seperatle PR.

TODOs:

Ready to review
Automated Tests
Docs

sebgoa · 2018-03-07T09:41:21Z

@nikhita it has been a while, but you might be interested by this PR and we would love to get an external review :)

we know, it is big...

corresponding auto-generated informer, listers, clienset and deep copy generated function

triggers CRD objects.

…ller

"kubeless function deploy" and move ingress object creation logic to http trigger controller

Co-authored-by: Andres <andres.mgotor@gmail.com> Co-authored-by: Tuna <ng.tuna@gmail.com>

Co-authored-by: Tuna <ng.tuna@gmail.com>

Co-authored-by: Tuna <ng.tuna@gmail.com> Co-authored-by: Andres <andres.mgotor@gmail.com>

andresmgot · 2018-03-08T08:38:59Z

cmd/kubeless/function/deploy.go

@@ -138,11 +140,6 @@ var deployCmd = &cobra.Command{
 			logrus.Fatal(err)
 		}

-		cpu, err := cmd.Flags().GetString("cpu")


where has this gone?

andresmgot · 2018-03-08T08:45:48Z

cmd/kubeless/function/deploy.go

+				Namespace: ns,
+			}
+			cronJobTrigger.ObjectMeta.Labels = map[string]string{
+				"created-by": "kubeless",


we should probably add labels here and in the http trigger for function=funcName?

andresmgot · 2018-03-08T08:47:30Z

cmd/kubeless/function/deploy.go

+			}
+			kafkaTrigger.ObjectMeta.Labels = map[string]string{
+				"created-by": "kubeless",
+				"function":   funcName,


does this mean that we have a relation 1:1 for trigger objects functions? That is fine for me now, I just want to clarify if that is the intention

I think it means a N:1 trigger:functions (ie: multiple triggers can point to the same function)... Which we have anyway, given that we have a functionSelector below that (presumably) only matches the multiple instances/versions of a single Function. (or at least, I can't think of a useful example where the functionSelector could be used to sprays requests across a wide range of Functions).

sorry for the confusion, above label setting pointed by @andresmgot has no meaning. I will remove it. Below line is what matters.

kafkaTrigger.Spec.FunctionSelector.MatchLabels = f.ObjectMeta.Labels

This is to keep status-quo of below user experiance of kubeless CLI with what it is today. Lables used for the function are the once specified for FunctionSelector

kubeless function deploy pubsub-nodejs --trigger-topic s3-nodejs --runtime nodejs6 --handler pubsub-nodejs.handler --from-file nodejs/hellowithdata.js

Through the Kafka trigger CRD object/API, you can still express the N:1 releationship between functions:kafka triggers. Function could be part of multiple Kafka triggers as well.

andresmgot · 2018-03-08T08:51:05Z

cmd/kubeless/route/routeDelete.go

+			}
+		}
+
+		httpTrigger.Spec.RouteName = routeName


this line is not necessary, isn't it? shouldn't we erase the route name value?

andresmgot · 2018-03-08T09:34:40Z

docker/controller-manager/Dockerfile

@@ -0,0 +1,7 @@
+FROM bitnami/minideb:jessie
+
+LABEL maintainer="Nguyen Anh Tu <tuna@bitnami.com>"


we need to set as maintainers the kubernetes group, I am not sure the exact maintainer we need to use but we should check this

andresmgot · 2018-03-08T09:50:50Z

kubeless.jsonnet

-        "httpImage": "kubeless/python@sha256:0f3b64b654df5326198e481cd26e73ecccd905aae60810fc9baea4dcbb61f697",
-        "pubsubImage": "kubeless/python-event-consumer@sha256:1aeb6cef151222201abed6406694081db26fa2235d7ac128113dcebd8d73a6cb",
-        "initImage": "tuna/python-pillow:2.7.11-alpine"
+        "runtimeImage": "andresmgot/python@sha256:cdba8db2ab40c2e1b954f133c4b3c2dc8f63044e490bcccee88621a61d2d06d5",


I need to move these images to the kubeless repo if they are going to be the final status

I have not see any issues so far. should be good to move to kubeless repo.

andresmgot · 2018-03-08T10:11:25Z

pkg/apis/kubeless/v1beta1/http_trigger.go

+// HTTPTriggerSpec contains func specification
+type HTTPTriggerSpec struct {
+	FunctionName  string         `json:"function-name"` // Name of the associated function
+	ServiceSpec   v1.ServiceSpec `json:"service"`


maybe out of scope for this PR but should we take the oportunity to change this for a Service instead of ServiceSpec to be able to set annotations?

Whereas I feel the opposite ;) why are we exposing the Service at all? I think an HTTP trigger should talk about HTTP host/port/path route that will be created (and optional TLS details) .. and that's about all.

Yes but some people need the flexibility to define some annotations in order to use some external services (slack link). Also note that this would be for advanced use only, the normal usage would be using the kubeless CLI with the flags --port --hostname...

andresmgot · 2018-03-08T10:23:05Z

pkg/controller/kafka_trigger_controller.go

+		}
+
+		for _, function := range functions {
+			if err != nil {


this err != nil seems misplaced

andresmgot · 2018-03-08T10:23:42Z

pkg/controller/kafka_trigger_controller.go

+	}
+
+	for _, function := range functions {
+		if err != nil {


the same about this err

andresmgot · 2018-03-08T10:27:24Z

pkg/event-consumers/kafka/kafka-consumer.go

+			}
+		case err, more := <-consumer.Errors():
+			if more {
+				logrus.Fatalf("Error: %s\n", err.Error())


does this cause a crash in the consumer (and probably the shared pod) if there is an error with a message?

andresmgot · 2018-03-08T14:52:05Z

pkg/event-consumers/kafka/kafka-consumer.go

+
+	timestamp := time.Now().UTC()
+	req.SetHeader("event-id", fmt.Sprintf("kafka-consumer-%s-%s-%s", funcName, ns, timestamp.Format(time.RFC3339Nano)))
+	req.SetHeader("event-type", "application/json")


we cannot assume that the message is a json

anguslees · 2018-03-16T08:36:40Z

cmd/kubeless/function/call.go

@@ -19,18 +19,28 @@ package function
 import (
 	"bytes"
 	"fmt"
+	"math/rand"


UnixNano() only gives us a few bytes of entropy between competing controllers that start at similar times, which will lead to more frequent event id collisions than we otherwise expect. I suggest using crypto/rand instead, to draw on an a better entropy source.

anguslees · 2018-03-16T08:40:08Z

cmd/kubeless/function/call.go

+func getRandString(n int) string {
+	b := make([]byte, n)
+	for i := range b {
+		b[i] = letterBytes[rand.Intn(len(letterBytes))]


Is it important that event-ids are only from this quite-restricted base36 character set?
(I expected to see eg standard or url-safe base64 here)

not really, is there an easy (short) way of doing that? There may be a way I am unaware of.

import ( "encoding/base64" "crypto/rand" ) func getRandString(n int) (string, err) { b := make([]byte, n) if _, err := rand.Read(b); err != nil { return "", err } return base64.RawURLEncoding.EncodeToString(b), nil }

anguslees · 2018-03-16T08:40:47Z

cmd/kubeless/function/call.go

@@ -86,7 +96,7 @@ var callCmd = &cobra.Command{
 			req = req.AbsPath(svc.ObjectMeta.SelfLink + ":" + port + "/proxy/")
 		}
 		timestamp := time.Now().UTC()
-		req.SetHeader("event-id", fmt.Sprintf("cli-%s-%s-%s", version.VERSION, version.GITCOMMIT, timestamp.Format(time.RFC3339Nano)))
+		req.SetHeader("event-id", fmt.Sprintf("kubeless-cli-%s", getRandString(11)))


why the prefix?

(Am I missing something? Is it important/useful to include the client identifier in the event id rather than putting that information in some other header?)

the function will receive only these headers (at least in the event parameter), that's why I am adding that prefix. It is true that there is the event-namespace header that includes the emitter info so this is "duplicated". Would it be meaningful/useful if we have here just a random string?

Unlike the other event headers, I think the event-id is going to be logged/stored/hashed/indexed by everything that looks at these event traces. If we can (eg) halve the number of bytes required for that, then everything becomes that tiny bit more efficient. (In particular, naive storage sharding can become quite confused by having variation only in the suffix).

It's a micro-optimisation at this point, for sure. But I dream big :)

anguslees · 2018-03-19T02:49:33Z

kubeless-rbac.jsonnet

@@ -24,7 +24,7 @@ local controller_roles = [
  {
    apiGroups: ["kubeless.io"],
    resources: ["functions", "kafkatriggers", "httptriggers", "cronjobtriggers"],
-    verbs: ["get", "list", "watch", "update"],
+    verbs: ["create", "get", "list", "watch", "update", "patch"],


I see we use patch (which is essentially equivalent to update) for managing finalizers. When does the controller serviceaccount need to create these resources?

Which resources? Trigger objects?

the resources in this rbac rule (functions and triggers)

Got it. You are right. create should not be there. Will fix it.

anguslees · 2018-03-19T02:57:19Z

pkg/utils/k8sutil.go

+	if len(patch) == 0 || string(patch) == "{}" {
+		return nil
+	}
+	_, err = kubelessClient.KubelessV1beta1().KafkaTriggers(original.Namespace).Patch(original.Name, types.MergePatchType, patch)


Just repeating earlier comment: These patches need to be applied against a particular resource version (with retry on conflict detection) - otherwise we can race against another change between reading the "original" version and applying the patch.

So both the update/patch has to be done against the latest resource version? Do you recall any core controllers of kube-controller-manger does this retry on conflict detection logic. All I see is they just do update with out retry mostly.

See all the calls to k8s.io/client-go/util/retry.RetryOnConflict. I presume the other Update calls just propagate the conflict error up the stack, at worst doing a full requeue and error retry later.

The example of a Patch I've just looked at (pkg/controller/volume/expand/cache.UpdatePVSize) does a patch that is idempotent, and generated+used within a single simple function, so the call to CreateTwoWayMergePatch() is just an expensive way to calculate a tiny/known patch object. This use of Patch() is fine since the resulting patch doesn't actually depend on the original object anymore. If you have another specific example in mind, then I'd be happy to look into it.

I needed to stare at the kubeless code a lot longer to work out if the kubeless patches where also idempotent. In particular, things like having time-separated calls to previousFunction, err := utils.GetFunctionCustomResource(kubelessClient, ...) and updatedFunction, err := getFunctionDescription(cli, ...) makes me very nervous .. but it seems cli isn't actually used(?) in the latter. Staring harder at the getFunctionDescription() object, some specific constructs like the loop that incrementally appends volumes/volumeMounts for each secret will clearly produce the wrong patch if something else also adds a volume between when previousFunction was fetched and Patch() is called.
My point with the comparison to UpdatePVSize is that concluding that kubeless' use of Patch was not safe took about 20mins of looking for specific issues, since the code between Get+Patch is quite lengthy and complex. This is not a good pattern to follow for robust code (and also actually wrong in this case).

TL;DR; I will revert the last commit that does 3-way merge patches and keep the update.

things like having time-separated calls to previousFunction, err := utils.GetFunctionCustomResource(kubelessClient, ...) and updatedFunction, err := getFunctionDescription(cli, ...) makes me very nervous ..

Agree. getFunctionDescription has grown big. And now that we have deployment sepc etc as part of Function spec, it's getting complex. But its CLI only change. If some object changes while cli command execution in flight, cli fails.

My point with the comparison to UpdatePVSize is that concluding that kubeless' use of Patch was not safe took about 20mins of looking for specific issues, since the code between Get+Patch is quite lengthy and complex

Actually from controllers use of patch is pretty straightforward and quick. All controllers are doing is updating Finalizer. Anyway, my intention was to avoid update on top of changed object resulting in the object has been modified; please apply your changes to the latest version and try again. I guess i will just keep the retry logic on update failure.

All the core controllers simply perform updates, perhaps they wil eventually sync to desired state.

https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/persistentvolume/pv_controller.go#L822

https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go#L518

https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/deployment/sync.go#L516

It also seems use of 3-way merge Patch may be overkill for controller. First, very likely all controller will be dealing is object meta and status. It really does not have same conflicts that updates to Spec would need that kubectl/api-server has to deal with.

So to summarise, i will revert last commit. Update is what seems more logical for controller, but update alone is not sufficient but would need retry on resource version conflict.

anguslees · 2018-03-19T08:37:25Z

Actually from controllers use of patch is pretty straightforward and quick. All controllers are doing is updating Finalizer.

Just to drive home the point that it's not about code length, but that the read-buildpatch-patch pattern is dangerous and hard to get right (and thus almost certainly wrong in long code), all the AddFinalizer/RemoveFinalizer functions were also unsafe to use with Patch.

Again, consider a race against another client that is also adding/removing finalizers:

      A                         B
 read original object
 add to in-mem finalizer array
 generate patch
                        read original object
                        add to in-mem finalizer array
                        generate patch
 Patch() replaces finalizer array
                        Patch() replaces finalizer array

The final object has lost the change that A tried to make to the finalizer array.

.. and to be clear: Patch() is fine, if used on a patch that is idempotent (was not derived from a particular snapshot of the object).

murali-reddy · 2018-03-19T09:53:14Z

but that the read-buildpatch-patch pattern is dangerous and hard to get right (and thus almost certainly wrong in long code), all the AddFinalizer/RemoveFinalizer functions were also unsafe to use with Patch.

I am not sure how else we can achieve patch without read-buildpatch-patch.

We cannot use strategic merge patch with CRD's. JSON merge patch replace arrays so we endup in the race conditions you mentioned.

I am inclined to keep Update with retry at least its very controlled change with no unseen side affect.

anguslees · 2018-03-19T10:27:55Z

I am inclined to keep Update with retry at least its very controlled change with no unseen side affect.

Agreed! For all the reasons you describe.

…633) * nodejs, remove runtime internal data from sandbox, avoid code concat * update according to comments * keep compatibility with module.exports * adjust kubeless.jsonnet * Move images to the kubeless repository

This reverts commit 358f8cb.

murali-reddy · 2018-03-21T05:26:31Z

@andresmgot @anguslees I believe all issues are addressed. Fixed failing GKE test run as well. Please take a look and see if anything that is a MUST fix for this PR to get LGTM.

Also note that PR #630 will cover new cli for trigger create/update/delete/update and docs.

andresmgot

My +1 for merging this 🎉
let's continue adding further improvements in #630 or other PRs if necessary

murali-reddy · 2018-03-22T03:51:49Z

Thanks for the review. If there are any suggestions I will follow them up in #630.

anguslees · 2018-03-22T03:45:08Z

pkg/event-consumers/kafka/kafka-consumer.go

@@ -84,6 +85,12 @@ func createConsumerProcess(broker, topic, funcName, ns, consumerGroupID string,
 	}
 }

+func isJSON(s string) bool {


Where is the unittest?
This function only works for JSON objects. (That may or may not be ok?)

let me submit a different PR for covering this file with unit tests

anguslees · 2018-03-22T03:56:14Z

cmd/kubeless/function/function.go

@@ -133,7 +134,7 @@ func getContentType(filename string, fbytes []byte) string {
 	return contentType
 }

-func getFunctionDescription(cli kubernetes.Interface, funcName, ns, handler, file, deps, runtime, runtimeImage, mem, cpu, timeout string, envs, labels []string, secrets []string, defaultFunction kubelessApi.Function) (*kubelessApi.Function, error) {
+func getFunctionDescription(cli kubernetes.Interface, funcName, ns, handler, file, deps, runtime, runtimeImage, mem, cpu, timeout string, port int32, headless bool, envs, labels []string, secrets []string, defaultFunction kubelessApi.Function) (*kubelessApi.Function, error) {


What's the usecase for varying port/headless here?

I think the Service we're creating here is an internal trigger->function implementation detail, so altering the port doesn't make much sense - and if making it headless is better/required in some way, then surely we should do that unconditionally for all Functions...

In particular, if we allow changing this here, then I think every trigger needs to be modified to deal with surprise values of ports, and to ensure that load is still balanced across replicas appropriately for headless services. I raise this is a reason that we should not support this variation here.

What's the usecase for varying port/headless here?
I think the Service we're creating here is an internal trigger->function implementation detail, so altering the port doesn't make much sense - and if making it headless is better/required in some way, then surely we should do that unconditionally for all Functions...

The port/headless change is for people that doesn't want to use a kubeless trigger and want to execute the function with their own methods. It is not so "internal".

In particular, if we allow changing this here, then I think every trigger needs to be modified to deal with surprise values of ports

Agree, triggers should retrieve that information before calling the function (if possible).

It is not so "internal".

Rather than exposing this internal detail, and thus requiring it to be more flexible than is needed purely for kubeless' own use, and thus pushing complexity onto all the other parts of kubeless - I propose we keep the internal Service as internal and don't support this flexibility here.

If users want a headless Service, or a different port, or some other functionality (type=LoadBalancer?), then I propose we instead ask those users to create a new Service. This new Service can have whatever special bits they want and just use the same function label selector + targetPort=8080. We could even create a ServiceTrigger that did that for them...

If no one wants (lets me check in the slack and community call) may be we should just remove it. Its been pain to keep supporting it.

anguslees · 2018-03-22T03:57:52Z

take a look and see if anything that is a MUST fix for this PR to get LGTM

I can't answer that, while the contents of this PR keeps changing in new and surprising ways.

murali-reddy and others added 13 commits March 7, 2018 16:30

CRD API objects for functions, http, kafka, cronjob triggers and

d1c51f0

corresponding auto-generated informer, listers, clienset and deep copy generated function

CRD controllers for the function, kafka triggers, cronjob triggers, http

1bf648f

triggers CRD objects.

new server side binaries kubeless-controller-manager and kafka-contro…

10705cb

…ller

updated CLI to incorporate implict creation of triggers with

989f10b

"kubeless function deploy" and move ingress object creation logic to http trigger controller

updated to build scripts

7d4b74e

updated to manifests

655da7f

Co-authored-by: Andres <andres.mgotor@gmail.com> Co-authored-by: Tuna <ng.tuna@gmail.com>

Updates to examples used in integration tests

1fabea6

Co-authored-by: Andres <andres.mgotor@gmail.com> Co-authored-by: Tuna <ng.tuna@gmail.com>

kafka even consumer with new go kafka lib dependencies

416d27a

Co-authored-by: Tuna <ng.tuna@gmail.com>

update vendor libs

41d136a

Co-authored-by: Tuna <ng.tuna@gmail.com>

update docker files

ed682eb

Co-authored-by: Tuna <ng.tuna@gmail.com> Co-authored-by: Andres <andres.mgotor@gmail.com>

updates to utils and langruntime

20d9485

update .travis.yaml

004c4a9

delete old controller docker file

72afaab

murali-reddy force-pushed the function-trigger-runtimes branch from bea0ddf to 72afaab Compare March 7, 2018 11:01

better kafka even consumer logging

44ed20b

murali-reddy force-pushed the function-trigger-runtimes branch 3 times, most recently from 1f51065 to 99222b4 Compare March 8, 2018 09:35

fix travis ci issue related to kafka broker

343e5d5

murali-reddy force-pushed the function-trigger-runtimes branch from 99222b4 to 343e5d5 Compare March 8, 2018 10:21

andresmgot reviewed Mar 8, 2018

View reviewed changes

murali-reddy and others added 2 commits March 8, 2018 18:33

keep the old GKE version 1.7.12-gke.1

bb47001

Use bitnami-bot docker account

2d78977

andresmgot force-pushed the function-trigger-runtimes branch from 8201cf8 to 2d78977 Compare March 8, 2018 14:35

andresmgot reviewed Mar 8, 2018

View reviewed changes

Andres added 4 commits March 8, 2018 16:19

Fix http request method for the Kafka consumer

180a178

Push Kafka image

5d769cd

Fix content-type in consumer HTTP message

b70c3ab

Fix GKE version

53ef2e7

anguslees reviewed Mar 16, 2018

View reviewed changes

Review

c0f4ab4

andresmgot force-pushed the function-trigger-runtimes branch from 15148c0 to c0f4ab4 Compare March 16, 2018 09:44

use 3-way merge Patch only to update the CRD objects

358f8cb

murali-reddy force-pushed the function-trigger-runtimes branch 3 times, most recently from de9e4e9 to 358f8cb Compare March 18, 2018 17:15

anguslees reviewed Mar 19, 2018

View reviewed changes

cscheiber and others added 4 commits March 19, 2018 11:30

Revert "use 3-way merge Patch only to update the CRD objects"

764b8c8

This reverts commit 358f8cb.

Avoid reflect.DeepEqual for deployment comparison

0215d4f

Fix empty body for NodeJS functions

25d300c

andresmgot force-pushed the function-trigger-runtimes branch from 8724ad3 to 25d300c Compare March 20, 2018 09:18

fix GKE CI failures due to upstream issue kubernetes/kubernetes#60538

200b867

murali-reddy force-pushed the function-trigger-runtimes branch from 63226d8 to 200b867 Compare March 20, 2018 14:50

Andres added 3 commits March 21, 2018 12:28

Fix error catching in Python runtime. Remove old files

5fe1681

Send message as jsons if necessary in kafka-consumer

cd56789

Move serviceSpec to function object. Fix service modifications

fd63962

andresmgot force-pushed the function-trigger-runtimes branch from e174ec2 to fd63962 Compare March 21, 2018 12:06

andresmgot approved these changes Mar 21, 2018

View reviewed changes

murali-reddy merged commit cf3f36e into master Mar 22, 2018

anguslees reviewed Mar 22, 2018

View reviewed changes

murali-reddy deleted the function-trigger-runtimes branch April 3, 2018 03:36

andresmgot mentioned this pull request Apr 9, 2018

Decouple triggers and runtimes #354

Closed

		@@ -0,0 +1,7 @@
		FROM bitnami/minideb:jessie

		LABEL maintainer="Nguyen Anh Tu <tuna@bitnami.com>"

decoupling Function, Trigger and Runtimes #620

decoupling Function, Trigger and Runtimes #620

Conversation

murali-reddy commented Mar 7, 2018 • edited Loading

sebgoa commented Mar 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andresmgot Mar 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anguslees Mar 16, 2018 • edited Loading

Choose a reason for hiding this comment

anguslees Mar 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anguslees Mar 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anguslees Mar 19, 2018 • edited Loading

Choose a reason for hiding this comment

murali-reddy Mar 19, 2018 • edited Loading

Choose a reason for hiding this comment

anguslees commented Mar 19, 2018 • edited Loading

murali-reddy commented Mar 19, 2018 • edited Loading

anguslees commented Mar 19, 2018

murali-reddy commented Mar 21, 2018 • edited Loading

andresmgot left a comment

Choose a reason for hiding this comment

murali-reddy commented Mar 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anguslees Mar 22, 2018 • edited Loading

Choose a reason for hiding this comment

murali-reddy Mar 22, 2018 • edited Loading

Choose a reason for hiding this comment

anguslees commented Mar 22, 2018 • edited Loading

murali-reddy commented Mar 7, 2018 •

edited

Loading

andresmgot Mar 12, 2018 •

edited

Loading

anguslees Mar 16, 2018 •

edited

Loading

anguslees Mar 16, 2018 •

edited

Loading

anguslees Mar 16, 2018 •

edited

Loading

anguslees Mar 19, 2018 •

edited

Loading

murali-reddy Mar 19, 2018 •

edited

Loading

anguslees commented Mar 19, 2018 •

edited

Loading

murali-reddy commented Mar 19, 2018 •

edited

Loading

murali-reddy commented Mar 21, 2018 •

edited

Loading

anguslees Mar 22, 2018 •

edited

Loading

murali-reddy Mar 22, 2018 •

edited

Loading

anguslees commented Mar 22, 2018 •

edited

Loading