Save runtimes in configmaps (#575)

* Working prototype

* Added kubeless-config yaml

* Cleanup

* Working after cleanup

* Modify the client to read the configmap

 - The configmap needs to be read by the client

* Made changes based on comments

 - The langruntimes is now part of the controller
   data structure so that it can be passed on to
   k8sutils.go

- The langruntimes reads the configmap on the
  start of the controller.

- Currently monitoring of the configmap is missing.
  So if the configmap is changed the controller needs
  to be restarted.

- The kubeless binary has been updated to read the
  configmap everytime the kubeless binary is used to
  deploy/update the function

- Updated the unit tests

* Dont repeat configmap initialization for every test.

* Updated the documentation for adding custom runtimes.

* Update the jsonnet to pass the runtime information as configmaps

Also remove kubeless-config.yaml
Add a new get-server-config flag to print the server configuration
Remove checks from unittest without RBAC. Since controller
fails to read the config file.

* Changes based on PR comments

 - Removed the debug messages from libtest
 - Added unit tests for imagesecrets
 - Moved the common code into langruntimetestutils.go
 - Updated the Readme with information on new switch
   kubeless function.

* Made changes based on feed from PR

* Resolving merge conflicts

* Rebased to latest master

 - Resolved the merge conflicts
 - Reworked langruntimes

* Update the documentation to reflect new behaviour

*  Made changes based on PR Comments

 - Made get-server-config a command/
 - Fixed Readme to reflect changes.

README.md

@@ -67,6 +67,13 @@ functions.k8s.io   CustomResourceDefinition.v1beta1.apiextensions.k8s.io
 $ kubectl get functions
 ```
 
+If you have installed kubeless into some other namespace (which is not called kubeless) or changed the name of the config file from kubeless-config to something else, then you have to export the kubeless namespace and name of kubeless config as environment variables before using kubless cli. This can be done as follows:
+
+```console
+$ export KUBELESS_NAMESPACE=<name of namespace>
+$ export KUBELESS_CONFIG=<name of config file>
+```
+
 NOTE: Kafka statefulset uses a PVC (persistent volume claim). Depending on the configuration of your cluster you may need to provision a PV (Persistent Volume) that matches the PVC or configure dynamic storage provisioning. Otherwise Kafka pod will fail to get scheduled. Also note that Kafka is only required for PubSub functions, you can still use http triggered functions. Please refer to [PV](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) documentation on how to provision storage for PVC.
 
 You are now ready to create functions.
@@ -212,6 +219,14 @@ $ kubeless topic delete another-topic
 $ kubeless topic ls
 ```
 
+You can also see the list of supported runtimes:
+
+```console
+$ kubeless get-server-config
+INFO[0000] Current Server Config:
+INFO[0000] Supported Runtimes are: python2.7, python3.4, python3.6, nodejs6, nodejs8, ruby2.4, dotnetcore2.0
+```
+
 ## Examples
 
 See the [examples](./examples) directory for a list of various examples. Minio, SLACK, Twitter etc ...

cmd/kubeless/function/deploy.go

@@ -18,6 +18,7 @@ package function
 
 import (
 	"io/ioutil"
+	"os"
 	"strings"
 
 	kubelessApi "github.com/kubeless/kubeless/pkg/apis/kubeless/v1beta1"
@@ -26,13 +27,33 @@ import (
 	"github.com/robfig/cron"
 	"github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
 var deployCmd = &cobra.Command{
 	Use:   "deploy <function_name> FLAG",
 	Short: "deploy a function to Kubeless",
 	Long:  `deploy a function to Kubeless`,
 	Run: func(cmd *cobra.Command, args []string) {
+		cli := utils.GetClientOutOfCluster()
+		controllerNamespace := os.Getenv("KUBELESS_NAMESPACE")
+		kubelessConfig := os.Getenv("KUBELESS_CONFIG")
+
+		if len(controllerNamespace) == 0 {
+			controllerNamespace = "kubeless"
+		}
+
+		if len(kubelessConfig) == 0 {
+			kubelessConfig = "kubeless-config"
+		}
+		config, err := cli.CoreV1().ConfigMaps(controllerNamespace).Get(kubelessConfig, metav1.GetOptions{})
+		if err != nil {
+			logrus.Fatalf("Unable to read the configmap: %v", err)
+		}
+
+		var lr = langruntime.New(config)
+		lr.ReadConfigMap()
+
 		if len(args) != 1 {
 			logrus.Fatal("Need exactly one argument - function name")
 		}
@@ -74,9 +95,9 @@ var deployCmd = &cobra.Command{
 			logrus.Fatal(err)
 		}
 
-		if runtime != "" && !langruntime.IsValidRuntime(runtime) {
+		if runtime != "" && !lr.IsValidRuntime(runtime) {
 			logrus.Fatalf("Invalid runtime: %s. Supported runtimes are: %s",
-				runtime, strings.Join(langruntime.GetRuntimes(), ", "))
+				runtime, strings.Join(lr.GetRuntimes(), ", "))
 		}
 
 		handler, err := cmd.Flags().GetString("handler")
@@ -151,7 +172,6 @@ var deployCmd = &cobra.Command{
 			logrus.Fatal("You must specify handler for the runtime.")
 		}
 
-		cli := utils.GetClientOutOfCluster()
 		defaultFunctionSpec := kubelessApi.Function{}
 		defaultFunctionSpec.Spec.Type = "HTTP"
 		defaultFunctionSpec.ObjectMeta.Labels = map[string]string{
@@ -178,7 +198,7 @@ var deployCmd = &cobra.Command{
 }
 
 func init() {
-	deployCmd.Flags().StringP("runtime", "", "", "Specify runtime. Available runtimes are: "+strings.Join(langruntime.GetRuntimes(), ", "))
+	deployCmd.Flags().StringP("runtime", "", "", "Specify runtime")
 	deployCmd.Flags().StringP("handler", "", "", "Specify handler")
 	deployCmd.Flags().StringP("from-file", "", "", "Specify code file")
 	deployCmd.Flags().StringSliceP("label", "", []string{}, "Specify labels of the function. Both separator ':' and '=' are allowed. For example: --label foo1=bar1,foo2:bar2")

cmd/kubeless/function/function_test.go

@@ -24,11 +24,10 @@ import (
 	"reflect"
 	"testing"
 
-	"k8s.io/api/core/v1"
-
 	kubelessApi "github.com/kubeless/kubeless/pkg/apis/kubeless/v1beta1"
 	"k8s.io/api/apps/v1beta2"
 	"k8s.io/api/autoscaling/v2beta1"
+	"k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/intstr"

cmd/kubeless/function/update.go

@@ -18,6 +18,7 @@ package function
 
 import (
 	"io/ioutil"
+	"os"
 	"strings"
 
 	"github.com/kubeless/kubeless/pkg/langruntime"
@@ -26,13 +27,33 @@ import (
 	"github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
 	"github.com/spf13/pflag"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
 var updateCmd = &cobra.Command{
 	Use:   "update <function_name> FLAG",
 	Short: "update a function on Kubeless",
 	Long:  `update a function on Kubeless`,
 	Run: func(cmd *cobra.Command, args []string) {
+		cli := utils.GetClientOutOfCluster()
+		controllerNamespace := os.Getenv("KUBELESS_NAMESPACE")
+		kubelessConfig := os.Getenv("KUBELESS_CONFIG")
+
+		if len(controllerNamespace) == 0 {
+			controllerNamespace = "kubeless"
+		}
+
+		if len(kubelessConfig) == 0 {
+			kubelessConfig = "kubeless-config"
+		}
+		config, err := cli.CoreV1().ConfigMaps(controllerNamespace).Get(kubelessConfig, metav1.GetOptions{})
+		if err != nil {
+			logrus.Fatalf("Unable to read the configmap: %v", err)
+		}
+
+		var lr = langruntime.New(config)
+		lr.ReadConfigMap()
+
 		if len(args) != 1 {
 			logrus.Fatal("Need exactly one argument - function name")
 		}
@@ -66,9 +87,9 @@ var updateCmd = &cobra.Command{
 			logrus.Fatal(err)
 		}
 
-		if runtime != "" && !langruntime.IsValidRuntime(runtime) {
+		if runtime != "" && !lr.IsValidRuntime(runtime) {
 			logrus.Fatalf("Invalid runtime: %s. Supported runtimes are: %s",
-				runtime, strings.Join(langruntime.GetRuntimes(), ", "))
+				runtime, strings.Join(lr.GetRuntimes(), ", "))
 		}
 
 		triggerHTTP, err := cmd.Flags().GetBool("trigger-http")
@@ -154,7 +175,6 @@ var updateCmd = &cobra.Command{
 		if port != nil && (*port <= 0 || *port > 65535) {
 			logrus.Fatalf("Invalid port number %d specified", *port)
 		}
-		cli := utils.GetClientOutOfCluster()
 		f, err := getFunctionDescription(cli, funcName, ns, handler, file, funcDeps, runtime, topic, schedule, runtimeImage, mem, timeout, triggerHTTP, headless, port, envs, labels, secrets, previousFunction)
 		if err != nil {
 			logrus.Fatal(err)
@@ -175,7 +195,7 @@ var updateCmd = &cobra.Command{
 }
 
 func init() {
-	updateCmd.Flags().StringP("runtime", "", "", "Specify runtime. Available runtimes are: "+strings.Join(langruntime.GetRuntimes(), ", "))
+	updateCmd.Flags().StringP("runtime", "", "", "Specify runtime")
 	updateCmd.Flags().StringP("handler", "", "", "Specify handler")
 	updateCmd.Flags().StringP("from-file", "", "", "Specify code file")
 	updateCmd.Flags().StringP("memory", "", "", "Request amount of memory for the function")

cmd/kubeless/getServerConfig/getServerConfig.go

@@ -0,0 +1,43 @@
+package getServerConfig
+
+import (
+	"os"
+	"strings"
+
+	"github.com/kubeless/kubeless/pkg/langruntime"
+	"github.com/kubeless/kubeless/pkg/utils"
+	"github.com/sirupsen/logrus"
+	"github.com/spf13/cobra"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+//GetServerConfigCmd contains first-class command for displaying the current server config
+var GetServerConfigCmd = &cobra.Command{
+	Use:   "get-server-config",
+	Short: "Print the current configuration of the controller",
+	Long:  ``,
+	Run: func(cmd *cobra.Command, args []string) {
+		cli := utils.GetClientOutOfCluster()
+		controllerNamespace := os.Getenv("KUBELESS_NAMESPACE")
+		kubelessConfig := os.Getenv("KUBELESS_CONFIG")
+
+		if len(controllerNamespace) == 0 {
+			controllerNamespace = "kubeless"
+		}
+
+		if len(kubelessConfig) == 0 {
+			kubelessConfig = "kubeless-config"
+		}
+		config, err := cli.CoreV1().ConfigMaps(controllerNamespace).Get(kubelessConfig, metav1.GetOptions{})
+		if err != nil {
+			logrus.Fatalf("Unable to read the configmap: %v", err)
+		}
+
+		var lr = langruntime.New(config)
+		lr.ReadConfigMap()
+
+		logrus.Info("Current Server Config:")
+		logrus.Infof("Supported Runtimes are: %s",
+			strings.Join(lr.GetRuntimes(), ", "))
+	},
+}

cmd/kubeless/kubeless.go

@@ -22,6 +22,7 @@ import (
 
 	"github.com/kubeless/kubeless/cmd/kubeless/autoscale"
 	"github.com/kubeless/kubeless/cmd/kubeless/function"
+	"github.com/kubeless/kubeless/cmd/kubeless/getServerConfig"
 	"github.com/kubeless/kubeless/cmd/kubeless/route"
 	"github.com/kubeless/kubeless/cmd/kubeless/topic"
 	"github.com/kubeless/kubeless/cmd/kubeless/version"
@@ -37,7 +38,7 @@ func newRootCmd() *cobra.Command {
 		Long:  globalUsage,
 	}
 
-	cmd.AddCommand(function.FunctionCmd, topic.TopicCmd, version.VersionCmd, route.RouteCmd, autoscale.AutoscaleCmd)
+	cmd.AddCommand(function.FunctionCmd, topic.TopicCmd, version.VersionCmd, route.RouteCmd, autoscale.AutoscaleCmd, getServerConfig.GetServerConfigCmd)
 	return cmd
 }
 

docs/implementing-new-runtime.md

@@ -13,49 +13,38 @@ All the structures that implement  language support are coded in the file `langr
 
 If you want to extend it and make another language available it is necessary to change the following components:
 
-## 1. Change the Const block string structure
+## 1. Update the kubeless-config configmap
 
-In this block of constants declaration, you sould create an entry pointing to the repository of the Docker container image for the runtime of the new language.
+In this configmap there is a set of runtime images. You need to update this set with an entry pointing to the repository of the Docker container image for the runtime of the new language.
 
 Usually there are three entries - one for HTTP triggers, another for event based functions and another one for the Init container that will be used to install dependencies in the build process. If your new runtime implementation will support only HTTP triggers, then create only two entries as follows:
 
-In the example below, the `dotnetcore2Http` and `dotnetcore2Init` entry was created, pointing to an image of a Docker container running the .netcore runtime.
+In the example below, a custom image for `dotnetcore` has been added. You can optionally add `imagePullSecrets` if they are necessary to pull the image from a private Docker registry.
 
 ```patch
-const (
+runtime-images
 ...
-        ruby24Pubsub   = "bitnami/kubeless-ruby-event-consumer@sha256:938a860dbd9b7fb6b4338248a02c92279315c6e42eed0700128b925d3696b606"
-+        dotnetcore2Http   = "mydocker/kubeless-netcore:latest"
-+        dotnetcore2Init   = "mydocker/kubeless-netcore-build:latest"
++        - ID: "dotnetcore"
++		  versions:
++		  - name: "dotnetcore2"
++		    version: "2.0"
++			httpImage: "mydocker/kubeless-netcore:latest"
++           initImage: "mydocker/kubeless-netcore-build:latest"
++           imagePullSecrets:
++           - imageSecret: "Secret"
++         depName: "requirements.xml"
++         fileNameSuffix: ".cs"
 ``` 
 
-## 2. Include a runtime variable
+Restart the controller after updating the configmap.
 
-Each runtime has a variable for it and needs also to be treated by the function `init()` as shown below.
-
-```patch
-var pythonVersions, nodeVersions, rubyVersions, dotnetcoreVersions []runtimeVersion
-
-func init() {
-...
-+	dotnetcore2 := runtimeVersion{version: "2.0", httpImage: dotnetcore2Http, pubsubImage: "", initImage: dotnetcore2Init}
-+	dotnetcoreVersions = []runtimeVersion{dotnetcore2}
-
-	availableRuntimes = []RuntimeInfo{
-		{ID: "python", versions: pythonVersions, DepName: "requirements.txt", FileNameSuffix: ".py"},
-		{ID: "nodejs", versions: nodeVersions, DepName: "package.json", FileNameSuffix: ".js"},
-		{ID: "ruby", versions: rubyVersions, DepName: "Gemfile", FileNameSuffix: ".rb"},
-+		{ID: "dotnetcore", versions: dotnetcoreVersions, DepName: "requirements.xml", FileNameSuffix: ".cs"},
-	}
-```
-
-## 3. Add the build instructions to include dependencies in the runtime
+## 2. Add the build instructions to include dependencies in the runtime
 
 Each runtime has specific instructions to install its dependencies. These instructions are specified in the method `GetBuildContainer()`. About this method you should know:
  - The folder with the function and the dependency files is mounted at `depsVolume.MountPath`
  - Dependencies should be installed in the folder `runtimeVolume.MountPath`
 
-## 4. Update the deployment to load requirements for the runtime image
+## 3. Update the deployment to load requirements for the runtime image
 
 Some languages require to specify an environment variable in order to load dependencies from a certain path. If that is the case, update the function `updateDeployment()` to include the required environment variable:
 
@@ -72,7 +61,7 @@ func UpdateDeployment(dpm *v1beta1.Deployment, depsPath, runtime string) {
 
 This function is called if there are requirements to be injected in your runtime or if it is a custom runtime.
 
-## 5. Add examples
+## 4. Add examples
 
 In order to demonstrate the usage of the new runtime it will be necessary to add at least three different examples:
  - GET Example: A simple example in which the function returns a "hello world" string or similar.
@@ -81,7 +70,7 @@ In order to demonstrate the usage of the new runtime it will be necessary to add
 
  The examples should be added to the folder `examples/<language_id>/` and should be added as well to the Makefile present in `examples/Makefile`. Note that the target should be `get-<language_id>`, `post-<language_id>` and `get-<language_id>-deps` for three examples above.
 
-## 6. Add tests
+## 5. Add tests
 
 For each new runtime, there should be integration tests that deploys the three examples above and check that the function is successfully deployed and that the output of the function is the expected one. For doing so add the counterpart `get-<language_id>-verify`, `post-<language_id>-verify` and `get-<language_id>-deps-verify` in the `examples/Makefile` and enable the execution of these tests in the script `test/integration-tests.bats`: