[ED-3649] merge release-6.0.0 changes to 7.0.0 (#3998)

* Fix syntax (#3894)

* Updated vars for cron schedule (#3895)

* Updated vars for cron schedule

* Update main.yml

* [ED-2793] Fix: release-6.0.0 installation issues  (#3900)

* Update Jenkins version

* Fix job parameter

* Fix error: You need to install jmespath prior to running json_query filter

* Update main.yml with cassandra (#3845) (#3901)

Co-authored-by: gohilamariappan <41056032+gohilamariappan@users.noreply.github.com>

* ED-2838: ##Release-6.0.0 new variables (#3899)

* ED-2838 Release-6.0.0 new variables

* Added with comments

* ED-2838 jenkins josb update and new variables

* added storage endpoint detials

* [ED-2793] Fix installation issues (#3905)

* Update maven repo url

* Add missing jobs

* Fix docker creds secrets issue when having specialcharacters

* [ED-2793] Add OCI specific vars (#3912)

* Add OCI specific vars

* Add mobile deeplink url var

* Add ml_core_internal_access_token for private repo template

Add ml_core_internal_access_token for private repo template

* Update cassandra jenkins job config (#3927)

* saving changes (#3935)

Co-authored-by: ankitshahu <ankit@matchlog.delivery>

* #OB-553 fix: Analytics service issue fixes for OCI

* #OB-553 fix: Analytics service issue fixes for OCI

* ED-3534:- Adoption Changes in ML service (#3963)

* saving changes

* added Upload api endpoint in public kong

---------

Co-authored-by: ankitshahu <ankit@matchlog.delivery>

* ED-3079: deploy nginx to support cors for s3 compatible object storage (#3915)

* nginx-cors

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* hostname changed

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* test

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* test

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* added nginx-cors-public chart

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* test

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* corrected proxy conf

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* corrected syntax

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* changed nodeport

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* removed duplicate

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* nodeport change

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* nodeport change

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* changed public ip

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* test

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* removed nginx-cors ansible roles

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* added jenkins job for nginx-cors-public deployment

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* updated variable in values.j2

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* testing public ingress along with s3 cors

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* testing with condition for csp

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* removed nginx-cors-public

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

---------

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>

* Added new kafka topics in main.yml and config.j2 (#3970)

* Issue #CO-824: dynamic redirect URL

* Issue #CO-824: dynamic redirect URL

* Issue #CO-824: SSO Isse condition update

* Issue ED-0000 report service env update

* Issue ED-0000 report service env update (#3977)

* Issue ED-0000 report service env update

* Issue ED-0000 report service env update (#3981)

* ED-3534:- Adoption Changes in ML service (#3983)

* helm chart for ml-analytics

* Update and rename values.yaml to values.j2

* Update main.yml

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Delete kubernetes/helm_charts/core/ml-analytics-service/templates/NOTES.txt

* Update deployment.yaml

* Update deployment.yaml

* Update deployment.yaml

* Update values.j2

* Update deployment.yaml

* Update deployment.yaml

* Update ml-analytics-service.conf

* Update deployment.yaml

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Updated ml-analytics-service.conf

* added new kafka topics for observations

* Update ml-analytics-service.conf

---------

Co-authored-by: raghupathi <raghupathi.g@optit.in>
Co-authored-by: Raghupathi Guduri <raghupathiguduri@gmail.com>

* Issue #IQ-750 fix: added script for inquiry schema upload

* Issue #IQ-750 fix: updated jenkins file

* added the faust related changes

* Issue #IQ-750 fix: updated jenkins script

* added the value to ml_analytics_faust_applications_list

* Update secrets.yml

* ED-3534:- Adoption Changes in ML service (#3993)

* updated druid ports

* updated druid ports

* key changes

* key changes

---------

Co-authored-by: vivek <user@PWA177.local>

* Add release-6.0.0 missing vars

* Add release-6.0.0 missing vars (#3995)

* Add inquiry-schema-upload Jenkinsfile

---------

Signed-off-by: Deepak Devadathan <deepak.devadathan@oracle.com>
Co-authored-by: Raghupathi Guduri <raghupathiguduri@gmail.com>
Co-authored-by: gohilamariappan <41056032+gohilamariappan@users.noreply.github.com>
Co-authored-by: Prasath Sivasubramaniyan <tsprasath91@gmail.com>
Co-authored-by: Akash Shah <aks301190@gmail.com>
Co-authored-by: Ankit Shahu <ankit.shahu@hotmail.com>
Co-authored-by: ankitshahu <ankit@matchlog.delivery>
Co-authored-by: sowmya-dixit <sowmyadixit7@gmail.com>
Co-authored-by: Deepak Devadathan <deepak.devadathan@oracle.com>
Co-authored-by: Prashant Gunjeti <153811259+prashanthShiksha@users.noreply.github.com>
Co-authored-by: Shubham Bansal <shubhambansal@TI-MAC-070.local>
Co-authored-by: Rajeev Sathish <rajeev.sathish@tarento.com>
Co-authored-by: Vivek M <125434153+Vivek-M-08@users.noreply.github.com>
Co-authored-by: raghupathi <raghupathi.g@optit.in>
Co-authored-by: Kumar Gauraw <er.krgauraw@gmail.com>
Co-authored-by: vivek <user@PWA177.local>

ansible/artifacts/sunbird/login/resources/js/telemetry_service.js

@@ -3159,6 +3159,7 @@ var handleGoogleAuthEvent = () => {
         redirectToLib();
     }
 };
+
 var redirectToPortal = (redirectUrlPath) => { // redirectUrlPath for sso and self signUp
     const curUrlObj = window.location;
     var redirect_uri = getValueFromSession('redirect_uri');
@@ -3171,9 +3172,7 @@ var redirectToPortal = (redirectUrlPath) => { // redirectUrlPath for sso and sel
             const redirect_uriLocation = new URL(redirect_uri);
             if (client_id === 'android' || client_id === 'desktop') {
                 window.location.href = sessionUrlObj.protocol + '//' + sessionUrlObj.host + redirectUrlPath + updatedQuery;
-            } else if(client_id === 'portal' && 
-            redirectUrlPath === '/sign-in/sso/select-org' && 
-            (redirect_uri.includes('dock.sunbirded.org') || redirect_uri.includes('dockstaging.sunbirded.org'))) {
+            } else if(client_id === 'portal' && redirectUrlPath === '/sign-in/sso/select-org' && (sessionUrlObj.host !== redirect_uriLocation.host)) {
                 window.location.href = sessionUrlObj.protocol + '//' + sessionUrlObj.host + redirectUrlPath + updatedQuery;
             } else {
                 window.location.href = redirect_uriLocation.protocol + '//' + redirect_uriLocation.host +

ansible/roles/kong-api/defaults/main.yml

@@ -8300,6 +8300,27 @@ kong_apis:
     config.required: false
     config.enabled: false
 
+- name: uploadFileToCloud
+  uris: "{{ cloud_service_prefix }}/mlcore/v1/files/upload"
+  upstream_url: "{{ ml_core_service_url }}/v1/cloud-services/files/upload"
+  strip_uri: true
+  plugins:
+  - name: jwt
+  - name: cors
+  - "{{ statsd_pulgin }}"
+  - name: acl
+    config.whitelist:
+    - cloudUrlsCreate
+  - name: rate-limiting
+    config.policy: local
+    config.hour: "{{ medium_rate_limit_per_hour }}"
+    config.limit_by: credential
+  - name: request-size-limiting
+    config.allowed_payload_size: "{{ small_request_size_limit }}"
+  - name: opa-checks
+    config.required: false
+    config.enabled: false
+
 - name: listPlatformPrograms
   uris: "{{ user_extension_prefix }}/mlcore/v1/programsByPlatformRoles"
   upstream_url: "{{ ml_core_service_url }}/v1/user-extension/programsByPlatformRoles"

ansible/roles/ml-analytics-service/defaults/main.yml

@@ -138,3 +138,4 @@ ml_analytics_batch_cron_hour: "{{ ml_batch_cron_hour | default('18') }}"
 ml_analytics_nvsk_cron_minute: "{{ ml_nvsk_cron_minute | default('30') }}"
 ml_analytics_nvsk_cron_hour: "{{ ml_nvsk_cron_hour | default('7') }}"
 ml_analytics_nvsk_cron_weekday: "{{ ml_nvsk_cron_weekday | default('4') }}"
+ml_analytics_faust_applications_list: "'observations/py_observation_streaming observations/', 'observations/py_observation_evidence_streaming observations/', 'survey/py_survey_streaming survey/', 'survey/py_survey_evidence_streaming survey/'"

ansible/roles/ml-analytics-service/templates/config.j2

@@ -263,3 +263,8 @@ store = {{ ml_analytics_reports_store }}
 [ML_CORE_SERVICE_URL]
 
 url = {{ ml_analytics_core_service }}
+
+
+[SERVICES]
+
+faust_applications_list = {{ ml_analytics_faust_applications_list }}

ansible/roles/stack-sunbird/defaults/main.yml

@@ -264,6 +264,7 @@ service_env:
   ml-reports-service: ../../../../ansible/roles/stack-sunbird/templates/ml-reports-service.env
   ml-core-service: ../../../../ansible/roles/stack-sunbird/templates/ml-core-service.env
   ml-projects-service: ../../../../ansible/roles/stack-sunbird/templates/ml-projects-service.env
+  ml-analytics-service: ../../../../ansible/roles/stack-sunbird/templates/ml-analytics-service.conf
   inbound: ../../../../ansible/roles/stack-sunbird/templates/inbound.env
   outbound:
     - ../../../../ansible/roles/stack-sunbird/templates/outbound.env

ansible/roles/stack-sunbird/templates/analytics_api_service.conf

@@ -201,10 +201,10 @@ public {
   }
 }
 cloud_storage_type="{{cloud_service_provider}}"
-storage.key.config="{{cloud_public_storage_accountname}}"
-storage.secret.config="{{cloud_public_storage_secret}}"
-public.storage.key.config="{{cloud_public_storage_accountname}}"
-public.storage.secret.config="{{cloud_public_storage_secret}}"
+storage.key.config="cloud_storage_key"
+storage.secret.config="cloud_storage_secret"
+public.storage.key.config="public_cloud_storage_key"
+public.storage.secret.config="public_cloud_storage_secret"
 cloud_storage_endpoint_with_protocol="{{cloud_storage_url}}"
 metrics.time.interval.min=30
 cache.refresh.time.interval.min=1440

ansible/roles/stack-sunbird/templates/ml-core-service.env

@@ -87,4 +87,8 @@ ID={{ml_core_service_id | default(env_name+"."+sunbird_instance+".ml.core.servic
 TELEMETRY_ON_OFF={{manage_learn_telemetry_on_off | default("ON")}}
 
 #Topic name for telemetry 
-TELEMETRY_TOPIC={{telemetry_raw_topic_name | default(env_name+".telemetry.raw")}}
+TELEMETRY_TOPIC={{telemetry_raw_topic_name | default(env_name+".telemetry.raw")}}
+
+#Public base url of sunbird
+PUBLIC_BASE_URL={{ml_core_public_base_url | default("")}}
+

ansible/roles/stack-sunbird/templates/ml-survey-service.env

@@ -86,3 +86,4 @@ TELEMETRY_ON_OFF={{manage_learn_telemetry_on_off | default("ON")}}
 
 #Topic name for telemetry 
 TELEMETRY_TOPIC={{telemetry_raw_topic_name | default(env_name+".telemetry.raw")}}
+

ansible/roles/stack-sunbird/templates/sunbird_report-service.env

@@ -29,3 +29,13 @@ sunbird_cache_ttl={{sunbird_cache_ttl}}
 sunbird_report_sas_expiry_in_minutes=3600
 sunbird_dataservice_url={{sunbird_dataservice_url}}
 
+# 6.0.0
+
+sunbird_cloud_storage_provider={{cloud_service_provider}}
+cloud_private_storage_accountname={{cloud_private_storage_accountname | default("")}}
+cloud_private_storage_secret={{cloud_private_storage_secret | default("")}}
+cloud_storage_privatereports_bucketname={{cloud_storage_privatereports_bucketname | default("reports")}}
+cloud_storage_resourceBundle_bucketname={{cloud_storage_label_bucketname | default("label")}}
+cloud_private_storage_region={{cloud_private_storage_region | default("ap-south-1")}}
+cloud_private_storage_project={{cloud_private_storage_project | default("")}}
+cloud_private_storage_endpoint={{cloud_private_storage_endpoint | default("")}}

kubernetes/helm_charts/core/analytics/templates/deployment.yaml

@@ -35,14 +35,14 @@ spec:
           value: {{ .Values.env.min_heap | quote }}
         - name: MAX_HEAP
           value: {{ .Values.env.max_heap | quote }}
-        - name: azure_storage_secret
-          value: {{ .Values.env.azure_private_account_secret | quote }}
-        - name: azure_storage_key
-          value: {{ .Values.env.azure_private_account_name | quote }}
-        - name: public_azure_storage_secret
-          value: {{ .Values.env.azure_public_account_secret | quote }}
-        - name: public_azure_storage_key
-          value: {{ .Values.env.azure_public_account_name | quote }}
+        - name: cloud_storage_secret
+          value: {{ .Values.env.cloud_private_account_secret | quote }}
+        - name: cloud_storage_key
+          value: {{ .Values.env.cloud_private_account_name | quote }}
+        - name: public_cloud_storage_secret
+          value: {{ .Values.env.cloud_public_account_secret | quote }}
+        - name: public_cloud_storage_key
+          value: {{ .Values.env.cloud_public_account_name | quote }}
         - name: _JAVA_OPTIONS
           value: -Dlog4j2.formatMsgNoLookups=true
         envFrom:

kubernetes/helm_charts/core/analytics/values.j2

@@ -8,10 +8,10 @@ env:
   javaoptions: {{analytics_java_mem_limit|default('-Xmx600m')}}
   min_heap: {{analytics_min_heap_limit|default('-Xms1g')}}
   max_heap: {{analytics_max_heap_limit|default('-Xmx2g')}}
-  azure_private_account_secret: {{ cloud_private_storage_secret }}
-  azure_private_account_name: {{ cloud_private_storage_accountname }}
-  azure_public_account_secret: {{ cloud_public_storage_secret }}
-  azure_public_account_name: {{ cloud_public_storage_accountname }}
+  cloud_private_account_secret: {{ cloud_private_storage_secret }}
+  cloud_private_account_name: {{ cloud_private_storage_accountname }}
+  cloud_public_account_secret: {{ cloud_public_storage_secret }}
+  cloud_public_account_name: {{ cloud_public_storage_accountname }}
 replicaCount: {{analytics_replicacount|default(1)}}
 repository: {{analytics_repository|default('sunbird-analytics-service')}}
 image_tag: {{ image_tag }}

kubernetes/helm_charts/core/ml-analytics-service/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

kubernetes/helm_charts/core/ml-analytics-service/Chart.yaml

@@ -0,0 +1,24 @@
+apiVersion: v2
+name: ml-analytics-service
+description: A Helm chart for Kubernetes
+
+# A chart can be either an 'application' or a 'library' chart.
+#
+# Application charts are a collection of templates that can be packaged into versioned archives
+# to be deployed.
+#
+# Library charts provide useful utilities or functions for the chart developer. They're included as
+# a dependency of application charts to inject those utilities and functions into the rendering
+# pipeline. Library charts do not define any templates and therefore cannot be deployed.
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "1.16.0"

kubernetes/helm_charts/core/ml-analytics-service/templates/_helpers.tpl

@@ -0,0 +1,62 @@
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "ml-analytics-service.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "ml-analytics-service.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "ml-analytics-service.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "ml-analytics-service.labels" -}}
+helm.sh/chart: {{ include "ml-analytics-service.chart" . }}
+{{ include "ml-analytics-service.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "ml-analytics-service.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "ml-analytics-service.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "ml-analytics-service.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "ml-analytics-service.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}

kubernetes/helm_charts/core/ml-analytics-service/templates/configmap.yaml

@@ -0,0 +1,11 @@
+---
+{{- $keys := .Files.Glob "keys/*" }}
+{{ if $keys }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ml-analytics-access-keys 
+type: Opaque
+data:
+{{ (.Files.Glob "keys/*").AsSecrets | indent 2 }}
+{{ end }}

kubernetes/helm_charts/core/ml-analytics-service/templates/deployment.yaml

@@ -0,0 +1,41 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ .Chart.Name }}
+  namespace: {{ .Values.namespace }}
+  annotations:
+    reloader.stakater.com/auto: "true"
+spec:
+  replicas: {{ .Values.replicaCount }}
+  strategy:
+     rollingUpdate:
+        maxSurge: {{ .Values.strategy.maxsurge }}
+        maxUnavailable: {{ .Values.strategy.maxunavailable }}
+  selector:
+    matchLabels:
+      app: {{ .Chart.Name }}
+  template:
+    metadata:
+      labels:
+        app: {{ .Chart.Name }}
+    spec:
+{{- if .Values.imagepullsecrets }}
+      imagePullSecrets:
+      - name: {{ .Values.imagepullsecrets }}
+{{- end }}
+      volumes:
+      - name: {{ .Chart.Name }}-config
+        configMap:
+          name: {{ .Chart.Name }}-config
+      containers:
+      - name: {{ .Chart.Name }}
+        image: "{{ .Values.dockerhub }}/{{ .Values.repository }}:{{ .Values.image_tag }}"
+        imagePullPolicy: Always
+        resources:
+{{ toYaml .Values.resources | indent 10 }}
+        ports:
+        - containerPort: {{ .Values.network.port }}
+        volumeMounts:
+        - name: {{ .Chart.Name }}-config
+          mountPath: /opt/sparkjobs/ml-analytics-service/config.ini
+          subPath: ml-analytics-service.conf

kubernetes/helm_charts/core/ml-analytics-service/templates/service.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ .Chart.Name }}
+  namespace: {{ .Values.namespace }}
+  labels:
+    app: {{ .Chart.Name }}
+spec:
+  ports:
+  - name: http-{{ .Chart.Name }}
+    protocol: TCP
+    port: {{ .Values.network.targetport }}
+  selector:
+    app: {{ .Chart.Name }}

kubernetes/helm_charts/core/ml-analytics-service/values.j2

@@ -0,0 +1,21 @@
+namespace: {{ namespace }}
+imagepullsecrets: {{ imagepullsecrets }}
+dockerhub: {{ dockerhub }}
+
+replicaCount: {{ml_analytics_replicacount|default(1)}}
+repository: {{ml_analytics_repository|default('ml-analytics-service')}}
+image_tag: {{ image_tag }}
+resources:
+  requests:
+    cpu: {{ml_analytics_cpu_req|default('1000m')}}
+    memory: {{ml_analytics_mem_req|default('1000Mi')}}
+  limits:
+    cpu: {{ml_analytics_cpu_limit|default('2')}}
+    memory: {{ml_analytics_mem_limit|default('2048Mi')}}
+network:
+  port: 7000
+  targetport: 7000
+strategy:
+  type: RollingUpdate
+  maxsurge: {{ ml_core_maxsurge|default('25%') }}
+  maxunavailable: {{ ml_core_maxunavailable|default('25%') }}

kubernetes/helm_charts/core/nginx-public-ingress/templates/configMap.yaml

@@ -6,6 +6,10 @@ metadata:
 data:
   proxy-default.conf: |
 {{ .Values.proxyconfig | indent 4 }}
+{{- if eq .Values.csp "oci" }}
+  cors-proxy-default.conf: |
+{{ .Values.corsproxyconfig | indent 4 }}
+{{- end }}
   compression.conf: |
 {{ .Values.compressionConfig | indent 4 }}