Liveness Probes failing on port 8081

[dcasati]

Describe the bug
Liveness probes hitting port 8081 on the pega-web and pega-stream pod are failing. I believe the issue is here is related to the liveness probe port being defined on /opt/pega/conf/server.xml but not on /usr/loca/tomcat/conf/server.xml -

pegauser@pega-web-6f787bbb7b-w7jjr:/opt/pega/config$ diff -uNp  /usr/local/tomcat/conf/server.xml server.xml 
--- /usr/local/tomcat/conf/server.xml	2020-11-12 15:41:13.000000000 +0000
+++ server.xml	2021-03-30 21:14:19.087433168 +0000
@@ -68,7 +68,13 @@
     -->
     <Connector port="8080" protocol="HTTP/1.1"
                connectionTimeout="20000"
-               redirectPort="8443" />
+	       redirectPort="8443" />
+
+    <!-- facilitates liveness check via separate port -->
+    <Connector port="8081" protocol="HTTP/1.1"
+               connectionTimeout="20000"
+	       redirectPort="8443" 
+	       maxThreads="1"/>
     <!-- A "Connector" using the shared thread pool-->
     <!--
     <Connector executor="tomcatThreadPool"
@@ -158,14 +164,22 @@
         <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
         -->
 
+        <Valve className="org.apache.catalina.valves.RemoteIpValve"
+          protocolHeader="x-forwarded-proto" />
+
         <!-- Access log processes all example.
              Documentation at: /docs/config/valve.html
              Note: The pattern used is equivalent to using pattern="common" -->
         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
                prefix="localhost_access_log" suffix=".txt"
-               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
+               pattern="%{X-Forwarded-For}i %h %l %u %t &quot;%r&quot; %s %b" 
+               resolveHosts="false" />
+
+        <Valve className="org.apache.catalina.valves.ErrorReportValve"
+               showReport="false" 
+               showServerInfo="false" />
 
       </Host>
     </Engine>
   </Service>
-</Server>
+</Server>
\ No newline at end of file

To Reproduce

Running curl to check if the endpoint being probed is live on port 8081:

pegauser@pega-web-6f787bbb7b-w7jjr:/$ curl http://localhost:8081/prweb/PRRestService/monitor/pingService/ping
curl: (7) Failed to connect to localhost port 8081: Connection refused
pegauser@pega-web-6f787bbb7b-w7jjr:/$ 

The same command now on port 8080 - works as expected:

pegauser@pega-web-6f787bbb7b-w7jjr:/$ curl http://localhost:8080/prweb/PRRestService/monitor/pingService/ping
{"node_type":["WebUser"],"health":[{"last_reported_time":"2021-03-31T17:01:00.239Z","test_name":"HTML-Stream-Check","status":"success"},{"last_reported_time":"2021-03-31T17:01:00.239Z","test_name":"StaleThreadHealthCheck","status":"success"},{"last_reported_time":"2021-03-31T17:01:00.239Z","test_name":"ServiceRegistry-Check","status":"success"}],"state":"healthy","node_id":"pega-web-6f787bbb7b-w7jjr"}pegauser@pega-web-6f787bbb7b-w7jjr:/$ 

Expected behavior
Either implement an endpoint that listens on port 8081 (preferable) or re-use port 8080 for the liveness probe (likely less preferable)

Chart version
pega-1.4.6

$ helm ls
NAME  	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART     	APP VERSION
mypega	mypega   	1       	2021-03-30 21:08:05.400597966 +0000 UTC	deployed	pega-1.4.6	           

Server (if applicable, please complete the following information):

• OS: Ubuntu 18.04
• Environment: Azure Kubernetes Service

Additional context
For now, I was able to change the probe from port 8081 to 8080. This then avoids the pods being restarted after a while

diff -r -uNp pega/templates/_pega-deployment.tpl /tmp/pega/templates/_pega-deployment.tpl
--- pega/templates/_pega-deployment.tpl	2021-03-29 23:33:55.989713709 +0000
+++ /tmp/pega/templates/_pega-deployment.tpl	2021-03-31 16:50:28.875759725 +0000
@@ -188,7 +188,7 @@ spec:
         livenessProbe:
           httpGet:
             path: "/{{ template "pega.applicationContextPath" . }}/PRRestService/monitor/pingService/ping"
-            port: 8080
+            port: 8081
             scheme: HTTP
           initialDelaySeconds: {{ $livenessProbe.initialDelaySeconds | default 0 }}
           timeoutSeconds: {{ $livenessProbe.timeoutSeconds | default 20 }}
@@ -225,7 +225,7 @@ spec:
         livenessProbe:
           httpGet:
             path: "/{{ template "pega.applicationContextPath" . }}/PRRestService/monitor/pingService/ping"
-            port: 8080
+            port: 8081
             scheme: HTTP
           initialDelaySeconds: {{ $livenessProbe.initialDelaySeconds | default 200 }}
           timeoutSeconds: {{ $livenessProbe.timeoutSeconds | default 20 }}

[APegaDavis]

Hi @dcasati -- are you using a docker-pega-web-ready image that includes this PR (looks like either latest or 2.1.0-pegasystems-1666 tags)? If /opt/pega/config/server.xml it will replace the default tomcat version in /usr/local/tomcat/conf: https://github.com/pegasystems/docker-pega-web-ready/pull/118/files#diff-048a1a5a8856efac5b2321f63380f0b63c018074a59fc008da40ba1d9596340aR206

[dcasati]

Hi @APegaDavis -

I'm using pega:8.5.1 . I've inspected the docker-entrypoint.sh and that's definitely not what I have. Feel free to close this issue here if updating to a newer version is the way to go :)

[pega-talba]

Hi @APegaDavis -

I'm using pega:8.5.1 . I've inspected the docker-entrypoint.sh and that's definitely not what I have. Feel free to close this issue here if updating to a newer version is the way to go :)

While this is great to hear, we also shouldn't be requiring newer docker-pega-web-ready images in patch releases of pega-helm-charts. So, we'll ensure this works as-is for 1.4.7, and then start a new minor version that leverages 8081 and notes a minimum image version.