[CI] Randomize RPCService's port and handle NoSuchProcess error prope…

…rly (#690)

* [CI] Randomize RPCService's port in test_container_rpc.sh

We should avoid the port collisions between test cases as much as possible.
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1934/ shows the port collision to us.

* Fix my mistake

* Handle psutil.NoSuchProcess error properly

Current Jenkins CI shows the following errors to us.
```
 [unittest_rpc_container] Traceback (most recent call last):
 [unittest_rpc_container] File ../python/rpc_test_container.py, line 1, in <module>
 [unittest_rpc_container] import rpc
 [unittest_rpc_container] File /clipper/containers/python/rpc.py, line 20, in <module>
 [unittest_rpc_container] import clipper_admin.metrics as metrics
 [unittest_rpc_container] File /clipper/clipper_admin/clipper_admin/metrics/__init__.py, line 6, in <module>
 [unittest_rpc_container] if not server.redis_daemon_exist():
 [unittest_rpc_container] File /clipper/clipper_admin/clipper_admin/metrics/server.py, line 134, in redis_daemon_exist
 [unittest_rpc_container] process_names = [psutil.Process(pid).name() for pid in pids]
 [unittest_rpc_container] File /usr/local/lib/python2.7/dist-packages/psutil/__init__.py, line 338, in __init__
 [unittest_rpc_container] self._init(pid)
 [unittest_rpc_container] File /usr/local/lib/python2.7/dist-packages/psutil/__init__.py, line 378, in _init
 [unittest_rpc_container] raise NoSuchProcess(pid, None, msg)
 [unittest_rpc_container] psutil._exceptions.NoSuchProcess: psutil.NoSuchProcess no process found with pid 162
```

* Fix location

clipper_admin/clipper_admin/metrics/server.py

@@ -131,7 +131,15 @@ def redis_daemon_exist():
     # situation is that we are in a container without any other python2
     # process.
     pids = psutil.pids()
-    process_names = [psutil.Process(pid).name() for pid in pids]
+    process_names = []
+
+    for pid in pids:
+        try:
+            name = psutil.Process(pid).name()
+        except psutil.NoSuchProcess:
+            name = None
+        process_names.append(name)
+
     return 'redis-server' in process_names
 
 

containers/test/test_container_rpc.sh

@@ -22,7 +22,10 @@ then
 fi
 
 success=false
-rpc_service_port=17000  # for test only
+
+PORT_RANGE_START=10000
+PORT_RANGE_END=20000
+RPC_SERVICE_PORT=`perl -e "print int(rand($PORT_RANGE_END-$PORT_RANGE_START)) + $PORT_RANGE_START"`
 
 function clean_up {
     # Perform program exit housekeeping
@@ -48,8 +51,8 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 cd $DIR
 
 # Start python rpc test container
-echo "Starting python RPC test container..."
-python ../python/rpc_test_container.py --rpc_service_port $rpc_service_port &
+echo "Starting python RPC test container... (port:$RPC_SERVICE_PORT)"
+python ../python/rpc_test_container.py --rpc_service_port $RPC_SERVICE_PORT &
 
 # Deprecate JVM containers
 # cd ../jvm
@@ -68,20 +71,21 @@ cd $DIR/../../
 cd container
 make container_rpc_test
 container_uptime_seconds=180
-./container_rpc_test -t $container_uptime_seconds -p $rpc_service_port &
+echo "Starting cpp RPC test container... (port:$RPC_SERVICE_PORT)"
+./container_rpc_test -t $container_uptime_seconds -p $RPC_SERVICE_PORT &
 
 sleep 10s
 
 cd $DIR/../../debug/src/benchmarks
 make rpctest
-echo "Executing RPC test (first iteration)..."
 REDIS_PORT=$1
-./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $rpc_service_port
+echo "Executing RPC test (first iteration)... (redis port:$REDIS_PORT, rpc_service_port:$RPC_SERVICE_PORT)"
+./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $RPC_SERVICE_PORT
 redis-cli -p $REDIS_PORT "flushall"
 echo "Sleeping for 5 seconds..."
 sleep 5s
-echo "Executing RPC test (second iteration)..."
-./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $rpc_service_port
+echo "Executing RPC test (second iteration)... (redis port:$REDIS_PORT, rpc_service_port:$RPC_SERVICE_PORT)"
+./rpctest --num_containers=2 --timeout_seconds=30 --redis_port $REDIS_PORT --rpc_service_port $RPC_SERVICE_PORT
 redis-cli -p $REDIS_PORT "flushall"
 echo "TEST PASSED!"
 success=true