Remove zookeeper watches when entries are removed from ZooCache

[keith-turner]

When something is removed from zoocache also remove the watch. Currently when an entry is removed from zoocache explicitly its watch is just left there. This is follow on work from #5143.

Originally posted by @keith-turner in #5143 (comment)

[dlmarion]

@keith-turner - did you see that WatchTheWatchCountIT is also failing in main? I added the following to see what was being watched. It seems as if every ZooKeeper session (Accumulo process?) has a Watcher set on every tserver. Seemed odd to me.

diff --git a/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java b/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
index e8a55081c3..2308e774ee 100644
--- a/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
+++ b/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
@@ -291,7 +291,7 @@ public class MiniAccumuloClusterImpl implements AccumuloCluster {
       zooCfg.setProperty("clientPort", config.getZooKeeperPort() + "");
       zooCfg.setProperty("maxClientCnxns", "1000");
       zooCfg.setProperty("dataDir", config.getZooKeeperDir().getAbsolutePath());
-      zooCfg.setProperty("4lw.commands.whitelist", "ruok,wchs");
+      zooCfg.setProperty("4lw.commands.whitelist", "ruok,wchs,wchp");
       zooCfg.setProperty("admin.enableServer", "false");
       zooCfg.store(fileWriter, null);
 
diff --git a/test/src/main/java/org/apache/accumulo/test/functional/WatchTheWatchCountIT.java b/test/src/main/java/org/apache/accumulo/test/functional/WatchTheWatchCountIT.java
index b90d8b7c6e..fd89e21612 100644
--- a/test/src/main/java/org/apache/accumulo/test/functional/WatchTheWatchCountIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/functional/WatchTheWatchCountIT.java
@@ -88,6 +88,15 @@ public class WatchTheWatchCountIT extends ConfigurableMacBase {
       log.info("Number of watchers to be expected in range: ({}, {}), currently have {}", MIN, MAX,
           total);
 
+      if (total >= MAX) {
+        try (Socket socket = new Socket(hostAndPort.getHost(), hostAndPort.getPort())) {
+          socket.getOutputStream().write("wchp\n".getBytes(UTF_8), 0, 5);
+          byte[] buffer = new byte[10485760];
+          int n = socket.getInputStream().read(buffer);
+          String response = new String(buffer, 0, n, UTF_8);
+          System.out.println(response);
+        }        
+      }
       assertTrue(total > MIN && total < MAX, "Expected number of watchers to be contained in ("
           + MIN + ", " + MAX + "), but actually was " + total);
 

[keith-turner]

@keith-turner - did you see that WatchTheWatchCountIT is also failing in main? I added the following to see what was being watched. It seems as if every ZooKeeper session (Accumulo process?) has a Watcher set on every tserver. Seemed odd to me.

No I had not seen that, I can look into it.