indexer hangs because of interactive process (Bugzilla #13885)

[vladak]

status NEW severity normal in component indexer for ---
Reported in version unspecified on platform ANY/Generic
Assigned to: Trond Norbye

On 2010-01-14 11:45:31 +0000, Vladimir Kotal wrote:

When I started mirroring OpenSSL CVS repository (anonymous@cvs.openssl.org:/openssl-cvs) over SSH I did not setup the host keys. This resulted in the indexer to hang because of ssh client asked for the host keys (the classical yes/no question to verify the fingerprint).

The indexer should probably detect this sort of hang which is preventing it from proceeding.

I don't know whether this is just a problem with CVS indexer or it's common also for the other SCMs.

This is happening with 0.8.1.

On 2010-07-26 19:32:02 +0000, Knut Anders Hatlen wrote:

I see the same problem if I first check out the OpenSSL sources and then remove cvs.openssl.org from ~/.ssh/known_host before I try to index the sources.

I was hoping this could be solved by calling getOutputStream().close() on the Process object (and make it fail instead of hanging), but it seems like the input/output from the ssh process is not redirected along with the input/output from the parent process. Instead, it goes directly to the terminal. I found that even writing "yes\n" to the cvs process didn't help, since it was not forwarded to the ssh sub-process.

On 2010-07-27 09:45:05 +0000, Vladimir Kotal wrote:

(In reply to comment # 1)

I see the same problem if I first check out the OpenSSL sources and then remove
cvs.openssl.org from ~/.ssh/known_host before I try to index the sources.

I was hoping this could be solved by calling getOutputStream().close() on the
Process object (and make it fail instead of hanging), but it seems like the
input/output from the ssh process is not redirected along with the input/output
from the parent process. Instead, it goes directly to the terminal. I found
that even writing "yes\n" to the cvs process didn't help, since it was not
forwarded to the ssh sub-process.

A robust solution would be to set a time out (if no data are produced by the process) and then kill it.

[vladak]

We could use something like this http://stackoverflow.com/questions/5138946/process-builder-and-process-in-java-how-to-execute-a-process-with-a-timeout however it might not work for some SCMs, e.g. I just checked svn log on hanging server and it installed its own signal handler for SIGTERM/SIGINT. I suspect command.destroy() will default to SIGTERM (http://stackoverflow.com/questions/10630303/java-process-destroy-source-code-for-linux) and there is no easy way to kill it with SIGKILL (http://stackoverflow.com/questions/2950338/how-can-i-kill-a-linux-process-in-java-with-sigkill-process-destroy-does-sigte).

[vladak]

For completeness, the svn log command timed out after 3 minutes or so by itself which is acceptable. Nonetheless, this will help with commands which do not progress because they require some sort of interactive input.

The fix should reside in Executor.java:exec()

130        try {
131            process = processBuilder.start();
132
133            final InputStream errorStream = process.getErrorStream();
134            final SpoolHandler err = new SpoolHandler();
135            Thread thread = new Thread(new Runnable() {
136
137                @Override
138                public void run() {
139                    try {
140                        err.processStream(errorStream);
141                    } catch (IOException ex) {
142                        if (reportExceptions) {
143                            OpenGrokLogger.getLogger().log(Level.SEVERE,
144                                    "Error during process pipe listening", ex);
145                        }
146                    }
147                }
148            });
149            thread.start();
150

setup the Timer here

151            handler.processStream(process.getInputStream());
152
153            ret = process.waitFor();
154            process = null;
155            thread.join();
156            stderr = err.getBytes();