Copying big files is slow (compared to kubectl)

[hypnoce]

Is your task related to a problem? Please describe

On a local network, kubectl cp is network bounded when copying files :

# Copying a 800MB file
$ time kubectl cp pod_name:/path/to/big/file ./big_file
tar: Removing leading `/' from member names
 
real    0m7.548s
user    0m2.456s
sys     0m3.299s

Using the java kube client, time is much higher. It is cpu bounded :

  public static void main(String[] args) {
    DefaultKubernetesClient kubeClient = new DefaultKubernetesClient();
    final PodResource<Pod> pod = kubeClient.pods().withName("pod_name");
    pod.file("/path/to/big/file").copy(Paths.get("./big_file")); // takes 77s (10x !)
  }

I'm running java11 but it does not seem to be the problem. After a first analysis, it comes from Piped(In/Out)putStream that makes heavy use of locks.
Removing use of pipes drops the download time from ~77s to ~11s (7x gain).

Describe the solution you'd like

Refactor the code to make the network receiving thread directly write in the file instead of using piped(in/out)putstreams.

Describe alternatives you've considered

I've considered forking the kubectl command but it becomes very complex to pass the correct kubeconfig from java to CLI

Additional context

No response

[shawkins]

Linking to #3856 - to remove that listener we'll need to expose the ability to directly write not through piped streams. This would directly expose the read.

[hypnoce]

I'm not sure how the listener refactoring can help here. We can directly pass the necessary outputstream (File, Base64...) via the writingOutput method.
I quickly tried something (keeping the behavior that calling inputstream.close() closes the execwatch as well) : hypnoce@5d4cd2d
WDYT ?

[shawkins]

I'm not sure how the listener refactoring can help here.

Sorry too many layers. I was thinking about the pipe handling down there, but yes passing the desired output directly in from the PodOperationImpl will work. The existing PodOperationImpl with pipes would allow for a non-blocking copy wrt the calling thread - but of course it is just blocking so that's not needed.

Please open a pr. The only thing that may need a tweak is this future hypnoce@5d4cd2d#diff-a2903aa41f3a870d827a008517a92eda7a4abdb34d3c4660d48b08240145bcc7R537 - you can just do:

    cp.whenComplete((o, t) -> {
      if (cp.isCancelled()) {
        ...
      }
    });
    return cp;