[Workspace API] Cannot create/delete file in Theia if another websocket uses 'filesystem' service

[gorshkov-leonid]

I integrate my web application with Eclipse Che to get/create/update/delete content through websocket connection. I use "/services" websocket and "filesystem" service in the way as Eclipse Che does it. This way worked in Eclipse Che 7.7.0, but does not work properly in 7.15.0

When my socket is connected then I cannot create and delete files in Eclipse Che. Although a read and update work. A new socket somehow affects the functionality of Eclipse Che.

I thought that problem is in Eclipse Che(see eclipse-che/che#17443), but found out that it is reproduced on Theia.

I wrote test-client to demonstrate the problem:

test.zip

import com.google.common.util.concurrent.Uninterruptibles;
import io.vertx.core.Vertx;
import io.vertx.core.http.*;

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicReference;

public class Main {
    public static void main(String[] args) throws IOException {

        HttpClientOptions options = new HttpClientOptions();
        HttpClient client = Vertx.factory.vertx().createHttpClient(options);

        final Semaphore semaphore = new Semaphore(1);

        aquire(semaphore);

        AtomicReference<WebSocket> wsr = new AtomicReference<>();

        client.webSocket(3000, "localhost", "/services", resp -> {
            if (resp.cause() != null) {
                resp.cause().printStackTrace();
                System.exit(-1);
            }

            WebSocket websocket = resp.result();
            wsr.set(websocket);
            websocket
                    .handler(data -> System.out.println("Message: " + data.toString(StandardCharsets.UTF_8)))
                    .closeHandler(aVoid -> System.out.println("Websocket closed normally."))
                    .endHandler(aVoid -> System.out.println("Websocket ended normally."))
                    .exceptionHandler(e -> {
                        e.printStackTrace();
                        System.exit(-1);
                    });

            semaphore.release();
        });
        Executors.newFixedThreadPool(2).execute(() -> {
            aquire(semaphore);
            sendMsgWait(wsr.get(), "{\"kind\":\"open\",\"id\":2,\"path\":\"/services/filesystem\"}");
        });
        System.in.read();
    }

    private static void sendMsgWait(WebSocket webSocket, String msg) {
        try {
            webSocket.writeTextMessage(msg);
            Thread.sleep(3000);
        } catch (InterruptedException e) {
            System.out.println("Can't sleep");
            System.exit(-1);
        }
    }

    private static void aquire(Semaphore semaphore) {
        if (!Uninterruptibles.tryAcquireUninterruptibly(semaphore, 15, TimeUnit.SECONDS)) {
            System.out.println("Can't acquire semaphore");
            System.exit(-1);
        }
    }
}

Theia version

• Theia 1.3.0

OS

• Windows 10
• Linux Alpine 3.10

Steps to reproduce

1. Run theia
   • docker run -it -p 3000:3000 -v "<path>:/home/project:cached" theiaide/theia
2. Build test-application
   • mvn clean install
3. Run test-application
   • java -jar ./target/test-0.0.1-SNAPSHOT-jar-with-dependencies.jar
   • logs:

   Message: {"kind":"ready","id":2}
   

4. Try to create/delete a file in the workspace
   • AR: File is not created/deleted
   • logs:

   Message: {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"willCreate\",\"params\":\"/home/theia/.theia/logs/20200720T144411/host\"}"}
   

5. Stop application(CTRL + C).
6. Try to create/delete a file in the workspace
   • AR: File is created/deleted normally

---

Maybe this issue related to

• plugin: add workspace file api #7718

[vince-fugnitto]

@gorshkov-leonid I'm not quite sure I understand the use-case or the goal, why do you want to interface to internal services from your web-application? I don't think the internals are meant to be interfaced in such a way, these APIs are meant for Theia extensions. Can you perhaps clarify the overall goal of what you are trying to achieve?

[gorshkov-leonid]

@vince-fugnitto sure, some details are needed
I am writing a web application with its own UI (not in javascript) that uses other microservices. Via this application user eventually generates some source code and I want this code to be editable in IDE. That's why I separated heavy logic apart from Theia and expected that I would save/edit it via some API. That's why integration API is needed.

[paul-marechal]

Ok, so first things first I don't think we really support third parties accessing the websocket endpoints, and if someone does he needs to intimately understand and follow the current state of it.

Now regarding your issue: the patch you linked is the culprit indeed. What it does is that it waits for listeners of the willCreate event. When building an app with VS Code API support, such a handler is always hooked, and it tries to send an RPC message to what should be the extension host to give it time to process said event. In your case, you can see the request come (Message: {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"willCreate\",\"params\":\"/home/theia/.theia/logs/20200720T144411/host\"}"}) but you are not responding, leaving the system hanging.

(See relevant line)

[gorshkov-leonid]

@marechal-p thank you for explanation. But I mean there are two clients do not work:

1. Eclipse Che connected first
2. test-app connected second

Che works unless test-app connected. And what I cannot I understand, that why when I create file via Eclipse Che, that confirmation is got by test-app? It seems problem is common.

But I tried to answer with id received in willCreate (I expected 97 but received 0)

--> {"kind":"open","id":2,"path":"/services/filesystem"}
<-- Message: {"kind":"ready","id":2}
 
--> {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":97,\"method\":\"createFile\",\"params\":\"file:///home/project/55.txt\"}"}
<-- Message: {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"willCreate\",\"params\":\"file:///home/project/55.txt\"}"}
 
--> {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":0,\"result\":null}"}

Also, I decided that it could be error and answered with id 97.

--> {"kind":"open","id":2,"path":"/services/filesystem"}
<-- Message: {"kind":"ready","id":2}
 
--> {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":97,\"method\":\"createFile\",\"params\":\"file:///home/project/55.txt\"}"}
<-- Message: {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"willCreate\",\"params\":\"file:///home/project/55.txt\"}"}
 
--> {"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":97,\"result\":null}"}

This all did not help me

[gorshkov-leonid]

And I noticed, that my client got

{"kind":"data","id":2,"content":"{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"willCreate\",\"params\":\"file:///home/project/55.txt\"}"}

But Che got

{"kind":"data","id":3,"content":"{\"jsonrpc\":\"2.0\",\"id\":8,\"method\":\"willCreate\",\"params\":\"file:///home/project/55.txt\"}"}

That is in the case when creation was called only in Che.

Why Eclipse Che got 8 the 9 then 10, but my test-app gots always 0? Neither 0 nor 8 do not work.

[paul-marechal]

IIUC the first id is for a given ws logical channel (roughly one per service) and the json-rpc id is per request (it increments every time a message is sent). I just tried opening two Theia tabs (emulating two apps hooked on a ws each) and adding files still worked. I'd recommend you to keep looking for differences between the way you handle messaging and how Theia does.

[paul-marechal]

After looking a bit you might be blocking on this:

theia/packages/filesystem/src/node/node-filesystem.ts

Line 267 in 5de8eee

await this.client.willCreate(uri);

[gorshkov-leonid]

I can make a conclusion about how It works. If I create file in Eclipse Che that all clients will receive request willCreate. In My case Eclipse Che where I called creation received a request that looks normally (with iterated id) but test-app always receives a confirmation message with id=0. That looks like that all operation create must be accepted by all clients. It is strange.

I think if file is created in one client that there no need to send confirmation by other clients.

[paul-marechal]

but test-app always receives a confirmation message with id=0

This is normal because you only send one message, so it doesn't need to increment anything. Che/Theia makes a bunch of requests, hence why you see the number increasing.

I quickly tried to do what you are trying, and it must be something you are doing wrong when responding, because it works with the following NodeJS snippet (disclaimer: it's ugly)

const WebSocket = require('ws')

const ws = new WebSocket('ws://localhost:3000/services')

ws.on('open', () => {
    console.log('open')
    send(ws, JSON.stringify({
        id: 0,
        kind: 'open',
        path: '/services/filesystem',
    }))
})

ws.on('message', recv => {
    const parsed = JSON.parse(recv)
    console.log(`received: ${recv}`)
    switch (parsed.kind) {
        case 'ready': {
            console.log(`channel #${parsed.id} is ready.`)
            send(ws, JSON.stringify({
                id: 0,
                kind: 'data',
                content: JSON.stringify({
                    id: 0,
                    jsonrpc: '2.0',
                    method: 'createFile',
                    params: 'file:///d%3A/sources/node-pty/src/Untitled.txt'
                })
            }))
        } break
        case 'data': {
            const sub = JSON.parse(parsed.content)
            if ('method' in sub) {
                if (sub.method === 'willCreate') {
                    send(ws, JSON.stringify({
                        id: 0,
                        kind: 'data',
                        content: JSON.stringify({
                            id: sub.id,
                            jsonrpc: '2.0',
                            result: null,
                        })
                    }))
                }
            }
        } break
    }
})

function send(ws, message) {
    console.log(`sent: ${message}`)
    ws.send(message)
}

/** Execution logs:
open
sent: {"id":0,"kind":"open","path":"/services/filesystem"}
received: {"kind":"ready","id":0}
channel #0 is ready.
sent: {"id":0,"kind":"data","content":"{\"id\":0,\"jsonrpc\":\"2.0\",\"method\":\"createFile\",\"params\":\"file:///d%3A/sources/node-pty/src/Untitled.txt\"}"}
received: {"kind":"data","id":0,"content":"{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"willCreate\",\"params\":\"file:///d%3A/sources/node-pty/src/Untitled.txt\"}"}
sent: {"id":0,"kind":"data","content":"{\"id\":0,\"jsonrpc\":\"2.0\",\"result\":null}"}
received: {"kind":"data","id":0,"content":"{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"didCreate\",\"params\":[\"file:///d%3A/sources/node-pty/src/Untitled.txt\",false]}"}
*/

When I don't respond to the incoming willCreate request, nothing happens since the backend waits for its frontends to finish processing the event.

I think if file is created in one client that there no need to send confirmation by other clients.

cc @akosyakov, I don't have an opinion here.

[gorshkov-leonid]

@marechal-p Thank you for example ⭐ I made some wrong conclusions because of invalid requests (mixed up a path to project). Id is incremented as expected. Eclipse Che always accepts willCreate request. All would be fixed if test-app accepted this request.

As a result, as you say, all work 👌 : But it is not intuitively clear that creation and deletion should be approved by all clients. I have no questions more - It is an internal design and I will use it at our own risk.

[akosyakov]

I think if file is created in one client that there no need to send confirmation by other clients.

it's fixed in #7908, but there is no FileSystem json-rpc endpoint anymore