Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move output endpoint from Che master container to a separate one #11459

Closed
garagatyi opened this issue Oct 3, 2018 · 4 comments
Closed

Move output endpoint from Che master container to a separate one #11459

garagatyi opened this issue Oct 3, 2018 · 4 comments
Labels
kind/task Internal things, technical debt, and to-do tasks to be performed. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@garagatyi
Copy link

Description

When we were designing Che 6 we added an endpoint for pushing different kinds of output from a workspace.
It was supposed that we can move this out of the Che master for the scalability reasons.
Now we see the significant load from logs which leads to OOMs of Che master.
So, we finally need to do this move.
We will need to add it for logs ONLY, not for the cases when we sent statuses.
We will need to add separation of statuses/logs to the plugin broker.
Broker: link
Installer: link, link, link, link

Reproduction Steps

OS and version:

Diagnostics:

@garagatyi garagatyi added kind/task Internal things, technical debt, and to-do tasks to be performed. team/platform labels Oct 3, 2018
@garagatyi
Copy link
Author

@skabashnyuk @sleshchenko FYI

@garagatyi garagatyi changed the title Move output consumer from Che master container to a separate one Move output endpoint from Che master container to a separate one Oct 3, 2018
@garagatyi
Copy link
Author

It looks like clients do not use links they were supposed to use to get logs from Che master. Instead of that, they construct websocket link themselves and use it. The link they construct happens to be correct for now just because of the behavior of JSON_RPC that a client can connect to any link and use any JSON_RPC method.
The correct way of getting logs is to get the link with name from workspace retrieved from Che master.
Here is how workspace-loader constructs a link instead. I believe that we have the same issue for factory accepting page (with the crane) and maybe some others which I can't recall at the moment.
I believe that we will need to separate logs brokering to a separate service we need to start work on fixing clients ASAP.
@ashumilova @evidolob @l0rd

@garagatyi
Copy link
Author

I experimented with it a bit. Here are the changes in Che plugin broker to use a separate endpoint for the output https://github.com/eclipse/che-plugin-broker/compare/logsEndpoint?expand=1
Here are basic changes in Che https://github.com/eclipse/che/compare/master...garagatyi:outputPropagator?expand=1. But we stream some logs directly from Che master here and here and here
We would need to either send those logs to a remote service from Che master or stop streaming them at all.
Here is a sample of such a service I prepared. But it doesn't start correctly for some reason and I haven't figured out why yet.

@che-bot
Copy link
Contributor

che-bot commented Sep 7, 2019

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

@che-bot che-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 7, 2019
@che-bot che-bot closed this as completed Sep 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/task Internal things, technical debt, and to-do tasks to be performed. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Development

No branches or pull requests

3 participants