RFE: Expose usage metrics via Che Server API

[aditya-konarde]

Description

As an engineer working on a maintaining a hosted Eclipse Che Server, I would like to have access to metrics on the the number of provisioned Che accounts on the Server. Currently, this can be obtained with queries to the backing database. The request here is to expose these metrics via a metrics endpoint, so that they can be consumed directly by a monitoring system.

The metrics I am looking at are:

• Number of provisioned Che accounts . [Metric] Number of users | [Metric] Number of users #12093
• Number of Che Workspaces Create REST method to get all workspaces with filter | Create REST method to get all workspaces with filter #12091
• Number of Che workspace per user OR the average number of workspaces per user Create REST method to get all workspaces with filter #12091

The developers might be able to add in more metrics which are needed for development/troubleshooting. Exposing these will definitely help with a faster feedback loop for the developers, and help us look at trends in the usage for better capacity planning.

CC @ibuziuk

Reproduction Steps

Not applicable

OS and version:
Not applicable

Diagnostics:
Not applicable

[l0rd]

@aditya-konarde I am afraid we don't have such a concept of a "super user" or "admin" in Che but @skabashnyuk may provide more details.

[skabashnyuk]

@aditya-konarde @l0rd we do have a concept of "super user" or "admin". We call it user with "manageSystem" permission.

Number of provisioned Che accounts

Small remark. If Che is connected to Keycloak or other OIDC provider it will mean a number of users who at least once used Che.

Number of Che Workspaces

ok

Number of Che workspace per user OR the average number of workspaces per user

We might need pagination here.

I'll try to make a prototype of json so we can make sure we are talking about the same thing.

[mshaposhnik]

So it might be like:
GET /api/metrics?metric_type=total_accounts
GET /api/metrics?metric_type=total_workspaces
GET /api/metrics?metric_type=average_workspaces
with the JSON output in form:

{
 "total_workspaces": "136",
 "running_workspaces: "62",
 "timestamp" : "2018-10-23 13:34:55"
}

[aditya-konarde]

@fche fyi^

[aditya-konarde]

Best if we use a prometheus-style format: https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-format-example

[skabashnyuk]

@aditya-konarde Interesting idea. I need to synchronise this approach with @l0rd @ibuziuk and @davidfestal . AFAIK they doing(going to do) something similar. See more redhat-developer/rh-che#950.

[ibuziuk]

Number of provisioned Che accounts
Number of Che Workspaces
Number of Che workspace per user OR the average number of workspaces per user

Those metrics are really interesting and nice to have, but they will not really help from maintaining perspective. IMO, the most important metric that we could currently get is list of workspaces that are running more than X minutes (with info about user / namespace). I believe it should be the first usage metric exposed via Che Server API

[skabashnyuk]

@ibuziuk @aditya-konarde FYI

• Create REST method to get all workspaces with filter | Create REST method to get all workspaces with filter #12091
• [Metric] Number of workspaces by state | [Metric] Number of workspaces by state #12092
• [Metric] Number of users | [Metric] Number of users #12093
• [Metric] Workspace related metrics | [Metric] Workspace related metrics #12096

[ibuziuk]

@skabashnyuk based on our SLA discussion I have created the following issues:

• [Metric] Number of failed workspace startup [Metric] Number of failed workspace startup #12242
• [Metric] API error rate (number of 500 responses from che server) [Metric] API error rate (number of 500 responses from che server) #12244

Please, consider those issues as a priority from the overall list of metrics.