Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Misleading message from CHE server when a container stopped #14781

Closed
musienko-maxim opened this issue Oct 4, 2019 · 8 comments
Closed

Misleading message from CHE server when a container stopped #14781

musienko-maxim opened this issue Oct 4, 2019 · 8 comments
Assignees
@metlos
Labels
kind/enhancement A feature request - must adhere to the feature request template. severity/P1 Has a major impact to usage or development of the system. status/in-progress This issue has been taken by an engineer and is under active development.
Milestone
7.6.0

Comments

@musienko-maxim
Copy link
Contributor

musienko-maxim commented Oct 4, 2019 •
edited by sleshchenko
Loading

In some specific cases we can get a container that may be stopped unexpectedly just after start.
In such case, workspace pod is considered as non-ready and it does not service traffic.
So, after workspace start timeout (like 5-8 minutes) we got an exception Server 'theia' in container 'theia-ide3fl' not available but theia started just fine but there is an user-defined container that is completed.

We faced it this problem with a factory from quarkus workshop:
The quay.io/rhdevelopers/tutorial-tool image was stooped with exit 0 and we got unclear message from CHE server like: Server 'theia' in container 'theia-ide3fl' not available

Since quay.io/rhdevelopers/tutorial-tool is updated and does not fail anymore the following Devfile can be used to reproduce the case

apiVersion: 1.0.0
metadata:
  name: quarkus-workshop
projects:
  - name: quarkus-tutorial
    source:
      type: git
      location: 'https://github.com/redhat-developer-demos/quarkus-tutorial.git'
components:
  - id: redhat/vscode-yaml/latest
    type: chePlugin
  - id: redhat/vscode-xml/latest
    type: chePlugin
  - id: redhat/java8/latest
    type: chePlugin
    preferences:
      java.configuration.maven.userSettings: /opt/developer/.m2/settings.xml
  # Tool that allows to build java application including Quarkus
  - alias: tools
    type: kubernetes
    mountSources: true
    referenceContent: |
      apiVersion: v1
      kind: List
      items:
      - apiVersion: v1
        kind: Pod
        metadata:
          name: tutorial-tools
          labels:
            app: tutorial-tools
        spec:
          containers:
          - name: tutorial-tools
            image: quay.io/rhdevelopers/tutorial-tools:0.0.2
            imagePullPolicy: IfNotPresent
            env:
              - name: MAVEN_MIRROR_URL
                value: 'http://nexus.rhd-workshop-infra:8081/nexus/content/groups/public'
            workingDir:  /projects
            resources:
              limits: 
                memory: 2Gi
              requests:
                memory: 2Gi
    entrypoints:
       - parentName: tutorial-tools
         command: ['echo']
         args: ['hello']
    selector:
      app: tutorial-tools
@musienko-maxim musienko-maxim added the kind/enhancement A feature request - must adhere to the feature request template. label Oct 4, 2019
@che-bot che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Oct 4, 2019
@slemeur
Copy link
Contributor

slemeur commented Oct 4, 2019

I faced exactly the same issue with this devfile from @DonSchenck:

But the reason of the error was different. It was missing some parameter for the .Net 3 image.

metadata:
  name: dotnet-web
projects:
  - name: dotnet-web-simple
    source:
      location: 'https://github.com/che-samples/dotnet-web-simple'
      type: git
      branch: netcoreapp2.2
attributes:
  persistVolumes: 'false'
components:
  - id: redhat-developer/che-omnisharp-plugin/latest
    memoryLimit: 1024Mi
    type: chePlugin
    alias: omnisharp
  - id: redhat-developer/netcoredbg-theia-plugin/latest
    memoryLimit: 512Mi
    type: chePlugin
    alias: netcoredbg
  - mountSources: true
    endpoints:
      - name: 5000/tcp
        port: 5000
    memoryLimit: 512Mi
    type: dockerimage
    volumes:
      - name: dotnet
        containerPath: /home/user
    alias: dotnet
    image: 'mcr.microsoft.com/dotnet/core/sdk:3.0'
apiVersion: 1.0.0
commands:
  - name: build
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/dotnet-web-simple'
        type: exec
        command: dotnet build
        component: dotnet
  - name: Launch .NET Web Application on Debug mode
    actions:
      - referenceContent: |
          {
            "version": "0.2.0",
              "configurations": [
                {
                    "type": "netcoredbg",
                    "request": "launch",
                    "program": "/projects/dotnet-web-simple/bin/Debug/netcoreapp3.0/web.dll",
                    "args": [],
                    "name": ".NET Core Launch (console)",
                    "stopAtEntry": false,
                    "console": "internalConsole"
                }
            ]
          }
        type: vscode-launch
  - name: run
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/dotnet-web-simple'
        type: exec
        command: dotnet run
        component: dotnet
  - name: update dependencies
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/dotnet-web-simple'
        type: exec
        command: dotnet restore
        component: dotnet

@tolusha
Copy link
Contributor

tolusha commented Oct 7, 2019

@musienko-maxim
Unfortunately I have the error:

0/1 nodes are available: 1 Insufficient cpu.
Error: Failed to run the workspace: "Unrecoverable event occurred: 'FailedScheduling', '0/1 nodes are available: 1 Insufficient cpu.', 'workspaceo5uabgic7a5trg6w.tutorial-tools-97bd77dc4-zgfd7'"

@tolusha tolusha added severity/P1 Has a major impact to usage or development of the system. and removed status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. labels Oct 7, 2019
@sleshchenko
Copy link
Member

@tolusha Devfile in the PR description is updated to make easier to reproduce it.

@tolusha
Copy link
Contributor

tolusha commented Oct 7, 2019

@sleshchenko
and now workspace starts without errors ;)

@sleshchenko
Copy link
Member

@tolusha Thanks for trying one more time and notifying me. A small typo is fixed now and the workspace should not start anymore.

@l0rd l0rd added this to the Backlog - Platform milestone Oct 10, 2019
@nickboldt
Copy link
Contributor

Does this impact CRW 2 too? If so can we get this into the queue for 7.3.2/7.4?

@tsmaeder tsmaeder mentioned this issue Oct 31, 2019
Closed
@skabashnyuk skabashnyuk mentioned this issue Nov 1, 2019
Closed
12 tasks
@sleshchenko sleshchenko mentioned this issue Nov 1, 2019
Closed
@sleshchenko
Copy link
Member

Does this impact CRW 2 too?

Yes, it's actual for CRW as well.

@sleshchenko
Copy link
Member

sleshchenko commented Nov 4, 2019 •
edited by sparkoo
Loading

We can see that there are no corresponding events, so it can not be handled with unrecoverable events mechanism.

Events 
items:
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:49Z
  involvedObject:
    apiVersion: v1
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195082"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:49Z
  message: Successfully assigned che/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    to localhost
  metadata:
    creationTimestamp: 2019-11-01T02:01:49Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7750da58312
    namespace: che
    resourceVersion: "195089"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7750da58312
    uid: 9115fcf4-fc4b-11e9-9feb-080027d24256
  reason: Scheduled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: default-scheduler
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:52Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorcox}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:52Z
  message: pulling image "eclipse/che-theia-endpoint-runtime-binary:next"
  metadata:
    creationTimestamp: 2019-11-01T02:01:52Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e775902d7b29
    namespace: che
    resourceVersion: "195102"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e775902d7b29
    uid: 92636f7b-fc4b-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:53Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorcox}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:53Z
  message: Successfully pulled image "eclipse/che-theia-endpoint-runtime-binary:next"
  metadata:
    creationTimestamp: 2019-11-01T02:01:53Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e775fecd2453
    namespace: che
    resourceVersion: "195111"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e775fecd2453
    uid: 937f54cf-fc4b-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:53Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorcox}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:53Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T02:01:53Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7760482791f
    namespace: che
    resourceVersion: "195112"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7760482791f
    uid: 938d3ab0-fc4b-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:54Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorcox}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:54Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T02:01:54Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7761a1a572a
    namespace: che
    resourceVersion: "195113"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7761a1a572a
    uid: 93c4459a-fc4b-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:56Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{tutorial-tools}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:56Z
  message: Container image "quay.io/rhdevelopers/tutorial-tools:0.0.2" already present
    on machine
  metadata:
    creationTimestamp: 2019-11-01T02:01:56Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e77695c40a88
    namespace: che
    resourceVersion: "195126"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e77695c40a88
    uid: 95014c18-fc4b-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:56Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{tutorial-tools}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:56Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T02:01:56Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e776ae89563f
    namespace: che
    resourceVersion: "195128"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e776ae89563f
    uid: 95406bf1-fc4b-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:57Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{tutorial-tools}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:57Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T02:01:57Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e776be4a42c7
    namespace: che
    resourceVersion: "195131"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e776be4a42c7
    uid: 9568aaa9-fc4b-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:57Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execui8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:57Z
  message: pulling image "docker.io/eclipse/che-machine-exec:next"
  metadata:
    creationTimestamp: 2019-11-01T02:01:57Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e776bedecf4e
    namespace: che
    resourceVersion: "195132"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e776bedecf4e
    uid: 956b55aa-fc4b-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:59Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execui8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:59Z
  message: Successfully pulled image "docker.io/eclipse/che-machine-exec:next"
  metadata:
    creationTimestamp: 2019-11-01T02:01:59Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777327684f7
    namespace: che
    resourceVersion: "195139"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777327684f7
    uid: 96958d8b-fc4b-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:59Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execui8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:59Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T02:01:59Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7774422d8c7
    namespace: che
    resourceVersion: "195145"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7774422d8c7
    uid: 96bf4bfa-fc4b-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:59Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execui8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:59Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T02:01:59Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777501fb75b
    namespace: che
    resourceVersion: "195147"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777501fb75b
    uid: 96ddeaf3-fc4b-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:59Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-ideyk8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:59Z
  message: pulling image "docker.io/eclipse/che-theia:next"
  metadata:
    creationTimestamp: 2019-11-01T02:01:59Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e77750afac94
    namespace: che
    resourceVersion: "195148"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e77750afac94
    uid: 96e09fa2-fc4b-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:01Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-ideyk8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:01Z
  message: Successfully pulled image "docker.io/eclipse/che-theia:next"
  metadata:
    creationTimestamp: 2019-11-01T02:02:01Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777c3b72382
    namespace: che
    resourceVersion: "195155"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777c3b72382
    uid: 9806ee2e-fc4b-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:01Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-ideyk8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:01Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T02:02:01Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777e16af4fc
    namespace: che
    resourceVersion: "195159"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777e16af4fc
    uid: 9851d4c6-fc4b-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:02Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-ideyk8}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:02Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T02:02:02Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777f07ba157
    namespace: che
    resourceVersion: "195161"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777f07ba157
    uid: 987879bf-fc4b-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:02Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yamlnlp}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:02Z
  message: pulling image "docker.io/eclipse/che-remote-plugin-node:next"
  metadata:
    creationTimestamp: 2019-11-01T02:02:02Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777f0cd6093
    namespace: che
    resourceVersion: "195162"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e777f0cd6093
    uid: 98796758-fc4b-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:04Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yamlnlp}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:04Z
  message: Successfully pulled image "docker.io/eclipse/che-remote-plugin-node:next"
  metadata:
    creationTimestamp: 2019-11-01T02:02:04Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7785f0a93ce
    namespace: che
    resourceVersion: "195169"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7785f0a93ce
    uid: 99a0eb11-fc4b-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:04Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yamlnlp}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:04Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T02:02:04Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7789036c197
    namespace: che
    resourceVersion: "195172"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7789036c197
    uid: 9a135c79-fc4b-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:05Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yamlnlp}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:05Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T02:02:05Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e778a64e105d
    namespace: che
    resourceVersion: "195175"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e778a64e105d
    uid: 9a61e176-fc4b-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:05Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmlos2}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:05Z
  message: pulling image "docker.io/eclipse/che-remote-plugin-runner-java11:next"
  metadata:
    creationTimestamp: 2019-11-01T02:02:05Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e778a6664093
    namespace: che
    resourceVersion: "195176"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e778a6664093
    uid: 9a75f492-fc4b-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:07Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmlos2}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:07Z
  message: Successfully pulled image "docker.io/eclipse/che-remote-plugin-runner-java11:next"
  metadata:
    creationTimestamp: 2019-11-01T02:02:07Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7791d12614c
    namespace: che
    resourceVersion: "195183"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7791d12614c
    uid: 9b876f59-fc4b-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:08Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmlos2}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:08Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T02:02:08Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e779563b78b7
    namespace: che
    resourceVersion: "195187"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e779563b78b7
    uid: 9c2e62fb-fc4b-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:08Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmlos2}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:08Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T02:02:09Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7797d5e4f18
    namespace: che
    resourceVersion: "195190"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7797d5e4f18
    uid: 9c9973c9-fc4b-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:08Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-javas8r}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:08Z
  message: pulling image "docker.io/eclipse/che-remote-plugin-runner-java8:next"
  metadata:
    creationTimestamp: 2019-11-01T02:02:09Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7798026a7bc
    namespace: che
    resourceVersion: "195191"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e7798026a7bc
    uid: 9ca85fda-fc4b-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:02:10Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-javas8r}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj
    namespace: che
    resourceVersion: "195087"
    uid: 90a0ba5a-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:02:10Z
  message: Successfully pulled image "docker.io/eclipse/che-remote-plugin-runner-java8:next"
  metadata:
    creationTimestamp: 2019-11-01T02:02:10Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e779f103fcfa
    namespace: che
    resourceVersion: "195202"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj.15d2e779f103fcfa
    uid: 9d98be2d-fc4b-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:49Z
  involvedObject:
    apiVersion: apps/v1
    kind: ReplicaSet
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d
    namespace: che
    resourceVersion: "195073"
    uid: 90796969-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:49Z
  message: 'Created pod: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj'
  metadata:
    creationTimestamp: 2019-11-01T02:01:49Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d.15d2e774f738197b
    namespace: che
    resourceVersion: "195085"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d.15d2e774f738197b
    uid: 90dff3b8-fc4b-11e9-9feb-080027d24256
  reason: SuccessfulCreate
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: replicaset-controller
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:41Z
  involvedObject:
    apiVersion: v1
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192757"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:41Z
  message: Successfully assigned che/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    to localhost
  metadata:
    creationTimestamp: 2019-11-01T01:52:41Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f5674887cf
    namespace: che
    resourceVersion: "192762"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f5674887cf
    uid: 4a4e7ea9-fc4a-11e9-9feb-080027d24256
  reason: Scheduled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: default-scheduler
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:45Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorxx0}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:45Z
  message: pulling image "eclipse/che-theia-endpoint-runtime-binary:next"
  metadata:
    creationTimestamp: 2019-11-01T01:52:45Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f64c7af73b
    namespace: che
    resourceVersion: "192776"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f64c7af73b
    uid: 4ca41f54-fc4a-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:47Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorxx0}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:47Z
  message: Successfully pulled image "eclipse/che-theia-endpoint-runtime-binary:next"
  metadata:
    creationTimestamp: 2019-11-01T01:52:47Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f6d9c4dfb2
    namespace: che
    resourceVersion: "192784"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f6d9c4dfb2
    uid: 4e0b7832-fc4a-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:48Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorxx0}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:48Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T01:52:48Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f6ffc71c60
    namespace: che
    resourceVersion: "192786"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f6ffc71c60
    uid: 4e8f428f-fc4a-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:49Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.initContainers{remote-runtime-injectorxx0}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:49Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T01:52:49Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f73be0bf6d
    namespace: che
    resourceVersion: "192788"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f73be0bf6d
    uid: 4f2955b9-fc4a-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:53Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{tutorial-tools}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:53Z
  message: Container image "quay.io/rhdevelopers/tutorial-tools:0.0.2" already present
    on machine
  metadata:
    creationTimestamp: 2019-11-01T01:52:53Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f8471fc942
    namespace: che
    resourceVersion: "192810"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f8471fc942
    uid: 51a88569-fc4a-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:54Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{tutorial-tools}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:54Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T01:52:54Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f87a4abb4a
    namespace: che
    resourceVersion: "192816"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f87a4abb4a
    uid: 522babf6-fc4a-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:55Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{tutorial-tools}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:55Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T01:52:55Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f8947b57b5
    namespace: che
    resourceVersion: "192820"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f8947b57b5
    uid: 526e4aed-fc4a-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:55Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execkhe}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:55Z
  message: pulling image "docker.io/eclipse/che-machine-exec:next"
  metadata:
    creationTimestamp: 2019-11-01T01:52:55Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f895480724
    namespace: che
    resourceVersion: "192821"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f895480724
    uid: 527147b3-fc4a-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:57Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execkhe}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:57Z
  message: Successfully pulled image "docker.io/eclipse/che-machine-exec:next"
  metadata:
    creationTimestamp: 2019-11-01T01:52:57Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f906e7aafb
    namespace: che
    resourceVersion: "192829"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f906e7aafb
    uid: 53931bf4-fc4a-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:57Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execkhe}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:57Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T01:52:57Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f9163a805b
    namespace: che
    resourceVersion: "192831"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f9163a805b
    uid: 53ba8cb2-fc4a-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:57Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{che-machine-execkhe}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:57Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T01:52:57Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f929646cf7
    namespace: che
    resourceVersion: "192833"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f929646cf7
    uid: 53ebd0df-fc4a-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:57Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-idey10}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:57Z
  message: pulling image "docker.io/eclipse/che-theia:next"
  metadata:
    creationTimestamp: 2019-11-01T01:52:57Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f92a056c2d
    namespace: che
    resourceVersion: "192834"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f92a056c2d
    uid: 53ed3628-fc4a-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:59Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-idey10}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:59Z
  message: Successfully pulled image "docker.io/eclipse/che-theia:next"
  metadata:
    creationTimestamp: 2019-11-01T01:52:59Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f99d1291da
    namespace: che
    resourceVersion: "192843"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f99d1291da
    uid: 5513abc6-fc4a-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:00Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-idey10}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:00Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T01:53:00Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f9c95340e9
    namespace: che
    resourceVersion: "192848"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f9c95340e9
    uid: 558597ef-fc4a-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:01Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{theia-idey10}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:01Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T01:53:01Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f9f7b153f4
    namespace: che
    resourceVersion: "192853"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6f9f7b153f4
    uid: 560a2998-fc4a-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:01Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yaml0aj}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:01Z
  message: pulling image "docker.io/eclipse/che-remote-plugin-node:next"
  metadata:
    creationTimestamp: 2019-11-01T01:53:01Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fa0bf9e169
    namespace: che
    resourceVersion: "192854"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fa0bf9e169
    uid: 562ff737-fc4a-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:04Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yaml0aj}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:04Z
  message: Successfully pulled image "docker.io/eclipse/che-remote-plugin-node:next"
  metadata:
    creationTimestamp: 2019-11-01T01:53:04Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fab646840c
    namespace: che
    resourceVersion: "192866"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fab646840c
    uid: 57e3887d-fc4a-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:05Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yaml0aj}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:05Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T01:53:05Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fb0dd2204d
    namespace: che
    resourceVersion: "192872"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fb0dd2204d
    uid: 58c38990-fc4a-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:07Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-yaml0aj}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:07Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T01:53:07Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fb7793de7f
    namespace: che
    resourceVersion: "192881"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fb7793de7f
    uid: 59da833f-fc4a-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:07Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmllre}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:07Z
  message: pulling image "docker.io/eclipse/che-remote-plugin-runner-java11:next"
  metadata:
    creationTimestamp: 2019-11-01T01:53:07Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fb77de9630
    namespace: che
    resourceVersion: "192882"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fb77de9630
    uid: 59e76705-fc4a-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:09Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmllre}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:09Z
  message: Successfully pulled image "docker.io/eclipse/che-remote-plugin-runner-java11:next"
  metadata:
    creationTimestamp: 2019-11-01T01:53:09Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fbe9f5ec7e
    namespace: che
    resourceVersion: "192890"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fbe9f5ec7e
    uid: 5af9d27a-fc4a-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:10Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmllre}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:10Z
  message: Created container
  metadata:
    creationTimestamp: 2019-11-01T01:53:10Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fc1bea1ea1
    namespace: che
    resourceVersion: "192896"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fc1bea1ea1
    uid: 5b7727b9-fc4a-11e9-9feb-080027d24256
  reason: Created
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:12Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-xmllre}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:12Z
  message: Started container
  metadata:
    creationTimestamp: 2019-11-01T01:53:12Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fc8978542e
    namespace: che
    resourceVersion: "192901"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fc8978542e
    uid: 5cea4473-fc4a-11e9-9feb-080027d24256
  reason: Started
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:12Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-java2cf}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:12Z
  message: pulling image "docker.io/eclipse/che-remote-plugin-runner-java8:next"
  metadata:
    creationTimestamp: 2019-11-01T01:53:13Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fc8dec9c1c
    namespace: che
    resourceVersion: "192902"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fc8dec9c1c
    uid: 5d2ea752-fc4a-11e9-9feb-080027d24256
  reason: Pulling
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:14Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{vscode-java2cf}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:53:14Z
  message: Successfully pulled image "docker.io/eclipse/che-remote-plugin-runner-java8:next"
  metadata:
    creationTimestamp: 2019-11-01T01:53:15Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fd048e8507
    namespace: che
    resourceVersion: "192910"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6fd048e8507
    uid: 5e4f220a-fc4a-11e9-9feb-080027d24256
  reason: Pulled
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Normal
- apiVersion: v1
  count: 22
  eventTime: null
  firstTimestamp: 2019-11-01T01:53:23Z
  involvedObject:
    apiVersion: v1
    fieldPath: spec.containers{tutorial-tools}
    kind: Pod
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst
    namespace: che
    resourceVersion: "192759"
    uid: 4a4151c5-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:57:55Z
  message: Back-off restarting failed container
  metadata:
    creationTimestamp: 2019-11-01T01:57:55Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6ff424271d2
    namespace: che
    resourceVersion: "194055"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst.15d2e6ff424271d2
    uid: 05a3014f-fc4b-11e9-9feb-080027d24256
  reason: BackOff
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: kubelet
    host: localhost
  type: Warning
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:41Z
  involvedObject:
    apiVersion: apps/v1
    kind: ReplicaSet
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b
    namespace: che
    resourceVersion: "192753"
    uid: 4a3c2291-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:41Z
  message: 'Created pod: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst'
  metadata:
    creationTimestamp: 2019-11-01T01:52:41Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b.15d2e6f566112705
    namespace: che
    resourceVersion: "192761"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b.15d2e6f566112705
    uid: 4a4b27f1-fc4a-11e9-9feb-080027d24256
  reason: SuccessfulCreate
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: replicaset-controller
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:26Z
  involvedObject:
    apiVersion: apps/v1
    kind: ReplicaSet
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b
    namespace: che
    resourceVersion: "194920"
    uid: 4a3c2291-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:26Z
  message: 'Deleted pod: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b-9hgst'
  metadata:
    creationTimestamp: 2019-11-01T02:01:26Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b.15d2e76f84ee592e
    namespace: che
    resourceVersion: "194927"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b.15d2e76f84ee592e
    uid: 82eac267-fc4b-11e9-9feb-080027d24256
  reason: SuccessfulDelete
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: replicaset-controller
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T01:52:41Z
  involvedObject:
    apiVersion: apps/v1
    kind: Deployment
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools
    namespace: che
    resourceVersion: "192752"
    uid: 4a396cba-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T01:52:41Z
  message: Scaled up replica set workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b
    to 1
  metadata:
    creationTimestamp: 2019-11-01T01:52:41Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools.15d2e6f561951ead
    namespace: che
    resourceVersion: "192754"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools.15d2e6f561951ead
    uid: 4a3e0e2f-fc4a-11e9-9feb-080027d24256
  reason: ScalingReplicaSet
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: deployment-controller
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:25Z
  involvedObject:
    apiVersion: apps/v1
    kind: Deployment
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools
    namespace: che
    resourceVersion: "194919"
    uid: 4a396cba-fc4a-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:25Z
  message: Scaled down replica set workspacemqq71m0z7nwrrp1n.tutorial-tools-b574cc58b
    to 0
  metadata:
    creationTimestamp: 2019-11-01T02:01:25Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools.15d2e76f7cdf4ded
    namespace: che
    resourceVersion: "194922"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools.15d2e76f7cdf4ded
    uid: 82d7942b-fc4b-11e9-9feb-080027d24256
  reason: ScalingReplicaSet
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: deployment-controller
  type: Normal
- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: 2019-11-01T02:01:48Z
  involvedObject:
    apiVersion: apps/v1
    kind: Deployment
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools
    namespace: che
    resourceVersion: "195070"
    uid: 9065f900-fc4b-11e9-9feb-080027d24256
  kind: Event
  lastTimestamp: 2019-11-01T02:01:48Z
  message: Scaled up replica set workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d
    to 1
  metadata:
    creationTimestamp: 2019-11-01T02:01:48Z
    name: workspacemqq71m0z7nwrrp1n.tutorial-tools.15d2e774d9276af9
    namespace: che
    resourceVersion: "195076"
    selfLink: /api/v1/namespaces/che/events/workspacemqq71m0z7nwrrp1n.tutorial-tools.15d2e774d9276af9
    uid: 9093d0b5-fc4b-11e9-9feb-080027d24256
  reason: ScalingReplicaSet
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: deployment-controller
  type: Normal
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

But Pod yaml contains the corresponding information we could use:

...
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2019-11-01T02:01:55Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2019-11-01T02:01:49Z'
      message: 'containers with unready status: [tutorial-tools]'
      reason: ContainersNotReady
      status: 'False'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: null
      message: 'containers with unready status: [tutorial-tools]'
      reason: ContainersNotReady
      status: 'False'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2019-11-01T02:01:49Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        docker://a72d32481f58cf4a4f9de5d8acc39920872e26af63749d56b805a43bde39c96d
      image: 'quay.io/rhdevelopers/tutorial-tools:0.0.2'
      imageID: >-
        docker-pullable://quay.io/rhdevelopers/tutorial-tools@sha256:632b26ded49b4e0654d57f4ab0fc2252941e8d5828fb290ff5e459fa842e3eef
      lastState:
        terminated:
          containerID: >-
            docker://a72d32481f58cf4a4f9de5d8acc39920872e26af63749d56b805a43bde39c96d
          exitCode: 0
          finishedAt: '2019-11-01T02:03:40Z'
          reason: Completed
          startedAt: '2019-11-01T02:03:40Z'
      name: tutorial-tools
      ready: false
      restartCount: 4
      state:
        waiting:
          message: >-
            Back-off 1m20s restarting failed container=tutorial-tools
            pod=workspacemqq71m0z7nwrrp1n.tutorial-tools-8546cdd68d-b8gmj_che(90a0ba5a-fc4b-11e9-9feb-080027d24256)
          reason: CrashLoopBackOff
    ...
  phase: Running
  podIP: 172.17.0.12
  qosClass: Burstable
  startTime: '2019-11-01T02:01:49Z'

So, we should watch workspaces pods and make sure every container is ready before proceeding to the servers check phase.
Here is a list of all possible conditions types https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions.
But instead of just waiting for some conditions - we could early fail when there are containers statuses with restart count more than 3...

@skabashnyuk skabashnyuk mentioned this issue Nov 27, 2019
Closed
9 tasks
@skabashnyuk skabashnyuk modified the milestones: Backlog - Platform, 7.6.0 Nov 27, 2019
@l0rd l0rd mentioned this issue Nov 28, 2019
Open
@metlos metlos added the status/in-progress This issue has been taken by an engineer and is under active development. label Dec 2, 2019
@metlos metlos self-assigned this Dec 2, 2019
metlos added a commit to metlos/che that referenced this issue Dec 10, 2019
@metlos
Fixes eclipse-che#14781 - Add additional diagnostic for failure when …
929c7a1 
…K8s pronounces

a pod running. It still might have failed if one of the containers has
terminated.

Signed-off-by: Lukas Krejci <lkrejci@redhat.com>
@metlos metlos mentioned this issue Dec 10, 2019
Merged
@metlos metlos closed this as completed in 01d03b2 Dec 13, 2019
@skabashnyuk skabashnyuk changed the title Misleading message form CHE server when a container stopped Misleading message from CHE server when a container stopped Dec 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@metlos metlos

Labels
kind/enhancement A feature request - must adhere to the feature request template. severity/P1 Has a major impact to usage or development of the system. status/in-progress This issue has been taken by an engineer and is under active development.
Projects
None yet
Milestone
7.6.0
Development

No branches or pull requests
9 participants
@metlos @nickboldt @l0rd @skabashnyuk @slemeur @musienko-maxim @tolusha @sleshchenko @che-bot