Create a CI job that would check RhChe with dependency to nightly upstream

[garagatyi]

Issue problem:
RhChe depends on tags of upstream Che. We usually update the tag after each new upstream release. But when we try to move to a new tag we quite often see incompatibilities that weren't addressed in RhChe after changes in upstream. Often this happens after upstream Che is released and fixes can be applied only in a new bugfix release. We also find out new problems too late to be able rapidly deliver new features on prod.
We can improve this situation by creating a CI job that creates PR with current upstream nightly and runs it each day or two to check whether RhChe compiles against this version of the upstream and whether functional tests show no regression.

I'm not sure how our CI checks PRs right now but in case it merges PR branch to master on each run we can create a CI job that would just trigger regular PR check on a specific PR that we would maintain just for that purpose.
Another improvement we can do is to have a CI job for creating a PR:

• greps current RhChe version and removes branch uptream-check--SNAPSHOT, e.g. uptream-check-6.11.0-SNAPSHOT
• curl che pom from github and gets current nightly
• check whether there is a branch in RhChe with a name: uptream-check-, e.g. uptream-check-6.12.0-SNAPSHOT
• if there is no branch - create it
• change upstream dependency version
• if the local repo is dirty (job runs first time for this particular upstream version) commit/push it to the repo
• create PR if it is not created yet
• comment [test] in the PR (credentials should be whitelisted on regular PR CI)

This CI would maintain fresh PR for each upstream version so we can have a discussion regarding errors for a particular Che version in a PR and we would have a PR for each version.

Red Hat Che version:

version: (help/about menu)

• I can reproduce it on latest official image

Reproduction Steps:

Describe how to reproduce the problem

Runtime:

runtime used:

• minishift (include output of minishift version)
• OpenShift.io
• Openshift Container Platform (include output of oc version)
• k8s (include output of kubectl version)

[garagatyi]

@l0rd @ibuziuk @davidfestal @amisevsk @rhopp @Katka92 @ScrewTSW WDYT?

[l0rd]

@garagatyi such a job already exist https://ci.codenvycorp.com/view/All/job/rh-che-ci-master/

[garagatyi]

@l0rd AFAIK it just checks that RhChe compiles, but not that it passes RhChe specific deployment and functional tests. Whereas it is not critical but it would make us almost 100% sure that we can gracefully migrate to the new version without any issues.
@riuvshin can you add me as a receiver of notifications when job mentioned by Mario fails?

[l0rd]

After the build CI deploys it and creates a workspace. At least this is what it was doing originally. Anyway +1 but more than create a new job this issue should be about improving existing one right?

[garagatyi]

Yeah, we can improve the existing one. Right now it just builds RhChe but doesn't deploy it and doesn't run tests against it. Since we already have all the needed stuff to run tests on RhChe side it might be better to move the job here rather than changing the job on Codenvy CI side.
@rhopp @riuvshin WDYT?

[garagatyi]

We can also live with the build only and close this issue, but more we automate - more real work could be done.

[rhopp]

@garagatyi That should be possible. I've created issue for that: https://github.com/redhat-developer/che-functional-tests/issues/368 and we'll try to include it in next sprint.

[garagatyi]

@rhopp Thanks! Please, inform @riuvshin in case you implement this to put down Codenvy CI job and save some resources.

[garagatyi]

@rhopp from what I see centos job failed on December 5th when compiling RhChe against upstream 6.15-SNAPSHOT.
But before 5th it was succeeding but using upstream 6.14.2. I don't quite understand how it figures out which upstream version to use.
Can you elaborate?

[rhopp]

@garagatyi This has been broken since this commit and fixed by this commit.

The compatibility test failed to get upstream version, so it continued using current one. This is a bug in the script. It should fail in such case.

@Katka92 Could you please change the compatibility test to fail, if if fails to get upstream version?

[Katka92]

@rhopp Sure, I'll work on it today. Thank you for notifying me.

[Katka92]

Fix was pushed: #1136

[Katka92]

Hi guys, I'm starting to work on the enhanced version of this CI job.
What we have now:

• change dependency to latest upstream SNAPSHOT
• builds and push to quay
• pulls image to dev cluster
• runs tests against it

What I'm going to add:

• add branch for SNAPSHOT
• create PR for fixing regression related with compatibility (maybe create PR only if something fails?)
• send mail if some regression is found

[garagatyi]

@Katka92 Maybe I'm missing something but I don't remember receiving emails from the current job. But I still do receive emails from the old one on the codenvy side. Is it on purpose or not?

[garagatyi]

Oh, so dumb. Now I see that email is among improvements to be made.
Great plan! I think we can have PR even when the build is fine. In this case, we would be able to check something manually easily. It is important because there are always some nasty corner cases that can't be covered by tests.
Also, want to mention that we might have a failing build way later than this PR is created. Including second or third break of a build upstream even when we fixed the first issues. So, CI job should respect a situation when PR already exists and instead of creating a new branch and PR it should use an existing one (especially because it might have fixes committed by us).
We can actually avoid email if we create a PR. Users subscribed to Github notification would receive it in any case. And to make it more addressed we can add mentions of our Github users to the comment or description of the PR.

[Katka92]

@garagatyi Thank you for your opinion. As you said - in that case the email is probably not needed. On the other hand - we can add sending mails anytime if we find that we miss it.

[ibuziuk]

@ScrewTSW assigning to you since you are working on compatibility CI job

[ibuziuk]

@ScrewTSW is this done?

[ibuziuk]

@ScrewTSW this issue is in the Done column. Are you going to close the issue or some work is still expected?

[Katka92]

I think we can close it.