Issue with devon-ide installation

[vapadwal]

There is an issue with devon-ide when we tried to run setup.bat.

Deveon-ide is not able to download the software from India CG network.

we get the below error while running setup,bat

_C:\Users\vapadwal>cd D:\projects\myproject-ide

C:\Users\vapadwal>D:

D:\projects\myproject-ide>devon

*** ATTENTION ***
Your devon-ide is missing the software at D:\projects\myproject-ide\software
Please run the following command to complete your IDE setup:
devon ide setup [<settings-url>]
devon-ide environment variables have been set for D:\projects\myproject-ide

D:\projects\myproject-ide>devon ide setup
Found git repository /d/projects/myproject-ide/settings - updating via git
Can not update local git repository: /d/projects/myproject-ide/settings

*** Setting up java ***
Starting installation from https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_windows-x64_bin.zip to /d/projects/myproject-ide/software/java
Downloading openjdk-11.0.2_windows-x64_bin.zip from https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_windows-x64_bin.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

******** ATTENTION ********
ERROR: Failed to download openjdk-11.0.2_windows-x64_bin.zip from https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_windows-x64_bin.zip
We are sorry for the inconvenience. Please check the above errors, resolve them and try again.
Exit code was 60
ERROR: command 'java setup' failed with exit code 60

*** ATTENTION ***
Your devon-ide is missing the software at D:\projects\myproject-ide\software
Please run the following command to complete your IDE setup:
devon ide setup [<settings-url>]
devon-ide environment variables have been set for D:\projects\myproject-ide_

But when change to other network(tried with home network) than India CG network, its able to install everything and setup it.

This is the same problem we had faced while trying it with oomph

[markusschuh]

I see several options here:

1. Curl could use the "--insecure" flag for download. That is a no-go in my point of view, since it would allow any control on downloads - without a chance for anybody to detect, if something unexpectedly has happened during download. (risk of malware)

2. .In company environments any access to internet could be relevant for overall company security - so sometimes special requirements may need to be fulfilled, before the internet access is possible with a specific tool. Your special network situation and the exact doing, how to enable this setup access, will need to be discussed company internally. In difference to alternative "0" this will only allow a specific additional trusted CA,
   Any related reconfiguration is nothing that should be part of the open source devon-ide.

3. The more generic part is, that if curl is telling here, what the potential root cause might be and how it can be fixed. So if the outgoing https access to internet is intercepted by a security control and downloads are checked for malware, this will replace the issuer of the original internet certificates. curl can be reconfigured to trust this new issuer also, I doubt also, such a reconfiguration should be handled within devon-ide itself. It should be configured on the host, where the setup runs. devon-ide might give some hints on that inside own documentation - but of course such documentation already exist inside the public available curl documentation. I'll give my own summary at the bottom of this comment,

4. If the download were completely impossible (not "only" failing due to certificate error, but blocked in any case, this were an area, where devon-ide should provide a fallback method. And it already does, see Ability to configure software downloads via settings config #98, which allows to prefill the software repository from an offline archive. This just urgently needs more testing and maybe some more documentation.

How to enable another - non-official CA - as additional trusted CA for curl:

The exact steps may vary between operating system and also there are different approaches, e.g. depending if the user has the right to modify curl installation directory or not. I shortly describe a specific case here, which has its challenges:
Using curl in git bash on Windows. Not modifying the original ca-bundle.crt of curl, but referring to a modified copy in user controlled directory:

The challenges in this case are:

1. Under windows - and this includes the curl inside git bash - curl will not read custom option values from a $HOME/.curl but instead look for $HOME/_curlrc (with a fallback, if $HOME isn't set)
2. curl (in git bash for windows) expects Windows path, so a syntax like "C:" - not "/c/" - must be used inside the config files.
3. Environment variables inside curl config files were not expanded (neither "~"), so if personal settings like trusted CA certificates shall be read from $HOME, this must be expanded.

So for curl used inside a git bash shell, you would add some my_trusted_ca.pem via:

1. mkdir "$HOME/certs"
2. cp "$EXEPATH/mingw64/ssl/certs/ca-bundle.crt" "$HOME/.certs/my-ca-bundle.crt"
3. cat my_trusted_ca.pem >> "$HOME/.certs/my-ca-bundle.crt"
4. echo $(cygpath --windows "$HOME")/.certs/my-ca-bundle.crt > "$HOME/_curlrc"

[hohwille]

@vapadwal Thanks for testing. This is what I wanted to figure out. So if your proxy prevent such downloads it can not work this way.
However, a solution could be to create a custom software repo:
https://github.com/devonfw/devon-ide/blob/master/documentation/software.asciidoc#repository

[markusschuh]

Or try
set CURL_SSL_BACKEND=Schannel

This should help at least, when the URLs - in the same network constellation - work in Chrome. And the
curl --version
shows a parenthesized "Schannel" .
It is - based on the errors - less "a proxy blocking the download", but instead a non-harmonized CA trust setup between proxy and local curl installation.

[vapadwal]

@vapadwal Thanks for testing. This is what I wanted to figure out. So if your proxy prevent such downloads it can not work this way.
However, a solution could be to create a custom software repo:
https://github.com/devonfw/devon-ide/blob/master/documentation/software.asciidoc#repository

tried this approach.

steps:

1. updated the settings\devon\devon.properties with DEVON_SOFTWARE_REPOSITORY(http:\localhost:8080)
2. Uploaded the java and maven tgz files in local tomcat to test it
3. Java needed to be put in tomcat - in dir java\11.0.2\java-11.0.2-windows and file name needed to change to java-11.0.2-windows.tgz
4. Maven needed to be put in tomcat in dir maven\3.6.0\maven-3.6.0-all and file name needed to change to maven-3.6.0-all.tgz
   5. updated needed in functions file , added a slash in between, otherwise it was not downloading
   doDownload "${download_url}""/""${filename}"

Note: point 5 is the change required

[vapadwal]

Or try
set CURL_SSL_BACKEND=Schannel

This should help at least, when the URLs - in the same network constellation - work in Chrome. And the
curl --version
shows a parenthesized "Schannel" .
It is - based on the errors - less "a proxy blocking the download", but instead a non-harmonized CA trust setup between proxy and local curl installation.

Tried this one also but still the same error.

[hohwille]

@vapadwal thanks for testing and feedback.
So if I understood the results properly the repository approach is working besides a bug you were facing. We would need to create a central repository inside CG network and rested this from India but this is good news so far.

updated needed in functions file , added a slash in between, otherwise it was not downloading

doDownload "${download_url}""/""${filename}"

There might indeed be a bug and maybe it was working for the moment but the filename has to be a separate second argument and the actual bug then seems to be that the download_url variable is wrong.

[hohwille]

I fixed the error with the download from the repository so now the repository structure should also be as specified in the docs.

[hohwille]

@vapadwal for the record - can you tell if you can download the software failing to download with devon-ide from the internet with your browser (Chrome) as asked by @markusschuh ?
If no then it is obvious that we do not need to try tweaks for CURL, etc. as a workaround anyhow.

[vapadwal]

Yes we can download the softwares like java/maven etc from chrome but not devon-ide
tried the link given by devon-ide
https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_windows-x64_bin.zip
http://ftp.fau.de/apache/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz

[markusschuh]

I‘m able to connect to CG India VPN and then to resolve the certificate issues for curl based on my comments. Older curl version may need more complex configuration than newer ones, so not any configuration works for everyone. Nevertheless you should not try to implement „fixes“ for this environmental dependencies into devon-ide. Having the option to install from local repositories is helpful for other environments anyway - so it is here also.

[vapadwal]

Able to install java, maven, eclipse, vscode via local repository.

one small issue while installing eclipse , it was not able to download plugins from the internet from cg network

But able to add plugin manually or via add plugin

Cobigen already have an issue #84

`.....................
Installing eclipse plugin de.bastiankrol.startexplorer.feature from http://basti1302.github.com/startexplorer/update/

******** ATTENTION ********
WARNING: Installation of plugin de.bastiankrol.startexplorer.feature failed. Please try to install manually.

Installing eclipse plugin com.ess.regexutil from http://regex-util.sourceforge.net/update/

******** ATTENTION ********
WARNING: Installation of plugin com.ess.regexutil failed. Please try to install manually.

Installing eclipse plugin net.sf.mmm.eclipse.templatevariables.feature from https://m-m-m.github.io/eclipse-templatevariables/latest

******** ATTENTION ********
WARNING: Installation of plugin net.sf.mmm.eclipse.templatevariables.feature failed. Please try to install manually.

Installing eclipse plugin com.github.spotbugs.plugin.eclipse from https://spotbugs.github.io/eclipse/

******** ATTENTION ********
WARNING: Installation of plugin com.github.spotbugs.plugin.eclipse failed. Please try to install manually.

Installing eclipse plugin net.sf.eclipsecs from http://eclipse-cs.sf.net/update/

******** ATTENTION ********
WARNING: Installation of plugin net.sf.eclipsecs failed. Please try to install manually.

***** ERROR *****
CobiGen still has no public update site - see #84
doInstallEclipsePlugin https://github.com/devonfw/tools-cobigen/releases com.devonfw.cobigen.eclipse.feature.feature.group,com.devonfw.cobigen.tempeng-freemarker,com.devonfw.cobigen.htmlplugin,com.devonfw.cobigen.javaplugin,com.devonfw.cobigen.jsonplugin,com.devonfw.cobigen.openapiplugin,com.devonfw.cobigen.propertyplugin,com.devonfw.cobigen.textmerger,com.devonfw.cobigen.tsplugin.com.devonfw.cobigen.tempeng-velocity,com.devonfw.cobigen.xmlplugin
***** ERROR *****
TM Terminal does not have an update-site but only a marketplace site.
We currently have no clue how to automatically install plugins from marketplace
Until this issue is resolved, you need to manually install TM Terminal from the below marketplace URL:
http://marketplace.eclipse.org/content/tm-terminal
doInstallEclipsePlugin https://TODO org.eclipse.tm.terminal.feature.feature.group,org.eclipse.tm.terminal.view.feature.feature.group,org.eclipse.tm.terminal.control.feature.feature.group,org.eclipse.tm.terminal.connector.ssh.feature.feature.group,org.eclipse.tm.terminal.connector.telnet.feature.feature.group
Running command: command -v eclipse
/d/projects/myproject-ide/software/eclipse/eclipse
Succeeded to verify installation of Eclipse (command)
Running command: cat '/d/projects/myproject-ide/software/eclipse/.eclipseproduct'
name=Eclipse Platform
id=org.eclipse.platform
version=4.10.0
Succeeded to run command cat
Created script eclipse-main.bat
devon-ide environment variables have been set for D:\projects\myproject-ide_`

[vapadwal]

Some more issues found

neither of command works
(ng,node,npm,yarn)

D:\projects\myproject-ide>devon npm setup

******** ATTENTION ********
ERROR: Undefined devon command: node
Not found at /d/projects/myproject-ide/scripts/command/node.
We are sorry for the inconvenience. Please check the above errors, resolve them and try again.
devon-ide environment variables have been set for D:\projects\myproject-ide

D:\projects\myproject-ide>devon yarn setup
Running command: brew install yarn
/d/projects/myproject-ide/scripts/command/../functions: line 52: brew: command not found

******** ATTENTION ********
ERROR: Failed to run command brew
We are sorry for the inconvenience. Please check the above errors, resolve them and try again.
Exit code was 127
devon-ide environment variables have been set for D:\projects\myproject-ide

D:\projects\myproject-ide>devon ng setup
Running command: npm install -g @angular/cli@latest --unsafe
/d/projects/myproject-ide/scripts/command/../functions: line 52: npm: command not found

******** ATTENTION ********
ERROR: Failed to install angular-cli (npm)
We are sorry for the inconvenience. Please check the above errors, resolve them and try again.
Exit code was 127
devon-ide environment variables have been set for D:\projects\myproject-ide

D:\projects\myproject-ide>devon node setup
Unknown command node
devon-ide environment variables have been set for D:\projects\myproject-ide

Regarding java 10 and 11,

When we create and import devon4j project, it does not work with Java 10 or 11
we get the below warning

[WARNING] The POM for com.devonfw.java.modules:devon4j-security:jar:3.0.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details

the spring boot security package doesn't get install and we have compilation errors
[ERROR] /D:/projects/myproject-ide/company/api/src/main/java/com/company/general/service/api/rest/SecurityRestService.java:[9,45] package org.springframework.security.web.csrf does not exist
[ERROR] /D:/projects/myproject-ide/company/api/src/main/java/com/company/general/service/api/rest/SecurityRestService.java:[29,3] cannot find symbol

It works with Java 8 though

[hohwille]

Thanks for your feedback again.
All regarding to node is expected as nobody has implemented this issue yet:
#54
For the issue you found with yarn and brew, I created this new issue:
#125

Also the issue with Java11 is already known. This is about to be fixed in devon4j 3.1.0 and once released this will also be resolved in devon-ide that simply uses the latest available version of the archetype.

[hohwille]

IMHO all problems discussed in this issue have either been fixed, work as designed or new concrete issues have been created to track them. Hence, I am closing. Thanks again for the valuable feedback and helping to make devon-ide better.