HTTP proxy support not working properly

[sven-binnig]

As a developer, I want create a new project.

I'm working with a windows-10-pc behind a proxy.
The command ide --debug create test-project ends with an error message
could not clone [url] to [path] because you are offline

But I'm not offline - I'm only behind a proxy.
Setting up the environment variables http_proxyand https_proxydoesn't help.
I short look into the code makes it obvious, that no proxy configuration is used to determine, if I'm online or offline.

As a result of this, I cannot setup a project-folder.

[sven-binnig]

Additionally: please make sure to handle this:
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[sven-binnig]

2nd addition belongs to the .gitconfig
Setting up a project include clone the url for the installable tools.
This only works for me while temporaryly setting http.sslVerify to false

[sven-binnig]

After I have downloaded the zip archive of the code I tried to implement a dirty fix, that tries to take care about the existance of an proxy. I'm sure the this fix doesn't fit the needs of code-quality of this projekt. Beside this, I just want to share what I have done:

public boolean isOnline() {

    if (this.online == null) {
      // we currently assume we have only a CLI process that runs shortly
      // therefore we run this check only once to save resources when this method is called many times
      try {
        int timeout = 1000;
        //open a connection to github.com and try to retrieve data
        //getContent fails if there is no connection
        URL url = new URL("https://www.github.com");
        ProxyContext proxyContext = new ProxyContext(this);
        Proxy httpsProxy = proxyContext.getProxy(url.toExternalForm());
        URLConnection connection;
        if(proxyContext.equals(Proxy.NO_PROXY)){
          connection = url.openConnection();
        } else {
          // this is very evil - but I need to fix the 'PKIX'-Issue quickly 
          System.setProperty("javax.net.ssl.trustStore", "C:/Windows/win.ini");
          System.setProperty("javax.net.ssl.trustStoreType", "Windows-ROOT");
          connection = url.openConnection(httpsProxy);
        }
        connection.setConnectTimeout(timeout);
        connection.getContent();
        this.online = Boolean.TRUE;
      } catch (Exception ignored) {
        this.online = Boolean.FALSE;
      }
    }

[hohwille]

@sven-binnig thank you so much for your active contribution and analysis of our product in beta phase and sorry for the bugs you are facing. 👍
I created story #245 for proxy support that was implemented with PR #338.
However, in #245, I wrote:

I am unsure if this already supports a download via an HTTP proxy by passing JVM arguments (-D....proxy=...) or not.

It seems that this tiny information was overlooked.
I finally took a few minutes to research this and found this one:
https://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html

http.proxyHost: the host name of the proxy server
http.proxyPort: the port number, the default value being 80.
http.nonProxyHosts: a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by '|'. The patterns may start or end with a '*' for wildcards. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.

So IMHO this ProxyContext solution might be a little over-engineered.
Launching our program ideasy with -Dhttp.proxyHost=${http_proxy} might already do the trick for all network traffic in our product in your case.
Could you try to call this:

ideasy -Dhttp.proxyHost=${http_proxy} -v

or in case of CMD (if you insist on still using it ;) ):

ideasy -Dhttp.proxyHost=%http_proxy% -v

And see if that offline warning goes away?
If yes we might have a much simpler solution for the proxy support.
However, also your suggested change to use ProxyContext in online check would make sense.
BTW:

// this is very evil - but I need to fix the 'PKIX'-Issue quickly

This is an evil habit of ugly company VPN tools that should be forbidden by law!
But users suffering with such problems would need a solution and java is not using the truststore of the OS where the VPN tool may have hoocked into and hacked yourself by planting his faked certificate into.

My idea for both KISS and high flexibility is to simply introduce a variable IDE_OPTIONS that users can define manually (e.g. on Windows via Environment variables) or via custom scripts to add JVM options to ideasy.
There you could add stuff like -Djavax.net.ssl.truststore=... without the need that we need hacks in our code.
We only change our wrapper scripts (ide and ide.bat) to honor this variable. See #792 for that.

[sven-binnig]

@hohwille
I vote for #792
And I will try ideasy -Dhttp.proxyHost=%http_proxy% -v right now

[sven-binnig]

Results:
Environment variable is set.

C:\projects>ideasy -Dhttp.proxyHost=%http_proxy% -v
2024.12.001-beta-SNAPSHOT

Without any strange warning.

Creating a new project with still have the same result:

C:\projects>ideasy -Dhttp.proxyHost=%http_proxy% create test-project
Creating new IDEasy project in C:\projects\test-project
←[96mDirectory C:\projects\test-project already exists. Do you want to continue?←[0m
←[96mOption 1: yes←[0m
←[96mOption 2: no←[0m
1
←[35mStart: Clone settings repository←[0m
Missing your settings at C:\projects\test-project\settings and no SETTINGS_URL is defined.
Further details can be found here: https://github.com/devonfw/IDEasy/blob/main/documentation/settings.asciidoc
Please contact the technical lead of your project to get the SETTINGS_URL for your project.
In case you just want to test IDEasy you may simply hit return to install the default settings.
Settings URL [https://github.com/devonfw/ide-settings.git]:

Step 'Clone settings repository' ended with failure.
←[91mStep 'ide (create,test-project)' failed: com.devonfw.tools.ide.cli.CliOfflineException: Could not clone https://github.com/devonfw/ide-settings.git to C:\projects\test-project\settings because you are offline.←[0m
←[91m Step 'Clone settings repository' failed: unexpected error←[0m
←[91m2 step(s) failed out of 2 steps.←[0m
←[91mCould not clone https://github.com/devonfw/ide-settings.git to C:\projects\test-project\settings because you are offline.←[0m

The warning is still existent ...

or in case of CMD (if you insist on still using it ;) ):

Should I better use (Git)Bash?
And why?

[sven-binnig]

@hohwille
OK - perhabs jansi works better with bash?

[hohwille]

@sven-binnig great news: I just merged PR #811
From tomorrow on we will have it included in our latest SNAPSHOT release.
To upgrade you existing installation to the latest SNAPSHOT however has a little hurdle:
You need to manually edit your .bashrc (and in case you have zsh also your .zshrc) and remove the entries created by our setup script. Simply search for any line containing ide (such as source .../_ide/autocompletion, alias ide=... and just ide).
Since we are still in BETA and have a smaller user-basis we did not see the need to automate this for now.
After you removed this, rerun the setup script.

BTW: With the new approach we can easily roll out new changes to your shells without again modifying bashrc.

[hohwille]

If you have done that, please test if proxy support is now working fine.
Documentation is also updated: https://github.com/devonfw/IDEasy/blob/main/documentation/proxy-support.adoc

[sven-binnig]

@hohwille First: sorry for the delay - a cold catched me and forced me to lay down my head

I have downloaded the latest snapshotide-cli-2024.12.001-beta-20241123.023620-5-windows-x64.tar.gz and did a complete new setup with it.

Afterward I tried to create a new project.
I defined the environment variable IDE_OPTIONS with my proxy settings and trust store definitions.

But I still get the same messages as before:

$ ide create test-project
Creating new IDEasy project in C:\projects\test-project
Start: Clone settings repository
Missing your settings at C:\projects\test-project\settings and no SETTINGS_URL is defined.
Further details can be found here: https://github.com/devonfw/IDEasy/blob/main/documentation/settings.asciidoc
Please contact the technical lead of your project to get the SETTINGS_URL for your project.
In case you just want to test IDEasy you may simply hit return to install the default settings.
Settings URL [https://github.com/devonfw/ide-settings.git]:

Step 'Clone settings repository' ended with failure.
Step 'ide (create,test-project)' failed: com.devonfw.tools.ide.cli.CliOfflineException: Could not clone https://github.com/devonfw/ide-settings.git to C:\projects\test-project\settings because you are offline.
 Step 'Clone settings repository' failed: unexpected error
2 step(s) failed out of 2 steps.
Could not clone https://github.com/devonfw/ide-settings.git to C:\projects\test-project\settings because you are offline.

Error: IDEasy failed with exit code 23

[sven-binnig]

Sorry - I have downloaded the wrong SNAPSHOT.
I am downloading ide-cli-2024.12.001-beta-20241130.023556-10-windows-x64.tar.gz an will retry ....

[sven-binnig]

The result remains the same

And also strange:
ide --version gives SNAPSHOT as version identifier.

[sven-binnig]

The last step I have done (as always) with CMD
When I use git bash the result is different:

$ ide create test-project
Creating new IDEasy project in C:\projects\test-project
Start: Clone settings repository
Missing your settings at C:\projects\test-project\settings and no SETTINGS_URL is defined.
Further details can be found here: https://github.com/devonfw/IDEasy/blob/main/documentation/settings.asciidoc
Please contact the technical lead of your project to get the SETTINGS_URL for your project.
In case you just want to test IDEasy you may simply hit return to install the default settings.
Settings URL [https://github.com/devonfw/ide-settings.git]:

Cloning into '.'...
fatal: unable to access 'https://github.com/devonfw/ide-settings.git/': Could not resolve host: github.com
Running command 'C:\Program Files\Git\mingw64\bin\git.exe' with arguments 'clone' '--recursive' 'https://github.com/devonfw/ide-settings.git' '--config' 'core.autocrlf=false' '.'
failed with exit code 128!
Git failed to clone https://github.com/devonfw/ide-settings.git into C:\projects\test-project\settings.
Successfully updated settings repository.
Templates folder is missing in settings repository.
Start: Install or update software
Successfully ended step 'Install or update software'.
Cannot find repositories folder nor projects folder.
Successfully created new project 'test-project'.
Successfully completed ide (create,test-project)

The try to clone the standard settings from github ends with:

Running command 'C:\Program Files\Git\mingw64\bin\git.exe' with arguments 'clone' '--recursive' 'https://github.com/devonfw/ide-settings.git' '--config' 'core.autocrlf=false' '.'
failed with exit code 128!
Git failed to clone https://github.com/devonfw/ide-settings.git into C:\projects\test-project\settings.

I will retry this with an URL of our git ...

[sven-binnig]

Using a "local" settings url works.
But when I try to install intellij, I got the same problem with cloning the tools-url-repos.

[sven-binnig]

@hohwille
In AbstractIdeContext .isOnline() (line 549 ff.) I still found:
URLConnection connection = new URL("https://www.github.com").openConnection();

There is also a method public URLConnection openConnection(Proxy proxy) -> why not using this method when proxy is configured?
In my private work around this works....

[hohwille]

@sven-binnig thanks for all your feedback.
Howver, let us focus on this issue first instead of discussing tons of other things.
I just wanted to know if the proxy auto-configuration now works.
Just call this

ideasy -t status

And let me know the output. The most important thing is if you are now considered as being online.
Also look for something like this:

$ export https_proxy=https://norwegian.woods.no/
$ ideasy -t status
...
Found environment variable https_proxy=https://norwegian.woods.no/
System property was set to https.proxyHost=norwegian.woods.no
System property was set to https.proxyPort=443
...
IDE_ROOT is set to D:\projects
IDE_HOME is set to D:\projects\IDEasy
You are offline. Check your internet connection and potential proxy settings.
...

Since I am not sitting behind a real proxy, my configuration was just a test and that breaks stuff since I configured nonsense.
However, from the trace logs I can see what the new proxy autoconfiguration did and that it worked as I expected.
My proxy was configured and therefore I went offline since I cannot reach an HTTP proxy server on norwegian.woods.no:443.
This is what I would like to test on your end and see if you get online behind your proxy.
If that works the downloads should still work.
Also you are now able to use the IDE_OPTIONS variable as documented.

[hohwille]

And also strange:
ide --version gives SNAPSHOT as version identifier.

Yes, bug #830 already fixed and new SNAPSHOT released. Welcome to SNAPSTHOT testing ;)

[hohwille]

Settings URL [https://github.com/devonfw/ide-settings.git]:

Cloning into '.'...
fatal: unable to access 'https://github.com/devonfw/ide-settings.git/': Could not resolve host: github.com
Running command 'C:\Program Files\Git\mingw64\bin\git.exe' with arguments 'clone' '--recursive' 'https://github.com/devonfw/ide-settings.git' '--config' 'core.autocrlf=false' '.'
failed with exit code 128!
Git failed to clone https://github.com/devonfw/ide-settings.git into C:\projects\test-project\settings.

This is not a problem of IDEasy. You are behind a proxy and you need to get your configuration sane for your setup.
Blame the people that force you to work behind a proxy and do not provide you with correct configuration out of the box.
What we are calling is this:

"C:\Program Files\Git\mingw64\bin\git.exe" clone --recursive https://github.com/devonfw/ide-settings.git --config core.autocrlf=false .

inside the newly created settings folder in your project.
This command is working for normal people.
If it is not working for you, you need to study how to make git work behind your strange proxy setup.
See e.g. https://git-scm.com/docs/git-config#EXAMPLES or https://stackoverflow.com/questions/128035/how-do-i-pull-from-a-git-repository-through-an-http-proxy

[hohwille]

Using a "local" settings url works.
But when I try to install intellij, I got the same problem with cloning the tools-url-repos.

This has nothing to do with this bug ticket about HTTP proxy support.
Please discuss in #766 - we are already working on it and hope to fix it this week and still being able to do the 2024.12.001 release by the end of this week, but we face various unexpected problems so fingers crossed.

[sven-binnig]

ide sttaus without setting proxy configuration via IDE_OPTIONS:

$ ideasy status
Running command 'C:\Program Files\Git\mingw64\bin\git.exe' with arguments 'rev-parse' 'HEAD'
failed with exit code 128!
fatal: not a git repository (or any of the parent directories): .git
Failed to get the local commit hash.
Skipping git fetch on C:\projects\test-project\settings because we are offline.
IDE_ROOT is set to C:\projects
IDE_HOME is set to C:\projects\test-project
You are offline. Check your internet connection and potential proxy settings.

[sven-binnig]

ide sttaus with setting proxy configuration via IDE_OPTIONS:

...
No http proxy configured.
No https proxy configured.
Skipping git fetch on C:\projects\test-project\settings because we are offline.
Skipped git fetch.
IDE_ROOT is set to C:\projects
IDE_HOME is set to C:\projects\test-project
You are offline. Check your internet connection and potential proxy settings.

[sven-binnig]

ide sttaus without setting proxy configuration via IDE_OPTIONS but via environment variable (without export)

...
No http proxy configured.
No https proxy configured.
Skipping git fetch on C:\projects\test-project\settings because we are offline.
Skipped git fetch.
IDE_ROOT is set to C:\projects
IDE_HOME is set to C:\projects\test-project
You are offline. Check your internet connection and potential proxy settings.

[hohwille]

In AbstractIdeContext .isOnline() (line 549 ff.) I still found:
URLConnection connection = new URL("https://www.github.com").openConnection();

There is also a method public URLConnection openConnection(Proxy proxy) -> why not using this method when proxy is configured?

Oh, so you want to join our development team ;)
Feel free to provide pull-requests with suggested changes.
Thanks for your suggestion but I removed this explicit Proxy conversion on purpose since it will not solve all your edge-case problems. What if your proxy requires authentication? There are Java system properties to solve all these problems. The approach to manually build a Proxy configuration in our code and configure the proxy explicitly will only take away flexibility because then we have to implement everything ourselves that Java has already solved. I am therefore convinced that the previous approach was leading to a dead end and my new approach is the correct way to go.

[hohwille]

ide sttaus without setting proxy configuration via IDE_OPTIONS but via environment variable (without export)

Thanks for testing. Did you read the documentation I linked above:
https://github.com/devonfw/IDEasy/blob/main/documentation/proxy-support.adoc

A variable that you define in your current shell without exporting is private and invisible to sub-processes. It is therefore impossible for git, curl, wget or ideasy to access your https_proxy variable.
For real you shall not even manually export such variable in your bash but globally set it in your windows environment variables.

[hohwille]

Sorry, I always take for granted that developers using IDEasy are aware how environment variables work.
If the following link may help you with your problem, we are happy to also add it to our proxy documentation:
https://ftr.elevio.help/en/articles/171-configure-the-proxy-with-user-specific-environment-variables

[sven-binnig]

ide status with setting proxy configuration via env var and export:

...
Found environment variable http_proxy=http://xxxx:yyyy
System property was set to http.proxyHost=xxx
System property was set to http.proxyPort=yyy
Found environment variable https_proxy=http://xxxx:yyyy
System property was set to https.proxyHost=xxx
System property was set to https.proxyPort=yyy
Skipping git fetch on C:\projects\test-project\settings because we are offline.
Skipped git fetch.
IDE_ROOT is set to C:\projects
IDE_HOME is set to C:\projects\test-project
You are offline. Check your internet connection and potential proxy settings.

I have made values for host and port "unreadable" - but I have used our company-values

[hohwille]

I have made values for host and port "unreadable" - but I have used our company-values

OK. That is what I wanted to know. Too bad it is still not working.
Could you test the same with another tool other than IDEasy:

$ curl https://github.com/devonfw/ide-settings

If that is also failing your configuration is still incorrect. In case it gives you the HTML output of that URL page, then the configuration is sane but IDEasy still has a bug.

[sven-binnig]

Output curl

$ curl https://github.com/devonfw/ide-settings
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
curl: (35) schannel: next InitializeSecurityContext failed: CRYPT_E_NO_REVOCATION_CHECK (0x80092012) - Die Sperrfunktion konnte keine Sperrpr▒fung f▒r das Zertifikat durchf▒hren.

[hohwille]

OK, that explains why you are still offline in IDEasy. You additionally cannot get trusted TLS (HTTPS) connections due to your corporate proxy.
So next you need to create and configure a truststore:
https://github.com/devonfw/IDEasy/blob/main/documentation/proxy-support.adoc#truststore

This is not an easy beginners tasks but unfortunately we cannot solve this for you. Please try to utilize tutorials on the web how to get this solved. Maybe you need to involve the support of your company. If git, curl, etc. is also not working they should have an answer how a developer should be able to work in your company.

[sven-binnig]

But in my IDE_OPTIONS i have utilized a trust store with
-Djavax.net.ssl.trustStore=C:/Windows/win.ini -Djavax.net.ssl.trustStoreType=Windows-ROOT

[hohwille]

But in my IDE_OPTIONS i have utilized a trust store with -Djavax.net.ssl.trustStore=C:/Windows/win.ini -Djavax.net.ssl.trustStoreType=Windows-ROOT

Nice try, but does C:\Windows\win.ini really exist on your computer?
If it exists, isn't it an *.ini file (text format with sections in angled brackets)?
See also https://en.wikipedia.org/wiki/WIN.INI
The file you are referring to must be a java keystore file.
Please read the instructions I linked about keystores.
Your config looks more like the first hit you found on stackoverflow or got from chatgpt but IMHO it does not make any sense.

[hohwille]

The first thing you need is the "fake" TLS certificate of your proxy.
You can try to get it out of your browser if it is also using the same proxy.
Open the "lock" icon and inspect the certificate. In Firefox looks like this:

Check that your certificate belongs to your proxy and not really originates from the website your are browsing.
Then export that certificate to a file.

Then create a new Java keystore and import the certificate.
Then change your IDE_OPTIONS to point to that keystore file on your disc and to the passphrase you defined when creating the keystore with keytool.

[hohwille]

Your config looks more like the first hit you found on stackoverflow or got from chatgpt but IMHO it does not make any sense.

Indeed from stackoverflow: https://stackoverflow.com/a/59056537/1506881
It got quite some votes so it cannot be nonsense but maybe this approach only worked in older Java versions or is not supported by GraalVM.

UPDATE: A little hard to find, but documented officially in Oracles Java specs:
https://docs.oracle.com/en/java/javase/21/security/oracle-providers.html#GUID-4F1737D6-1569-4340-B140-678C70E63CD5__GUID-BD31F123-97D0-4F61-84D4-645461A12C00

Actually if you are using this trustStoreType the trustStore location is actually ignored. So you can configure any nonsense here like win.ini that does not seem to exist in recent Windows.

[hohwille]

https://www.graalvm.org/latest/reference-manual/native-image/dynamic-features/CertificateManagement/

[sven-binnig]

from https://www.graalvm.org/latest/reference-manual/native-image/dynamic-features/CertificateManagement/

During the image building process, the native-image builder captures the host environment’s default TrustStore and embeds it into the native executable

I read it in that way, that nevertheless which setting I do make, it will be ignored.
Correct?

[sven-binnig]

@hohwille

Actually if you are using this trustStoreType the trustStore location is actually ignored. So you can configure any nonsense here like win.ini that does not seem to exist in recent Windows.

Yes- I really don't know when I discovered that hack. But it workes stable, when I face the truststore-issue with java processes.

If I understood the article about certifate management and graalvm/native-image correctly, the definitions about truststore via the variable IDE_OPTIONS will have no effect ...

[hohwille]

I read it in that way, that nevertheless which setting I do make, it will be ignored.
Correct?

IMHO not. You have to read on. This is only saying what the static default behaviour is.
The next section explains:

The certificate file can also be changed dynamically at run time via setting the javax.net.ssl.trustStore* system properties.

That is exactly what we are trying to do.
Did you create a custom truststore file as I suggested and retest it this way?

[hohwille]

We will try to simulate the problem with wiremock as proxy.
However, we cannot simulate that we do not have direct Internet access so we might miss problems were something bypasses the proxy and gets data directly from the Internet without using the proxy.

[sven-binnig]

Sorry - I was busy the last days - but I will try as soon as possible.

[sven-binnig]

While testing #783 I have realized, that only defining the proxy in the IDE_OPTIONS is not sufficent.
When I export HTTPS_PROXY I could download the settings an the url an setup an project with ide create.
But a tool setup with ide setup <tool> oder ide <tools> is still not working with the message, that I am offline.

[hohwille]

While testing #783 I have realized, that only defining the proxy in the IDE_OPTIONS is not sufficent. When I export HTTPS_PROXY I could download the settings an the url an setup an project with ide create. But a tool setup with ide setup <tool> oder ide <tools> is still not working with the message, that I am offline.

IMHO this is because of the story #525 that is kind of a contradicting or at least conflicting demand.
However, we need to separate things clearly:

• being able to configure a HTTP proxy server and according "JVM options" (initial topic of this story)
• your concrete problem with TLS (javax.net.ssl.SSLHandshakeException: PKIX path building failed)
• the check if we are online or offline (I hope it also works fine in combination with HTTP proxy but this is still to be verified)
• other use-cases like installing and running Intellij

If for some reason in your specific setup IDEasy cannot properly create an HTTPS connection and therefore thinks it is offline, you are fundamentally blocked and face lots of problems. However, that does not mean that anything is wrong with the functionality of launching Intellij in general (as discussed in #783) but all comes down to the same root problem.

[hohwille]

I have created #897 since the general configuration via IDE_OPTIONS was implemented and is working.
Also the proxy auto-configuration via variables like HTTPS_PROXY or NO_PROXY environment variables has been implemented and tested.
Therefore I am closing this issue and further problems about the additional TLS specific bugs should be further discussed in #897.