Downloads after NVD init don't seem to be using proxy settings correctly #6800

DocMoebiuz · 2024-07-04T07:41:38Z

Describe the bug
We have a corporate proxy in place and I am providing the settings including proxy user and proxy pass through the JAVA_TOOL_OPTIONS as described in the documentation.

This works for the NVD updates, but as soon as I get to the the point where it wants to init the retireJS repo or download the publishedSupressions.xml, then I receive a 407 error from the proxy.

Version of dependency-check used
The problem occurs using version 10.0.1

Log file

[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 240 minutes
[ERROR] Failed to initialize the RetireJS repo
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to initialize the RetireJS repo
        at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:152)
        at org.owasp.dependencycheck.data.update.RetireJSDataSource.update(RetireJSDataSource.java:95)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
        at org.owasp.dependencycheck.App.runScan(App.java:262)
        at org.owasp.dependencycheck.App.run(App.java:194)
        at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json' to 'C:\actions-runner\_work\NSDT\dependency-check\data\jsrepository.json'; Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:100)
        at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:150)
        ... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
        ... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
        ... 11 common frames omitted
[WARN] Failed to update hosted suppressions file, results may contain false positives already resolved by the DependencyCheck project
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to update the hosted suppressions file
        at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:156)
        at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.update(HostedSuppressionsDataSource.java:87)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
        at org.owasp.dependencycheck.App.runScan(App.java:262)
        at org.owasp.dependencycheck.App.run(App.java:194)
        at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml' to 'C:\actions-runner\_work\NSDT\dependency-check\data\publishedSuppressions.xml'; Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:83)
        at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:154)
        ... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
        ... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
        ... 11 common frames omitted
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
org.owasp.dependencycheck.data.update.exception.UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
        at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update(KnownExploitedDataSource.java:93)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
        at org.owasp.dependencycheck.App.runScan(App.java:262)
        at org.owasp.dependencycheck.App.run(App.java:194)
        at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
        at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update(KnownExploitedDataSource.java:80)
        ... 6 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
        ... 8 common frames omitted
[WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
[ERROR] Unable to continue dependency-check analysis.
[ERROR] One or more fatal errors occurred
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR] No documents exist

To Reproduce
Steps to reproduce the behavior:

Run dependency check with proxy parameters provided through JAVA_TOOLS_OPTIONS
Wait for the NVD to finish
See error

Expected behavior
The process should download additional resources without error using the same proxy config.

Additional context
I have successfully downloaded the files manually, so it is not the proxy that blocks these specific URLs

The text was updated successfully, but these errors were encountered:

chadlwilson · 2024-07-04T09:05:37Z

The correct env var is JAVA_TOOL_OPTIONS not JAVA_TOOLS_OPTIONS. Are you sure you've configured it correctly?

Your log above shows the below, so it never tried to call NVD.

[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 240 minutes

DocMoebiuz · 2024-07-04T09:56:38Z

It was a typo - I am using JAVA_TOOL_OPTIONS
NVD is skipped because it was downloaded less than 4 hours ago. That's what I am saying, the NVD download works... the downloads thereafter dont.

chadlwilson · 2024-07-04T10:04:06Z

Well the error implies it is talking to your proxy, hence getting Proxy returns "HTTP/1.1 407 Proxy Authentication Required" so folks would probably need to see the specific arguments you are sending (redacted) and how you are sending them (Gradle? Maven? Standalone? did it used to work but no longer does? can you force it to update NVD and show a log which shows NVD working but other data sources not?).

Depending on your proxy configuration, there are many reasons this could happen, e.g the proxy happens to whitelist or allow through the NVD API host, but not cisa.gov or github.io etc. Would need to see the args and a more complete log showing the NVD API working to tell.

DocMoebiuz · 2024-07-04T10:28:54Z

i can download from cisa.gov or github.io with the same proxy settings from the command line, so this rules out that the proxy is having some specific issues with these URLs.

chadlwilson · 2024-07-04T10:39:49Z

OK, that's useful info, but folks would still need to know what you're supplying to ODC and which specific variant you are running.

DocMoebiuz · 2024-07-04T11:11:25Z

i am not sure what you mean with "supplying to ODC"? - I have a dotnet project and I am trying to run the dependency-check.bat CLI on windows. I followed the CLI documentation which explains how to pass in the proxy relevant settings via the JAVA_TOOL_OPTIONS.

It seems to me as if the NVD download requests are using the proxy information, whereas all other requests don't. I have to run the CLI cmd with --disableRetireJS --disableKnownExploited so that I get passed those.

chadlwilson · 2024-07-04T11:41:51Z

What is the specific value of JAVA_TOOL_OPTIONS you are using and how are you exposing it to the CLI? What commands are you using? How can someone else replicate your problem reliably without having to guess what you are doing and replicate your precise proxy or runtime environment (which is not disclosed)?

While you may be right that a core proxy functionality is fundamentally broken, given the very wide user base of ODC it seems more likely that you’re doing something wrong, there is something specific to your environment that is wrong - or the documentation is misleading.

If you don’t explain precisely what you’re doing or include logs that show what you say is happening (NVD API working), we can’t easily get anywhere.

DocMoebiuz · 2024-07-04T11:51:09Z

My JAVA_TOOL_OPTIONS are like this, and set as environment variables:

-Dhttps.proxyHost=${{ vars.PROXY_HOST }} -Dhttps.proxyPort=${{ vars.PROXY_PORT }} -Dhttps.proxyUser=${{ secrets.PROXY_USER }} -Dhttps.proxyPassword=${{ secrets.PROXY_PASS }}

The CLI tool is started like this:
..\dependency-check\bin\dependency-check.bat --scan .\Code --format HTML --project "myproject.sln" --out .\report --nvdApiKey $env:NVD_API_KEY --disableRetireJS --disableKnownExploited

As I already stated before, the NVD downloads went fine with these settings. If I drop the --disableRetireJS or --disableKnownExploited option then I would see the error log entries from my first post.

jeremylong · 2024-07-04T11:58:22Z

Does this work for you (of course updating the proxy?

curl --proxy "http://user:pwd@127.0.0.1:1234" "https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json"

chadlwilson · 2024-07-04T12:10:45Z

As I already stated before, the NVD downloads went fine with these settings. If I drop the --disableRetireJS or --disableKnownExploited option then I would see the error log entries from my first post.

Yes, you stated/told, but you didn't show via logs. I wouldn't ask to see if I didn't think it was useful to eliminate problems or assumptions about how ODC works, this is why I asked:

can you force it to update NVD and show a log which shows NVD working but other data sources not?).

Anyway, good luck to you.

DocMoebiuz · 2024-07-04T12:51:20Z

Does this work for you (of course updating the proxy?

curl --proxy "http://user:pwd@127.0.0.1:1234" "https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json"

Yes, that's working, but I had to switch to another proxy url to make it work with curl whereas it was working fine with Invoke-WebRequest -Uri $url -OutFile $output -Verbose. I will now double check one more time and maybe use the --purge command to verify whether the NVD can be downloaded (as suggested by @chadlwilson)

Did I mention that I hate proxies 😄

DocMoebiuz · 2024-07-04T12:54:33Z

Unforunately, it's the same result like before, now also showing that the NVD downloads work properly:

4:47:38,161 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
14:47:38,161 |-INFO in ch.qos.logback.classic.joran.action.ContextNameAction - Setting logger context name as [dependency-check]
14:47:38,161 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
14:47:38,161 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [console]
14:47:38,[17](actions/runs/6374661/job/18382986#step:6:18)7 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.commons.jcs] to ERROR
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.hc] to ERROR
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
14:47:38,192 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [console] to Logger[ROOT]
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@5bb21b69 - Registering current configuration as safe fallback point

[INFO] Checking for updates
[INFO] NVD API has 28 records in this update
[INFO] Downloaded 28/28 (100%)
[INFO] Completed processing batch 1/1 (100%) in 233ms
Error:  Failed to initialize the RetireJS repo
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to initialize the RetireJS repo
	at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:152)
	at org.owasp.dependencycheck.data.update.RetireJSDataSource.update(RetireJSDataSource.java:95)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
	at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
	at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
	at org.owasp.dependencycheck.App.runScan(App.java:262)
	at org.owasp.dependencycheck.App.run(App.java:194)
	at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json' to 'C:\actions-runner\_work\NSDT\dependency-check\data\jsrepository.json'; Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
	at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
	at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:100)
	at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:150)
	... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
	at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
	at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
	at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
	... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
	at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:[18](https://github/actions/runs/6374661/job/18382986#step:6:19)5)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
	at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
	... 11 common frames omitted
[WARN] Failed to update hosted suppressions file, results may contain false positives already resolved by the DependencyCheck project
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to update the hosted suppressions file
	at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:156)
	at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.update(HostedSuppressionsDataSource.java:87)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
	at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
	at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
	at org.owasp.dependencycheck.App.runScan(App.java:262)
	at org.owasp.dependencycheck.App.run(App.java:[19](https://actions/runs/6374661/job/18382986#step:6:20)4)
	at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml' to 'C:\actions-runner\_work\NSDT\dependency-check\data\publishedSuppressions.xml'; Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
	at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
	at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:83)
	at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:154)
	... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
	at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
	at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
	at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
	... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
	at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
	at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:[20](https://github/actions/runs/6374661/job/18382986#step:6:21)6)
	... 11 common frames omitted

jeremylong · 2024-07-04T13:20:38Z

...I had to switch to another proxy url to make it work with curl

Sounds like an environment issue? possibly one of the proxies blocks access to some resources? I know some users have had to work with the networking team to allow the connections on the proxy.

DocMoebiuz · 2024-07-05T07:27:42Z

Well, as I wrote already: I can curl the URLs that are failing from the command line with the same options from JAVA_TOOL_OPTIONS. It can't be the proxy preventing access... it seems to me as if the NVD requests pick up the proxy settings but for the other "modules" like the RetireJS they aren't.

jeremylong · 2024-07-05T10:25:28Z

You can also host these files inside your network - see https://jeremylong.github.io/DependencyCheck/data/index.html

DocMoebiuz · 2024-07-09T08:01:39Z

So, looking at this page: https://jeremylong.github.io/DependencyCheck/data/proxy.html

Is it fair to assume that if JAVA_TOOL_OPTIONS is set correctly, all subsequent requests by dependency-check will use those? I am asking since I am starting dependency check in a github action which always has it's own set of env variables, etc.

Also, the page https://jeremylong.github.io/DependencyCheck/data/proxy.html mentions legacy proxy config. I am wondering if that is required if JAVA_TOOL_OPTIONS is set correctly.

jeremylong · 2024-07-09T10:29:41Z

Yes, correctly setting the JAVA_TOOL_OPTIONS should be used for all requests. The legacy options were left there to make it easier for upgrades.

DocMoebiuz · 2024-08-07T14:12:06Z

So I kept digging into this. I finally decided to start a local px instance to simply use localhost:3128 without authentication. Result: it is working. As soon as I revert to directly using the corporate proxy including auth username and password it will fail again.

So for some reason the authentication information is not used in all requests correctly or there are some other settings that are not configured correctly when using BASIC authentication for proxy.
I couldn't find any additional options that would allow me to specify more details.

BTW I also learned that it doesn't make a difference whether I am using JAVA_OPTS or JAVA_TOOL_OPTIONS.

aikebah · 2024-08-31T17:04:27Z

Could it also be that your network team has different rules for nvd.nist.gov (seen by them as a trusted source; anonymous proxy access allowed) versus github (seen by them as a risky site, so people should authenticate on the proxy (and maybe even be part of certain user-groups) before being granted access to information stored there)

After all it's not uncommon for enterprise proxies to have various categories of sites with different access grants from the internal network.

aikebah · 2024-08-31T18:50:14Z

Managed to reproduce locally with a customized squid-proxy docker container at localhost. I'll dive deeper into it.

DocMoebiuz · 2024-08-31T19:14:26Z

That's great news! thanks for further looking into it.

aikebah · 2024-09-01T18:49:32Z

Authenticating proxy even fails when using the legacy configuration parameters. Only when I force the setting for jdk.http.auth.tunneling.disabledSchemes early on (by making it part of settings initialisation) the legacy parameters work. The JAVA_TOOL_OPTIONS does not work for those cases still, which is logical, as the https.proxyPassword and https.proxyUser are system properties that are not consulted by the JVM (fully in line with the documentation of the proxy properties on ), they appear to be useful extensions to the set by Apache HTTPClient.

@jeremylong I propose to try and swap out our custom HTTP client code customizing the standard JVM classes for the various resource downloads by the Apache HTTPClient. Need to see how that is best done, but expect I would be able to get that done somewhere this week; would be a good additional candidate for the 11.0.0 release I think

jeremylong · 2024-09-02T11:48:41Z

100% agree. I've been using the HTTPClient in the open-vulnerability-client. If you can work on this - that would be great.

aikebah · 2024-09-08T14:27:25Z

Think I was a bit too optimistic regarding the amount of work to get rid of all other ways and consistently use HTTPClient within a weeks time. Nevertheless things are moving forward piece by piece.

DocMoebiuz added the bug label Jul 4, 2024

aikebah self-assigned this Sep 1, 2024

aikebah mentioned this issue Sep 7, 2024

feat: Use Apache HTTPClient for downloads of public resources #6949

Merged

aikebah added this to the 11.0.0 milestone Sep 18, 2024

aikebah closed this as completed Oct 20, 2024

github-actions bot locked as resolved and limited conversation to collaborators Dec 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Downloads after NVD init don't seem to be using proxy settings correctly #6800

Downloads after NVD init don't seem to be using proxy settings correctly #6800

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

jeremylong commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

jeremylong commented Jul 4, 2024

DocMoebiuz commented Jul 5, 2024

jeremylong commented Jul 5, 2024

DocMoebiuz commented Jul 9, 2024 •

edited

Loading

jeremylong commented Jul 9, 2024

DocMoebiuz commented Aug 7, 2024 •

edited

Loading

aikebah commented Aug 31, 2024

aikebah commented Aug 31, 2024

DocMoebiuz commented Aug 31, 2024

aikebah commented Sep 1, 2024

jeremylong commented Sep 2, 2024

aikebah commented Sep 8, 2024 •

edited

Loading

Downloads after NVD init don't seem to be using proxy settings correctly #6800

Downloads after NVD init don't seem to be using proxy settings correctly #6800

Comments

DocMoebiuz commented Jul 4, 2024 • edited Loading

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024 • edited Loading

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024 • edited Loading

jeremylong commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

DocMoebiuz commented Jul 4, 2024 • edited Loading

DocMoebiuz commented Jul 4, 2024 • edited Loading

jeremylong commented Jul 4, 2024

DocMoebiuz commented Jul 5, 2024

jeremylong commented Jul 5, 2024

DocMoebiuz commented Jul 9, 2024 • edited Loading

jeremylong commented Jul 9, 2024

DocMoebiuz commented Aug 7, 2024 • edited Loading

aikebah commented Aug 31, 2024

aikebah commented Aug 31, 2024

DocMoebiuz commented Aug 31, 2024

aikebah commented Sep 1, 2024

jeremylong commented Sep 2, 2024

aikebah commented Sep 8, 2024 • edited Loading

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

DocMoebiuz commented Jul 4, 2024 •

edited

Loading

DocMoebiuz commented Jul 9, 2024 •

edited

Loading

DocMoebiuz commented Aug 7, 2024 •

edited

Loading

aikebah commented Sep 8, 2024 •

edited

Loading