[Ingest Manager] Kibana does not take into account proxy for package download

[MacPower]

Kibana version:
7.8.0
Elasticsearch version:
7.8.0

Original install method (e.g. download page, yum, from source, etc.):

I used ECK 1.1.2

Describe the bug:

Kibana server with ingest manager enable won't take into account proxy environment variable for package download.

Steps to reproduce:
1.
config :

xpack.ingestManager.enabled: true

Add proxy env variable

        - name: kibana
          env:
          - name: HTTP_PROXY
            value: http://***:3128
          - name: HTTPS_PROXY
            value: http://*****:3128
          - name: NO_PROXY
            value: 172.20.0.1:443,169.254.169.254,.cluster.local

Start kibana.

Expected behavior:

Kibana starting with ingest manager enable.

Errors in browser console (if relevant):

Version: 7.8.0
Build: 31997
Error: Start lifecycle of "ingestManager" plugin wasn't completed in 30sec. Consider disabling the plugin and re-start.
withTimeout/</<@https://kibana-test.*******r/31997/bundles/commons.bundle.js:3:1153520

Provide logs and/or server output (if relevant):

Kibana server logs:

"Error connecting to package registry: request to https://epr-experimental.elastic.co/search?package=endpoint&internal=true failed, reason: connect ETIMEDOUT 151.101.122.217:443"}

Any additional context:

I deployed this kibana instance in a network environment who does not allow internet connection without proxy settings.
I tried the url with curl and the proxy and it works.

Thanks in advance.

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[kkh-security-distractions]

working in the banking industry. There is no chance of allowing direct connections from Kibana to the web.

Please fix this

[ruflin]

@kkh-security-distractions Can you share a bit more details on what the requirements for your environment are? Does it just need to go through a proxy or does it need to run inside your environment? The more details the better.

[tomrade]

I think the basics are they want the kibana server process to talk to the package registry via a proxy server ( they have set this in their container env)
Im here with the same issue. We have a security monitoring environment with no direct internet and have to use a http(s) proxy for all outbound requests.

I can see a change to add proxy support for alerting actions here ( however im guessing this will just be actions right now)
#74289

[MacPower]

@kkh-security-distractions Can you share a bit more details on what the requirements for your environment are? Does it just need to go through a proxy or does it need to run inside your environment? The more details the better.

Hello,

Yes we need to through a proxy server for every external HTTP requests.
So basically we just need to talk with the package registry via a proxy to be able to download the required package.

I use environnement proxy variables in my container, I wish them to be used for the artifacts download. It can also be a specific parameter in the Kibana configuration file, it does not matter.

Without the ability to use a proxy server, I think many companies like bank (I am in telecom) will not be able to use those nice features of Ingest Manager.

I hope I have been cleared, fell free to ask if I wasn't.

[iorfix]

I have two different scenarios:

1. in lab environment, all outside communications are proxied
2. in production environment, direct outside communications are not allowed at all. Internal repositories (Redhat Satellite) are used for RPM images.
   I agree with @MacPower concerns.

[Grunticus03]

I can confirm this is an issue with Kibana installed on Windows Server 2012 R2. We have a production environment where all 80/443 traffic must go through a web proxy. I've run into this problem with a variety of the Elastic Stack applications. It's very surprising that, for an enterprise targeted application, this isn't something that is baked into the advanced settings page or at the very least, a setting in the app config file.

[iorfix]

I see there is the property xpack.ingestManager.registryUrl used to specify a different registryUrl. Is there a documented way to have a local registryUrl, and in what this should be different from the default one? How to enroll packages into the registry?

[ruflin]

@iorfix This is undocumented by design for now as at the moment we only use this for testing. You find a bit more on this here but keep in mind, this is not supported.

[tomrade]

I wonder if we could use that URL with a proxy pass or similar in Apache+NGINX to then redirect out to the net

[MacPower]

I wonder if we could use that URL with a proxy pass or similar in Apache+NGINX to then redirect out to the net

I thought about that, like a man in the middle attack, with a proxy forwarder. I did not had the time to try also this is https, problem with certificate can occur with https.

[niempy]

we will try the same.
+1 if this is solved. We have multiple customers and also hosting a multi tenant platform. All behind proxies :(

[akshat5195]

Hi,

Is this issue resolved yet?
We are trying to open Ingest Manager via Kibana.
We have internet access only via proxy
getting below mentioned error
Error connecting to package registry at https://epr-7-9.elastic.co/search?package=endpoint&internal=true&experimental=true&kibana.version=7.9.1: request to https://epr-7-9.elastic.co/search?package=endpoint&internal=true&experimental=true&kibana.version=7.9.1 failed, reason: getaddrinfo ENOTFOUND epr-7-9.elastic.co epr-7-9.elastic.co:443

[ruflin]

@akshat5195 No, the issue is still open. We will close it when it is completed.

[akshat5195]

Do we have any ETA? It will be in the same release (7.9.1)

[ruflin]

We have now an open PR to add proxy support here: #78648 Would be great if some contributors on this issue could have a look at the PR to see if that solves their current issue.

[wolframhaussig]

@ruflin I was unable to find the artifact in the referenced pull request. Is there a location where I can download it? I only found the Typescript files in the PR and I seem to need the transpiled Javascript files

[jfsiii]

@wolframhaussig Until the PR merges, you'll have to checkout the PR branch, then build Kibana locally and yarn start with one of the environment variables mentioned in the description.

After it merges (and some build/publish delay) I believe it'll be available as a SNAPSHOT image on https://artifacts-api.elastic.co/v1/search/8.0-SNAPSHOT/kibana