Promoting integrations packages to snapshot #133

ycombinator · 2020-07-14T18:45:14Z

Promotes all available packages from elastic/integrations#176 to snapshot registry.

Script used:

find . -iname manifest.yml -depth 2 | sort -u | grep -v base | xargs grep '^version' | while read l; do package=$(echo $l | awk -F\/ '{print $2}'); version=$(echo $l | awk -F\: '{print $3}'); dir=$package/${version/ /}; mkdir -p ~/development/github/package-storage/packages/$dir;  cp -r $package/* ~/development/github/package-storage/packages/$dir/; done

ph · 2020-07-14T18:53:32Z

LGTM, waiting on CI https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fpackage-storage/detail/PR-133/1/pipeline

elasticmachine · 2020-07-14T19:01:59Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Started by user Shaunak Kashyap, Replayed #7]
Start Time: 2020-07-16T15:50:23.331+0000
Duration: 8 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22
Skipped	0
Total	22

ycombinator · 2020-07-14T19:04:16Z

Looks like CI failed because the Kibana docker container didn't come up: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fpackage-storage/detail/PR-133/1/tests

ycombinator · 2020-07-14T19:19:09Z

Hmmm, doesn't look like the CI failure is ephemeral. Investigating...

ycombinator · 2020-07-14T19:27:19Z

I re-ran the test command locally.

docker-compose -f snapshot.yml -f local.yml up --force-recreate --remove-orphans --build

I was able to reproduce the failure starting up the Kibana container. Checking the container's logs:

» docker logs 9f063e93fc0b
{"type":"log","@timestamp":"2020-07-14T19:23:43+00:00","tags":["warning","plugins-discovery"],"pid":9,"message":"Expect plugin \"id\" in camelCase, but found: beats_management"}
{"type":"log","@timestamp":"2020-07-14T19:23:43+00:00","tags":["warning","plugins-discovery"],"pid":9,"message":"Expect plugin \"id\" in camelCase, but found: triggers_actions_ui"}
[BABEL] Note: The code generator has deoptimised the styling of /usr/share/kibana/x-pack/plugins/canvas/server/templates/pitch_presentation.js as it exceeds the max of 500KB.
{"type":"log","@timestamp":"2020-07-14T19:24:26+00:00","tags":["fatal","root"],"pid":9,"message":"{ Error: [config validation of [xpack.ingestManager].epm]: definition for this key is missing\n    at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:62:19)\n    at ConfigService.validateAtPath (/usr/share/kibana/src/core/server/config/config_service.js:188:19)\n    at MapSubscriber.getDistinctConfig.pipe.config [as project] (/usr/share/kibana/src/core/server/config/config_service.js:196:81)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:76:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25)\n    at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Subject.js:102:51)\n    at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:30:22)\n    at MapSubscriber.shareReplayOperation (/usr/share/kibana/node_modules/rxjs/internal/operators/shareReplay.js:45:32)\n    at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n    at MapOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:32:23)\n  cause:\n   { Error: definition for this key is missing\n       at ObjectType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:87:20)\n       at type.Type.schema.error (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:52:53)\n       at finish (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:529:50)\n       at type._validate (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:618:24)\n       at type._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:674:29)\n       at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n       at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:57:72)\n       at ConfigService.validateAtPath (/usr/share/kibana/src/core/server/config/config_service.js:188:19)\n       at MapSubscriber.getDistinctConfig.pipe.config [as project] (/usr/share/kibana/src/core/server/config/config_service.js:196:81)\n       at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n       at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n       at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n       at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n       at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n       at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n       at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:76:28) cause: undefined, path: [ 'epm' ] } }"}

 FATAL  Error: [config validation of [xpack.ingestManager].epm]: definition for this key is missing

@ph @skh any ideas what's going on here? Maybe something in https://github.com/elastic/package-storage/blob/master/testing/environments/kibana.config.yml needs to be changed?

skh · 2020-07-14T19:33:52Z

Yes.

xpack.ingestManager.epm.enabled is gone (it's always enabled now).
xpack.ingestManager.epm.registryUrl has been renamed to xpack.ingestManager.registryUrl.

ycombinator · 2020-07-14T20:03:15Z

Kibana config being updated via #134 and #137.

ycombinator · 2020-07-14T21:50:07Z

CI failures seem relevant. Investigating...

ycombinator · 2020-07-14T22:42:41Z

I looked into why the tests are failing in CI. First I ran mage -debug testIntegration locally and it failed exactly in the same way that it's failing in CI. Some excerpts from the failure:

--- FAIL: TestSetup (118.98s)
    --- FAIL: TestSetup/install-packages (0.00s)
        --- FAIL: TestSetup/install-packages/system-0.5.0 (2.20s)
            main_integration_test.go:113:
                        Error Trace:    main_integration_test.go:113
                        Error:          Not equal:
                                        expected: 200
                                        actual  : 500
                        Test:           TestSetup/install-packages/system-0.5.0
        --- FAIL: TestSetup/install-packages/mysql-0.1.3 (2.21s)
            main_integration_test.go:113:
                        Error Trace:    main_integration_test.go:113
                        Error:          Not equal:
                                        expected: 200
                                        actual  : 400
                        Test:           TestSetup/install-packages/mysql-0.1.3
...

So next I spun up the same test environment as spun up by the mage testIntegration command:

docker-compose -f snapshot.yml -f local.yml up --force-recreate --remove-orphans --build

And then manually tried to execute the line in the test that was failing:

package-storage/testing/main_integration_test.go

Lines 102 to 113 in 8ac7650

    
           req, err = http.NewRequest("POST", "http://elastic:changeme@localhost:5601/api/ingest_manager/epm/packages/"+p, nil) 
        
           if err != nil { 
        
           	t.Error(err) 
        
           } 
        
           req.Header.Add("kbn-xsrf", "ingest_manager") 
        
           resp, err = http.DefaultClient.Do(req) 
        
           if err != nil { 
        
           	t.Error(err) 
        
           } 
        
           defer resp.Body.Close() 
        
           assert.Equal(t, 200, resp.StatusCode)

» curl -v -u "elastic:changeme" -H 'kbn-xsrf: ingest_manager' -X POST 'http://localhost:5601/api/ingest_manager/epm/packages/aws-0.2.0'


*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 5601 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Server auth using Basic with user 'elastic'
> POST /api/ingest_manager/epm/packages/aws-0.2.0 HTTP/1.1
> Host: localhost:5601
> Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==
> User-Agent: curl/7.64.1
> Accept: */*
> kbn-xsrf: ingest_manager
>
< HTTP/1.1 500 Internal Server Error
< kbn-name: kibana
< kbn-license-sig: 0a1db41c0e967fd9779bbfd96f19fa005a6b06b466ce863e095469b5ade9442f
< kbn-xpack-sig: ecd16da9370615908a145c6674799607
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 104
< Date: Tue, 14 Jul 2020 22:34:14 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
{"statusCode":500,"error":"Internal Server Error","message":"Cannot read property 'query' of undefined"}* Closing connection 0

» curl -v -u "elastic:changeme" -H 'kbn-xsrf: ingest_manager' -X POST 'http://localhost:5601/api/ingest_manager/epm/packages/mysql-0.1.3'


*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 5601 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Server auth using Basic with user 'elastic'
> POST /api/ingest_manager/epm/packages/mysql-0.1.3 HTTP/1.1
> Host: localhost:5601
> Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==
> User-Agent: curl/7.64.1
> Accept: */*
> kbn-xsrf: ingest_manager
>
< HTTP/1.1 400 Bad Request
< kbn-name: kibana
< kbn-license-sig: 0a1db41c0e967fd9779bbfd96f19fa005a6b06b466ce863e095469b5ade9442f
< kbn-xpack-sig: ecd16da9370615908a145c6674799607
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 103
< Date: Tue, 14 Jul 2020 22:34:42 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
{"statusCode":400,"error":"Bad Request","message":"Cannot install or update to an out-of-date package"}* Closing connection 0

Still investigating what's going on but I don't know these waters too well. @ph @skh if the above rings any bells for you, please let me know!

ph · 2020-07-14T22:44:45Z

@neptunian is that message coming from the refactor of the installation?

ycombinator · 2020-07-14T22:47:03Z

@ph I wonder if it has anything to do with the fact that all integrations packages (except system) are now experimental and this line in the integration test was not asking for experimental packages from the registry:

package-storage/testing/main_integration_test.go

Line 131 in 8ac7650

resp, err := http.Get("http://localhost:8080/search")

As a test I've added ?experimental=true to that line in f27dafd. Let's see if that helps get rid of the errors for non-system packages at least.

ycombinator · 2020-07-14T22:55:15Z

Looks like with f27dafd things are a bit better. Previously we had 10 failing tests in CI, now we have 8.

Looking at each of the failures, we got rid of all the 400 errors. Now we have a bunch of 500 errors (like we did in the previous run with the system package) instead 😂.

ycombinator · 2020-07-15T00:26:29Z

testing/main_integration_test.go

@@ -128,7 +128,7 @@ type Package struct {
 }

 func getPackages(t *testing.T) ([]string, error) {
-	resp, err := http.Get("http://localhost:8080/search")
+	resp, err := http.Get("http://localhost:8080/search?experimental=true")


@ph Does this change make sense to you?

ycombinator · 2020-07-15T00:35:57Z

I'm able to reproduce the latest 500 errors locally as well.

$ cd testing/environments
$ docker-compose -f snapshot.yml -f local.yml up --force-recreate --remove-orphans --build

$ curl -v -u "elastic:changeme" -H 'kbn-xsrf: ingest_manager' -X POST 'http://localhost:5601/api/ingest_manager/epm/packages/nginx-0.2.0'
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 5601 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Server auth using Basic with user 'elastic'
> POST /api/ingest_manager/epm/packages/nginx-0.2.0 HTTP/1.1
> Host: localhost:5601
> Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==
> User-Agent: curl/7.64.1
> Accept: */*
> kbn-xsrf: ingest_manager
>
< HTTP/1.1 500 Internal Server Error
< kbn-name: kibana
< kbn-license-sig: 91c2e97c925a585852cdfa94a54de4224940abd2eae669bff92ea801713b9a9b
< kbn-xpack-sig: f863b5537e5d51a0545642252d273ca1
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 104
< Date: Wed, 15 Jul 2020 00:30:21 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
{"statusCode":500,"error":"Internal Server Error","message":"Cannot read property 'query' of undefined"}* Closing connection 0

This is the same error we are getting with the above API call for the following packages:

aws-0.2.0
cisco-0.2.0
kafka-0.2.0
netflow-0.2.0
nginx-0.2.0
system-0.5.0

However, the same API call succeeds for the following packages:

apache-0.1.0
endpoint-0.9.0
iis-0.1.0
kubernetes-0.1.0
log-0.2.0
mongodb-0.1.0
mysql-0.2.0
postgresql-0.1.0
prometheus-0.1.0
rabbitmq-0.1.0
redis-0.2.0
snapshot-0.0.1
windows-0.1.0
zookeeper-0.1.0

skh · 2020-07-15T16:13:39Z

{"statusCode":500,"error":"Internal Server Error","message":"Cannot read property 'query' of undefined"}

This error comes from this method: https://github.com/elastic/kibana/blob/master/src/plugins/discover/server/saved_objects/search_migrations.ts#L24

It happens on searches that contain a searchSourceJSON that contains unescaped JSON.

I currently happens only on searches, because the piece of code that crashes when it encounters these searchSourceJSON instances only runs on searches.

It only happens now because the migration in question has been added quite recently.

The underlying questions are, with the amount of unescaped searchSourceJSON occurences we have in all kibana assets, all inherited from beats modules

were these assets broken for years?
did libbeats do some processing on these assets?
did Kibana behavior become stricter?

ycombinator · 2020-07-15T16:20:27Z

Thanks so much for digging into this error @skh. To try and narrow down the surface area of what might've changed to cause this error now, I'm going to test these packages with an older Kibana.

ycombinator · 2020-07-15T17:32:16Z

I tried running the same curl command as before with Kibana 7.8.0 and got back the same 500 error response. So this indicates that it's not a recent change in Kibana that is causing the failures here.

ycombinator · 2020-07-15T17:54:18Z

One of the packages that's failing the tests is aws. This package exists in the production branch so I spot-checked various Kibana asset JSON files there. They are all using escaped searchSourceJSON values.

However, looking at the same files in the integrations repo, searchSourceJSON is not escaped there.

So it appears that somehow when the packages were copied from the integrations repo to the package-storage repo in the past, escaping was being done as part of that process.

Digging around some more I found this PR: #66. Indeed, one of the changes in there is to "encode dashboards" and you can clearly see the escaping (aka encoding) of searchSourceJSON and other kibana object fields in the diff. Digging some more I found elastic/package-registry#506, which modifies the generator to do the encoding, as necessary.

So we need to do the same encoding/escaping in this PR here now.

ycombinator · 2020-07-15T19:21:49Z

Thanks to @ph and @jonathan-buttner for pointing me to https://github.com/elastic/integrations/blob/master/magefile.go#L253. I ran that target and copied over the generated files (minus the base package) to this PR: 4005c0c.

ycombinator · 2020-07-15T19:43:07Z

Hmmm, now we're up to 21 CI test failures 😞 . Investigating...

ycombinator · 2020-07-15T19:53:36Z

Regarding the new 500 errors, they are happening because of this underlying error:

{"statusCode":500,"error":"Internal Server Error","message":"[x_content_parse_exception] [1:2197] [data_stream_template] unknown field [timestamp_field]"}* Closing connection 0

@skh pointed me to elastic/kibana#71727, which was merged ~4 hours ago. It is likely the latest 8.0.0-SNAPSHOT Docker image of Kibana does not yet contain the changes from this PR, which is why we're seeing these errors.

I will try again in ~24 hours.

ycombinator · 2020-07-16T15:50:12Z

New build of Kibana is available. Re-running CI job...

The code to generate example packages was removed some time ago but the docs were forgotten. This cleans it up.

ycombinator requested review from ph and a team July 14, 2020 18:48

This was referenced Jul 14, 2020

Updating kibana config #134

Merged

Updating kibana config elastic/integrations#177

Merged

Promoting integrations packages to snapshot

9e2ba06

ycombinator force-pushed the promote-integrations-packages branch from de786d2 to 9e2ba06 Compare July 14, 2020 20:41

Query experimental packages from registry

f27dafd

ycombinator commented Jul 15, 2020

View reviewed changes

Including changes from elastic/integrations#166

22893f3

Fixing encoding in Kibana assets

4005c0c

ph approved these changes Jul 16, 2020

View reviewed changes

ycombinator merged commit a7a308b into elastic:snapshot Jul 16, 2020

ycombinator deleted the promote-integrations-packages branch July 16, 2020 19:45

thomasneirynck pushed a commit to thomasneirynck/package-storage that referenced this pull request Sep 28, 2021

Remove generate example package docs (elastic#133)

9c8a57e

The code to generate example packages was removed some time ago but the docs were forgotten. This cleans it up.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Promoting integrations packages to snapshot #133

Promoting integrations packages to snapshot #133

ycombinator commented Jul 14, 2020

ph commented Jul 14, 2020

elasticmachine commented Jul 14, 2020 •

edited

Loading

Build stats

Test stats 🧪

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

skh commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ph commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020 •

edited

Loading

ycombinator Jul 15, 2020

ycombinator commented Jul 15, 2020 •

edited

Loading

skh commented Jul 15, 2020

ycombinator commented Jul 15, 2020 •

edited

Loading

ycombinator commented Jul 15, 2020 •

edited

Loading

ycombinator commented Jul 15, 2020

ycombinator commented Jul 15, 2020 •

edited

Loading

ycombinator commented Jul 15, 2020

ycombinator commented Jul 15, 2020

ycombinator commented Jul 16, 2020

Promoting integrations packages to snapshot #133

Promoting integrations packages to snapshot #133

Conversation

ycombinator commented Jul 14, 2020

ph commented Jul 14, 2020

elasticmachine commented Jul 14, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

skh commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ph commented Jul 14, 2020

ycombinator commented Jul 14, 2020

ycombinator commented Jul 14, 2020 • edited Loading

ycombinator Jul 15, 2020

Choose a reason for hiding this comment

ycombinator commented Jul 15, 2020 • edited Loading

skh commented Jul 15, 2020

ycombinator commented Jul 15, 2020 • edited Loading

ycombinator commented Jul 15, 2020 • edited Loading

ycombinator commented Jul 15, 2020

ycombinator commented Jul 15, 2020 • edited Loading

ycombinator commented Jul 15, 2020

ycombinator commented Jul 15, 2020

ycombinator commented Jul 16, 2020

elasticmachine commented Jul 14, 2020 •

edited

Loading

ycombinator commented Jul 14, 2020 •

edited

Loading

ycombinator commented Jul 15, 2020 •

edited

Loading

ycombinator commented Jul 15, 2020 •

edited

Loading

ycombinator commented Jul 15, 2020 •

edited

Loading

ycombinator commented Jul 15, 2020 •

edited

Loading