[Ingest Manager] When an integrations is uninstalled we should invalidate the cache

[ph]

If integration is uninstalled, Kibana wipes also the cache so it can be freshly installed again from the registry (which might contain a new version).

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@skh Can you take a look to see if this still an issue? The problem was that if you ever installed a package and you try to install a new one If would pick up the cached version instead of the newly available package form the registry.

[skh]

I've tested this with a locally running package registry, with a package foo in versions 1.0.0 and 1.0.1.

Version 1.0.0 of this package is available in testdata/package in the package-registry repository, version 1.0.1 was created by me on the fly by copying the package and changing the directory name and the version in the manifest.yml file to 1.0.1.

After adding the new package, only the registry was restarted, not kibana.

UI

Installing the package, then adding a new version of a package and restarting the package registry results in the following UI behaviour:

• The integration details page shows that an update to the integration is available, and offers a button to update the integration:
  
• After uninstalling the old version of integration in the UI, while still on the details page of the old version, the "Install assets" option is not offered:
  
• While still on the details page of the now uninstalled old version, clicking the "Add Foo" button in the upper right in order to install the integration and immediately add it to a configuration installs the correct, recent version, but without announcing to the user that a newer version was available and used.
• Navigating away from the details page of the uninstalled old version and back to the integration details page only shows the recent version, and clicking "Install Foo assets" installs the recent version of the integration:
  

I believe the UI is behaving correctly. It does silently switch to a newer version in one case (written in bold above), but I think that happens intentionally.

API

Installing the package, then adding a new version of a package and restarting the package registry results in the following API responses:

• curl -u "elastic:changeme" -X GET http://localhost:5601/api/ingest_manager/epm/packages/foo-1.0.0 works as before, it returns information about the installed package. The response contains a latestVersion field that indicates that there is a newer package available.

• curl -u "elastic:changeme" -X DELETE http://localhost:5601/api/ingest_manager/epm/packages/foo-1.0.0 -H "kbn-xsrf: lalala" correctly deletes the package

• Trying to install the old version with curl -u "elastic:changeme" -X POST http://localhost:5601/api/ingest_manager/epm/packages/foo-1.0.0 -H "kbn-xsrf: lalala" results in the error {"statusCode":400,"error":"Bad Request","message":"Cannot install or update to an out-of-date package"}

• Installing the new version with curl -u "elastic:changeme" -X POST http://localhost:5601/api/ingest_manager/epm/packages/foo-1.0.1 -H "kbn-xsrf: lalala" successfully installs the package.

• It is not possible to install the package without giving the version: curl -u "elastic:changeme" -X POST http://localhost:5601/api/ingest_manager/epm/packages/foo -H "kbn-xsrf: lalala" results in the error {"statusCode":500,"error":"Internal Server Error","message":"Invalid Version: undefined"}. This error could be improved and return a 400 status pointing out that the version must be given, instead of a 500.

Overall, I think the API behaves correctly as well.

---

@ph do you remember what you did exactly to trigger this behavior?

[skh]

Closing, please reopen if the problem can still be reproduced.

[ruflin]

@skh The question was more about installing the same version multiple times. So if I install foo-1.0.0, delete foo-1.0.0 and install foo-1.0.0 again, will it use any locally cached data / files or not? This is for development purpose to be able to quickly iterated on a package and making sure no local caches are left over if a package is deleted.

[skh]

@ruflin Thanks for the added information, reopening.

[skh]

@ruflin would it be acceptable to make the cache deletion happen when force:true is set on the POST request for the installation, instead of when a package is uninstalled?

I'm not sure it's beneficial to always remove the package from the cache, as packages in the registry shouldn't change their content without a version bump.

[ruflin]

I personally prefer to have it on uninstall as I think we should "clean up" the system if a package is uninstalled. Having it on force too makes a lot of sense.

I don't see much use of keeping the cache around if a package is removed. If someone tried out 10 packages and then only works with 2, he should not still need memory for the other 8.

[jfsiii]

@skh @ruflin I created #76306 to expose the function that would support this.

We could expand it to include using it in a route handler service, or do that in a separate PR. I just wanted to provide the function to call.

[ruflin]

@jfsiii Great to have it. Would be nice to have a follow up PR, it will help package development.

[skh]

@jfsiii Thanks! I'll open a PR to use it.