[Fleet] Use epm-packages saved object as main source of truth for installed packages

[skh]

Currently, when a list of packages is requested, roughly the following happens:

• Get list of all available packages from the registry
• Get saved objects of type epm-packages
• Add the property “installed: (false|true)” to each package in the registry response
• Return the list of packages for further handling

This flow needs to be changed so that packages from all sources are considered:

• Get saved objects of type epm-packages
• Get list of available packages from the registry
• Get list of available packages from ES package storage index (once it exists, see the section below this)
• Merge these lists in the correct way (whatever this is, sometimes we only want installed ones, sometimes all)
• Return the list of packages for further handling

In addition, whenever information about one package is requested, this should check the saved object first to find out the installation source, and then collect and return the package info from the corresponding place.

In this case EPM API currently acts as a proxy for the registry as well, so it is possible to look up the package info for an uninstalled package. We’ll probably want to keep that behaviour.

This has effects on other areas of Fleet, which need to be tested and adjusted if necessary:

• UI to list available and installed integrations (Additional question here: do we want to visibly point out the installation source, e.g. to list uploaded integrations in a separate tab?)
• Agent policies: which integrations are offered to be added to a policy
• Every package operation: regeneration of Kibana index patterns looks for installed packages
• possibly more

---

Possible scenarios (support status for 7.11 needs to be defined)

An uploaded package does not exist in the registry

• Use case: Developer is working on a brand-new package and uploaded it for testing
• Use case: User has created and uploaded a custom package
• This package should be shown in the list of installed packages
• The package details page for this package should mention that this is an uploaded package

An uploaded package does exist in the registry

• Uploaded package version is older than the registry package
  • Use case: User has downgraded the package manually to fix a problem (e.g. because the original package has disappeared, and the current one has a bug)
  • The package should be shown in the list of installed packages
  • The package details page for this package should mention that this is an uploaded package and that a newer version exists in the registry, and offer to upgrade to the registry version
• Uploaded package version is the same as the registry package
  • Use case: User has fixed a problem in the package and uploaded a custom version
  • Use case: Developer / support agent has provided the user with a custom version to fix a problem
  • The package should be shown in the list of installed packages
  • The package details page for this package should mention that this is an uploaded package and offer reinstallation of the registry package
• Uploaded package version is newer than the registry package
  • Use case: Developer is working on a newer version and uploaded it for testing
  • Use case: Developer / Support has provided the user with a newer version to fix a problem
  • The package should be shown in the list of installed packages
  • The package details page for this package should mention that this is an uploaded package and offer reinstallation / downgrade of the registry package

[elasticmachine]

Pinging @elastic/ingest-management (Feature:Fleet)

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jfsiii]

Linking #70860 and #66777 for awareness

[neptunian]

@skh The reason why we added the add the property “installed: (false|true)” to each package is because the requirement was to not fetch the version they have installed, but the latest version available, and adding the installed attribute was a way of indicating they have some version of this package installed and and it may not be the latest, and they need to update it. This behaviour might be a bit unexpected, as clicking on an installed integration means possibly not looking at the integration version you have installed and instead looking at the latest, but that was the requirement. So in regards to the installed integrations list page, we'd have to keep some of this logic, unless we want to change it to actually show/link them to the versions they have installed. With the need to find the latest up to date package in order to check for upgradeability, we'd need to check all sources for the most up to date version of the package and fetch those, not the version they have installed in the saved object.

[skh]

@ruflin Can you comment?

[skh]

@ruflin @neptunian I have added a list of scenarios and possible behavior to the initial description, as a starting point for discussion.

[neptunian]

In my previous comment, I had it a bit backward. We do send the user to the version they have installed, however to get the latest version available we do still need to fetch for the latest packages because in the list, it was desired that we use the latest version's logo (probably will not change from version to version much) but send them to the version they have installed. We were also going to have some kind of "update" notification on the logos themselves in the list, so we would need to know the latest version and the version they have installed.

[skh]

@neptunian Totally agreed, I don't see a reason not to fetch the latest versions from the registry.

[ruflin]

Proposal LGTM. Could we simplify it at first but not allowing to upload a package with the same or older version and just reject it? Like this we only have to take care of the case of a newer package for now. Will this simplify things or is the effort the same anyways?

[neptunian]

Proposal LGTM. Could we simplify it at first but not allowing to upload a package with the same or older version and just reject it? Like this we only have to take care of the case of a newer package for now. Will this simplify things or is the effort the same anyways?

@ruflin I think it would be much easier to only have this case. And I think this case could be used to cover the use cases of the others, for the most part.

I have a general question regarding covering all the above use cases: How much do we expect the use cases listed above where we are going to have Support send packages to users to fix broken packages? If this is mainly for developers, it seems like it'd be simpler to not allow uploaded packages to upgrade, downgrade (we don't currently allow downgrading of any packages) and reinstall to and from registry packages. Basically to keep them separate from each other to the user. That means uploaded packages could not have the same name as a package in the registry. They could upload test packages and test upgrading and force downgrading and reinstalling them, but they don't overlap with registry packages. I mention this because I think there's quite a bit of complexity that will be introduced when they overlap and I'm not sure how necessary that is. If we are only going to cover the case of uploading new packages as @ruflin mentioned, it is not as complex.

Otherwise a few issues I want to mention:

From what I understand an uploaded package is automatically an installed package. Which I assume would mean that once a user upgraded/downgraded/uninstalled that package, it will be deleted from the storage index. So the user has to keep uploading the test package after it performs an action like this?

If the uploaded package is automatically installed, it would mean multiple versions of an uploaded package could not exist at the same time because a user can only have one package installed at a time. It would be nice if the user could uploaded multiple test packages (that are not automatically installed), and test upgrading between them. I don't think this is possible if they can only have one version of an uploaded package at a time. They would only be able to test upgrading from a registry package.

I can't recall why an uploaded package has to be installed. Is it possible for it to exist in an index of uploaded packages (where multiple versions could exist) to be installed and removed but still remain in the index? Though this would require we provide a way to delete it from the index. This index could be separate from the index where we store the installed packages.

In reference to the above use cases that @skh listed, it mentions downgrading, we don't currently support downgrading of packages, but it looks like we are wanting to introduce downgrading in the event a user uploaded a newer package than the one in the registry or in the event that a user wants to downgrade to a package they uploaded. Perhaps we should just get rid of this restriction.

[ruflin]

I hope the case where we provide a user with a custom package will be very rare. Instead what I expect is that we rollout a new version of the package quickly or let the user downgrade. So the experience is mainly focused on developers for now and users want to upload their own packages which should not conflict with the registry.

I'm putting aside the non conflict package as I assume this will "just work". For the conflicting package, I wonder if we can simplify it. Lets take the case of nginx development:

• nginx 1.2.3 was released in the registry
• nginx 1.2.4 the developer is working on
• nginx 1.2.2 is also already released and available in the registry

Now first, the developer wants to test if 1.2.4 works. Local setup is used and nginx 1.2.4 is uploaded and tested. No interaction with the registry. Next, the upgrade should be tested from 1.2.3. The dev installs 1.2.3 from the registry and then "uploads" 1.2.4 to have the migration to it tested. Now they also want to test from 1.2.2 to 1.2.4, same flow is used. If the dev also wants to test downgrade, they downgrade to 1.2.3 and the 1.2.4 package is removed. To upgrade again, it must be uploaded again. All the automation for this is on the developer side and not inside Kibana. In Kibana we always only have a single version of package available. As soon as we have local storage of package, things look a bit different. But then again it behaves the same as if these packages would be available in the registry.

[skh]

I can't recall why an uploaded package has to be installed. Is it possible for it to exist in an index of uploaded packages (where multiple versions could exist) to be installed and removed but still remain in the index? Though this would require we provide a way to delete it from the index. This index could be separate from the index where we store the installed packages.

The original feature #70582 is to install a package by direct upload.

Yes, we could separate upload and installation, and offer a way for the user to manage their uploaded packages and choose which one(s) to install, but that would be a different feature.

That means uploaded packages could not have the same name as a package in the registry. They could upload test packages and test upgrading and force downgrading and reinstalling them, but they don't overlap with registry packages. I mention this because I think there's quite a bit of complexity that will be introduced when they overlap and I'm not sure how necessary that is. If we are only going to cover the case of uploading new packages as @ruflin mentioned, it is not as complex.

I would prefer not to include a restriction on the package names. The saved object epm-packages already has a property install_source so it is always known which source the installed package came from.

[skh]

I hope the case where we provide a user with a custom package will be very rare.

I think this is one of the more important aspects of this functionality, it will increase the options we have to react to problems in production environments.