[Fleet] Support bundling packages as zip archives with Kibana source

[kpollich]

Summary

Ref #112095

Adds baseline support for bundling packages as .zip archives in the Kibana source code and installing them during setup.

In this first pass implementation, we're looking to satisfy a few requirements:

• Allow "stack-aligned" packages like APM and Synthetics to ship their package assets with Kibana instead of depending on EPR
• Add stability to Fleet's setup by reducing (and eventually removing in [Fleet] Make default integration install explicit #121628) its dependency on an accessible EPR service

This PR does the following in pursuit of the goals above:

• Adds a new build task that fetches a set of packages from the Snapshot EPR service and downloads them to a gitignored fleet/server/bundled_packages directory
  • apm
  • elastic_agent
  • endpoint
  • fleet_server
  • synthetics
• Adds a "preferred source" option to the bulkInstallPackages method that allows us to prefer installation from bundled packages on disk for preconfigured packages during setup

Testing Instructions

Before testing, run a yarn build to fetch bundled packages. You can isolate this task by commenting out other tasks in src/dev/build/build_distributables.ts.

To test, preconfigure an agent policy in your kibana.yml file that contains package_policies for each of the bundled packages, e.g.

# Packages still need to be included here in order to be installed during preconfiguration, we'll just
# always attempt to install them from disk during setup if possible
xpack.fleet.packages:
  - name: apm
    version: latest
  - name: synthetics
    version: latest

xpack.fleet.agentPolicies:
  - name: My preconfigured policy
    description: A preconfigured policy that contains all bundled packages - for testing!
    id: my-preconfigured-policy

    is_default: true
    is_default_fleet_server: true

    namespace: default

    package_policies:
      # APM and Synthetics will install from disk, not the registry
      - name: apm-1
        id: apm-1
        package:
          name: apm
      - name: synthetics-1
        id: synthetics-1
        package:
          name: synthetics

      # Elastic Agent and Fleet Server are default packages, so these are okay even though
      # we don't have anything in `xpack.fleet.packages`
      - name: elastic_agent-1
        id: elastic_agent-1
        package:
          name: elastic_agent
      - name: fleet_server-1
        id: fleet_server-1
        package:
          name: fleet_server

      # Endpoint won't be installed at all since it's not specified during preconfiguration, but
      # it is available on disk.

With the above setup, you should see log output along the lines of the following:

Setting up initial Fleet packages
kicking off bulk install of apm, synthetics, system, elastic_agent, fleet_server with preferred source of bundled
kicking off install of synthetics-0.7.0 from bundled package on disk
kicking off install of fleet_server-1.1.1 from bundled package on disk
kicking off install of apm-8.1.0-dev1 from bundled package on disk
kicking off install of system-1.7.0 from registry
kicking off install of elastic_agent-1.3.0 from bundled package on disk

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[kpollich]

This should be blocked by #121628, and we'll likely want to bundle the Fleet Server and Elastic Agent packages with the Kibana source as part of this PR.

[kpollich]

@elasticmachine merge upstream

[kpollich]

@elasticmachine merge upstream

[kpollich]

@elasticmachine merge upstream

[kpollich]

@elasticmachine merge upstream

[mtojek]

Why is it https://epr-snapshot.elastic.co, which is considered to be unstable, rather than production endpoint https://epr.elastic.co?

[kpollich]

Per @simitt, we're using the snapshot registry here to support bundling "in progress" development builds of packages like APM during development/testing, and because the APM package specifically isn't usually published ahead of time.

Maybe it'd be a better idea to pass a CLI flag here to control the EPR environment? yarn build --use-snapshot-epr or something along those lines to support development needs, but otherwise preferring the production environment. I'm not sure if this is suitable for APM's use case, though, so I'll let @simitt correct me if not.

[simitt]

Changing default to fetch from production, but supporting a parameter to fetch from staging or snapshot SGTM. I believe it is important to be able to package non-production dev packages during the dev cycle, but I also see why people are hesitant to always fetch from a non-production stage.

[kpollich]

I've implemented this in 61b8178 and included a note for a caveat as well that I'll reiterate here.

When using the snapshot registry, all checksums provided in fleet_package.json will no longer be considered valid because of differences in the package stored on the snapshot registry and those in production. It's also possible that a version that exists in production does not exist in snapshot - e.g. apm-8.0.0. Because of this, it's necessary to remove other bundled packages when using --use-snapshot-registry to test bundling of development packages to prevent build failures.

I think this should be fine to support use cases like local development/testing, and we'll leave the production versions and checksums (updated in the same commit above) checked in to the Kibana repo. Just wanted to make it clear that simply opting in via that CLI flag will cause the build to fail without manual updates to fleet_packages.json.

[kpollich]

We're punting on checksums in this PR for now due to concerns around their consistency with regards to EPR builds. See #124273 for info.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: a55aec3
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1142	1143	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	109.2KB	109.2KB	+37.0B

Unknown metric groups

API count

id	before	after	diff
fleet	1258	1259	+1

History

• 💔 Build #21132 failed af817e6
• 💔 Build #21118 failed 19cfa34
• 💔 Build #21072 failed 89438eb
• 💔 Build #21034 failed f53c637
• 💛 Build #20810 was flaky 28b5936
• 💔 Build #20779 failed 61b8178

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

[kibanamachine]

💔 Backport failed

The pull request could not be backported due to the following error:
There are no branches to backport to. Aborting.

How to fix

Re-run the backport manually:

node scripts/backport --pr 122297

Questions ?

Please refer to the Backport tool documentation