Allow HTTP metrics to run in bootstrap mode. Add ability to adjust timeouts for Fleet Server.

[blakerouse]

What does this PR do?

It allows the metrics endpoint to run during Fleet Server bootstrap mode. Adds timeouts (including negative for indefinite) for waiting on the Elastic Agent daemon and the Fleet Server bootstrap process.

Why is it important?

This is needed by Cloud to allow it to check the status of the Elastic Agent even when Fleet Server cannot complete bootstrap process. Cloud will set the timeout to be indefinite and the system will only check every 10 mins after the exponential backoff to see if it should continue.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes [elastic-agent] Elastic Agent shuts down when Fleet Server is unhealthy #28209

[mergify]

This pull request does not have a backport label. Could you fix it @blakerouse? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 86 min 52 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

[simitt]

@andresrc are you ok with backporting this as a fix to 7.15?

[simitt]

Generally looks good to me. I also want to test on ECE though, if you could hold back with merging until then.

[blakerouse]

/package

[jlind23]

@simitt @blakerouse did you have a chance to test it yet?

[simitt]

I tested and created elastic/fleet-server#763 as a follow up as the observed behavior was not quite the expected one, and the agent/fleet-server were very noisily logging the same errors.
Also, with the changes in this PR, the agent would always also return a fleet-server process with a pid in the /processes response, althought the fleet-server is not healthy.

[blakerouse]

I have the fix for elastic/fleet-server#763 here elastic/fleet-server#768. That will provide the behavior we need for this to work properly.

[blakerouse]

/package

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b http-metrics-in-bootstrap upstream/http-metrics-in-bootstrap
git merge upstream/master
git push upstream http-metrics-in-bootstrap

[simitt]

I retested with the fleet-server fix, and the agent and fleet-server work as expected now on cloud. The healthcheck endpoint is immediately exposed, the container is considered healthy, while fleet-server is still trying to start up. The agent returns status: STARTING for the fleet-server.

fleet-server still logs every ~5sec that it is waiting for the policy, but the agent logging is pretty silent.