Elastic Agent shouldn't log httpjson pagination completion as an Agent Error

[agmic]

I am not sure if this should be raised against Elastic Agent or the integration. Raising it here, but please move as necessary.

The Cisco Secure Endpoint integration (v2.24.2) collects event data via httpjson. The integration throws an error when it hits the last page, and this gets logged in the Elastic Agent error log with something similar to:
error processing response: template: :1:17: executing "" at <.last_response.body.metadata.links.next>: map has no entry for key "next"
This fills up the Elastic Agent dashboard with agent errors, as it is raised every time the Secure Endpoint data is polled (every 2 minutes/default) and the noise makes it more difficult to find actual errors.

I believe that this is occurring because fail_on_template_error is set to true.

The fail condition prompting this is at
integrations/packages/cisco_secure_endpoint/data_stream/event/agent/stream/httpjson.yml.hbs at main · elastic/integrations (github.com) line 36-40

response.pagination:
- set:
    target: url.value
    value: '[[ .last_response.body.metadata.links.next ]]'
    fail_on_template_error: true

I'm not sure how this should be fixed, because AIUI fail_on_template_error: true is the recommended configuration for pagination, but raising an error should maybe be reconsidered?

For confirmed bugs, please report:

• Version:
  Cisco Secure Endpoint integration v2.24.2
  Elastic Agent v1.18.0
  Elastic 8.11.2
• Operating System:
  Agent running on Windows Server 2019 Standard
• Discuss Forum URL:
  https://discuss.elastic.co/t/elastic-agent-shouldnt-log-httpjson-pagination-completion-as-an-error/353565
• Steps to Reproduce:
  This should happen on any run of the Secure Endpoint integration

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

Related to #34544.