[Agent] Agent upgrade attempt throws error on .asc files when using the 7.10 BC1 (or BC2) Agent and a 7.11 snapshot Kibana on cloud

[EricDavisX]

commit: 7.11 snapshot cloud install of whole stack, with 7.10 BC1 Agent
edavis-mbp:kibana_elastic edavis$ git show -s 29426f57db725bedc29ee9d7adb5efd2aa2f1983
Date: Mon Oct 12 12:35:00 2020 +0200

Agent hash 116336aa, the BC1 build from : elastic-agent-7.10.0-linux-x86_64.tar.gz
from the elastic /dev/issues/1496 release repo.
7.10.0-116336aa
https://staging.elastic.co/7.10.0-116336aa/downloads/beats/elastic-agent/elastic-agent-7.10.0-linux-x86_64.tar.gz

Browser version:
Chrome on macOS

Describe the bug:
Attempting to upgrade the Agent via the API almost works... it initiates the upgrade and downloads the new Agent and then fails on a .asc / gpg check. Michal says he was looking into this... I think. To be safe and sure I'm logging it.

Details of how to test are here, see details in main 'upgrade' Agent e2e-testing issue, here:
elastic/e2e-testing#341 (comment)

screenshot:

Error returned in Kibana Fleet Agent Activity Log:
Application: [90872928-e61d-43d6-95c8-d5181095588f]: State changed to FAILED: failed verification of agent binary: 2 errors occurred: * fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-47b32e/downloads/elastic-agent-7.11.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-47b32e/downloads/elastic-agent-7.11.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory * check detached signature: openpgp: invalid signature: hash tag doesn't match

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[EricDavisX]

I believe... Michal said he was looking at this, but not sure. He tried and didn't see this exact bug in 8.0 Kibana, so I was wondering if we needed to ensure that everything that was needed is backported to 7.10

[ph]

@michalpristas Can you update this issue with the status?

[EricDavisX]

I'm still seeing this bug today with the newer 7.10 BC2 Agent (on a newer 7.11 snapshot of Kibana on cloud). I'm wondering... has anyone seen this work from cloud yet? Maybe that's the nuance to resolve.

To add to details: I checked the file location on my test host against the error shown in the UI, and the .asc file isn't there as noted, but the .tar.gz file isn't there either. Meaning: Agent isn't downloading anything related to the api call made (at least not to this file location). I checked the error from the UI based test case and the Agent throws a 404 error when trying to download the new Agent file(s). In this case, it doesn't...implying that it found the file when starting the download. I don't knwo how to trace where they may have ended up such that the Agent then doesn't have them in the right disk location. them and

The Agent log indicates:

{"log.level":"debug","@timestamp":"2020-10-15T14:24:11.972-0400","log.origin":{"file.name":"application/handler_action_upgrade.go","file.line":24},"message":"handlerUpgrade: action 'action_id: ee74b995-f001-4ce6-913a-01481fa5eb4d, type: UPGRADE' received","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-10-15T14:24:11.972-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-10-15T14:24:11-04:00: type: 'STATE': sub_type: 'UPDATING' message: Application: [c31f8f54-07e0-4e5f-9d3d-79e8bcd8a9ed]: State changed to UPDATING: Update to version '7.11.0-SNAPSHOT' started","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-10-15T14:24:20.727-0400","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/fleet/agents/c31f8f54-07e0-4e5f-9d3d-79e8bcd8a9ed/acks","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-10-15T14:24:23.338-0400","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2020-10-15T14:24:23-04:00: type: 'ERROR': sub_type: 'FAILED' message: Application: [c31f8f54-07e0-4e5f-9d3d-79e8bcd8a9ed]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-81dc89/downloads/elastic-agent-7.11.0-SNAPSHOT-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-81dc89/downloads/elastic-agent-7.11.0-SNAPSHOT-darwin-x86_64.tar.gz.asc: no such file or directory\n\t* check detached signature: openpgp: invalid signature: hash tag doesn't match\n\n","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-10-15T14:24:23.338-0400","log.origin":{"file.name":"application/action_dispatcher.go","file.line":93},"message":"Failed to dispatch action 'action_id: ee74b995-f001-4ce6-913a-01481fa5eb4d, type: UPGRADE', error: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-81dc89/downloads/elastic-agent-7.11.0-SNAPSHOT-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-81dc89/downloads/elastic-agent-7.11.0-SNAPSHOT-darwin-x86_64.tar.gz.asc: no such file or directory\n\t* check detached signature: openpgp: invalid signature: hash tag doesn't match\n\n","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-10-15T14:24:23.338-0400","log.origin":{"file.name":"application/fleet_gateway.go","file.line":159},"message":"failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-81dc89/downloads/elastic-agent-7.11.0-SNAPSHOT-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-81dc89/downloads/elastic-agent-7.11.0-SNAPSHOT-darwin-x86_64.tar.gz.asc: no such file or directory\n\t* check detached signature: openpgp: invalid signature: hash tag doesn't match\n\n","ecs.version":"1.5.0"}

[EricDavisX]

Ah, I just scrolled thru chat and Michal was using cloud Kibana with local agent, so my theory is not likely helpful. I'm testing 8.0 Kibana cloud latest and will report again.

[EricDavisX]

I find the same errors when testing with 8.0 Cloud Kibana and a 7.10 Agent. I would be interested to see if Michal was testing with a 'dev' build with GPG check turned off - perhaps that would explain why his tests worked and mine do not?

[michalpristas]

checked asc files manually and they dont match.

gpg -v --verify elastic-agent-7.10.0-linux-x86_64.tar.gz.asc elastic-agent-7.10.0-linux-x86_64.tar.gz
gpg: armor header: Version: GnuPG v1
gpg: Signature made št  15 okt 06:00:17 2020 CEST
gpg:                using RSA key D27D666CD88E42B4
gpg: using pgp trust model
gpg: BAD signature from "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>" [unknown]
gpg: binary signature, digest algorithm SHA512, key algorithm rsa2048

pinged infra

[EricDavisX]

Infra agreed to sign a few build for us as needed, what we will need is the target build to be signed. so until their issue is fixed we wont be able to test this on just anu old build (from the ui). to test, we'll have to use the api and target the builds that are signed and have the feature support and all bug fixes. i'll post back which source_uri will work or when the Infra issue is solved for all builds.

[EricDavisX]

The Infra issue was fixed today and a Jenkins job is expected to be auto-run to do the correct (re)signing on all SNAPSHOT and BC builds from today on. I can't be sure about today's executions, but I know that the below builds when kicked off should be ok:
build 66 of 8.0 / master
build 17 of 7.10
build 61 of 7.x (7.11 snapshot)

So we can test on a 7.10 BC3 and attempt to upgrade (allowing expectations wrt other feature fixes coming in)

[EricDavisX]

we got all fixes in to 7.10 line, the BC4 or newer will have them all - recommend to use that build to test. and the 7.X and 8.0 unified builds failed, so we'll need the newer versions of those to pick up the fixes anyhow, but we should be able to test soon!

[EricDavisX]

I attempted this and did not see the .asc file error (I did see other things but those are separate and being workedon) - tested on 7.10 latest snapshot compiled over the weekend.