Scanning failure for non-amd64 Red Hat images created before June 2020 #435

knqyf263 · 2024-09-02T09:32:52Z

Description:

We've identified an issue in our scanning process for Red Hat container images.

Problem

Scanning fails for Red Hat container images that meet both of these criteria:

Created before June 2020
Architecture other than amd64

Root cause:

These older images lack /root/buildinfo/content_manifests, requiring NVR-to-CPE conversion.
Our current NVR-to-CPE mapping only includes amd64 architectures.

Impact

Incomplete vulnerability scanning for affected images

The text was updated successfully, but these errors were encountered:

atgreen · 2024-09-21T11:47:59Z

I'm getting this same error, pointing me at this Issue, despite the fact that the image DOES contain /root/buildinfo/content_manifests and was created in 2024.

green@fedora:~/ctest$ trivy image registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.14.0-1725667424
2024-09-21T07:45:26-04:00	INFO	[vuln] Vulnerability scanning is enabled
2024-09-21T07:45:26-04:00	INFO	[secret] Secret scanning is enabled
2024-09-21T07:45:26-04:00	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-09-21T07:45:26-04:00	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.55/docs/scanner/secret#recommendation for faster secret detection
2024-09-21T07:45:32-04:00	INFO	Detected OS	family="redhat" version="8.10"
2024-09-21T07:45:32-04:00	INFO	[redhat] Detecting RHEL/CentOS vulnerabilities...	os_version="8" pkg_num=375
2024-09-21T07:45:32-04:00	FATAL	Fatal error	image scan error: scan error: scan failed: scan failed: failed to detect vulnerabilities: unable to scan OS packages: failed vulnerability detection of OS packages: failed detection: redhat vulnerability detection error: failed to get Red Hat advisories: unable to find CPE indices. See https://github.com/aquasecurity/trivy-db/issues/435 for details

knqyf263 · 2024-09-23T05:43:48Z

@DmitriyLewen Can you please take a look?

DmitriyLewen · 2024-09-23T06:49:10Z

Hello @atgreen
It looks like the content_manifests file contains unknown repositories.

Unfortunately, I don't have access to this image.

Can you send me the /root/buildinfo/content_manifests/*.json file from this image?

UPD.
But Pyxis shows correct content sets:

➜ curl -X 'GET' \ 
  'https://catalog.redhat.com/api/containers/v1/images/id/66db9ee0a82efda2c54a9ec6' \
  -H 'accept: application/json' | jq .content_sets
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 47900  100 47900    0     0  66602      0 --:--:-- --:--:-- --:--:-- 66527
[
  "rhel-8-for-aarch64-baseos-rpms",
  "rhel-8-for-aarch64-appstream-rpms"
]

atgreen · 2024-09-23T09:42:59Z

out.txt

DmitriyLewen · 2024-09-23T10:12:08Z

It looks like several files contain incorrect content_sets:

====== /root/buildinfo/content_manifests/openshift-base-rhel8-container-v4.14.0-202408260910.p0.gf020942.assembly.stream.el8.json ======
{
    ...
    "content_sets": [
        "rhel-8-for-x86_64-appstream-rpms__8",
        "rhel-8-for-x86_64-baseos-rpms__8"
    ]
}====== /root/buildinfo/content_manifests/openshift-enterprise-base-container-v4.14.0-202408260910.p0.g03e5f40.assembly.stream.el8.json ======
{
    ...
    "content_sets": [
        "rhel-8-for-x86_64-appstream-rpms__8",
        "rhel-8-for-x86_64-baseos-rpms__8"
    ]
}====== /root/buildinfo/content_manifests/openshift-enterprise-cli-container-v4.14.0-202408260910.p0.g44b3ac2.assembly.stream.el8.json ======
{
    ...
    "content_sets": [
        "rhel-8-for-x86_64-appstream-rpms__8",
        "rhel-8-for-x86_64-baseos-rpms__8"
    ]
}====== /root/buildinfo/content_manifests/openshift-jenkins-2-container-v4.14.0-1725667424.json ====== 
{
    ...
    "content_sets": [
        "rhel-8-for-x86_64-baseos-rpms",
        "rhel-8-for-x86_64-appstream-rpms"
    ]
}====== /root/buildinfo/content_manifests/ubi8-container-8.10-901.1717584420.json ====== 
{
    ...
    "content_sets": [
        "rhel-8-for-x86_64-baseos-rpms",
        "rhel-8-for-x86_64-appstream-rpms"
    ],
    "image_contents": []
}

rhel-8-for-x86_64-baseos-rpms and rhel-8-for-x86_64-appstream-rpms are correct (see output of Pyxis).
rhel-8-for-x86_64-appstream-rpms__8 and rhel-8-for-x86_64-baseos-rpms__8 are wrong.

@knqyf263 you have more experience in this matter.
I have 2 question:

IIUC /root/buildinfo/content_manifests should contain one manifest file. Is it correct that this directory contains more than 1 file?
if yes - should we merge content_sets from these files?
https://github.com/aquasecurity/trivy/blob/37d549e5b86a1c5dce6710fbfd2310aec9abe949/pkg/fanal/analyzer/analyzer.go#L294-L296

knqyf263 · 2024-09-23T11:32:19Z

IIUC /root/buildinfo/content_manifests should contain one manifest file. Is it correct that this directory contains more than 1 file?

My understanding is that there should only be one file per layer. However, the image as a whole may contain multiple files. Therefore, it is necessary to look up the appropriate buildinfo for each package.
https://github.com/aquasecurity/trivy/blob/37d549e5b86a1c5dce6710fbfd2310aec9abe949/pkg/fanal/applier/docker.go#L53-L76

sayan-biswas · 2024-09-23T11:34:24Z

This is the content set we have used in our config.

aarch64:
- rhel-8-for-aarch64-baseos-rpms
- rhel-8-for-aarch64-appstream-rpms
ppc64le:
- rhel-8-for-ppc64le-baseos-rpms
- rhel-8-for-ppc64le-appstream-rpms
s390x:
- rhel-8-for-s390x-baseos-rpms
- rhel-8-for-s390x-appstream-rpms
x86_64:
- rhel-8-for-x86_64-baseos-rpms
- rhel-8-for-x86_64-appstream-rpms

knqyf263 · 2024-09-23T13:42:18Z

@sayan-biswas Thanks for sharing. We see some content sets, like rhel-8-for-x86_64-appstream-rpms__8. Is it legitimate or a mistake?

$ docker run --rm -it --entrypoint cat registry.redhat.io/ocp-tools-4/jenkins-agent-base-rhel8@sha256:b142a2461dcf8bc50cc6311808867c38a3de45c82145df648d34d37a75496d0f -- /root/buildinfo/content_manifests/openshift-enterprise-base-container-v4.15.0-202408260908.p0.g27f1695.assembly.stream.el8.json | jq .cont
ent_sets
[
  "rhel-8-for-aarch64-appstream-rpms__8",
  "rhel-8-for-aarch64-baseos-rpms__8"
]

AdmiralGT · 2024-10-16T13:59:14Z

I'm also seeing this error with the following content in content_manifests

[root@a481dbded621 /]# ls -l /root/buildinfo/content_manifests/
total 4
-rw-rw-r-- 1 root root 368 Oct 23  2023 ubi8-container-8.8-1067.1698056881.json
[root@a481dbded621 /]# cat /root/buildinfo/content_manifests/ubi8-container-8.8-1067.1698056881.json
{
    "metadata": {
        "icm_version": 1,
        "icm_spec": "https://raw.githubusercontent.com/containerbuildsystem/atomic-reactor/master/atomic_reactor/schemas/content_manifest.json",
        "image_layer_index": 0
    },
    "content_sets": [
        "rhel-8-for-x86_64-baseos-rpms",
        "rhel-8-for-x86_64-appstream-rpms"
    ],
    "image_contents": []
}

Based on the above comments, it looks like these are correct (and I only have a single file in content_manifests)

DmitriyLewen · 2024-10-17T08:14:29Z

I'm also seeing this error with the following content in content_manifests

Hmm... this is very strange...
Can you share this image?

AdmiralGT · 2024-10-17T09:00:07Z

I'm afraid not, but we've upgraded from v0.55.1 to v0.55.5 of trivy and that appears to have resolved the issue.

DmitriyLewen · 2024-10-17T09:25:43Z

Trivy doesn't have v0.55.5 version 😄

AdmiralGT · 2024-10-17T09:29:26Z

Trivy doesn't have v0.55.5 version 😄

Haha, I've not fully woken up yet, we've reverted to 0.55.0 (which does work) rather than 0.55.1 (which doesn't)

DmitriyLewen · 2024-10-17T09:36:23Z

[root@a481dbded621 /]# ls -l /root/buildinfo/content_manifests/
total 4
-rw-rw-r-- 1 root root 368 Oct 23 2023 ubi8-container-8.8-1067.1698056881.json
[root@a481dbded621 /]# cat /root/buildinfo/content_manifests/ubi8-container-8.8-1067.1698056881.json
{
"metadata": {
"icm_version": 1,
"icm_spec": "https://raw.githubusercontent.com/containerbuildsystem/atomic-reactor/master/atomic_reactor/schemas/content_manifest.json",
"image_layer_index": 0
},
"content_sets": [
"rhel-8-for-x86_64-baseos-rpms",
"rhel-8-for-x86_64-appstream-rpms"
],
"image_contents": []
}

Is this possible that manifest files for other layers have been deleted?

AdmiralGT · 2024-10-17T11:00:50Z

I don't believe so. Our build infrastructure templates our docker files but I couldn't find anything in the infrastructure to suggest it was deleting manifest files from other layers. I'll see if I can get out the templated docker file but that might not happen until next week I'm afraid.

DmitriyLewen · 2024-10-17T11:11:58Z

ok, i'll wait for the info.
i just can't think of any other ideas why you get an error with a valid context_sets

andersjdba · 2024-10-22T16:00:17Z

We started to see failures which directed to this issue after upgrading from trivy 0.55.1 to 0.56.1. We couldn't understand why we were seeing this and were at a bit of a loss.

Rightly or wrongly, we've always scanned our images in our CI in parallel. Out of curiosity more than anything, I reverted the parallel scan to run in serial and this error went away. I haven't had chance to look in to the "why" as yet but thought id share this incase it helps in anyway without muddying the waters.

knqyf263 mentioned this issue Sep 2, 2024

refactor(redhat-oval): return an error if the CPE is not found #428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scanning failure for non-amd64 Red Hat images created before June 2020 #435

Scanning failure for non-amd64 Red Hat images created before June 2020 #435

knqyf263 commented Sep 2, 2024

atgreen commented Sep 21, 2024 •

edited

Loading

knqyf263 commented Sep 23, 2024

DmitriyLewen commented Sep 23, 2024 •

edited

Loading

atgreen commented Sep 23, 2024

DmitriyLewen commented Sep 23, 2024 •

edited

Loading

knqyf263 commented Sep 23, 2024

sayan-biswas commented Sep 23, 2024

knqyf263 commented Sep 23, 2024

AdmiralGT commented Oct 16, 2024 •

edited

Loading

DmitriyLewen commented Oct 17, 2024

AdmiralGT commented Oct 17, 2024

DmitriyLewen commented Oct 17, 2024

AdmiralGT commented Oct 17, 2024

DmitriyLewen commented Oct 17, 2024

AdmiralGT commented Oct 17, 2024

DmitriyLewen commented Oct 17, 2024

andersjdba commented Oct 22, 2024

Scanning failure for non-amd64 Red Hat images created before June 2020 #435

Scanning failure for non-amd64 Red Hat images created before June 2020 #435

Comments

knqyf263 commented Sep 2, 2024

Description:

Problem

Root cause:

Impact

atgreen commented Sep 21, 2024 • edited Loading

knqyf263 commented Sep 23, 2024

DmitriyLewen commented Sep 23, 2024 • edited Loading

atgreen commented Sep 23, 2024

DmitriyLewen commented Sep 23, 2024 • edited Loading

knqyf263 commented Sep 23, 2024

sayan-biswas commented Sep 23, 2024

knqyf263 commented Sep 23, 2024

AdmiralGT commented Oct 16, 2024 • edited Loading

DmitriyLewen commented Oct 17, 2024

AdmiralGT commented Oct 17, 2024

DmitriyLewen commented Oct 17, 2024

AdmiralGT commented Oct 17, 2024

DmitriyLewen commented Oct 17, 2024

AdmiralGT commented Oct 17, 2024

DmitriyLewen commented Oct 17, 2024

andersjdba commented Oct 22, 2024

atgreen commented Sep 21, 2024 •

edited

Loading

DmitriyLewen commented Sep 23, 2024 •

edited

Loading

DmitriyLewen commented Sep 23, 2024 •

edited

Loading

AdmiralGT commented Oct 16, 2024 •

edited

Loading