unsupported attribute fork error during Trivy vm scan #6082

tmay-dg · 2024-02-08T00:21:05Z

tmay-dg
Feb 8, 2024

Description

We are receiving a strange unsupported attribute fork error on a Trivy scan of an AL2 AMI. I am creating a new AMI from a base AMI. I can scan the base AMI just fine, but when the new AMI is created it can no longer be scanned. There were no changes from the base AMI to the new AMI - other than spinning up an EC2 instance from the base and creating a new AMI from that.

Here is the full error that we receive:

2024-02-07T15:24:21.073Z	�[35mDEBUG�[0m	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2024-02-07T15:24:21.074Z	�[35mDEBUG�[0m	Ignore statuses	{"statuses": null}
2024-02-07T15:24:21.074Z	�[35mDEBUG�[0m	Timeout is set to less than 30 min - upgrading to 30 min for this command.
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	cache dir:  /home/jenkins/workspace/134a53aa-a31d-4661-a5d4-625e531fd5ba/trivy_scan/.trivy
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	Skipping DB update...
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	DB Schema: 2, UpdatedAt: 2023-11-15 18:11:55.656608033 +0000 UTC, NextUpdate: 2023-11-16 00:11:55.656607773 +0000 UTC, DownloadedAt: 2023-11-15 20:09:06.17699631 +0000 UTC
2024-02-07T15:24:21.078Z	�[34mINFO�[0m	Vulnerability scanning is enabled
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	Vulnerability type:  [os library]
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	The nuget packages directory couldn't be found. License search disabled
2024-02-07T15:24:21.413Z	�[34mINFO�[0m	Snapshot snap-0c607bedc500f133e found
2024-02-07T15:24:22.082Z	�[35mDEBUG�[0m	Missing virtual machine cache: sha256:f2e868d4114240a16c2b58297a995b1638facaa15954516d0eb8dc2f06da17c4
2024-02-07T15:24:22.233Z	�[35mDEBUG�[0m	Found partition: Linux
2024-02-07T15:24:23.279Z	�[35mDEBUG�[0m	Skipping directory: dev
2024-02-07T15:24:40.946Z	�[35mDEBUG�[0m	Skipping directory: proc
2024-02-07T15:24:40.946Z	�[35mDEBUG�[0m	Skipping directory: sys
2024-02-07T15:24:41.815Z	�[35mDEBUG�[0m	
00000000  49 4e 81 a0 03 03 00 00  00 00 00 00 00 00 00 be  |IN..............|
00000010  00 00 00 01 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  65 bc 98 da 3a e3 af 00  65 c3 9e f6 18 0d a4 bb  |e...:...e.......|
00000030  65 c3 9e f6 18 0d a4 bb  00 00 00 00 00 80 00 00  |e...............|
00000040  00 00 00 00 00 00 08 02  00 00 00 00 00 00 00 3d  |...............=|
00000050  00 01 1b 02 00 00 00 00  00 00 00 02 7f a4 92 a6  |................|
00000060  ff ff ff ff b0 36 f8 12  00 00 00 00 00 00 00 01  |.....6..........|
00000070  00 00 00 10 00 00 0d e3  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  65 bc 98 da 3a e3 af 00  00 00 00 00 00 00 23 82  |e...:.........#.|
000000a0  96 37 cd e4 52 7f 45 c3  8d 69 93 16 01 d9 ac f2  |.7..R.E..i......|
000000b0  00 01 00 01 00 00 00 00  00 00 00 00 90 60 00 01  |.............`..|
000000c0  00 00 00 00 00 00 02 00  00 00 00 15 d7 00 03 0b  |................|
000000d0  80 00 00 00 00 06 18 00  00 00 00 16 38 60 00 01  |............8`..|
000000e0  00 00 00 00 00 06 1a 00  00 00 00 16 38 80 00 a6  |............8...|
000000f0  80 00 00 00 00 07 66 00  00 00 00 16 4d 40 00 02  |......f.....M@..|
00000100  00 00 00 00 00 07 6a 00  00 00 00 16 4d 80 00 02  |......j.....M...|
00000110  80 00 00 00 00 07 6e 00  00 00 00 16 00 00 00 00  |......n.........|
00000120  00 00 04 ea 00 07 74 00  00 00 00 16 4e 20 00 01  |......t.....N ..|
00000130  80 00 00 00 00 07 76 00  00 00 00 16 4e 40 00 03  |......v.....N@..|
00000140  00 00 00 00 00 07 7c 00  00 00 00 16 4e a0 00 2e  |......|.....N...|
00000150  80 00 00 00 00 07 d8 00  00 00 00 16 54 60 04 14  |............T`..|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 90 a0 00 01  65 63 34 90 63 03 61 10  |........ec4.c.a.|
000001a0  06 00 0c 4c 02 53 47 49  5f 41 43 4c 5f 46 49 4c  |...L.SGI_ACL_FIL|
000001b0  45 00 00 00 06 00 00 00  01 ff ff ff ff 00 06 00  |E...............|
000001c0  00 00 00 00 04 ff ff ff  ff 00 04 00 00 00 00 00  |................|
000001d0  08 00 00 00 04 00 04 00  00 00 00 00 08 00 00 00  |................|
000001e0  0a 00 04 00 00 00 00 00  10 ff ff ff ff 00 04 00  |................|
000001f0  00 00 00 00 20 ff ff ff  ff 00 00 00 00 00 00 00  |.... ...........|

2024-02-07T15:24:41.816Z	�[33mWARN�[0m	Partition error: filesystem walk error: fs.Walk error: read directory /var/log/journal/ec27c465383160b521793f2f10de6fbd: failed to list directory entries inode: 30393: failed to parse inode 9090: parse inode format btree: parse bmbr block error: parse bmbr key-ptr error: unsupported attribute fork error
2024-02-07T15:24:41.818Z	�[33mWARN�[0m	No OS package is detected. Make sure you haven't deleted any files that contain information about the installed packages.
2024-02-07T15:24:41.818Z	�[33mWARN�[0m	e.g. files under "/lib/apk/db/", "/var/lib/dpkg/" and "/var/lib/rpm"
2024-02-07T15:24:41.818Z	�[34mINFO�[0m	Detected OS: amazon
2024-02-07T15:24:41.818Z	�[34mINFO�[0m	Detecting Amazon Linux vulnerabilities...
2024-02-07T15:24:41.818Z	�[35mDEBUG�[0m	amazon: os version: 2
2024-02-07T15:24:41.818Z	�[35mDEBUG�[0m	amazon: the number of packages: 0
2024-02-07T15:24:41.818Z	�[34mINFO�[0m	Number of language-specific files: 0

You may notice the DB is older in the output above - I have tried with the latest DB and have also run the scans with Trivy 0.49.1 - neither help with the issue. I am unsure how simply creating a new AMI from an existing AMI (with zero changes) could cause the scans to suddenly stop working.

Desired Behavior

I expected the bespoke/derivative AMI to be successfully scanned.

Actual Behavior

The bespoke/derivative AMI cannot be scanned.

Reproduction Steps

Unfortunately I can only trigger this behavior on our company "Gold" AL2 base AMI. Other Gold AMI's that we have do not exhibit the behavior. If I create a brand new AMI from Amazon's latest AL2, and then create an additional bespoke/derivative AMI from that, then I can scan just fine.

Target

Virtual Machine Image

Scanner

Vulnerability

Output Format

JSON

Mode

Standalone

Debug Output

/home/jenkins/tachyon/tools/installed/trivy/linux/x86-64/0.47.0/trivy vm --scanners vuln --format json --output /home/jenkins/workspace/134a53aa-a31d-4661-a5d4-625e531fd5ba/trivy_scan/trivy-ami-093e691d14b6e3ebf-results.json --cache-dir /home/jenkins/workspace/134a53aa-a31d-4661-a5d4-625e531fd5ba/trivy_scan/.trivy --skip-db-update --skip-java-db-update --offline-scan --exit-code 101 --debug ami:ami-093e691d14b6e3ebf --aws-region us-east-1
2024-02-07T15:24:21.073Z	�[35mDEBUG�[0m	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2024-02-07T15:24:21.074Z	�[35mDEBUG�[0m	Ignore statuses	{"statuses": null}
2024-02-07T15:24:21.074Z	�[35mDEBUG�[0m	Timeout is set to less than 30 min - upgrading to 30 min for this command.
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	cache dir:  /home/jenkins/workspace/134a53aa-a31d-4661-a5d4-625e531fd5ba/trivy_scan/.trivy
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	Skipping DB update...
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	DB Schema: 2, UpdatedAt: 2023-11-15 18:11:55.656608033 +0000 UTC, NextUpdate: 2023-11-16 00:11:55.656607773 +0000 UTC, DownloadedAt: 2023-11-15 20:09:06.17699631 +0000 UTC
2024-02-07T15:24:21.078Z	�[34mINFO�[0m	Vulnerability scanning is enabled
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	Vulnerability type:  [os library]
2024-02-07T15:24:21.078Z	�[35mDEBUG�[0m	The nuget packages directory couldn't be found. License search disabled
2024-02-07T15:24:21.413Z	�[34mINFO�[0m	Snapshot snap-0c607bedc500f133e found
2024-02-07T15:24:22.082Z	�[35mDEBUG�[0m	Missing virtual machine cache: sha256:f2e868d4114240a16c2b58297a995b1638facaa15954516d0eb8dc2f06da17c4
2024-02-07T15:24:22.233Z	�[35mDEBUG�[0m	Found partition: Linux
2024-02-07T15:24:23.279Z	�[35mDEBUG�[0m	Skipping directory: dev
2024-02-07T15:24:40.946Z	�[35mDEBUG�[0m	Skipping directory: proc
2024-02-07T15:24:40.946Z	�[35mDEBUG�[0m	Skipping directory: sys
2024-02-07T15:24:41.815Z	�[35mDEBUG�[0m	
00000000  49 4e 81 a0 03 03 00 00  00 00 00 00 00 00 00 be  |IN..............|
00000010  00 00 00 01 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  65 bc 98 da 3a e3 af 00  65 c3 9e f6 18 0d a4 bb  |e...:...e.......|
00000030  65 c3 9e f6 18 0d a4 bb  00 00 00 00 00 80 00 00  |e...............|
00000040  00 00 00 00 00 00 08 02  00 00 00 00 00 00 00 3d  |...............=|
00000050  00 01 1b 02 00 00 00 00  00 00 00 02 7f a4 92 a6  |................|
00000060  ff ff ff ff b0 36 f8 12  00 00 00 00 00 00 00 01  |.....6..........|
00000070  00 00 00 10 00 00 0d e3  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  65 bc 98 da 3a e3 af 00  00 00 00 00 00 00 23 82  |e...:.........#.|
000000a0  96 37 cd e4 52 7f 45 c3  8d 69 93 16 01 d9 ac f2  |.7..R.E..i......|
000000b0  00 01 00 01 00 00 00 00  00 00 00 00 90 60 00 01  |.............`..|
000000c0  00 00 00 00 00 00 02 00  00 00 00 15 d7 00 03 0b  |................|
000000d0  80 00 00 00 00 06 18 00  00 00 00 16 38 60 00 01  |............8`..|
000000e0  00 00 00 00 00 06 1a 00  00 00 00 16 38 80 00 a6  |............8...|
000000f0  80 00 00 00 00 07 66 00  00 00 00 16 4d 40 00 02  |......f.....M@..|
00000100  00 00 00 00 00 07 6a 00  00 00 00 16 4d 80 00 02  |......j.....M...|
00000110  80 00 00 00 00 07 6e 00  00 00 00 16 00 00 00 00  |......n.........|
00000120  00 00 04 ea 00 07 74 00  00 00 00 16 4e 20 00 01  |......t.....N ..|
00000130  80 00 00 00 00 07 76 00  00 00 00 16 4e 40 00 03  |......v.....N@..|
00000140  00 00 00 00 00 07 7c 00  00 00 00 16 4e a0 00 2e  |......|.....N...|
00000150  80 00 00 00 00 07 d8 00  00 00 00 16 54 60 04 14  |............T`..|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 90 a0 00 01  65 63 34 90 63 03 61 10  |........ec4.c.a.|
000001a0  06 00 0c 4c 02 53 47 49  5f 41 43 4c 5f 46 49 4c  |...L.SGI_ACL_FIL|
000001b0  45 00 00 00 06 00 00 00  01 ff ff ff ff 00 06 00  |E...............|
000001c0  00 00 00 00 04 ff ff ff  ff 00 04 00 00 00 00 00  |................|
000001d0  08 00 00 00 04 00 04 00  00 00 00 00 08 00 00 00  |................|
000001e0  0a 00 04 00 00 00 00 00  10 ff ff ff ff 00 04 00  |................|
000001f0  00 00 00 00 20 ff ff ff  ff 00 00 00 00 00 00 00  |.... ...........|

2024-02-07T15:24:41.816Z	�[33mWARN�[0m	Partition error: filesystem walk error: fs.Walk error: read directory /var/log/journal/ec27c465383160b521793f2f10de6fbd: failed to list directory entries inode: 30393: failed to parse inode 9090: parse inode format btree: parse bmbr block error: parse bmbr key-ptr error: unsupported attribute fork error
2024-02-07T15:24:41.818Z	�[33mWARN�[0m	No OS package is detected. Make sure you haven't deleted any files that contain information about the installed packages.
2024-02-07T15:24:41.818Z	�[33mWARN�[0m	e.g. files under "/lib/apk/db/", "/var/lib/dpkg/" and "/var/lib/rpm"
2024-02-07T15:24:41.818Z	�[34mINFO�[0m	Detected OS: amazon
2024-02-07T15:24:41.818Z	�[34mINFO�[0m	Detecting Amazon Linux vulnerabilities...
2024-02-07T15:24:41.818Z	�[35mDEBUG�[0m	amazon: os version: 2
2024-02-07T15:24:41.818Z	�[35mDEBUG�[0m	amazon: the number of packages: 0
2024-02-07T15:24:41.818Z	�[34mINFO�[0m	Number of language-specific files: 0

Operating System

Amazon Linux 2

Version

trivy --version
Version: 0.49.1
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-02-07 18:12:19.054469144 +0000 UTC
  NextUpdate: 2024-02-08 00:12:19.054468763 +0000 UTC
  DownloadedAt: 2024-02-07 19:49:05.760553154 +0000 UTC
Java DB:
  Version: 1
  UpdatedAt: 2024-02-07 00:43:51.227831333 +0000 UTC
  NextUpdate: 2024-02-10 00:43:51.227831212 +0000 UTC
  DownloadedAt: 2024-02-07 17:13:44.357117872 +0000 UTC

Checklist

Run trivy image --reset
Read the troubleshooting

knqyf263 · 2024-02-08T05:41:04Z

knqyf263
Feb 8, 2024
Maintainer

@masahiro331 Can you please look into it?

0 replies

tmay-dg · 2024-02-08T18:52:39Z

tmay-dg
Feb 8, 2024
Author

An update - I can resolve the scanning of the bespoke/derivative AMI by running the following prior to building the AMI:

rm -rf /var/log/journal/*

I don't understand why journal log file(s) would cause Trivy scans to fail though. It seems worrisome that someone could craft a special log message (or messages) that causes subsequent Trivy scans to not detect any OS packages (and therefore detect no vulnerabilities).

0 replies

aleliaert · 2024-05-17T18:25:37Z

aleliaert
May 17, 2024

Am concerned about this too. @masahiro331 @knqyf263 can you create an issue on this?

0 replies

masahiro331 · 2024-06-04T02:15:49Z

masahiro331
Jun 4, 2024

Sorry for the delay. I'll investigate.

0 replies

masahiro331 · 2024-06-22T14:17:39Z

masahiro331
Jun 22, 2024

@aleliaert @tmay-dg
It seems to occur when an inode configured with Btree has an attribute fork.
https://github.com/masahiro331/go-xfs-filesystem/blob/main/xfs/inode.go#L369-L375

The library I'm using doesn't support this, so I need to modify it, but I haven't been able to reproduce this Binary data.

It looks like it will take some time to fix it.
A possible workaround is to set flag like --skip-dir /var/log/ .

1 reply

Errahulaws Jul 19, 2024

@masahiro331 looks like this is still not fixed
also --skip-dirs does not seems to be working for VM scanning

Can we have this fix or any alternative

tmay-dg · 2024-10-25T16:29:45Z

tmay-dg
Oct 25, 2024
Author

Hello, just wondering if there is any sort of update on this issue?

0 replies

tburow · 2024-10-31T18:58:06Z

tburow
Oct 31, 2024

@masahiro331 - I thought the INODE problem was fixed in the go-xfs module when we first starting working with AL2023 - I remember working with you on that problem. did we have a regression on this? this issue renders the AMI image scan for AL2 and AL2023 useless.

0 replies

tburow · 2024-10-31T19:10:09Z

tburow
Oct 31, 2024

@masahiro331

Possibly overlooked something - Trivy seems to be puking on the docker virtual block device

2024-10-27T12:24:37.131Z WARN Partition error: filesystem walk error: fs.Walk error: read directory /var/lib/docker/volumes/backingFsBlockDev: failed to list directory entries inode: 8699708: failed to list entries: not found entries

looks like there is another thread with this same problem - #7071

4 replies

masahiro331 Nov 6, 2024

oh... I'll investigate.

masahiro331 Nov 6, 2024

Can I get the docker image?

tmay-dg Nov 6, 2024
Author

This isn't a Docker image - it's an AMI (AL2). The /var/lib/docker/volumes/backingFsBlockDev error that @tburow mentioned is a different than the error I posted originally. The /var/lib/docker/volumes/backingFsBlockDev error was from a scan of an AL2023 AMI

tburow Nov 6, 2024

yes to Tyler's point this is an AMI with Docker engine installed. apologies if I conflated the issue - but is related to #7071

recurrent theme also is that any FS related Error silently fails.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unsupported attribute fork error during Trivy vm scan #6082

{{title}}

Replies: 8 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

unsupported attribute fork error during Trivy vm scan #6082

Description

Desired Behavior

Actual Behavior

Reproduction Steps

Target

Scanner

Output Format

Mode

Debug Output

Operating System

Version

Checklist

Replies: 8 comments · 5 replies

knqyf263 Feb 8, 2024 Maintainer

tmay-dg Feb 8, 2024 Author

tmay-dg Oct 25, 2024 Author

tmay-dg Nov 6, 2024 Author

Replies: 8 comments 5 replies

knqyf263
Feb 8, 2024
Maintainer

tmay-dg
Feb 8, 2024
Author

tmay-dg
Oct 25, 2024
Author

tmay-dg Nov 6, 2024
Author