Capa v7.01 Not recognizing Linux x64 file

[C0d3R3ad3r]

Description

Files being analyzed: docker-credential-pass-v0.8.0.linux-amd64 and linmux x64 file 23bae09b5699c2d5c4cb1b8aa908a3af898b00f88f06e021edcb16d7d558efad (both are linux x64 files)

Trying to analyze a linux x64 file and with no options set, just running ./capa [file], capa returns an error:

malware@malware-VirtualBox:~/Downloads$ ./capa -f auto docker-credential-pass-v0.8.0.linux-amd64 
ERROR:capa:--------------------------------------------------------------------------------
ERROR:capa: Input file does not appear to target a supported OS.
ERROR:capa: 
ERROR:capa: capa currently only supports analyzing executables for some operating systems (including Windows and Linux).
ERROR:capa:--------------------------------------------------------------------------------

However, when running with options explicitly set, ./capa -f auto --os linux [file], capa analyzes the file correctly.

malware@malware-VirtualBox:~/Downloads$ ./capa -f auto --os linux docker-credential-pass-v0.8.0.linux-amd64 
┍━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑                                                                        
│ md5                    │ 0e1ec55106a036ef12bee7220c221b5e                                                   │
│ sha1                   │ a18043386ff5f0ccfd63d79f534af8c599cf6969                                           │
│ sha256                 │ a69ce71a6b5cef7aadf343c93e00e1ffc549d649bd011fbb39bfa38534484511                   │
│ analysis               │ static                                                                             │
│ os                     │ linux                                                                              │
│ format                 │ elf                                                                                │
│ arch                   │ amd64                                                                              │
│ path                   │ /home/malware/Downloads/docker-credential-pass-v0.8.0.linux-amd64                  │
┕━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙

┍━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
│ ATT&CK Tactic          │ ATT&CK Technique                                                                   │
┝━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
│ DEFENSE EVASION        │ Obfuscated Files or Information T1027                                              │
┕━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙

┍━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
│ MBC Objective               │ MBC Behavior                                                                  │
┝━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
│ CRYPTOGRAPHY                │ Encrypt Data::AES [C0027.001]                                                 │
│                             │ Encrypt Data::RC4 [C0027.009]                                                 │
│                             │ Generate Pseudo-random Sequence::RC4 PRGA [C0021.004]                         │
├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ DATA                        │ Check String [C0019]                                                          │
│                             │ Encode Data::Base64 [C0026.001]                                               │
│                             │ Non-Cryptographic Hash::FNV [C0030.005]                                       │
├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ DEFENSE EVASION             │ Obfuscated Files or Information::Encoding-Standard Algorithm [E1027.m02]      │
│                             │ Obfuscated Files or Information::Encryption-Standard Algorithm [E1027.m05]    │
┕━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙

┍━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
│ Capability                                            │ Namespace                                            │
┝━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
│ execute syscall instruction (20 matches)              │ anti-analysis                                        │
│ compiled with Go                                      │ compiler/go                                          │
│ encode data using Base64 (3 matches)                  │ data-manipulation/encoding/base64                    │
│ reference Base64 string                               │ data-manipulation/encoding/base64                    │
│ encrypt data using AES via x86 extensions (4 matches) │ data-manipulation/encryption/aes                     │
│ encrypt data using RC4 PRGA                           │ data-manipulation/encryption/rc4                     │
│ hash data using fnv                                   │ data-manipulation/hashing/fnv                        │
┕━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙

malware@malware-VirtualBox:~/Downloads$ file 23bae09b5699c2d5c4cb1b8aa908a3af898b00f88f06e021edcb16d7d558efad 
23bae09b5699c2d5c4cb1b8aa908a3af898b00f88f06e021edcb16d7d558efad: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

It looks like with this file (and a few other linux x64) files, capa fails to detect the file type and reports the error that the input file does not appear to target a supported OS

Steps to Reproduce

1. run capa with no options passing the file through on the command line

Expected behavior:

Expected behavoir would be for capa to recognize the elf file and analyze it

Actual behavior:

capa errors with "Input file does not appear to target a supported OS."

Versions

capa v7.01
remnux 20.04

Additional Information

I am not sure if this is a bug that is not identifying these linux x64 files or not, but the workaround is to explicitly specify the --os linux on the command line which I would think should not be necessary (unless dealing with shellcode)

[williballenthin]

Hi @C0d3R3ad3r

Thanks for reporting this issue with so many details.

Guessing the underlying OS for an ELF file is tricky - there's not a header that says "this is for Linux". We have a bunch of heuristics that help us, but they don't work perfectly. You can see them here:

capa/capa/features/extractors/elf.py

Line 959 in fde1de3

def detect_elf_os(f) -> str:

23bae09b5699c2d5c4cb1b8aa908a3af898b00f88f06e021edcb16d7d558efad is a statically linked ELF file compiled by GCC. There aren't any hints in the ELF format about this targeting Linux, there's only a .note.gnu.property that specifies the CPU needs a few particular ISA features.

Within the file strings, there are some hints:

• file paths, like /share/zoneinfo/ and /etc/localtime look like Linux paths, though BSD also uses /etc/localtime so its not very convincing.
• the string LINUX_2.6

I will see if I can figure out if LINUX_2.6 is used by some common runtime and we can signature this, or if its derived from the malware's source code.

All in all, I think you did the right thing: to determine the sample targets Linux and provide the hint to capa. Unfortunately, capa won't always be able to recognize the underlying OS, so these hints are required.

[williballenthin]

the LINUX_2.6 string comes from a manual symbol resolution implementation related to VDSO routines (__vdso_clock_gettime). VDSO is describe here: https://man7.org/linux/man-pages/man7/vdso.7.html

We could inspect the strings of the file and note any of the symbol and version strings, using these to infer that VDSO on Linux is used. I don't really like that we're just guessing based on strings, but I suppose it's better than nothing.

[williballenthin]

a69ce71a6b5cef7aadf343c93e00e1ffc549d649bd011fbb39bfa38534484511 is a statically linked ELF file compiled from Go. There also aren't hints in the ELF format about this targeting Linux, but there is Go metadata that we could use.

Since Go is so prevalent, I think we should extend our OS detection code to recognize ELF binaries compiled by Go. The GoReSym technique is found here: https://github.com/mandiant/GoReSym/blob/0860a1b1b4f3495e9fb7e71eb4386bf3e0a7c500/main.go#L150-L178

[williballenthin]

Actually, here's the algorithm we should prefer to use: https://github.com/mandiant/GoReSym/blob/0860a1b1b4f3495e9fb7e71eb4386bf3e0a7c500/buildinfo/buildinfo.go#L105

https://github.com/mandiant/GoReSym/blob/0860a1b1b4f3495e9fb7e71eb4386bf3e0a7c500/buildinfo/buildinfo.go#L283-L295

[C0d3R3ad3r]

Hopefully, these suggestions can be implemented in the next update to CAPA. For now, I can use the --os option to have capa analyze the files. Also, thank you for the quick response.

I wasn't sure how capa determined the file type. I would think something like the file command or custom file determination scheme would easily pick up that the file was a 64 bit elf file. In the meantime, I can script something up to use the file command and then pass the file along to capa with the proper --os options

[williballenthin]

A few minor comments here, not to be combative, but just for information:

would easily pick up that the file was a 64 bit elf file

We can tell the file is a 64-bit ELF (specifically, an ELF file with 64-bit little endian words), but this doesn't tell us the OS. While we can probably guess that most of these are for Linux, I'm afraid we'd lead too many people astray when encountering a FreeBSD file, for example. So, we try to be pretty sure, or we bail and ask the user to specify --os.

the file command

I went looking how how file/magic determines the OS, and it's surprisingly coarse. We already cover the logic that magic uses (to my knowledge), and have a few more heuristics. Though, if you find that magic does a better job, do share some details here and we can emulate it even better!

[williballenthin]

I have an implementation of OS detection for Go ELF files pending, should have that PR opened tomorrow. I think that will cover a lot of the edge cases that people see today. I'm glad that @C0d3R3ad3r mentioned it here.

[C0d3R3ad3r]

Not taken as combative at all. Just trying to find the best way to process these types of files. I have encountered a lot of 64 bit elf files and show that they are not supported but using the --os option, capa runs as expected. I do appreciate all the comments and like I said, I am just trying to find some workaround for now. I get what you are saying though.

[williballenthin]

Feel free to post as many sample hashes as you like into this chat - I'd be happy to triage them and add heuristics as feasible.

[C0d3R3ad3r]

As a quick fix, I wrote a python script which imports magic library so when I pass a file through to capa (so I can do this in bulk), I can compare strings looking for "ELF 64-bit" or "ELF 32-bit". It works for my purposes, but I hope in a later release of capa this can be automatic.

import magic
mime = magic.Magic(mime=False) // Setting to false outputs human readable ie. ELF 64-bit instead of application/x-executable

mimeType = mime.from_file(file_path)

    if (
        "ELF 64-bit" in mimeType
    ):
        format_args = ["--os", "linux"]

    if (
        "ELF 32-bit" in mimeType
    ):
        format_args = ["--os", "linux"]

'

Trivial, but for now it works.
Sample I tried was 443d71fbee450a75eadb258cc48289bc (upx unpack it)
Screenshot was just run from the command line