Problems running analysis on large program

[Bobbias]

I'm trying to use ooanalyzer on a relatively large program, specifically, the video game Enemy Nations, however the tool seems to crash while running analysis.

The game's source code was released, however, it appears to be missing at least one source file entirely (comprising at least 2 classes), and potentially more outside of the actual main executable.

I've been interested in modernizing the source code and creating an updated version of the game. However, the game appears to be missing source code in it's released form. I am not particularly experienced reverse engineering, although I'm not entirely unfamiliar with assembly. However, given that the source code I need to recover/recreate is largely comprised of a class which inherits several levels deep, and the game is quite large, I figured ooanalyzer was my best choice for getting some assistance even finding the functions in question.

However, upon running the first inspection, I realized that there will be some problems with this. The game uses smartheap as a replacement for new/free in the standard library, it also uses several DLLs for which there is no information in the database (miles sound system, as well as their own custom DLL for networking related stuff).

Here's a condensed list of the various warnings and errors I got before the program crashed:

Successor 0x00552A3B of 552A39: xor       eax, eax not found.
OOAN[WARN ]: No stack delta information for: VDMPlay.dll:vpSupportedTransports
[The above happens for every function in VDMPlay.dll, and mss32.dll, as well as several older windows functions]
...
FSEM[ERROR]: Analysis of function 0x00467940 failed: relative memory exceeded
[This happened a number of times]
OOAN[ERROR]: Unexpected second import on call at 0x0041E8DDold=Import: addr=61a194 name='USER32.dll:SendMessageA' delta=+0010(User) callers=[ ... ] new=Import: addr=61a1b0 name='USER32.dll:SetScrollPos' delta=+0010(User) callers=[ ... ]
[The above repeats, except the old name is 'USER32.dll:GetScrollPos', and the new name is 'USER32.dll:SendMessageA']
FSEM[ERROR]: Function 0x00474BF6 has no out edges.
OOAN[ERROR]: No new() methods were found.  Heap objects may not be detected.
OOAN[ERROR]: No delete() methods were found.  Object analysis may be impaired.
PLOG[WARN ]: Consistency checks failed.
PLOG[WARN ]: Class 0x55b450 inherits from 0x561eb4 at offsets 0 and 0x30
[The above 2 messages then repeat several times, with about 4 different classes mentioned]
OPTI[MARCH]: Prolog facts: [--#####--------] 45102Killed

I've seen mention of manually adding function information to the API database, so dealing with the stack delta issues shouldn't be too difficult. I can also increase memory limits, so hopefully I can solve that issue. But I have no idea what the other errors/warnings mean, or why the tool simply crashed.

[sei-eschwartz]

OOAnalyzer may be a good solution, although the success rate on large games has been mixed. If you haven't already, you may want to see if there is RTTI data for the classes you are looking for. If there is, other tools might be easier.

If the game is using a smart heap, you'll definitely want to manually annotate the new and delete methods. I wouldn't worry too much about unknown DLLs unless they seem related to the classes you are interested in.

Your log ends with "Killed", which suggests that OOAnalyzer was killed because your machine was out of memory. You can probably confirm this by looking at the dmesg logs. How much memory does your machine have?

[Bobbias]

It's good to hear that I'm not attempting something out of the realm of reality with this at least.

Looks like it was killed for OOM:

kern  :err   : [  +0.000286] Out of memory: Killed process 445 (ooanalyzer) total-vm:9958640kB, anon-rss:7656124kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:19440kB oom_score_adj:0
kern  :info  : [  +0.258044] oom_reaper: reaped process 445 (ooanalyzer), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

I'm running ooanalyzer in WSL2 and I've got 16GB of ram.

I'm not sure what you mean when you refer to manually annotating the new/delete methods. Are you referring to something related to ooanalyzer, or simply identifying and naming them in Ghidra? I haven't identified them as of yet while reversing stuff in ghidra.

[sei-eschwartz]

I'm running ooanalyzer in WSL2 and I've got 16GB of ram.

As a first step, make sure that all 16 GB of ram are available to WSL2. But even so, it's possible that 16GB is insufficient.

I'm not sure what you mean when you refer to manually annotating the new/delete methods. Are you referring to something related to ooanalyzer, or simply identifying and naming them in Ghidra? I haven't identified them as of yet while reversing stuff in ghidra.

I mean passing the --new-method=ADDR and --delete-method=ADDR arguments once you figure out where the new and delete methods are.