Rooter running even though route = none

[whavey]

I am attempting to use the beta virtualbox_websrv machinery and I think I am close. I am using an image built from the dockerfile mentioned in #30 by @ilyaglow . Everything seems to be working up to the VM starting. I docker-compose up and can browse to the web interface and submit a sample and the new vboxweb machinery looks like its about to start the VM but then immediately tries to stop it and I get the error above about the rooter socket not existing. That error happens when the the file /tmp/cuckoo-rooter doesnt exist according the the cuckoo core rooter module. But I thought that module shouldnt even be called and I that I dont need the rooter running if route = none in the routing.conf?

Not sure if the 2nd error about the Session not being locked is related.

[ilyaglow]

Hmm, can you give it a try again with parameter debug = True being set here https://github.com/blacktop/docker-cuckoo/blob/master/vbox/conf/virtualbox_websrv.conf#L21 ?

[whavey]

That is incredibly verbose, anything I should be looking for in particular?

[ilyaglow]

Hard to say, you can try to grep for results of IMachine_launchVMProcess SOAP call. Actually you can give a link to pastebin with all logs, or send me it as attachment to ilya at ilyaglotov.com. Not sure why rooter error pops up here though.

[whavey]

I tried to trim out the excess and just leave the main cuckoo debug output and just the soap call and responses.
https://pastebin.com/ggSQuUd8

It does look like maybe the waitforcompletion call after the launchVMProcess call returns an error: -2147467259

[whavey]

@blacktop Is the rooter supposed to be able to launch and maybe theres something wrong with the way the image was built? Or is the issue that something is causing the rooter to need to be started that shouldnt be?

[whavey]

I can get the rooter running by fixing some links and mounts (iptables and service in /usr/sbin and /sbin) and manually running cuckoo rooter in the container and then I dont get that error I get different errors from the rooter saying no such file or directory which I can troubleshoot but I dont understand why the rooter is having to do anything when route = none...

[blacktop]

Is this possibly something broken that might have been fixed upstream in cuckoo?

[blacktop]

@ilyaglow did your machinery PR ever get merged?

[ilyaglow]

No, it didn’t get merged, unfortunately. I thought about upstream changes, but if @whavey built my dockerfile with dirty patching it should be inherited from the basic cuckoo image, which was 2 months old. Anyway I’ll try to reproduce this issue today.

[ilyaglow]

Okay, I decided to make the whole setup from scratch and document it. Stay tuned.

[ilyaglow]

Just a quick notice - I think I figured out what was the cause of rooter error. Strangely it shows up because enable_net_trace call can set a filepath for a pcap that is not writable by vboxwebservice. So when machine is launched it fails immediately with a write file access denied error, and we can't see it because, well, I didn't handle that.

It seems that no pcap file on a specific folder makes rooter unhappy and send this error.

Still working out the kinks.

[whavey]

@ilyaglow any updates? I attempted to run it again using your patch from 0d6012b and building the image from @blacktop patch 9b86ada so I didnt have to use the dirty patch to build. Still had same issues so its probably what you mentioned above about the enable_net_trace permissions for the pcap file. Should I try placing a pcap file (dump.pcap?) somewhere with certain permissions?
Not trying to pester, just have deadlines.

[ilyaglow]

The manual is mostly done.

I’m a bit away from my working box, so here’s a couple things you should check:

1. remotevbox python package inside docker should be the latest version available (0.1.5) - there was a locking issue sometimes.
2. the user that starts VBoxWebService on host should have uid=1000 (as a cuckoo user inside docker container), and your cuckoo-storage shared between host and docker should be chown’ed by this uid.

[ilyaglow]

Actually, changing the uid of cuckoo user inside docker is probably easier than moving all your virtual machines to another user (in my experience).

[whavey]

Trying to understand this, so if I have a vboxweb user (the one specified in /etc/default/virtualbox) on the host with a uid of <vboxuser_uid> I need to have the cuckoo user inside the container have the same uid? Would the best way of going about that be to modify the docker-entrypoint script somewhere in this area?: https://github.com/blacktop/docker-cuckoo/blob/master/2.0/docker-entrypoint.sh#L118

[ilyaglow]

Trying to understand this, so if I have a vboxweb user (the one specified in /etc/default/virtualbox) on the host with a uid of <vboxuser_uid> I need to have the cuckoo user inside the container have the same uid?

Yes, exactly.

Would the best way of going about that be to modify the docker-entrypoint script somewhere in this area?

Still trying to figure out the best way to do it... But I like your idea. In #35 I proposed another approach: make uid configurable when you build an image and it is probably a wrong path, but anyway the existing Dockerfile just doesn't work for me somehow.

Did you change something to make it work properly?

[ilyaglow]

@whavey I've added additional checks from the machinery side, so rebuild your image. It should fail early if something is wrong.

[whavey]

I cloned from scratch and built from the file in the vbox dir and the only errors I got were for incompatibilities with some package versions and mitmproxy. Still built successfully though. Should I try and start the container now and see if its working or wait until #35 gets pulled?

[ilyaglow]

Oh, sorry, I meant the cuckoo will fail early, when you will start the container :)

You can try to checkout my vbox-work branch (aka #35), and change build args CUCKOO_UID in docker-compose.vbox.yml to your <vboxuser_uid>, then do docker-compose -f docker-compose.vbox.yml build and docker-compose -f docker-compose.vbox.yml up -d.

[whavey]

I must be losing my mind... I cant get it to build I keep getting: ERROR: Dockerfile parse error line 5: ARG requires exactly one argument.
I tried it even after a fresh pull not even modifying the UID to my own, Im running the exact command you have above. The syntax of the compose file looks right as far as I can tell I dont know whats going on

[whavey]

oh. in the dockerfile in vbox. line 5 should be: ARG CUCKOO_UID=25000 not ARG CUCKOO_UID 25000

[ilyaglow]

Fixed, thanks

[ilyaglow]

Pull latest changes and you will be able to uncomment and set environment variable CUCKOO_UID in vbox/config-file.env to change cuckoo uid inside container on entrypoint level.

[whavey]

Awesome. Im away from my computer for the weekend but last I tried from a fresh build I was getting errors about a database schema not matching what was expected and to run some cuckoo migrate command which I tried and it didn't work. Don't know if that sounds familiar but I'll try and get some screenshots to you and troubleshoot with the latest changrs more on monday.

…

On Fri, Jun 29, 2018, 5:02 PM Ilya Glotov ***@***.***> wrote: Pull latest changes and you will be able to uncomment and set environment variable CUCKOO_UID in vbox/config-file.env to change cuckoo uid inside container on entrypoint level. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQAwfSczhaLz8MfNlj1eS11yTmpugNo0ks5uBrHygaJpZM4UvwrK> .

[ilyaglow]

This is because the latest cuckoo version 2.0.6 has database schema incompatibilities with 2.0.5.

I think you can just remove db volumes with docker-compose -f docker-compose.vbox.yml down -v.

[whavey]

So Im not getting any errors anymore but the VM isnt getting started by the virtualbox_websrv machinery. I get "<vm_name> not ready yet" until it times out. If I start the VM manually the analysis starts up and runs no problem. Im wondering if I have some other permissions issues so I tried using the noauth approach that you talked about in the readme on your vbox-work branch but when I dont specify a user and password in the websrv conf file I get a critical cuckoo error: Wrong credentials supplied in virtualbox_websrv.conf configuration file.

I set the cuckoo uid via the config-env file. I imported my windows sandbox vm as root but registered it as my vbox user that runs the web service. Then I run docker-compose up on the compose file as root. Does that all sound ok?

[ilyaglow]

Did you update remotevbox machinery to the latest version?
I suppose you still has network dump write permission error, but it’s weird that the latest machinery version didn’t say a thing about it.

Do you have an access to the display on your host? Another way to debug this issue is to see if the analysis vm actually starts in Virtualbox Manager and if it immediately fails, you should try to run it manually and see the error message.

[whavey]

Did you update remotevbox machinery to the latest version?

Yes I just rebuilt the image so that pulled the latest I believe (0.1.5).

I suppose you still has network dump write permission error, but it’s weird that the latest machinery version didn’t say a thing about it.

I did get that error after the timeout. After the timeout there was a warning about trying to stop an already stopped VM and yes I see the dump.pcap permissions issues.

Do you have an access to the display on your host? Another way to debug this issue is to see if the analysis vm actually starts in Virtualbox Manager and if it immediately fails, you should try to run it manually and see the error message.

I do have access to the display and Ive run it with the virtualbox manager gui (run as vbox user) and itll switch from a saved state to powered on for half a second and then go to powered off. In the cuckoo logs I see:
acquired machine <vm_name>
obtaining vm <vm_name>
Restoring machine and powering it off
Enable network tracing
Start vm <vm_name>
Starting analysis on guest
and then it goes into <vm_name>:not ready yet until it times out
Interestingly when I try to start the vm manually after all of this it fails with an error saying: Netsniffer cannot open '.../<analysis_number>/dump.pcap' for writing. The directory must exist and be writable for the current user

[ilyaglow]

Okay, I've been there - it means that your user on the host, that starts vboxwebsrv doesn't have rights to write to a shared storage.

Let's check uids and permissions.
Could you show me the output of the following commands?

id -u your-vbox-user
ls -lan /your-shared-cuckoo-storage-on-host
ls -lan /your-shared-cuckoo-storage-on-host/analyses
ps naux | grep -i vboxweb

And while the docker-compose is up:

docker-compose -f docker-compose.vbox.yml exec cuckoo id -u cuckoo
docker-compose -f docker-compose.vbox.yml exec cuckoo ls -lan /cuckoo/storage
docker-compose -f docker-compose.vbox.yml exec cuckoo ls -lan /cuckoo/storage/analyses

[whavey]

2020 is my vbox user

2020 owns storage, storage/analyses, and storage/binaries but not the contents underneath storage and binaries. 1000 owns everything thats being created there.

ps naux shows 2020 started vboxweb but that was me trying to run it in the background manually as vboxuser instead of running the service. I usually do systemctl start vboxweb-service as root.

the cuckoo uid in the container is still showing 1000 and the perms in the container are the same as the host; 2020 owns storage and storage/{analyses,binaries} but 1000 owns the contents underneath.

[whavey]

my fault, the compose file was set to use the config env file from the 2.0 directory instead of vbox. The permissions in the container and on the host show 2020 as the owner for the contents under analyses now. The group is 1000 still. And Im still getting same issue with the VM powering on for a second then turning off

[ilyaglow]

So if you try to start the VM immediately after it turned off, it will show the same netsniffer error?

Probably you should double check what uid actually starts vboxweb-service / vboxweb in background mode and which one is set in vbox/conf/virtualbox_websrv.conf file. Even if it's a service it should drop privileges to a user defined in /etc/default/virtualbox and if it doesn't - that is the problem.

[whavey]

Sorry for the delay I have some updates.

Running vboxweb as a service appears to just not work, likely because of some permissions issues somewhere that I just am not having any luck troubleshooting. Even trying the noauth approach (setting the vboxwebauth setting to null) doesnt seem to help. I always get the netsniffer error after the VM starts and immediately dies.

However running vboxweb from command line as my vbox user (with the UID that matches what cuckoo runs as) seems to be... ok. I dont get the netsniffer error but that may be because the VM never tries to start. I can start it manually and the analysis runs beautifully then terminates and the machinery tries to stop the VM in cuckoo but fails saying: trying to stop and already stopped VM (even though its not stopped). So it seems like the machinery is not able to get the status correctly?

Just to verify:
as vbox user (uid matches cuckoo) 'vboxmanage list vms' shows my cuckoo vm so I believe that means the analysis vm is registered as that user

/etc/default/virtualbox has VBOXWEB_USER=my vbox user VBOX_HOST=my host ip thats running the cuckoo docker containers

vbox/conf/virtualbox_websrv.conf:

• url=ip of host running docker cuckoo
• user = my vbox user
• password = password of above
• remote_storage=path/to/storage (not using the default /mnt/cuckoo-storage but rather a directory I made with perms for my vbox user uid as owner for storage/{analyses,binaries})

[ilyaglow]

What virtualbox version do you use?

I suspect that on some distros/builds /etc/default/virtualbox is ignored and the service still runs as root. But okay, let's forget about it and try to use vboxwebsrv binary as a vbox user.

After you run vboxwebservice as a vbox user (vboxwebsrv --host <your-ip>) you should restart cuckoo container and see if it fails - it should if machinery won't pass initialize checks.

Also, check the fact that you can connect to and operate vboxwebservice from a cuckoo container:

docker-compose exec cuckoo ash
python -c 'import remotevbox; vbox = remotevbox.connect("http://your-vboxweb-ip:port", "user", "password"); print(vbox.list_machines()); vbox.disconnect();'

The last command should return a list of your vms.

[whavey]

Sooo close.
So I just wiped everything and started from scratch importing the ova as my vbox user. Before I had imported it as root and just registered it with vbox user, apparently thats not good enough. But now the machinery starts and stops the analysis vm and everthing works with using command line vboxwebsrv... except; it doesnt restore the snapshot. As far as I can tell it doesnt even attempt to restore the snapshot so the analysis vm starts running and I have to log in.

I have Snapshot 1 taken for the vm and thats what I have for snapshot in virtualbox_websrv.conf

[ilyaglow]

Seems like my fault. Will fix it, thanks!

Update:
All machines declared in virtualbox_websrv.conf should be restored when cuckoo container starts, but not when cuckoo starts machines. It works for me like that just fine, but as we can see it's a flawed logic.

[ilyaglow]

Updated remotevbox package and remotevbox-machinery cuckoo fork branch, so rebuild your images. Added more error handling and vm restoration.

As always everything works fine for me, but probably will brake on someone else's setup, so your feedback is really important!

I'm now thinking about how to write useful tests for the machinery, though...

[whavey]

getting error now on: if status == machine.POWEROFF or status == machine.ABORTED
attributerror: IMachine object has no attribute 'POWEROFF'.

wasnt getting that before and the only thing I changed was I rebuilt and swapped out the image

[ilyaglow]

You have old version of remotevbox package installed in the container. Did you rebuild cuckoo image with --no-cache switch? This is the proper way to rebuild an image.

[blacktop]

@whavey were you able to get this to work?

[whavey]

Sorry I went on vacation before I was able to fully verify but Im sure the issue was as ilyaglow said; I didnt build the image with the --no-cache switch.
At this point the issue I initial posted has been resolved and weve just been working on other issues. This issue could probably be marked as resolved and if I have other problems when I get back I can post a new issue.

[blacktop]

We could maybe move this conversation to @ilyaglow's PR #35?

[blacktop]

I'll close this issue out now.