Log in

No account? Create an account
vincent.jb.org incident -- explanation for non-sysadmins - Alex Belits
vincent.jb.org incident -- explanation for non-sysadmins
Last two entries were about the vincent.jb.org incident, and I guess, many people who had seen them, found most of the text to be unreadable technobabble. Since at this point the immediate problems are solved -- vincent.jb.org is up, and scammers lost their email addresses -- it makes sense to explain, what the hell happened, and why.

First, things and people involved:


vincent.jb.org is a small server that belongs to j_b. By "small" I mean PII 400 with 512M of RAM and two hard drives -- 100G and 250G. It runs a bunch of services for Jon, hosts some sites, and some people have shell accounts on it -- they can login over ssh (secure shell), edit their web pages and scripts, run irc and mail clients, etc. The OS for quite a while remains Debian Sarge, a "stable" branch of Debian Linux distribution. PII 400 is a slow CPU, however it's sufficient for the purposes of this box.

It is located at my lab, sitting on the same network as some servers and desktops that I maintain, however I usually don't touch vincent unless Jon asks me -- he maintains the box remotely. At the moment of the incident Jon was in the process of configuring RAID1, a configuration with redundant hard drives that is supposed to be able to survive a disk failure.

In the process of configuring the controller for the second hard drive Jon had to install a system kernel that came from "unstable" (or development) branch of Debian, and disable the automated upgrade for the kernel. Debian (and other Debian-like distributions) has an automated update mechanism that covers all packages including the kernel image, and it works as long as the sysadmin keeps using the package listed in the index stored at the distro's server. When sysadmin runs "apt-get update; apt-get dist-upgrade", new indexes are downloaded, versions of packages are compared, and if there are any updates, they are installed, replacing the old versions, and package manager keeps things consistent.

However if sysadmin installs something manually, or uses a version that is not in repository, automated updates ignore the package, and it becomes the sysadmin's responsibility to update it if there is any need for it. Jon installed a kernel package from "unstable" branch, so it was a Debian package, however it did not match anything in the "stable" branch that was tracked by auto-update.

PRCTL bug and kernel versions with it

PRCTL bug is a design error in a Linux kernel that existed in a range of versions between 2.6.13 (inclusive) and (exclusive). It allows privilege escalation -- a regular user can exploit this bug to become root, a system administrator. The version numbers are for "official" kernel releases -- distributions usually backport security-related fixes into earlier versions, so many distributions had "2.6.16.something-something" versions that have this bug, and updated versions with the fix were still "2.6.16.something-somethingelse". Update mechanism works even if version number is changed, because the latest version is referred by "kernel-image-2.6" package, however distribution maintainers usually don't update kernel versions in the "stable" branch unless they are sure that it won't break compatibility. Everyone who had automated updates enabled for the kernel, got the updated version soon after the bug was discovered.

Debian "stable" branch was (and still is) at the version 2.6.8, plus all security/bugfixes updates (so the bug was never introduced in it), however the version installed on vincent was 2.6.15 (and therefore was buggy), taken at some point from the "unstable" branch, and therefore placed outside of any auto-update mechanism.

Skr1pt kiddies

Skr1pt kiddies (usually spelled like this, imitating their manner of writing) is a category of stupid people, who think, they are hackers. They roam around the Internet looking for hosts that can be exploited using some limited set of tools, usually scripts that perform something narrowly predefined, and collect compromised accounts and hosts for bragging rights and minor mischief. They rarely understand, what their tools do, and they completely depend on other people publishing pre-made scripts, so I have no idea what is the attraction of that activity, however the fact is, they are numerous.

Exploit scripts and rootkits

Exploit scripts are programs that automatically exploit some vulnerability -- usually they are written with skr1pt kiddies in mind, so to make sure that they work on everything, they are made in a form of Unix shell script that builds and runs everything necessary, then drops users into a root shell, or whatever it is supposed to accomplish.

Rootkits are programs that usually are used after the system is compromised, to hide the compromise, create backdoors, record what other users are doing, etc. -- usually it involves replacements for system utilities, libraries or even parts of the kernel. In theory, a properly made rootkit can completely hide its existence while it is running by always returning whatever would happen if it, and the intruder, wasn't there.

Internet scammers

Everyone with an email address encountered those -- they send spam email pretending to be all kinds of things, trying to trick the victims into sending them money, revealing bank accounts and credit card numbers, password to their bank or Paypal accounts. One of the methods relevant to this incident is sending an email supposedly from bank, Ebay, Paypal, etc., asking to "confirm" something, with a link to supposedly the web site in question. Link leads to a page that looks like that, however it is located on a completely different server, and once the user enters his password or account information there, everything is sent to the scammer while the user gets a fake confirmation page, or gets redirected to the real site. User continues on his merry way, scammer collects a huge number of accounts.

Scammers obviously want to keep low profile -- if someone is stupid enough to send spam or run a fake Paypal site on a computer that can be easily traced back to him, he is likely to be caught. So either everything is done in a place that openly supports spam and scams of this kind (and the number of those places decreases), or scammer uses others' computers that he takes over.

One way to take control over a large number of computers is to write a virus or worm. In the past viruses and worms had no practical purpose, and either caused pointless destruction, or tried to remain invisible and do nothing but copying themselves. Now most of them have a purpose -- to allow their "master" run something on the infected computers. When someone wants to send millions of copies of some spam, he just tells thousands of "zombies" to occasionally send a message, and soon his work is done without any bandwidth of his own, with source addresses all over the world, and nothing pointing back to him.

For this purpose it doesn't matter what kind of computer is compromised -- if it managed to receive something, it can be safely assumed that it can send email. So writing a worm that works only on Windows, and distributes itself only through Internet Explorer, or Outlook Express (two programs "responsible" for the overwhelming majority of worm/virus traffic) instantly yields a massive army of victims(called "botnet", as a network of bots). But most of those "zombie" computers are useless for hosting a fake site -- they are end-users' desktop, hidden behind firewalls, and with no server software installed. Firewalls are useless for defense againsy worms that distribute themselves through email or wait for the victim to visit a web page, however they are "good" at making the target worthless for hosting, so desktops in a botnet usually send spam, run keyloggers, steal their user's passwords but don't host web sites.

Now, what happened with vincent:

Part 1 -- Skr1pt kiddie

Apparently a skr1pt kiddie from Romaina (or skr1pt kiddie who previously broken into a computer in Romania) was messing with computers that belong to one of the users with an account on vincent. I don't know, what in particular he was trying to accomplish, or how he managed to get that user's password, however the fact is, he managed to log in to that user's home box (it's unclear if he managed to get any administrative permissions there) and to four boxes at his work, where he used PRCTL exploit to get root permissions. He used a pre-made script taken from the site psikoma.host.sk in Slovakia.

He then logged into vincent.jb.org as that user whose computer he broken into. Most likely skr1pt kiddie used one of the compromised computers to get a password for the account on vincent, and then tried to connect with it everywhere where he had seen the user connecting to, so he managed to login to vincent in exactly the same way as the real user did. Except that connection was from Romania. If he tried to be sneaky, he would login to the user's home box, and from there to vincent, so vincent logs wouldn't show anything unusual, however everything indicates that he never tried to cover his tracks. I thought that it may be for some purpose, and there was something else that he did cover, and left the obvious things for me to find, but I have found nothing of the kind, he probably just didn't care at all.

Once on vincent, he downloaded PRCTL exploit from psikoma.host.sk, and ran it. Exploit worked, so skr1pt kiddie got root access. What skr1pt kiddie didn't understand, was that even though he was running his shell as root, shell continued recording his commands as a normal user, and left a full record of what skr1pt kiddie did.

He either wanted to collect other users' passwords, or maybe just to keep root access to vincent. So he downloaded another file from the same host -- ssh, patched to open some kind of a backdoor, keylogger or maybe something else -- I didn't check which of those things were in there. He ran a regular building procedure, and ended up with trojaned ssh binary that he was supposed to install in place of the real ssh. And he kinda did it -- copied the binary in place where the original ssh was. The problem is, he didn't know how to run it.

He only knows what commands to run, not what they really do, so after he ran what he thought, was a system startup scrips responsible for ssh, to check its status, he found that the file wasn't in place where he expected it. He knew that the file might be elsewhere, so he went looking for the file with the same name -- and found some. Tried to run them as if they were startup scripts, and it didn't work because they weren't.

I don't know if he understood how close he was at that point to either succeeding (if he managed to run a new ssh after killing his own shell -- say, using crontab) or to rendering the box inaccessible for all users including himself (if he ran the *right* startup script, that was incompatible with his version, as I have discovered later). But at this point he apparently decided that he is over his head, downloaded a rootkit from the same psikoma.host.sk and ran it. Rootkit worked, and user was logged out, probably because among other things, rootkit restarted ssh, replacing it with its own version. From that point nothing pointed to the original intruder -- it's possible that he used the backdoor later, or that he and the second intruder were the same person, however there is no indication of either.

Part 2 -- Scammers

When the rootkit installed itself, skr1pt kiddie's password was emailed to exploitko@yahoo.com, so rootkit's author (or most likely, a person who modified it last) also got access to vincent. Few hours after the initial exploit, someone logged in, downloaded fake paypal site and scripts from psikoma.org, and left the scam site running. Fake site imitated Paypal login and credit card entry, and even made the browser request some meaningless files from paypalobjects.com to give an impression of a real Paypal site. Scripts collected all form entries and emailed them to priv8.1337@yahoo.com, another scammers' address.

First I thought that psikoma.org had script templates, and particular address was inserted by the intruder after he downloaded the files, however further examination had shown that files at psikoma.host.sk (psikoma.org disappeared by then) had that address hardcoded, just like exploitko@yahoo.com was hardcoded in the rootkit. What means that psikoma.host.sk owner provides exploits and rootkits to skr1pt kiddies, secretly collects their passwords, then installs scam sites there.

Scamming the password-stealing skr1pt kiddies by stealing their passwords to make sites that scam Paypal customers to steal their passwords and credit cards -- evil squared! Or maybe it's evil cubed and stupidity squared.

I don't know how many spam emails were sent by that scammer (they definitely used some other mechanism because I haven't found any spamming scripts installed on vincent), and how many sites they operated, however the fake Paypal site sent 65 messages, on average one in 38 minutes. Considering how fake the "confirmation" emails look like, I would say, indeed a sucker is born every minute.

Part 3 -- Cleanup or Flatten and Reinstall

vincent was compromised for two days before we have noticed it -- Jon rebooted the server, ssh startup script (the real one) started ssh executable (the compromised one, from the first skr1pt kiddie's attempt), and it did not work because startup script used options that are incompatible with the installed version. There was another startup script installed by a rootkit, however it didn't run at all because rootkit assumed a completely different Linux distribution -- it put the startup commands in a place where Debian doesn't run them. So users were finally locked out, and scam site (handled by Apache server that was installed originally) continued working. I have logged in from the console, reinstalled the real ssh server, Jon logged in remotely, and we have discovered that the box is compromised.

Since we knew that intruders compromised the root account, it was obvious that the box had to be reinstalled -- we can never know, what did they modify, and how it can manifest in the future even if we removed everything that was easy to find. Even though I have soon discovered the shell logs, it was obvious that some things could happen with logging disabled, and having root access to the server, intruders (both the original one and the scammer) could alter those logs.

First of all, I had to make a copy of everything that has to be restored. Intruders could hide things inside users' directories and web sites, however nothing within users' directory is supposed to run as anything privileged, so even if I restore some altered files from a backup, at worst it can affect the same user's files, or web sites -- that I would already have a backup copy of elsewhere. And, of course, I should disable key files that allow someone to login remotely as a user who owns the key file -- those keys' counterparts could be stolen from other computers, or intruders could add their own keys.

Database files also had to be restored -- as far as I know, nothing on vincent used database to store executables, and database is only accessed from web sites' scripts. It could be altered to deface the sites, but that would be the extent of it. I only had to keep in mind that databases contain passwords used by those applications, and it would be a good idea to assume that intruders know those passwords and can eventually use them to mess with the databases, so passwords will have to be changed.

Configuration files and DNS zones had to be sorted manually -- some of them may contain backdoors and fake sites, however the amount of data to sift through is small if I exclude everything that can be left at the default configuration, or made from scratch.

So first I had to backup users' directories over the network to some other box, where I can safely store them while reinstalling vincent, then copy databases, configuration files, and then if time premits, everything else. Vincent is compromised, and even though I have replaced ssh with a clean copy, there are many things that can still affect it -- libraries, PAMs (pluggable authentication modules, responsible among other things for checking passwords), kernel, and various other things running in the system. So I can't use compromised box to make a backup copy of anything, I have to boot a clean system.

Fortunately booting a clean system on a compromised box is easy -- many Linux distributions produce "live CDs" that are similar to installer CDs, except they boot into a complete self-contained environment. Ubuntu Linux has one of those CDs (it is actually its installer CD, that allows user to run the installer from already-booted environment), so I have booted Ubuntu and used it to copy everything to a server. To preserve user IDs and permissions, I have archived everything into a tar file ("Tape Archive" -- yes, it was originally intended for tapes) that then on the fly was compressed and sent to a server -- no local storage involved. This took a while, and I have spent some of that time analyzing the files on vincent that I could at that point safely read from a known-good environment of the live CD.

After making a copy of everything, and gathering most of the knowledge about the compromise in the process of looking at the files, I had to wipe and reinstall vincent. Debian Sarge installer CD is not nearly as advanced as Ubuntu, and one of those "not advanced" features is the lack of support for the second IDE controller, where one of the drives was, so I had to move both drives to the old controller. Fortunately it was not old enough to cause any incompatibilities as long as Linux kernel driver was talking to it. Jon wanted to install RAID1, what means everything is "mirrored" between two drives, and if either of them will fail, one copy will continue working until a replacement drive is installed. Good idea, considering that I have seen many disk failures followed by some rather painful (and not always successful) recovery procedures. Debian installer allows the user to partition the drives, assemble RAID and assign the positions of the partitions in the filesystem hierarchy. After partitioning and formatting is done, it installs the "base system", a minimal configuration that can boot on its own, then reboots and contunues installing packages until it reaches the configuration that the user asked for. No real surprises there, except I had to copy identical boot partitions and bootloaders onto both hard drives to be able to move the second drive to its own controller -- controller's firmware, BIOS and Linux drivers apparently had different ideas about the order of drives, so it made sense to make them look the same until all control is passed to Linux.

"Network" Debian installer takes few packages from the boot CD, and the rest from a repository over the network, what was fine for my purposes because the lab has a fast enough connection to the Internet -- that was the reason why vincent was there in the first place. So after a while a default configuration of Sarge was installed, I have added pine mail reader, and moved the second drive to the second controller. Apache web server was version 1.3 in the original configuration, and by default Sarge included 2.0, so to keep things compatible I have downgraded it to 1.3 (that is also supported by Sarge). At that point I still had the default network configuration, with IP address taken from DHCP, so vincent wasn't accessible from outside until I have finished restoring its configuration.

Using the same ssh and tar I have copied home directories and database files back. Some users had numeric user and group IDs conflicting with IDs already allocated for various services, so I had to manually change them to unallocated values. I have made new entries for all users, and kept them locked, except for root, me (abelits) and jon. I went through the users' home directories, disabled remote login keys, and looked through startup/logout scripts, but found nothing out of the ordinary. Database restored without any problems, I have changed its passwords and enabled passwords where originally database could be used by local users without a password. DNS zones and configuration files seemed to be ok, so I have put them back, too, however I have left mail configuration to Jon.

After getting some sleep I have looked at the vincent configuration, and found that kernel is still at the initial version that was installed with a minimal system. Debian properly records a kernel package, however it's a specific package for that version, not a package that automatically updates when new releases are made. Auto-update procedure does nothing to it. It is also a "386" kernel, that is supposed to work on everything starting from Intel 486, and is not optimized for any particular model of the CPU. I have chosen the correct package, installed it, and it immediately resolved dependency for the latest Sarge kernel for PII, so everything got updated. I still had to manually copy the new kernel between the drives to satisfy BIOS stupidity, but that was the only additional manual operation that will be needed from this point on -- auto-update now handles everything except pine. Even Apache 1.3 is a separate branch from 2.0, so since I have chosen Apache 1.3, it will get all updates for Apache 1.3, and won't try to automatically install 2.0.

Jon restored the users' access with new passwords, fixed mail configuration, and vincent is running without any problems since then.

Tags: , ,

Leave a comment