What do you do if your linux server gets hacked?

Photo by brianfuller6385

So my linux server got hacked. The symptom was reports from my virtual private server hosting company (Linode VPS) that I was using more than 20% of my cpu. I am allowed to use 20%, or even 100% of my cpu. It’s just that that is unusual, and I have my server set to email me when I go above the 20%.

Well, when I checked, I found that I was actually using 100% of the cpu. (Which is above 20%…) I don’t have a webserver enabled on that machine, just email. So I got pretty lax. I was running Ubuntu 7.1 when Ubuntu is already up to 10.04. Not updating is a recipe for being hacked.

Security is an arms race. Computer systems have gotten complicated enough, that perfect security is an elusive goal. If you want to use your server for more than a few services, the permutations soon become astronomical. It quickly becomes true that no two servers are running the same configuration.

So update. Update the kernel. Update your packages. And backup your system, so you can get back up and running, when things go wrong.

Now, I did update the packages a few times. But not recently enough, apparently. Many updates are security updates. A hacker figures out a bug in a software package. A bug can be a potential weakness that might be exploited. When you upgrade, you get to take advantage of much of the great work package authors exert to fix known bugs. So the quicker and more often you update, the shorter the vulnerability window is left open for exploits.

So what do you do when your server gets hacked? Well, finding out what your vulnerability was can be difficult. Once you are hacked, the hackers have likely created new vulnerabilities. So even if you do figure out how they got in, it is too late. Closing that door, when there are hidden back doors, is as they say closing the barn door after the horses have left.

So look around for your curiosity. But don’t try to repair your system. I discovered on my system a perl process was running at 100%. I noted the process number and changed directory to /proc/thatProcessNumberInoted. Once in there, you can find out what directory the process is running in (cwd), and everything else about that job. My system was running something with the message “Enjoy FloodBot based on OverKill”. A quick google search tells me that is a server that aids in distributed denial of service attacks. I am sympathetic to WikiLeaks, and sometimes Anonymous. But I don’t like being volunteered without my consent.

So curiosity satisfied, the next step is to fix the problem. To fix the system, you want to start with a new install, and reinstall all your needed packages. What a pain, I know. But it is the only safe path.

I used a free one gig partition to install a new root partition. I used Ubuntu 10.04. For email, I reinstalled Exim4 using Aptitude.

Configuring the system can be a bit of hard work. The slightly risky, but worth it shortcut I took was to mount all of my old file systems as disks for my new system. The risky part, is I then copied the text configuration files for network setup and Exim4 virtual server configuration. What limits the risk, is this is a finite number of files, and I read them. I may have missed something, but I did create the configuration the first time, so I think I am pretty safe.

Facebook comments:

Leave a Reply