Is my server hacked?

Over the last two weeks, I’ve discussed how to prevent crackers from gaining access to your Linux computer. This week, we continue the series with ways you can tell if someone has cracked your machine.Script kiddies are the worse kinds of crackers, primarily because there are so many of them and most of them are unskilled. It is one thing to be cracked when you have put in all the correct patches, have a tested firewall, and run advanced intrusion detection actively on multiple levels. It is another when you are cracked because you were lazy and didn’t, for example, install the latest patch to BIND.

It’s embarrassing to be cracked because you weren’t paying attention. It’s aggravating to realize that some script kiddie downloaded one of many well known “root kits” or publicly available exploits, and is having a party with your CPU, storage, data, and bandwidth. How do these villains get started? The answer is usually with “warez,” which often consists of a root kit.

A root kit is a software package that a cracker uses to provide himself (and it’s usually a “he”) with root-level access on your machine. Once the cracker has root access on your machine, it is all over. The only method of recourse that is truly effective is to back up your data, wipe the disks, and reinstall the operating system. However, it is not always easy to discover that someone has taken over your machine.

Can you trust your ps command?

The first trick in finding a root kit is to run the command ps. Chances are that everything will look normal to you. Here is an example ps output:

1 ?        S      0:05 init
2 ?        SW     0:00 [kflushd]
3 ?        SW     0:00 [kupdate]
4 ?        SW     0:00 [kswapd]
5 ?        SW     0:00 [keventd]
6 ?        SW     0:00 [mdrecoveryd]
2655 ?        S      0:01 syslogd -m 0
2664 ?        S      0:00 klogd
2678 ?        S      0:01 identd -e -o
2685 ?        S      0:02 identd -e -o
2686 ?        S      0:56 identd -e -o
2688 ?        S      0:55 identd -e -o
2690 ?        S      0:01 identd -e -o
2696 ?        S      0:00 /usr/sbin/atd
2710 ?        S      0:00 crond
2724 ?        S      0:00 inetd

The real question is, however, “Is everything actually normal?” A common trick that a cracker will use is to replace the ps command. The replaced version will mask illicit programs running on your machine. To test this, check the size of your ps application. It is usually located in /bin/ps. On our Linux machines it is about 60 kilobytes. I recently encountered a root kit that had replaced the ps program. The compromised ps from the root kit was only 12 kilobytes in size.

Another obvious trick is the linking of root’s command history file to /dev/null. The command history file is used to track and log commands that are issued by a user when they log into a Linux machine. Crackers will redirect your history file to /dev/null so that you can not see what commands they were typing.

You can access your history file by typing history at your shell prompt. If you find yourself using the history command, and it does not display any previously used commands, take a look at your ~/.bash_history file. If the file is empty, perform a ls -l ~/.bash_history. When you perform the previous command you should see something similar to the following:

-rw-------    1 jd   jd   13829 Oct 10 17:06 /home/jd/.bash_history

However, you may see something like this:

lrwxrwxrwx    1 jd   jd   9 Oct 10 19:40 /home/jd/.bash_history -> /dev/null

If you see the above, the .bash_history file has been redirected to /dev/null. This is a dead giveaway. Take your machine off the Internet now, back up your data (if you can), and begin a reinstallation.

Look for unknown user accounts

While you are playing detective on your Linux machine, it is always smart to check for unknown user accounts. The next time you log into your Linux box, type the following command:

grep :x:0: /etc/passwd

The only line, I repeat, the only line that the grep command should return on a standard Linux installation is something similar to the following:


If your system returns more than one line with the previous grep command, you may have a problem. There should only be one user with the UID of 0 and if that grep command returns more than one line, you have more than one.

Charles Y.

Charles has been working as a webmaster since 1998. Since then, he has had his hands in thousands of websites and has helped millions get online through a company he partially owns called Web Host Pro.

Leave a Reply