A customer reported that a Linux machine used for ssh access (to in turn give telnet access to an ancient SCO machine) was refusing logins. I asked him to try logging in as root at the console; he was unable to do so.
When I arrived on site, I found that I could not login as he had said. I rebooted to single use mode and started peeking around. The machine had been hacked; there was little doubt about that. It's HOW it was hacked that bothers me,
First, there was no attempt to hide any evidence. I could see in wtmp and the secure logs that someone had logged in from a German ISP address, attained su status, and created a new su user for himself. He then changed root's password.
Fine so far, right? But then he did something very strange. He hand edited /etc/passwd and added "/nologin" at the end of each line except root and his own. This was what was preventing people from logging in.
Why do that?
My first thought was that this was just a disgruntled employee doing minor mischief. But when I went multi-user and started checking more, I found this:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
3 2614 root 3u IPv4 8033 TCP *:ircd (LISTEN)
That looks like the machine has been put into a botnet. I ran rkhunter but didn't find anything else unusual.
This is very odd. If you want the machine for a botnet, why disable the user logins, which only serves to immediately call attention to the machine?
Another oddity: this same issue happened several months earlier. That is, users could not login and the root password was changed. That time, the user access came back before I could get there and I had them boot to single user mode to change the root password. I wish I knew if an irc daemon was running then, but I attributed all of that to user error or a router glitch.
Could it be just an inept hacker? A "kiddie script" that disables logins? But why undo its work? And why redo it now?
And he DID redo it. The time stamps are plain: he did all this just days
ago. It makes no sense.
I suspect that this person got in because someone's home machine is already part of the botnet. I don't know how he attained escalated permission, but once you have physical access, all bets are off. We'll have to reinstall the machine, but if I can't identify the source, what's the point?
I don't know. I'm really not sure what to do. For the moment, I've locked down ssh so that only I can get on - I want to see if he does have another back door. But I'm also concerned about other machines in the network - any of these could be compromised also. So where do we go from here? I don't want to put this customer to a lot of expense for nothing, but the whole situation is disquieting.
It does offer a lesson though: when something odd like that happens, we
should take the time to look more deeply. If I had spotted that ircd months
ago, I'd have... what? I don't know. But still, I should have looked deeper then.