Hey Wiseass, Please Stop Hacking My Server

So I have this server I screw around with, throw XDA-types some free bandwidth and learn how to handle heavy traffic and to administer Linux, to experiment with tweaks before I drop them on this site (like the stock quote widget), this and that. It’s my digital mancave.

The other day I log in to see what’s doing with memcached which I installed to ease up the disk I/O activity (works great), I notice I’ve got recent timestamps on system binaries, wrong chmods, symbolic links in weird directions, netstat isn’t working, w, top, ps, basic everyday utilities. Netstat’s this thing that tells you about every TCP and UDP connection you’ve got, ports, processes attached to them. But the netstat I was looking at was only a few dozen bytes. So I cat netstat and get this:

Motd is the message of the day file, the thing you see when you log in through a text terminal, usually says something about the operating system. And here’s some guy inviting me to leave him a message to start a dialog of some sort, calling me by name, on this file on my system after I discover I’ve been rooted, implying that I should be confident that no matter what I try to do I won’t be able to prevent a subsequent reentry from someone who has taken an apparent persistent nefarious interest in me and my server.  And this line in the script he wrote I’m guessing is to make it harder for me to spot a back door if I were to start looking. If he wrote it himself, he’s pretty good; if he just copied it from some hacking website, he’s still good enough to be a threat with his crosshairs on me still.

What’s there to discuss man? What hints have we to swap? I am aware you did this and are at liberty to do it again. I concede perpetual defeat. If you’re thinking extortion, which I doubt, don’t bother, it’s a hobby server. If you’re in the market for some nonstop cat and mouse, I’m not — I’ll pull the plug. Is that what you want? You clearly know what you’re doing, I wish you’d channel that elsewhere. Please don’t turn this taunt into a haunt because I’ll just shutter the joint which would suck for me, you a little, and some guys cranking out roms on xda.

Creepy that it’s someone who knows me as I don’t know anyone socially, real life and online, off-hand who is clearly much better than I am at maintaining the upper hand in what appears to be a game or someone making a point or perhaps trying to teach me a lesson. I try to match the file timestamps against the auth.logs to get this guy’s IP but he covered his tracks, deleted the logs. So I cat the .bash_history, which he did not delete either because he neglected to or to just taunt me, and began to see that what he did after he was in was rather quick and surgical, no fumbling around.

The damage that I could find didn’t strike me as malicious, just a byproduct of the things he installed, including a backdoor daemon and possibly a phone-home thing I found after managing to restore the tainted system files (including those of the sysvinit-utils, like lastb, which I would otherwise have been able to use for clues on who did what to these files). Meaning I don’t think he lit out to break anything unsubtle, hopefully not use my system for anything illegal, just wanted to get my attention.

The right thing to do in this situation is to wipe out the server with a fresh install and put it back together one piece at a time. A few problems with that, I don’t have the time and even if I did, this guy is good and either because I enabled the wrong Apache module or used MySQL instead of Postgres or made the wrong entry in php.ini, all he has to do is be inclined to get back in and I bet he would, resulting in a lot of wasted time. It’s futile.

I don’t purport to be an expert but I’ve thrown the book at this thing to repair the damage and mitigate the vectors of attack, checked everything in init.d and crontab, watched netstat closely, got rid of extraneous daemons, tougher passwords, “hardening” the server, running a bunch of things that listen for sketchy activity and alert me, my own self-penetration testing with things like Nessus remotely, eyeballing what processes are running when I turn everything I use off, server-side with things like Snort and Nagios, .. I guess I should restrict logins to my usual IPs too, better do that now. I think he got in through MySQL somehow, I guess I could try to figure out how to unload all my databases into Postgres and set up everything I use that uses SQL to use Postgres. Maybe he’s not that great and just has a site of MySQL injection updates bookmarked. But I doubt that; I think he’s both good at this and enjoys it.

Or I can hope that he’ll read my response in the MOTD (or this article) and decide it’s getting old (he popped in again recently, this time piping the .bash_history into /dev/null):

Yes I’m aware that asking someone who rooted your server to stop busting your balls as a means of securing your system is not a means of securing your system, but either he obliges and I take his word for it or I have to pull the plug on this thing. I’m aware that writing this article isn’t too bright either, but I think it’s pretty good material. I’m also aware it’s bad etiquette to the rest of the Internet not to format here (or turn off the lights permanently), but c’mon.

Maybe it’s one of you guys. If so, please stop eating up my weekends trying to figure this out. I mean, good job, I get it, you know your way around a server better than I do. I’m confounded yet impressed. And to the rest of you, please don’t get any ideas, this is a big enough pain in the balls, I just wanted to vent here and maybe warn those of you with a server to make backups regularly, to be very selective with respect to things you install that rely on PHP and SQL, not to run anything you don’t need, resist the urge to go for the unstable distribution of your operating system and to remember that if you’ve got an IP you are always a target either to bots or to a friend like mine. If you just want to host some websites, go with shared hosting administered by professionals, don’t spring for dedicated, colo or a vps just to make things more interesting. But if you must, and you don’t want to offer yourself to guys like this with your legs spread quite as wide as mine are, forget Linux and maybe go with OpenBSD.

Doug Simmons