In the past, securing SSH on the public internet has been pretty much as easy as (a) keep your OS patched, (b) don’t let root log in with a password, and (c) run fail2ban to stop brute-force attacks.
Unfortunately, it looks like the bad guys have finally figured out how to put their bots to work running distributed SSH brute-force attacks. If so, then fail2ban is no longer going to be good enough, and more sophisticated (and inconvenient) measures are going to be needed.
Prior to December 1, the five machines I maintain with SSH servers accessible to the public have been probed by an average of 13 different IP addresses per day. On December 1, they were probed by 109 different IP addresses, a 738% increase over the prior average. On December 2 and 3, they were probed by 79 and 72 different IP addresses. Not as high as the first day, but still quite a jump!
I saw this increase across the board on five different machines on four distinct networks run by four different network service providers. I’ve been in correspondence with someone at the SANS Internet Storm Center who says he’s seen a similar spike on machines he maintains.
It seems clear to me that someone is engaging in a distributed brute-force attack trying to break into servers as root via ssh.
Since this particular attack is targeted at the root user, you’re safe for the time being as long as you don’t allow root to log in with a password. But it’s only a matter of time before they start attempting distributed brute-force attacks of non-root accounts. When that happens, blocking individual IP addresses with a series of failed login attempts is no longer going to be sufficient.
If you maintain a server whose SSH port is open to the public, please let me know the details if you’re seeing a similar attack on your server (you can post a comment here or email me. In case it is useful, here is the script I have been using to collect and display data from the machines I maintain.
UPDATE: It looks like it’s dying down. As of December 8, SSH brute-force attempts from distinct IP addresses are at or near their pre-spike levels:
Either somebody’s managed to put a stop to whatever was executing this attack, or the attackers have gone back to the drawing board and are tweaking their bots in preparation for the next attack. :-/
I manage about 80 Linux servers and see this on a daily basis. The attacks have increased over the years and are only getting worse. This has resulted in me blocking ssh from outside and only allowing certain ip’s to access via ssh. This is highly effective if you can do it. fail2ban is helpful, but not a complete solution. I found after removing port 22 some botnets will begin to attack other services like POP3 and IMAP. These cannot be blocked because most email users are using dynamic IP addresses. This is where fail2ban and bfd are useful. This is not just an SSH problem but a problem with any services which allow a user to authenticate. SSH, POP3, IMAP, FTP, etc. They are typically using a list of common users and common passwords. root, bin, user, test, temp, guest, joe, bob, david, etc. I have see email accounts broken into with weak passwords set by users and used to send spam through webmail applications.
1. Block all unused services and never allow SSH from any IP’s. If they can hack into a user account, they next step will be to login via ssh and attempt local root exploits. If you block SSH entirely it makes it EXTREMELY difficult to gain root access.
2. Run fail2ban or bfd on other services which cannot be closed such as POP3, IMAP and FTP.
I’ve seen no sign of a spike on my port-22 open-to-all (challenge-response only) OpenSSH. I had a clearly distributed spike around Sep 22, but nothing since other than the usual idiots on a single IP address pounding for ages. Even that is much less extreme than it used to be: a few years ago they’d keep it up for weeks, but now they give up after only fifty or sixty attempts.
Perhaps their bots are less stupid than they used to be, and rapidly come to the conclusion that my SSH daemon is not accepting password-interactive from the Internet, so drop it from later scans?
I run ssh on a non-standard port on 17 machines. I rarely (never?) see malicious login attempts. There has been no change or increase in probes in the past month. While running on the standard port (22?) I used to see hundreds of attempts a day.
HTH
LF
I also have observed a lot of distributed IP address SSH login attempts starting in early November. I disabled password authentication years ago and currently use certificates. My initial probe observations were the same as the others. The log entries showed an initial login request about an hour before the ‘main event’. The source address was different for each attempt and the usernames were all in alphabetical order. In late November, there were a few other attempts that tried the same list. My PF rule that blacklists IP addresses after too many failures wasn’t effective. The bad guys are not attempting the root login much anymore, but are using common usernames.
I’ve noticed a slight spike lately, but it doesn’t worry me over-much.
Even allowing root logins remotely by password via SSH, if the passwords are fairly strong and fail2ban/denyhosts block each source address in the cloud of attack bots after, say, 10 tries, unless the cloud is aggressively targeting you it still won’t add up to enough attempts to actually brute force the passwords. Some of the servers I run are medium sized e-commerce servers and I’m still never seeing over about 100 actual attempts per day.
By the way, I am seeing clever attempts at usernames other than “root”, based on the virtual sites hosted on each server.
And of course the suggestion to deny direct root login remotely, or require certificate authentication, is very good. I’m just saying that, at the moment, I’m not seeing enough change in the levels of this ongoing problem (search for my name and ssh brute force and you’ll see I was involved in some of these discussions 7 years ago, sigh).
And further of course the poster is absolutely right that we suffer common damage from the very high number of badly configured/ badly secured/ badly patched systems and servers which become the next attack bot. We do need, at a political and economic level, to help governments and industry self-regulatory bodies push for more effective reactive practices, e.g. auto-block non-commercial subscribers when an IDS thinks they’re infected, auto-warn commercial subscribers of same with auto-block to follow if fixes are not fairly quickly forthcoming, etc.
Good luck to all of us…
I’ve seen activity from distributed spambots (four different addresses attempting to send the same email in the space of a second or so), but haven’t seen distributed ssh attacks yet. That said I use denyhosts which might be stopping some of the attempts before they start. My server has ssh open but with password authentication disabled.
One data point that might be of interest: all the attacks I’ve seen start with a silent connect-disconnect, then the main attack follows a few hours or even days later.
I was reading the RISKS digest and saw your note, and checked the auth.log file on my home machine which is being probed even as I write this … the log is now 800KB per day, up from 150KB/day or so in mid November. I use it perhaps once a week, so that’s virtually all probing. I guess I should close port 22, but it’s not a very high security system, more for my own convenience.
Maybe someone sent out a virus designed to wake into a botnet on Dec 1st?
I am currently getting probed from a Russian data center, and from a DSL connection in Bangalore. Usernames mostly root but it is also trying things like “student”.
I have an SSH server that’s running all the time, on a nonstandard port, and also configured to only accept public key exchanges (no passwords). In the last two years, I’ve had about 800 total attempts to log in, but none recently (I made a couple invalid attempts after I read your note, just to make sure my logging was working). It may be because of the other protections, but I’m not seeing what you are.
When I do get errors, it’s almost never an attempt to get directly into root. The most common tries were “12345” (hey, that’s the same password as my luggage!) “test”, and “oracle”.
And that’s why I only ever use telnet.
Nate, I hope you were trying to make everyone laugh out loud by writing that. If so, you succeeded.
My home server (on Comcast) shows a spike, despite blocking addresses after too many failures — 89 on Dec 1 and 62 on Dec 2 vs 30 or so for previous days. I didn’t break it down to unique addresses.
I also notice all the breakin attempts show a client id string with some version of libssh or libssh2; the only ones claiming to be OpenSSH were my actual logins. Not that that’s useful to filter on if legitimate site users want to use libssh, or the bad guys catch this and start claiming to be using OpenSSH…
You’re only vulnerable if you permit root password logins *and* your root password is brute-force guessable. The botnets can hammer my servers all day long; I’m not worried about it.
Your servers may be safe, but other people’s servers getting broken into has a spillover affect that impacts all of us, most notably because those compromised servers can be used as bots to engage in additional attacks. So this is worth worrying about even if you’re smart enough to know how to protect your servers.
Even if your root password is not easily guessable, it’s a bad idea to allow remote connections to it. You really shouldn’t allow password-based ssh logins to ANY external account, it’s just begging for trouble.
The first order of business in getting a new shell is to set up a keypair, and then disabling password-based logins.