As a side note, don’t be cute and pick port 221 or 2222 or 22022 or whatever that’s got “22” in it.

I know that sounds silly but the slightly less stupid bots are written by people who understand people do things like that and account for them, and thus port 2222 isn’t actually better than 22, or whatever.

Thanks! I did mention this briefly, although I belong to the school that “since I am anyway banning IPs that fail authentication a few times, it’s not worth changing the port”. I think that it’s a valid thing especially if you ingest logs somewhere, but if you do don’t choose 2222! I have added a link to shodan in the post, which shows that almost everybody who changes port, changes to 2222!

SSH is designed to be exposed to the Internet. I mean, fine, if people want to use weak security-through-obscurity stuff on top of it, it probably isn’t going to actively hurt anything, but there are a few people on here who I would say really seem excessively worried about SSH relative to anything else.

Keep your system up-to-date. Use key-based authentication. There are a lot of eyeballs on sshd’s authentication stuff. If someone finds a zero-day exploit in sshd, they are probably not going to be expending it on you and risking having it discovered unless you are some kind of specially-valuable target. Anyone of the sort who is going to be in possession of a zero-day exploit in sshd is very, very unlikely to be stymied by you moving your service to another port.

But people are trying to brute force passwords.

Okay. So? They’re looking for poorly-configured or never-configured devices using password authentication that are exposed to the Internet with default passwords, which you aren’t doing, right?

There are 65,536 ports to test. Something like nmap can scan these for open ports at a pretty good clip in parallel. There is protocol-fingerprint software that, given an open port, can recognize a service running on it via a service fingerprint. SSH has an extremely obvious fingerprint, so you don’t even need any fancy software to identify the service, given a port, and anything that does even the slightest investigation is going to know that, yes, that port has sshd connected to it.

$ nc localhost 22
SSH-2.0-OpenSSH_9.7p1 Debian-7

Someone building a database of what’s out there on the Internet is going to have no difficulty at all in finding your nonstandard-port sshd. There are probably many databases out there that list your server already. Hell, I bet that there are people who track how frequently people update their sshds.

Now, let’s go back to the password-brute-forcing crowd doing their scans for systems with password auth enabled and “root” and a blank password or whatever.

If you’re using pubkeys, password auth isn’t an issue in the first place. And the effort required to brute-force even a very bad, single-English-word password, much less a pubkey is a whole lot larger than the effort required to brute-force find an sshd running on an unusual port. The likelihood that someone is going to brute-force even a good password by just hammering at an sshd with their bot is not high. The vast bulk of what I see is people looking for “didn’t bother to set a password or used one of about three passwords” systems.

Like, healthy caution, sure, but I have a long list of things that I’d be vastly more-concerned about from a system compromise standpoint than some random person out there bashing their way through my sshd instance. What software do you run on your computer? Do you trust the eyes on any open-source software to be sufficient to catch insertion of bad software? I have about 4,400 packages installed on my desktop. I know even a little bit about the developers on maybe 10 of them. Do you run any closed-source software? I have a huge Steam library, virtually all of which are closed source. Do you trust the providers of that software? Do you trust them not only not to compromise your system but to have secured their systems against compromise? What about mods for those games? What about the systems used by the people who write those? What about your browser? What about your video drivers that some random website might touch in your browser via WebGL?

There is one neat trick: don’t expose SSH.

There is still not a reason anyone has been able to give for 99% of self-hosters to expose SSH.

If you need to access your machine via ssh while on the go. Wireguard to your local network, use SSH. Done. Unless you are running an always-up public facing site, the amount of times you have to access your machine that can’t wait until after work is very low anyway.

Bots will scan all ports. That is just how it works. Less than 22, but you will still get spammed. Why force your computer to go through the fail2ban loop and take up resources when it is simply not needed at all and you can block it on another machine?

That is basically the essence of this post too! Except crowdsec is used to do what fail2ban does + some light form of WAF (without spinning another machine - which is not strictly needed for a WAF, you can use owasp modsecurity-ready proxies).

AFAIK I know that SSH has MaxAuthTries and LoginGraceTime, but all it does is terminating the SSH session (I.e. slow down at most), it won’t block the IP via firewall or configuration.

Not sure if there is a recent feature that does the same.

Hey, the short answer is yes, you can.

I would elaborate a little more:

• First, you have the problem of sourcing the data. In essence, Crowdsec won’t be able to go and fetch those logs for you dynamically, but can go and take those logs from a file (you can do a dirty solution like a sidecar deployment) or from a stream. You can deploy crowdsec in multiple modes, and you can have many instances that talk to each other. You can also simply have some process tailing the pod logs and sending them to a file crowdsec has access to or serving them as a stream (see https://doc.crowdsec.net/docs/data_sources/intro).
• The above means that it doesn’t really matter whether you run Crowdsec inside your cluster (it does have a Helm chart) or on the host. Ultimately all it matters is that crowdsec has access to your pods logs (for example, the logs of your ingress controller).
• The next piece is the remediation component. What do you want crowdsec to do, once it is able to detect bad IPs? If you want to just add IPs to the firewall, then it might make more sense running it on the host(s) you use in ingress, if you want to add the IPs to network policies you can do it, but you need to develop your own remediation components. I am planning to write a remediation component that will add the IPs to Hetzner firewall, some other systems are already supported, but this would be a way to basically block the IPs outside your cluster. For nginx ingress controller there is already a pre-made remediation component .

In practice I personally would choose a simple setup where the interesting logs are just forwarded (in Syslog format for example) to a single crowdsec instance. If you have ingress from a single node, I’d go for running it on the host and banning via firewall, if you have multiple ingress nodes, then I would run it inside the cluster and ban via a loadBalancer/cloud firewall/whatever you have in front.

In essence, I would spend some time to think about your preferences, and it might take a little bit to make the setup clean, but I think you have plenty of flexibility to do what you prefer. Let me know if you want to bounce some more ideas!

It is bad for the open web and constantly asks you to verify you are human

Fair question. What I meant is that suggesting that would have made the whole post 10 lines long and not worth doing. So I avoided such suggestions that completely change the threat model.

It’s not useless to avoid a good security posture (although you might have concerns of a monopoly gatekeeping the internet, TLS traffic inspection privacy concerns etc.), on the contrary makes everything I have written about here redundant (+ provide more, like DDoS protection) as you are outsourcing the security controls.

Yeah, what I mean is that it’s useless using ports like 2222, that’s like the unofficial SSH port! Bots are generally harmless (once you move to key auth), and you get functional the same result with the automatic IP ban on failed auth, minus the bother to change client configurations to your custom port. Anyway, if someone does want cleaner logs, changing port works :)

Oh, it looks like! Something went wrong with Zola build and I must have not noticed. Thanks a lot for pinging me about that, I will fix it today!

EDIT: Fixed! That’s what you get when you forget to bump the Docker image version after you upgrade zola version locally with a breaking change in the config! :) Thanks for letting me know, it would have taken me a long time to see it was broken!