I socially engineered myself
For several years I managed a VPS running dokku, which is a very useful piece of software that allows you to deploy Docker containers, mostly web applications, with minimal effort. It automatically manages exposed ports, certificates, databases, and a bunch of other things if you want to.
I can't recommend it enough if you want git-push-and-forget-style deployment. It's not overwhelmingly complex compared to other tools in the same space and it helped me host lots of small experiments and useful tools:
- Frantisek, a Markov-chain-based Kafka bot that I built
- ntfy.sh, which is a self-hostable service to send push messages to your own phone
- Cryptpad, which is an end-to-end encrypted collaborative notepad / spreadsheet / file sharing app
- Echoip to find out the IP address of a single-board computer in my living room so the DNS record of a domain pointing to it could be kept up to date
- All my friends, which was a web app that I created so my friends could send me selfies at the start of the Covid-19 pandemic. I then used these to create 3D models of their skulls.
- A datasette that hosted a bunch of data I personally found interesting
- … and many more over time
The reason that this reads a bit like an obituary is because these services are not running anymore.
Running servers is easy
What I liked about running these services is that it proved, time and time again, that running something on the web is not as hard as it is sometimes made to look. Dokku is not a trivial piece of software, but it did many useful things, and it cost me almost zero maintenance. Again: if you're looking to have a reproducible git-push-to-deploy, Heroku-like workflow, consider Dokku more than anything else. It's a joy to use and it is simple enough to understand how it plumbs together the tools underneath.
I am not an expert in server security, I mostly know the basics: Don't use standard ports, do at least some basic monitoring, use strong keys / passwords, don't run unnecessary software, keep the software you run up-to-date, etc. Keep the attack surface as small as possible. Over the last four years or so, the longer the server ran, the more I was surprised how not-so-hard-at-all it was.
Stopping servers is easy, too
fail2ban is another really useful tool: It continuously scans your logs to see suspicious patterns like repeatedly failed login attempts. When it detects such patterns it updates your firewall to ban the IPs these attempts are coming from. When I first looked at fail2ban's activity, I was really not expecting to see that my VPS attracted enough automated login attempts to have a couple of hundred, sometimes a thousand bans triggered per day. None of the services running on it were well known, so it really surprised me and created a sense of a vague threat in the background.
Last weekend this feeling of threat suddenly became very real.
A couple of services were unreachable for the past two weeks and I didn't find the time to care about it. I did not find it very important and thought the problem was probably hidden somewhere in some configuration that I tinkered with. Finally last weekend, one morning I was hungover and found the time to take care of it. I tried to ssh into the server to fix it, and I got told that my credentials are invalid.
I panicked, because I didn't change the key that I used to authenticate myself on this server for quite a while. I tried different ssh settings, directly using the servers IP address… nothing worked.
It for sure must have been something I installed. I took so little care of the server once I saw it is really not-so-hard-at-all to run it, of course it must have fallen on my foot after a while. I took the hard but necessary decision to shut the server down quickly.
Of course it is sad to let go of what you like, but it is better than to pay for someone that uses your hardware to mine Bitcoin or send spammy e-mails to scam people that don't know better.
A belated epiphany
The next evening I received an e-mail from healthchecks.io informing me that backups on this server have been missing for two days, and maybe I should go check.
Of course I knew that backups weren't running, I stopped them when I burned the entire thing to the ground. Still, thinking about it for a while it didn't really make sense to me: Why were only some of the services stopped? Why would an attacker keep me from logging in and cause so much visible disruption, but keep other things running as they were? That's when it dawned on me: There was no attacker.
In my half-awake state, I forgot that I changed the login a couple of months earlier because I thought it was too easy to guess. I put myself into a situation where I thought I had to act quickly, even more so after not taking the time to fix some of the services for weeks. I did not take the time to consider the situation calmly, and made a harmful decision to wipe out everything to make sure bad would not come to worse.
I socially engineered myself.
An undoubtedly positive aspect of this is that I still was not hacked and so I'm still, even though I don't want to jinx it by challenging anyone to prove me wrong, of the opinion that it's surprisingly easy to host some services yourself. Another good side of it is that at some point in the past I was smarter than last weekend and set up backups that ran reliably until the end. I'll take this opportunity to re-evaluate the way I host my experiments and recreate what I liked, and humbly accept the soft glory of maintenance and care.