Slight outage today


I accidentally let the certificate expire for the site so we were seeing 503 errors earlier and the forum was unreachable. I’m sorry about that!

On the good side @darix used the downtime to go ahead and migrate the forums over to our new server (I mentioned that in my last blog post about LibreArts).

The fact that you’re reading this here now means that it’s all up and running again. :slight_smile:

This is a good thing as it means we’re now one step closer to consolidating services and saving costs while giving us room for growth.
For comparison, we were previously using three services:

  • webhost (static site files, main website)
  • Linode (forum)
  • Amazon S3 (forum file storage)

Together these are costing us around $85/month. (The bulk of that is data transfer from the Amazon S3 bucket for forum assets.)
That Linode server is only 2CPU, 4GB RAM, 80GB storage.

We’ve now spun up a new machine at Hetzner to handle everything for us that’s about 25% less expensive.
(8CPU/16Cores, 64GB RAM, 2x8TB storage).

We had already migrated the main website (and others that we host for various projects) and were planning on moving the forums soon. Then the outage happened and @darix figured “why not”. :smiley: We were already down, so let’s just migrate while it’s out.

So we (he) did.

That means all that’s left is to finish migrating data off S3 and make sure it’s connected properly to the forums. After that we’ll entirely be on our own servers. (With much more horsepower). In fact, you should already notice the forum feels a bit snappier than before.

Sorry again for the downtime! And thank you @darix for getting us up and running again (and migrated!) so quickly!

(And yes, we had already automated the generation of Let’s Encrypt certificates for the new infra - so that shouldn’t be an issue any more… :wink: ).


:love_you_gesture: @darix thanks!!


Out of curiosity, did you consider switching out s3 for backblaze b2 in combination with cloudflare?

Can’t compete with a Hetzner Box but on the other hand it should offer better reliability. :slight_smile:

1 Like

we could also do cloudfront + s3 … but right now i would say we go for locally stored files and maybe use backblaze for backups?

1 Like

We do need an s3 compatible storage for backups!

I haven’t read the fine print but if serving assets from b2 via cloudflare is free you could consider just serving them from there as well. It’s obviously not ideal to couple the backup and production infrastructure like this but it could be a fairly simple setup and more reliable than betting on two disks in the same server. :slight_smile:

I don’t make infrastructure decision, but b2 would be backup only. We have enough disk space to last us a while on the new host and also have a lot of bandwidth.