Well that was fun!

I’m guessing a few of you have noticed that we’re finally baaack!

Thanks to all of the hard work these paste few weeks by @darix! (He’s got a topic started where he may provide some more technical details on what happened).

The short version is, we found a bug. It took our primary hardware offline (and all the vms that we run for the websites and this forum). On a side note, we’ve been operating for around 98 months straight without a significant outage - so around 99.5% uptime? :smiley: We were probably due for our first big uh-oh.

The reason for the long downtime is because I wasn’t as much help as I’d like to be getting new servers spooled up. So I see some training in my near future with darix to make sure I know better what’s going on and how to fix/break them more.

I wanted to thank everyone for being patient and for all of the kind words and support emails/tweets/etc. we got during the downtime. We really appreciate the love!

I’m really sorry for the downtime and I’ll work with darix to make sure we have a better bus factor spread for the future. Stay tuned to his topic in case he gets a moment and wants to post more details on the technical problem that wiped us off the internet for a little bit. :wink:

[edit] - I also realize that I should have done a much better job communicating to the community. This is my fault and I apologize for that. Life has a way of kicking you when you’re spinning a bunch of plates in the air. It’s not an excuse, just an observation. :frowning: I will do better!

58 Likes

You just don’t often realize what you have until it is gone… thanks for all your dedication

14 Likes

Yaaay! Thank you for your hard work and dedication! Glad to have Pixls.us back!

2 Likes

Thank you so much for getting my favorite forum back up and running!

3 Likes

That’s so true!

1 Like

Thanks for the hard work of all the people involved!
You know it’s bad when you see 503, and take it as a sign of progress. :wink:
In the past few days, I’ve also visited the darktable community on reddit, and a few times I just wanted to write something like ‘See [some post on discuss.pixls.us]’, only to realise it was not available. That certainly shows how useful a resource and community this site has become. So, thanks again!

4 Likes

Yep!

I concur - I was surprised by how often I kept missing it. Thanks @patdavid and @darix !

4 Likes

Woo-hoo!

@darix has rebuilt it!

Better than it was before!

Better…
Stronger…
Faster!

:wink:

3 Likes

I have missed you so much.
You don’t know how necessary something is until you don’t have it.
I know you have never stopped being there… but now I see you again.
Greetings to all.

7 Likes

Welcome back!

It was a dark moment when my browser refused to believe that pixls.us even existed any longer. But news trickled through — and here you are back again.

:grinning:

2 Likes

Did G’MIC coding. Basically that.

3 Likes

I didn’t know how addicted I was to this forum until it was gone :wink:.

Thanks for all the work running this site and bringing it back to life.

4 Likes

I want to copy/paste the words @priort used - they eloquently express what I - and I suspect many others - think: namely that I have no intention of complaining that the site was down for a while - it just made me realise how much I value and missed it. I was also feeling for you, knowing how much you were having to deal with. Congratulations on getting the site back up and working and my sincere thanks for the efforts of all those involved.

5 Likes

There is this saying “you don’t know your backup works until you did a full recovery”.

It was probably very pain- and stressful, but I’d say you all did an awesome job. Kudos.

1 Like

But hopefully for a whole lot less than the $6 million for Lee Majors aka Steve Austin

2 Likes

correct would be saying “finding out first why it broke so we do not run into the same issues straight away again”

4 Likes

This was by far the biggest delay. I suspect darix could have had us back up and running within a day or two but safer to determine why we went down in the first place so we didn’t have to do it all again. :slight_smile:

7 Likes

Thank you sssooooo much and welcome back to all of us

1 Like

You don’t miss something until it’s gone.
It seems that at least one thread of mine has been lost in the “darktable” software category. Has anyone else noticed this besides me? There was a good tutorial in there from a forum member on how to correct a vignette.

If you posted something within ~24 hours of the down time, there is a chance it was lost, sorry. We have nightly backups of the forum.