I have a WD MyCloudMirror and it is definitive nothing I would recommitment to anybody else. It runs with old software with known security holes …
I’m a linux guy. Retired software engineer. I don’t like database oriented systems like Darktable. I won’t go into why. I just don’t.
I keep a hand made file system hierarchy with descriptive names
In any given directory I can make keywords to search on with a symlink or an empty file:
…from a terminal window command line:
locate -i toucan
(there might be a half a dozen directories were i have toucan images. Now they’re easy to find)
The jpegs. If I make a jpeg from PIC_1234.NEF (nikon raw) I would call it something like:
PIC_1234_Bald-eagle.jpg That way I can always forever more match PIC_1234_Bald-eagle.jpg to the PIC_1234.NEF it was made from, usually but not always found one directory above the jpeg.
I have two 8 terabyte drives. One mirrors the other. OS on a flash drive.
Once a month or so I copy the backup directory to a usb drive. It takes all night so I do it
as I go to sleep. All three of those discs would be lost if the house burned down. I should fix that. But I likely won’t. I’m old retired and lazy now.
If my 8 terabyte drives fill up (62% full now) I will go to NAS system somehow. Building one from parts would be a fun project.
Finally (I promise) I taught a 3 hour photography class a few years ago, as part of a state-wide Audubon Festival. The number one problem the class reported was how to archive images. They came to learn about F-stops and image processing too. But they all seemed to be pulling their hair out trying to find lost images, or with trying to learn the menus of a complex commercial image archiving software system. KISS is the software designer’s best friend. Keep It Simple Stupid.
I’ve got a Synology DS918+ with four pretty garden variety 1 gb Western Digital drives. Nothing fancy.
All my images are stored on the NAS in a volume that I access from a number of machines on my home network both wired and wireless through a Ubiquiti Amplifi Wi-Fi mesh system.
The NAS is backed nightly up to a standalone USB drive via “Hyper Backup” .
Also, the images are backed up to my Amazon Prime (unlimited photo storage) using “Cloud Sync” again, every night.
This all works seamlessly and invisibly without my intervention.
All the working folders on my various computers and a “work folder” on the NAS are synchronized real time using “Resilio Sync”.
I like the Synology box. The support and application availability have been superb. Certainly much better than the ReadyNAS that the Synology replaced.
@doobs, I’d be struggling to fit very much on your NAS as described!!!
Terrabyte drives, maybe?
I’m not using a NAS right now, but I’m seriously thinking about it, as my disk usage has been exploding since I got a new 24 megapixel camera. I’m eating about 164GB a year in photos, and way more in video. Right now I have a home-made server with a 4TB drive and it’s about filled up. My workstation is similarly getting filled with those pictures, but I still have a while to go there, especially since I have the option of using Darktable’s “local copy” functionality.
I got a 8TB drive that I yanked in a friends’ server that serves as an offsite backup as well. All this is synchronized everywhere with git-annex, which is a little hard to use, but has served me very well over the years.
Still, I’ve been looking at NAS options, especially since I’ve replaced a part of that server. I just got a Vero 4k to serve as a home-cinema/set-top box, but it has basically no storage whatsoever. So I either need to add yet another external drive that dangles off of it, or try to fix the problem in the longer term and centralize storage in one unit, with multiple (possibly redundant) drives.
I’ve looked at many options. I know that Synology make good products, but I find them a tad expensive and I would prefer something that runs Linux and free software and preferably open hardware as well. This leaves me with the following options:
Then there’s also the option of building your own. The problem with that is you end up with a “tower” setup with hard drives hidden deep inside a machine: very few cases have “trays” that allow swapping drives in and out easily. That’s why I am thinking of getting a “real” NAS: each drive gets its tray, with a little light that tells you when it’s time to change it. There might be trays (like the Orico) which could enable this in a custom-made setup, but at that point, it’s basically a NAS as well since you don’t own the SATA controller.
Oh, and so far, I’ve bought normal desktop drives, except for the 8TB one where I specifically bought a “Seagate IronWolf 8TB NAS” (ST8000VN0022) - it was not much more expensive and it seems logical to buy a drive specifically for that purpose. The drives generally fill up before they start failing, but I do install a SMART monitoring tool and act on warnings the hard drives give me. Ideally, I’d have the main storage on RAID-1 so I could recover more easily from failures, but with redundant copies, I can always restore on a new drive when trouble comes, and uptime is not critical since I’m an amateur.
Long story short: I would love to hear more from people here about the hardware solutions they’re using.
The other two (well, 3 and 2) I do. But that one, I am not sure I understand. If you have reliable medium (e.g. RAID-1) and multiple copies, why do you need different media? I know you said you relaxed the requirement here:
But what’s the idea behind that principle anyways?
Different media has different lifespans and can survive different things. If I was super worried about floods, one of media types would be optical disk, like DVD or blue ray. Those can get wet and still function properly. Others have a longer lifespan, like tape backup disks.
I also think you should have cold backups, ones that are not always connected to machine and powered on. Even if you have the most badass raid setup, that doesn’t stop someone from hacking that system deleting everything, or your house from flooding/burning down
I wouldn’t bet my life on a muddy DVD, but point taken. I would assume offsite backups take care of that, but I guess that only works if they are in a different city/country, which raises the costs for me significantly (I just physically carried that 8TB drive to the other location, it would have taken months to transfer all that data out on this slow uplink).
That’s a good point too! I’ve been looking at how to make append-only git-annex repositories - joeyh implemented it, but my experience with it is hit and miss so far. Borg backups are even worse, unfortunately…
Problem with “cold” backups is they are inherently out of date if you don’t warm them up once in a while. That might mean a manual process, which might mean you will never do it. It’s how I did my offsites before, and I could spend months without doing them. I looked at the trade offs again and figured I would better risk an attacker wiping everything than losing months of work in a (more likely?) local crash.
ah well that’s why you have offsites though. Fires and floods don’t (usually!) follow your backup trail to destroy all copies of your files…
My offsite backup custodian is my son. He lives about 3 miles from us, and eats meals here regularly. I’ll occasionally tell him to bring the drive with him to dinner, and we’ll plug it in and let rsync run while we eat. The initial rsync took a day and a half; I think a few months worth takes about one dinner’s duration…
Oh, and he likes having access to the family photos, so he has motivation to initiate the above…
Indeed, manual management of a backup takes some attention and effort. Just have to find the scheme that works. The risk I figure I’m mitigating here is the risk of a structure fire. That here in Colorado could be due to forest fires as well as things we do around the house, and his location away from the woods (and us) mitigates both.
The risk I’m vexed by is the Yellowstone caldera, which is supposed to consume the USA’s Intermountain West at any moment now…
I have several drives that have the git annex repos fully encrypted. I don’t hook them up to a system for more than a few minutes to sync changes, then I rotate that drive off site. One of the offsite drives is at my gf’s work (since I work from home now), and the other is at a friends that is ~60 miles away. I don’t rotate the latter as often as the former, but at least some of my files will be saved if there is an absolute disaster.
First of all, no NAS for me. But in these last few days I’ve tested a setup to keep safe my photo library. For ‘safe’ I mean it’s better than a simple backup to an external USB drive using rsync (which is what I was doing before that – I bet it’s also the standard practice for 90% of the photographers out there only they will be using Timemachine on their Macbooks).
By the way, I was reading the brilliant post by @pittendrigh; Colin – I like your approach, it could’ve saved me literally months of hard work during the Great Migrations of 2014-5 (Aperture–>Lightroom) and 2018 (Lightroom–>Darktable). But anway, I’ve been using LR-style apps to process and organize my photo library for years now so that’s my way, I just trust that Darktable being opern source will be my trusty companion for the future.
Back to my backup strategy. I normally work on a laptop (Dell XPS 15) connected to external monitors and all the library sits on an external USB hard drive; it’s around 850 Gb of data, approx ~45k photos. I also have a spare pc at home which I use as media player (Linux with Kodi installed). I’ve fitted an additional internal 2Tb hard drive in the pc and then installed Syncthing on both the laptop and this pc. I sync the
.cache/darktable/ from the internal drive of my laptop and the directory
PHOTOS/ from the external usb drive. These directories are set as send only in Syncthing prefs on my laptop and receive only on the pc box.
I’m pretty happy so far, I can work the usual way on my laptop and then I can turn on the pc and it will silently backup everything new I’ve done. Versioning is also activated on both ends of Syncthing, I keep 5 copies of each modified file (and yes, this will soon make me hit the limit of my hard drive).
Next step is to buy two 4Tb drive (around 100 euros each) and put them in a raid enclosure, and use this enclosure to do a routine rsync backup. Even if I’m not too diligent about this step I’ll still have Syncthing running in the background with minimal input from me.
How does a “RAID enclosure” differ from a “NAS” for you? I know how they differ at the technical level, but the more I look at RAID/HDD enclosure/racks, the more I’m thinking they really look like NAS devices that connect through USB instead of Ethernet…
They’re mostly the same: a bunch of disks. A NAS will have its own CPU, ram, etc and often offers multiple services: CIFS, NFS, FTP, etc. A USB disk is a USB disk
Well that’s why I asked: the word used was not “USB disk”, it was “RAID enclosure”, and that’s in a very different league than “just” a “SATA/USB” adapter (which is what an enclosure usually is). A RAID enclosure has way more “brains” than just a single USB disk. I would particularly be concerned about the filesystem and RAID layout, for example: what happens when things fail and you need to take the disk out?
(There’s also the argument that a USB disk has it’s own CPU, RAM and could technically offer multiple services as well. It’s just we don’t usually think of it that way.)
Maybe I used the wrong term, I’m not sure. I thought that a NAS is something that lives on its own, with its little cpu and OS. The USB RAID enclosure that I bought is simply a glorified USB enclosure that allows me to connect two hard drives and use them either indipendetly (so the computer to which it’s attached sees 2 different drives) or as a RAID array (the thing I’m planning to do, two 4Tb drives giving me a total of 4Tb space but if one fails I have the other).
I think that this particular USB RAID enclosure is not very clever (I’m reading Alastair Reynolds’ Poseidon Children’s trilogy where machines become very clever indeed so I’m a bit worried to have another little brain waiting to prey on me). It may have a cpu, I don’t know, but it certainly needs to be connected to a computer to work.
The pertinent difference is “network”. For what it’s worth, I think the network transport of NAS adds a layer of complexity that may not be worth the trouble, depending on how you access these things.
There’s some sort of microprocessor in any of these devices, but it’s probably not useful to you for anything other than what the manufacturer programmed it to do. Edit: Oh, unless it’s a regular computer, then you can do whatever…
Meant to comment on this when I read it, got sidetracked… Be careful doing this. If you use two identical (make/model) drives purchased from the same place, you have a good chance of exposing yourself to subsequent failures of both, before you can deal with the first failure. This is a thing, I’ve seen it happen, and the worst case is complete loss of your data. Two different make/model drives of the same capacity will provide separation, probably enough to allow MTTR (Mean Time To Repair)…
You can actually buy hot swap bays for 3.5" drives that fit in a 5.25" drive bay giving you easy access with normal cases.
Thanks for the tip Glenn! I didn’t think about this. I was going to buy two Seagates, they are a bit faster than the WD (5900rpm), but I’ll drop one Seagate and get a WD (will they work fine together if they have slightly different speeds?).
Yes they’ll work fine a different speeds.