I use rsync all the time to copy files between NTFS and Linux (XFS). No problems with incremental copying. Though I use rsync from Cygwin.
I have an rsync-based script which copies just photos from this year, It includes the --deletete. I use it every time I upload from camera, have a culling session, import into dt or have an editing session.
I donāt think I have a problem with the ntfs and rsync, but I do find ntfs doesnāt take long to throw errors, so maybe I do.
Well, thatās not goodā¦
I have a couple of gb of lifetime pCloud space. This thread prompted me to try their backup system.
Two problems: One, is that one gives it a folder name. It has no concept of that folderās position in oneās tree. I could just-about live with that.
But I canāt live withā¦ I gave it a 238Gb file system directory to back up and let it run. Several hours later, it still wants 45 days at best!
My ISP service is 200mb/sec fibre. pCloud is managing 30-something kb.
Oh wellā¦ I tried.
2 Likes
@tankist02 @Thad_E_Ginathom, interested in seeing your command lines/parametersā¦
Downloaded their app and ran it. Little or nothing to configure.
I tried their pcloud-drive sync app a few years back. That was dismal, and stupid, in that it was hard to know if anything had actually synced.
I use pCloud to distribute photos. As I may have mentioned (forgive me if this is the umpteenth time) I photograph South-Indian classical concerts. I send pics to the artists and the organisers. After processing, I resize to 1920 and upload to pCloud, distributing the link.
rsync -avun (dry run)
rsync -avu
Sometimes a slower but more precise mode of using checksums:
rsync -cavu
1 Like
Time to reassess my storage use, and then revisit, again, my backup system!
I have nearly 4 terabytes of photos, with maybe one or two of other stuff. Now I come to look at it again, I canāt even afford cloud backup!
But My storage use is currently very wasteful. For the past few years, I have been photographing rw+jpeg. Then many batches get raw-processed, and the processed pics get downsized to 1920. I end up, with, for each picā¦
raw file
camera jpeg
dt processed jpeg
downsized copy.
Iām thinking to keep the raw file and xmp, plus the 1920-by-1280. Iāll discard the camera jpeg and processed jpeg. The latter can be reproduced at any time from darktable so long as I keep the raw; the 1920-by-1280 is fine for casual browsing (my monitor is only HD anyway)
Iām reaching the point where a 5Gb external hard disk wonāt be big enough for local backups. I need to do something.
Tedious, hard work to go through hundreds of directories, but I guess I have to make a start.
2 Likes
What OS are you using? On Linux you could use find to automatically find and delete files that arenāt raw or xmp. Or that are jpg, if that makes more sense. I guess there are tools on Windows that accomplish the same thing?
I could do the same thing, but at this point I settle for doing a better job with new files. Particularly ruthlessly and quickly culling images as soon as I import them. That cuts down 75% or more of my storage increase.
4 Likes
Yes, I am using Linux, and I do use scripts. For instance, for initial culling, I go through the camera jpegs and delete unwanted. a script then deletes the associated raw files and moves the raws into a directory called raw.
The camera produces .JPG files; dt produces .jpg files. thereās a hint to me as to how such a clean up could procede.
remove the JPG files. remove dt/.jpg files. Keep 1920/.jpg files.
(aside: one of the several things that I hate about this forum software is that it doesnāt allow āasteriskā but goes into italics instead /rant)
(another is that it doesnāt all less-than and greater-than)
I could have done all this āwith one handā twenty-five years ago, but now, not so much. Itās not impossible. I have a doubt that my files are all consistent enough to apply just one script.
Yes, I was a Unix man who did earn a living doing that stuff. Yes, I am now very, very rusty. But I havenāt completely forgotten :).
75% is a very good culling āscore.ā I seldom reach 30%. Asking for advice on another forum, I was told āConnect to your inner mad-axeman murderer!ā
But I have also started using rating in dt. Currently it is 1, 2 or 3 stars. I may have a second glance at 2-star, but mainly, only 3-star gets processed, saving a lot of time as well as disk space.
Early on, I came to the realization that alls I needed for ~90% of my captures was a 800x600 rendition. My wife uses these on facebook, and I post images for all sorts of reasons on web forums, and that size still just fits in those destinations. Itās also the realization that got me to switch to raw exclusively for capture: I wanted these small renders from the camera, and the camera wouldnāt do them. So now, I shoot raw only, then batch-develop them to those 800x600 āproofā images. For most, thatās as far as it goes.
With my hack software rawproc, the processing toolchain is stored in the ImageDescription tag of the render, so no sidecar file. Yay!!! So, every time I drag a render image file to the rawproc desktop icon, rawproc will find that toolchain and ask me if I want to open the raw and re-apply processing instead of just opening the render image. That proof render serves the function of that sidecar file used by other software, and the toolchain follows the image it created without additional fuss. With that, Iāll use the proof toolchain to make any other render I need, most often a full-resolution version, all I have to do is drag the proof render to rawproc, change the toolchain (usually, delete the resize tool), and save to a different destination file. That file also gets the toolchain that made it in the metadata, and so on.
Iāve occasionally thought about increasing the proof image size to 1080 or somesuch. Inertia keeps that from happeningā¦ 
2 Likes
Update
Following a trip to Oman, I have completely filled my 2TB external drive (and the second 2TB I use as backup). So the time has come to implement the new plan.
I have an additional requirement, namely that (one of) my (offline) backups needs to be unencrypted and without a password. Iām not concerned about theft of my family photos, but it is important that my family can access them without any technical hurdles or secret information in the event that something happens to me.
It looks like restic insists on both a password and encryption. But Iāve learned that a simple rsync script can provide incremental snapshots using code I understand, and with a result that will be accessible to my non-technical family members.
The script Iāll use is here:
My plan is:
- Add my new external 8TB HD to
digikamas a ānew collectionā and transfer my photos there; this will be my new primary storage - use the rsync script to (incrementally) back up the new HD to a second external HD on a monthly schedule (or after ingesting big batches of images from a trip)
- create annual incremental backups to a third HD that I will otherwise store at the office.
I donāt have the third HD yet, but will be able to use the two 2TB drives as non-incremental backups until I do.
This will likely take a few weekends to accomplish. Let me know if anything sounds wrong 
Thanks everyone for the interesting discussion!
3 Likes
I will just throw in this thread Cloud Backup Solutions for Europeans where I explained why we use restic (via resticprofile) and it has a base config to get you started.
also remember ā¦ if you never test restore and can actually notice if your backup got broken. do you even have a backup?
3 Likes
Honest question, how do you test restoring more space than you have available?
say you have 10TB total space, 8TB of photos and media, backed up twice, one remote. How do you test this without buying more storage and having 2x your total used space?
This seems to be a bit more difficult when using restic or other snapshot/encrypted solutions. With rsync -av --checksum --delete, you can guarantee that your files are the same in source as they are in the destination, and the restore should not even need to be tested in this case.
3 Likes
resticprofile profilename.check --read-data
I would add -n to your rsync cmdline if you dont plan to actually write to it and just see what would need to be sent for āis my backup okā. Which I think it doesnt actually do properly because it can corrupt in both places.
2 Likes
Got it, I guess it must verify the checksums of the remote data, thanks 
I will read more into it. I didnāt know this kind of check counted as a test restore
I wouldnāt say this is 100% equivalent. but at least you know it can read all the data in the backup and the integrity checks of restic passed.
to elaborate on why it isnāt 100%. For this you would need to verify
If you start a new system:
- do we have all the files we need.
- if we restore from the backup, do the apps work again?
yes it sounds at first more server focused. But imagine you do a backup of your ~/Documents but not of your ~/Drawings ā¦ if you now try to open your libreoffice document and export it again as a PDF, you will notice āoops we have referenced the inkscape things in ~/Drawingsā. Or you notice that to quickly being back on track to keep working on your libreoffice document, you should have also have a backup of your libreoffice settings.
oh and a package list that all your tools are installed again. (if you arenāt using things like ansible or saltstack for that)
I mean it depends on what your goal line is āJust get me back the data, I thought of preservingā or āget me back on track to make my deadlineā.
1 Like
If rsync is creating incremental backups based on file differences, the result should include a discrete copy of a file for every different version, shouldnāt it? Meaning, if the file is corrupted at some point, the backup will include the version pre-corruption and a new version with the corrupted version.
Is there a tool that does something more robust than checking if the file has changed?
I have my pictures backed up on an external 8tb hdd that is almost always off (i have also the same 8tb hdd in my desktop as a data disk), and till 2018 on blurayā¦ maybe when iāll have a bit more time iāll try to reach 2025 also on bluray, even if it is really annoying for 200 gb + every year (for 2024 instead, i have a mere 30gb or soā¦)
I often leave an md5sum of the files in order to be able to verify them.
But recently i have found out i have at least five 2,5" hdd ranging from 250gb to 1tb.
I think i could copy e.g. 2019 to one, 2020 to another one, and so on, much faster than bluray.
Let me say it this way ā¦ some very smart people write backup tools. They try to cover all the bases and make it really robust. IMHO the time of abusing rsync to make snapshots and so on are over. You will not gain anything from trying to reinvent the wheel. On top of this I like my backups encrypted, so that they can be saved securely off site.
2 Likes
In my opinion, share the images in their social media and make photo books of the best images from that year and/or significant vacations. While we think all of our images are masterpieces that someone will want to see, I think thatās not realistic. Having someone technical enough, with time and interest to go thru my collection sounds too far fetch. I strive to make the photos available and accessible now. The backup is for me in case of hardware failure or a house fire.