Backup and Archive Strategy

[rwslippey]

Hello everyone,

So I've recently noticed (okay, I've known for a while) that I am a complete digital mess. Files everywhere, you know how it goes.

So I'm taking to reorganizing my files right now and it brings up a question. I'd like some opinions on how you all would handle this.

I'm a photographer (business, but very low volume recently) so I've got RAW files that are used in my Lightroom Catalog (Adobe's digital management application). I obviously don't want anything to ever happen to these files. Up until this point I've kept my "photo library" on my desktop and replicated to my nas nightly. From their it goes to a crashplan server. A thought that recently popped into my head was that I might be going about this wrong. These RAW files will never change. (any edits are actually recorded in the database and used when loading this RAW file, so edits don't change the file at all) So my thoughts are that I should move the files to the nas and start some type of archival workflow.

The main reason I want to start some kind of archival workflow is because I thought if anything ever happens to my desktop (ransom ware mostly) my automated system would propagate all the way through my backup system, making it useless.

So I think I'm going to move the files to the nas and just tell Lightroom where to find them. My question is how would you go about this archival workflow. I'm kind of at a loss on this.

Any reason not to? How would you approach this? Incremental backups instead of a full on crashplan sync (where it could over right my good data with bad?)

Sorry if this doesn't make any sense. I makes sense in my head, but its been a long week.


Thanks

 

[joeschmuck]

Having experienced Ransom ware first hand, the one I saw encrypted all the files in My Documents and that was about it. But it was enough to re-image the entire computer. This was on a work computer for one of my employees. We had some data loss which took a long time to recover from. There was no network files attacked thankfully, that would have been very bad, however those are easily recoverable. What I learned here is to never store data in the My_Documents folder.

While I don't know how large your photo library is, my opinion is to manually backup your RAW files to DVD-R media periodically. This could be once a week or how ever often you desire. The goal is to ensure that if you are hit with some disabling issue, that even if the automated backups are affected, you still have some minimal loss.

Another option is a removable USB hard drive and to copy the files on that periodically.

Working a business has some responsibilities to the customer as you are aware. I'm sure there are photos that if you lost them, they could never be recreated, like wedding photos or maybe some special one time test.

I think the one way you can eliminate the issue is to air gap the computer you use for all your photos and editing. Copy the files to a USB hard drive, move them to a second computer on the internet and use crashplan to upload them to the offsite archive in the sky. This keeps your data as safe as possible. If you must use a single computer then ensure you have an outstanding firewall and anti-virus suite and it stays up to date, only use the computer for work and never visit an internet site that isn't directly work related (no web surfing). Email can be done my most ISPs via a web browser interface and that will help prevent infected email getting on your computer, provided you don't click on links in the emails except for ones you know for certain are real and that they pertain to your job. Delete all that other crap email that comes in. Trust me, No one named Veronica is going to give you $50,000.00 for cashing her $100,000.00 check from Idontexistville. And no Facebook, Instagram, whatever. It's a work computer, use it for work only. If you don't have a second computer, well it's time to get one if you are serious about keeping your data as safe as possible.

 

[Mirfster]

All in all this is a very good reason that the "Rule Of Three" works even against Ransom-ware. If the data is housed "On-Site", "Near-Site" and "Off-Site"; you are in a pretty good state of being able to recover.

But at the fundamental level, I totally agree that it starts with the End User and what the machine is used for. There are sound reasons why hardly anyone ever needs to be an Administrator of their assigned system. Personally, I do not even log on with Administrator Rights on my own desktops; nor does my Wife anymore. She is the worst at opening Click-Bait... ;)

 

[joeschmuck]

Wife anymore. She is the worst at opening Click-Bait... ;)

You made me laugh because I can relate.

 

[Stux]

An issue you have, is that local disk performance might be important to you, and getting that speed out of your NAS might be hard (which means expensive)

You could use an equivalent to rsync to copy the files to the NAS. You could then use snapshots. And you could then use an offsite copy.

In this approach, your NAS is the backup and crash plan is your offsite.

As long as you have 2 copies on site and one off, you're all good.

 

[Jailer]

nor does my Wife anymore. She is the worst at opening Click-Bait... ;)

You've obviously never met mine.

 

[rwslippey]

Thanks guys for all the replies....

I think the user level is pretty secure (or at least I though it was until last week) I'm VERY good about not using administrator accounts for regular use and click bait is pretty obvious. (well to me atleast, but as everyone here has stated, it isn't as obvious to my wife)

I caught something last week that got me. in Firefox I somehow got a click bait site and the only thing I could do was click on the actual link(which I didn't of course) call the number (you know the type, this is MicroSHAFT, you're computer is infected, call this number to fix it) I tried figuring our where the site was hosted but couldn't do anything but close the browser... so that's what I did. Scanned my computer wit and all. Long story short, I though we were good after multiple scans and then used my debit card to pay my cell phone bill online. I can't tell you weather that is where it was compromised or not but the next day someone got drunk on my dime in Brooklyn NY (The first and last time I was in Brooklyn was for a military funeral last December) So reload it is, scanned the NAS, scanned everything vigorously!

But it got me thinking... hence my post here.

Joe, your method seems the easiest and most straight forward and I think it might be the direction I'll go. I've seen many photographers use this method for asset management (most simply use external hard drives though) I'm sitting at about 700GB right now, so not too bad, but not CD or DVD sized. You're absolutely right about file storage, loss could be devastating. Right now I only have 2 weddings under my belt but going forward (have one in December) I obviously hope to get a few more.

Rule of thirds is kind of what I'm going for here... before Freenas I used a rotating hard drive method. I'd pop one drive in and it would copy everything (as it happens) to the drive. and swap the drive off-site every week or so. I still feel like this is the best method as you always "know" it is getting done. I'd REALLY like a way to do this with freenas to avoid having to run everything through my PC. Some type of direct hot swap bay that would fire a script to copy data to the hot swap drive would be great (is this possible?) I know freenas doesn't work well with USB. I'd always like to have the ability to directly connect that drive to a system and go if need be. So I'd prefer the files to copy directly as if it was a regular hard drive and not in any type of encrypted volume (like crashplan) or in a snapshot where I need to bring it back into the NAS....

Performance isn't really an issue. Lightroom works rather well with network storage (for photos) Import might take a little longer though. but once imported the files are loaded locally in preview form. Changes are never made directly to the files themselves. The changes are stored in a local database ( the Lightroom Catalog) as a set of instructions on how to interpret the file (increase exposure, contract, or whatever the changes I made were) The Lightroom Catalog will always be directly on my desktop for performance reasons)


sorry for the delay in this reply and my long winded reply.... I'm still trying to figure out my workflow with FreeNas and got really busy this weekend.
another thought in mind is how to incorporate my laptop imports into the mix (import on the road).... thoughts for another day I guess.

 

[joeschmuck]

700GB is quite a bit but if you break the files down by the customer or job, you may be able to get away with a single DVD per job or two, but I'm just guessing. You could then organize your files in a safe place and if an old customer needs a copy of something, well you won't have to restore everything but rather just the one or two DVDs. This will produce quite a few pieces of media but it's not that bad if you do it by customer. The down side is it will take time to create all the media but again, that is okay, just do a few a day. And then find some nice CD organizer to file your media away.

Importing file while on the road can be done provided you have a fast enough internet connection to upload your photos. You can create your own cloud server and upload to it. There is more work to do and you would have to expose your FreeNAS system to the internet so security will now be of concern. You may need to establish a really good firewall. I have a VPN Tunnel into my computer, I can also use SFTP which might be a better choice. I'm not the person to answer the best way to do it, I just know of a few ways that work and are touted as being secure.

 

[rwslippey]

700GB is quite a bit but if you break the files down by the customer or job, you may be able to get away with a single DVD per job or two, but I'm just guessing. You could then organize your files in a safe place and if an old customer needs a copy of something, well you won't have to restore everything but rather just the one or two DVDs. This will produce quite a few pieces of media but it's not that bad if you do it by customer. The down side is it will take time to create all the media but again, that is okay, just do a few a day. And then find some nice CD organizer to file your media away.

Thanks, That sounds like the best idea at this point. Fortunately, it'll just be an archive or the last line of defense so even if a customer needs something I should (in theory) still have it accessible through other means. Any idea how long DVDs last? I should probably add that 700GB is mostly RAW files (uncompressed) and the majority of it is from my first few years of photography where I did a lot of trying different types and did a lot more shooting. It also includes ALL of my daughter's first year of life (unfortunately, with a rather crappy camera...) so yes I LOVE this data! but most of the professional stuff didn't start until the last year or so.

Importing file while on the road can be done provided you have a fast enough internet connection to upload your photos. You can create your own cloud server and upload to it. There is more work to do and you would have to expose your FreeNAS system to the internet so security will now be of concern. You may need to establish a really good firewall. I have a VPN Tunnel into my computer, I can also use SFTP which might be a better choice. I'm not the person to answer the best way to do it, I just know of a few ways that work and are touted as being secure.

This is actually more of a Lightroom workflow issue. If I was on the road I wouldn't import directly to the NAS. I'd import locally and then move things around when I got back. I might want to look into some type of on the road backup method to push things to the NAS on the road, but I'm thinking BT sync for that? I do have a VPN server up and running that I use to connect sometimes when I need something off the NAS but not very frequently. I actually plan on rebuilding that soon.


thanks again

 

[joeschmuck]

Any idea how long DVDs last?

50+ years or 30 seconds in the hands of a child. Also you should burn them at 16X speed if possible, the slower the better. It allows deeper pits to be burned. And I would stay clear of Dual Layer DVDs, they are very problematic. When you create one of those, you need to verify all the data is valid each time. With regular DVD-R or DVD+R, once you trust them, they usually just work great. When I say trust them what I mean is some manufacture media will work great, some will not work great for a specific DVD burner. Once you find out what works all the time then stick with that brand. But when burning the media, something like this for work product, I would still verify the media (normally an option during media creation).

 

[Arwen]

One other option in burned disc media, are M-Disc;

http://www.mdisc.com/corporate/

They come in 4.7GB DVD, 25GB/50GB/100GB Blu-ray. Needs special burner for DVD,
but as I understand it, all Blu-ray burners support the 25GB M-Disc. Not the cheapest solution,
about $1 for DVD and $2 for 25GB Blu-ray. I trust these a bit more than stock DVD and
Blu-rays.

 

[rwslippey]

One other option in burned disc media, are M-Disc;

http://www.mdisc.com/corporate/

They come in 4.7GB DVD, 25GB/50GB/100GB Blu-ray. Needs special burner for DVD,
but as I understand it, all Blu-ray burners support the 25GB M-Disc. Not the cheapest solution,
about $1 for DVD and $2 for 25GB Blu-ray. I trust these a bit more than stock DVD and
Blu-rays.

I can't wait for these to hit mainstream

http://www.theverge.com/2016/2/16/11018018/5d-data-storage-glass

Sent from my SGH-M919 using Tapatalk

 

[rwslippey]

Looks like the Navy actually tested those M-Discs. Results were pretty darn good. Almost every other manufacturer had some data loss (even their so called "archival" quality)
Guess we know whose getting that contract

The next question is DVD or Blu-Ray?

 

[Mirfster]

"Superman Memory Crystals"... That is what I'm waiting on... ;)

 

[rwslippey]

"Superman Memory Crystals"... That is what I'm waiting on... ;)

I'm starting to think that anything is possible. Look where we have come over the last 100 years or even 10 years!

 

[Mirfster]

It already exists, just you and I aren't going to see if for quite some time.

 

[joeschmuck]

DVD-R and DVD+R in the 4.7GB size are well proven, no data loss so I'm not sure what the comments above were talking about, probably the BR discs. Look at this website forum in the Hardware section and CD/DVD/BluRay Writers. This will give you a lot of data on which drives work very well and which ones don't. There is also a section on Blank Media. A little research goes a long ways. This website can be a bit overwhelming with all the extra software things you can do and upgraded firmware but if you are serious about creating good media, it's worth the effort, or you can just buy the DVD writer I have, heck, the price has dropped significantly since I purchased mine so many years ago and mine is going strong. Also you will need to read some of the stickies to make heads and tails out of all the charts, otherwise you will be lost.

I have a BluRay reader and it can read DVD/CD media as well but it cannot write and that is what I purchased it for. I also have a Lite-On DVD writer which will burn a large variety of DVD and CD media flawlessly.

My favorite brands are Lite-On and LG when it comes to these devices. With my DVD Burner I can purchase a large spindle of fairly generic DVD+R media and they will burn very well. When cutting the media, always verify after the burn for anything you want to ensure was burned without error. If your media burning software doesn't offer that option, ask for help, it's probably buried somewhere.

Also, as for media longevity, you can purchase "Gold" media which is touted to last 100+ years. In 50 years this media type will likely be obsolete but you could go that route. When CD-R media first came out, gold media was damn near the standard in my area. I've got gold media going back about 25 years with photos on them. A CD-R burner was so expensive that I got my ISP to burn the files on the media for me, all I paid for was the cost of the media. Nice guys.

 

[Stux]

One way to backup your NAS is to replicate it to another FreeNAS. The backup target can be a simpler/cheaper one.

The benefit is the replication requires no human intervention. And that means it's automatic. And an automatic backup is one that actually happens.

 

[Arwen]

Looks like the Navy actually tested those M-Discs. Results were pretty darn good. Almost every other manufacturer had some data loss (even their so called "archival" quality)
Guess we know whose getting that contract

The next question is DVD or Blu-Ray?

Well, simply looking at price, 25GB Blu-ray M-Disc at $2, is 5 times the storage of the 4.7GB DVD M-Disc at
$1. Yet only double the price. For me, that means no more 4.7GB DVD M-Discs, unless I need to play them back
on a DVD only drive.

 

[joeschmuck]

One way to backup your NAS is to replicate it to another FreeNAS. The backup target can be a simpler/cheaper one.

The benefit is the replication requires no human intervention. And that means it's automatic. And an automatic backup is one that actually happens.

The only downside is if some files become infected with say ransomware then that could migrate automatically to the backup before you know it. This is why I backup important data like photos to DVD media periodically. You will never be 100% safe unless you have a air-gaped system but you at least do what you can to minimize the pain.