Warning

 

Close

Confirm Action

Are you sure you wish to do this?

Confirm Cancel
BCM
User Panel

Site Notices
Posted: 2/3/2021 8:03:12 PM EDT
My photo archive is ever growing and backing it all up continues to increase in difficulty, I was hoping somebody else has figured out a better solution than I have
Link Posted: 2/3/2021 8:06:48 PM EDT
[#1]
Home NAS
Link Posted: 2/3/2021 8:10:11 PM EDT
[#2]
Local SSD, with backups on two portale HDDs
Link Posted: 2/3/2021 8:11:13 PM EDT
[#3]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Home NAS
View Quote


I should clarify I think.

I'm interested in the complete system. From primary storage to backups.
Link Posted: 2/3/2021 8:15:25 PM EDT
[#4]
Many of mine are trapped on PHOTOBUCKET............
Link Posted: 2/3/2021 8:19:33 PM EDT
[#5]
I have stuff like that on a file server on my local network. I use iDrive cloud backup service to protect it in case the server crashes. If you really want to keep those pics, use a good cloud backup service.
Link Posted: 2/3/2021 8:20:18 PM EDT
[#6]
Link Posted: 2/3/2021 8:27:46 PM EDT
[#7]
What kind of data footprint are you talking?

Local NAS are great but without off-site storage (another physical location) it is incomplete.

In addition to local storage and removable drives I use a cloud service … and until a few weeks ago I'd have said AWS on a Glacial tier @ $4 per TB a month

I am now looking at https://tresorit.com/individuals as a start cuz noBezos when I can.

For stuff I shoot on my iPhone I just use Apple Photos and iCloud and it covers all the personal stuff.

For all client work (not just photos) I have a separate setup.

The stack looks like this:

Desktop (only copies if I am editing, not the original)
External Volume Disk or SSD Library on a removable dock (https://www.amazon.com/gp/product/B073W4YDFQ/)
+
Mirror of each External volume (so I buy 2 drives everytime I need new space, same dock as above)
+
Cloud mirror of External
Link Posted: 2/3/2021 8:31:49 PM EDT
[#8]
Link Posted: 2/3/2021 8:34:04 PM EDT
[#9]
............
Link Posted: 2/3/2021 8:42:53 PM EDT
[#10]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Seriously.

Delete that shit.

Nobody cares about your junior high school yearbook pics or your first dirt bike.

You can post your coolest teenage-adventure memory pic to FecesBook and you'll get three likes.
All from people who were not even remotely involved in the subject of the pic.

As for "family photos," try giving that stuff away to family. They don't give a shit. Ask me how I know.

Porn can be replaced.
View Quote


I don't think he was looking to have a probable hobby/pastime psychoanalyzed.
Link Posted: 2/3/2021 8:51:34 PM EDT
[#11]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


I don't think he was looking to have a probable hobby/pastime psychoanalyzed.
View Quote



Yeah, you're right.

Not General Discussion.

I apologize.
Link Posted: 2/3/2021 10:00:15 PM EDT
[#12]
Discussion ForumsJump to Quoted PostQuote History
Quoted:



Yeah, you're right.

Not General Discussion.

I apologize.
View Quote


lol you're good man, that was hilarious.
Link Posted: 2/3/2021 10:02:09 PM EDT
[#13]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


I should clarify I think.

I'm interested in the complete system. From primary storage to backups.
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
Home NAS


I should clarify I think.

I'm interested in the complete system. From primary storage to backups.


Can cost some coin. Synology has a solution that seems pretty simple. Get two NAS boxes from them. Put one at a friend/relative's house. Store your photos on the NAS and you can configure it to auto-backup to the remote NAS.
Link Posted: 2/3/2021 10:09:27 PM EDT
[#14]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
What kind of data footprint are you talking?

Local NAS are great but without off-site storage (another physical location) it is incomplete.

In addition to local storage and removable drives I use a cloud service … and until a few weeks ago I'd have said AWS on a Glacial tier @ $4 per TB a month

I am now looking at https://tresorit.com/individuals as a start cuz noBezos when I can.

For stuff I shoot on my iPhone I just use Apple Photos and iCloud and it covers all the personal stuff.

For all client work (not just photos) I have a separate setup.

The stack looks like this:

Desktop (only copies if I am editing, not the original)
External Volume Disk or SSD Library on a removable dock (https://www.amazon.com/gp/product/B073W4YDFQ/)
+
Mirror of each External volume (so I buy 2 drives everytime I need new space, same dock as above)
+
Cloud mirror of External
View Quote


About 4 TB of photos currently. I'm using a NAS for storage but backup is really where I'm struggling. Cloud storage pricing for 4 TB isn't insane, but it's not cheap either, and I only expect it to increase. Having backups on additional hardware seems more cost effective, but from my understanding cold storage of hard drives isn't reliable. A secondary off site NAS seems like the best option, but that requires roping somebody into letting me keep my stuff on their property and the inconvenience of that.
Link Posted: 2/3/2021 10:12:37 PM EDT
[#15]
1TB Samsung SSD
and many usb drives as backup
Link Posted: 2/4/2021 12:15:24 AM EDT
[#16]
I'm on a Mac, and use an external enclosure that holds 4 drives (either HDD or SSD).  Basically, the only thing on the internal Mac drive is the OS, and programs. The internal drive is cloned (bootable)  to one of the external drives each week.  All images go to the external drive, and are backed up with Time Machine.

If the house burns down.....so be it.
Link Posted: 2/4/2021 6:23:49 PM EDT
[#17]
I have HDD and SSDs. But I came across a Google search that SSD do wear out, so a note of warning and caution. I store the same in my house. IF I get hit by a meteor, I am in trouble. Actually, I should look into cloud storage though, in additon to my on-site backups
Link Posted: 2/4/2021 11:24:13 PM EDT
[#18]


I have a PCIE SATA III 8-port expansion card, which in conjunction with my motherboard, gives me a total 16 SATA drive connections.  So, I have 2 5-bay hot swap cages sitting below my desktop where mirrored drives and additional storage drives reside.  5 drives in the desktop, and 10 hotswap bays below it (4 currently occupied).  I have to leave at least one hotswap bay empty and do period backups to that bay, and pull the drive for safe keeping.

Just prior to a month long trip out west this fall I pulled my backups and stopped by my neighbor's house to drop off the Pelican case, but they weren't home.  Drove past the wife while she was jogging, but handing off a Pelican case to her out the window of the truck just seemed, well, sketchy.  So I just roadtripped with my backups in the truck.

One of these days, I need to move to cloud backup ... it's not the expense ... they can do it for about what it costs me to add my own physical backup drive space ... it's the transfer time that puts me off.  Can't imagine how long it would take to do the initial transfer of 12TB.

I dunno.  One day, I'll cloud the backups.  Until then, I'll be squirreling a Pelican case with backup drives.
Link Posted: 2/5/2021 12:34:50 AM EDT
[#19]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


I have a PCIE SATA III 8-port expansion card, which in conjunction with my motherboard, gives me a total 16 SATA drive connections.  So, I have 2 5-bay hot swap cages sitting below my desktop where mirrored drives and additional storage drives reside.  5 drives in the desktop, and 10 hotswap bays below it (4 currently occupied).  I have to leave at least one hotswap bay empty and do period backups to that bay, and pull the drive for safe keeping.

Just prior to a month long trip out west this fall I pulled my backups and stopped by my neighbor's house to drop off the Pelican case, but they weren't home.  Drove past the wife while she was jogging, but handing off a Pelican case to her out the window of the truck just seemed, well, sketchy.  So I just roadtripped with my backups in the truck.

One of these days, I need to move to cloud backup ... it's not the expense ... they can do it for about what it costs me to add my own physical backup drive space ... it's the transfer time that puts me off.  Can't imagine how long it would take to do the initial transfer of 12TB.

I dunno.  One day, I'll cloud the backups.  Until then, I'll be squirreling a Pelican case with backup drives.
View Quote


I hear ya on the cloud stuff. I was testing speeds with Amazon Photos and Crashplan Pro and it would take forever to get everything uploaded. Download would be faster, but it would still be awhile. I know Backblaze offers to send you your data on a hard drive (only 3.5TB at a time though) if you need to recover data, but I'm not a fan of their app for the personal backup and their B2 backup (which would be necessary for NAS use) gets expensive with lots of data. At your 12TB of photos B2 would be $60 a month. You could just buy a 12TB drive within a handful of months paying that.

My solution for awhile has been a Synology 2-bay NAS with two 6TB HDD's in raid 1 as my primary storage and a spare external hard drive as my backup. At about 4TB it has some legs left on it before I would need to upgrade the drives, but I decided to be proactive and figure out a better upgrade path. I put together an Unraid server from my old desktop and bought a pair of 10TB drives to put it in it for now. One parity drive and one data drive. I have 6 more 3.5" drive bays to add more HDD's down the line, two 2.5" bays as well. I have a PCIE SAS/SATA card in it as well to actually be able to hook up that many drives at some point. The motherboard had 8 SATA ports but only 4 were SATA 3.0. I can set it up to automatically backup to Crashplan, but it almost seems useless if it would take months to get everything back. Not that I would necessarily need my photos back ASAP, it would just suck having your internet bandwidth being occupied for months.

I'd be willing to just buy hard drives and make copies, or Blurays and make copies, but everything I've read seems to indicate that just storing those away for safekeeping is still fairly unreliable. It makes me think my safest and possibly most cost efficient solution would to just be building a 2nd Unraid server that I can put offsite and mirror my local server to. Based in the B2 subscription costs I'd pay off my hardware before the warranty on the drives ran out, I'd just rather not inconvenience a family member or friend by having them store my tower and have it leach their internet.
Link Posted: 2/5/2021 11:25:33 PM EDT
[#20]
Discussion ForumsJump to Quoted PostQuote History
Quoted:

I should clarify I think.

I'm interested in the complete system. From primary storage to backups.
View Quote
OK, here's my entire data flow - and remember that you asked for it...  

  1. Capture images on a memory card.
  2. Ingest images:
    • Collate "keeper" images onto my Workbench Drive (1TB NVMe SSD).
    • Backup "keepers" to Archive Drive (8TB 7200rpm HDD).
    • Format memory card.
  3. Edit images using the Workbench Drive.
  4. After edits are done, move (not copy) originals and any edits (e.g., raw files and derivative PSDs) to "Current Album" folder (named uniquely in format "Album_NNN") on Main Storage (2x 14TB 7200rpm HDD configured in RAID 1 [mirror]).
  5. After "Current Album" reaches ~90GB,
    • Copy "Current Album" to Archive Optical Disc (100GB M-Disc BluRay).
    • Copy "Current Album" to Archive Drive.
    • Delete "keepers" backup from Archive Drive. (I use separate folders for Backup [temporary] and Archive [permanent].) At this point, copies of these files exist on the Main Storage drive, on the Archive drive, and on an Archive Optical disc (not to mention the cloud).
  6. Backup process runs every day:
    • Cloud (Mon-Wed-Fri-Sat)
    • System Backup (Sun-Tue-Thu)

If you've kept count, I use 5 drives:
  1. Workbench - initial ingest and edits. This is the most data-intensive part of my photo management workflow and I use the fastest drive in my system for this phase.
  2. Main Storage - permanent storage of images after initial edits are complete. This is a large capacity drive (which pretty much mandates spinning media) that has slower performance than the Workbench.
  3. Archive Optical - permanent backup to second media type. This is a collection of BluRay discs.
  4. Archive - initial backups and permanent archive. Initial backups are kept at least until primary edits are complete. Archive is updated concurrently with creation of Archive Optical disc.
  5. System Backup - local backup image of my OS drive and Main Storage drive. I currently use Macrium Reflect with a Full/Differential/Incremental strategy. This results in what are effectively mountable virtual hard drive images I can use for restoring files locally.
I also use Code42 CrashPlan for backup up Main Storage to the cloud. This provides the off-site component of my "3-2-1" backup strategy (three copies of each file, two media types, one copy off site). I don't count this as a drive (since it's not part of my physical system), but mention it here for completeness because you said "primary storage to backups" and it is part of my backup strategy.  

This is probably more comprehensive (complex?) than most non-professional photographers may need. Most of the complexity comes from the "3-2-1" backup strategy and I started doing that religiously after having to explain to my wife how we lost years' worth of photos of her son because the cloud backup option I used at the time failed miserably when attempting to restore. (That's a story worthy of its own thread.)
Link Posted: 2/5/2021 11:58:27 PM EDT
[#21]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
OK, here's my entire data flow - and remember that you asked for it...  

  1. Capture images on a memory card.
  2. Ingest images:
    • Collate "keeper" images onto my Workbench Drive (1TB NVMe SSD).
    • Backup "keepers" to Archive Drive (8TB 7200rpm HDD).
    • Format memory card.
  3. Edit images using the Workbench Drive.
  4. After edits are done, move (not copy) originals and any edits (e.g., raw files and derivative PSDs) to "Current Album" folder (named uniquely in format "Album_NNN") on Main Storage (2x 14TB 7200rpm HDD configured in RAID 1 [mirror]).
  5. After "Current Album" reaches ~90GB,
    • Copy "Current Album" to Archive Optical Disc (100GB M-Disc BluRay).
    • Copy "Current Album" to Archive Drive.
    • Delete "keepers" from Archive Drive (separate folders for Backup [temporary] and Archive [permanent]).
  6. Backup process runs every day:
    • Cloud (Mon-Wed-Fri-Sat)
    • System Backup (Sun-Tue-Thu)

If you've kept count, I use 5 drives:
  1. Workbench - initial ingest and edits. This is the most data-intensive part of my photo management workflow and I use the fastest drive in my system for this phase.
  2. Main Storage - permanent storage of images after initial edits are complete. This is a large capacity drive (which pretty much mandates spinning media) that has slower performance than the Workbench.
  3. Archive Optical - permanent backup to second media type. This is a collection of BluRay discs.
  4. Archive - initial backups and permanent archive. Initial backups are kept at least until primary edits are complete. Archive is updated concurrently with creation of Archive Optical disc.
  5. System Backup - local backup image of my OS drive and Main Storage drive. I currently use Macrium Reflect with a Full/Differential/Incremental strategy. This results in what are effectively mountable virtual hard drive images I can use for restoring files locally.
I also use Code42 CrashPlan for backup up Main Storage to the cloud. This provides the off-site component of my "3-2-1" backup strategy (three copies of each file, two media types, one copy off site). I don't count this as a drive (since it's not part of my physical system), but mention it here for completeness because you said "primary storage to backups" and it is part of my backup strategy.  

This is probably more comprehensive (complex?) than most non-professional photographers may need. Most of the complexity comes from the "3-2-1" backup strategy and I started doing that religiously after having to explain to my wife how we lost years' worth of photos of her son because the cloud backup option I used at the time failed miserably when attempting to restore. (That's a story worthy of its own thread.)
View Quote


Thanks for spending the time to type all that up, I appreciate it.

What are you using for your archive drive? Just another 14TB drive?

What's the plan for when you fill that 14TB of space?

I've looked into the M-Discs, but at $12 per 100GB I'm not really sure they're the best method for me. Financially it seems to make more sense to buy HDD's, even if that means having to purchase hardware and setup a second NAS/unraid server.
Link Posted: 2/6/2021 12:22:38 AM EDT
[#22]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


Thanks for spending the time to type all that up, I appreciate it.

  1. What are you using for your archive drive? Just another 14TB drive?
  2. What's the plan for when you fill that 14TB of space?
  3. I've looked into the M-Discs, but at $12 per 100GB I'm not really sure they're the best method for me. Financially it seems to make more sense to buy HDD's, even if that means having to purchase hardware and setup a second NAS/unraid server.
View Quote

  1. Yes, the System backup drive is a single 14TB unit. Main Storage is a pair of 14TB drives configured in a mirrored RAID array. My Archive drive is currently an 8TB HDD and I treat this as a "hot backup" unit.
  2. Replace the 14TB drives with higher capacity units. A NAS may be introduced to the system at that point depending on the available size and cost of higher capacity drives.
  3. Regular BluRay discs can work at a lower cost point, for sure. I went with M-Discs primarily due to reputation and capacity. These discs are in addition to the hard drives and serve as disconnected local backups and a second media type mitigates risks (however small) such as fluid spills. HDDs can serve the purpose of disconnected backups if one accepts those risks (again, however small they may be). My strategy is based heavily on the one Peter Krogh describes in "The DAM Book" and summarized at DPBestFlow.org.
Link Posted: 2/6/2021 12:31:13 AM EDT
[#23]
Photos stored in Apple 'photos' app.
Synced thru iCloud with masters kept on local iMac hard drive.
Backed up to Carbonite
Time machine local backups to 4 rotated 4TB drives. 3 oldest in offsite safe deposit box.

45,000+ photos, about 500GB
Link Posted: 2/6/2021 12:39:19 AM EDT
[#24]
Discussion ForumsJump to Quoted PostQuote History
Quoted:

  1. Yes, the Archive drive is a single 14TB unit. Main Storage is a pair of 14TB drives configured in a mirrored RAID array.
  2. Replace the 14TB drives with higher capacity units. A NAS may be introduced to the system at that point depending on the available size and cost of higher capacity drives.
  3. Regular BluRay discs can work at a lower cost point, for sure. I went with M-Discs primarily due to reputation and capacity. These discs are in addition to the hard drives and serve as disconnected local backups and a second media type mitigates risks (however small) such as fluid spills. HDDs can serve the purpose of disconnected backups if one accepts those risks (again, however small they may be). My strategy is based heavily on the one Peter Krogh describes in "The DAM Book" and summarized at DPBestFlow.org.
View Quote


Yeah, the rough part about HDD's is they only go so big. I think it's going to be hard to keep getting big gains out of the format as well. It's kind of amazing how far they've come, but look at 2.5" drives, they had to go from 9.5mm to 15mm to get above 2TB.

That was my reasoning on going with a NAS/server with a lot of HDD bays. Building it myself also made sense from a hardware replacement standpoint.
Link Posted: 2/6/2021 12:53:00 AM EDT
[#25]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


Yeah, the rough part about HDD's is they only go so big. I think it's going to be hard to keep getting big gains out of the format as well. It's kind of amazing how far they've come, but look at 2.5" drives, they had to go from 9.5mm to 15mm to get above 2TB.

That was my reasoning on going with a NAS/server with a lot of HDD bays. Building it myself also made sense from a hardware replacement standpoint.
View Quote
Yep, just when it seems HDDs are about to hit a capacity wall, manufacturers introduce tech for much greater sizes - like HAMR and MAMR.
Link Posted: 5/1/2021 6:28:48 PM EDT
[#26]
What would be the ideal setup for a part time drone photography business?

I plan on focusing on agriculture and construction industry’s.
Link Posted: 5/2/2021 2:22:52 AM EDT
[#27]
I'm currently about to redo my whole setup.  Considering my internet is obscenely slow, I'm looking at alternative methods.

Medium term storage will be onto redundant drives/NAS.
Long term storage of important files will be archived to M-Disc and stored offsite.  Namely the images I sell the most of, important shots, etc.
Ideally once I get better internet, I'll be doing active file backups to backblaze or similar.

That said, I need to go through my archive and massively cull things first.  That's the huge time sink since I haven't culled files in a year or two.
Link Posted: 5/2/2021 2:26:36 AM EDT
[#28]
A USB portable hard drive
Link Posted: 5/2/2021 2:30:06 AM EDT
[#29]
Link Posted: 5/2/2021 3:13:30 AM EDT
[#30]
NAS server at home with RAID and weekly dumps to a NAS backup drive, portable drives in the safe, then portable offsite at a family members house about once a quarter.

I did the online storage for a couple years, don't exactly trust it anymore and I don't think both my house and my dad's house are going to burn down at the same time, not to mention I find it a hassle.  I try and keep it low effort, or I'll find a reason not to mess with it.
Link Posted: 5/2/2021 9:13:18 AM EDT
[#31]
Mine’s going to go exponential once I get out and start really shooting with the new camera. But that’ll be a good problem to have.
Link Posted: 5/4/2021 12:06:21 AM EDT
[#32]
My system tends to be old school.
Primary files on internal and external USB hard drives.
Backups on external USB hard drives.

Backups done by windows CMD scripts using XCOPY and ROBOCOPY.

My personal files stay on the hard drives and their backup drives.

My "pro" files are on the hard drives and their backups until the images are uploaded for their online viewing/ordering. Then those original files and their edits are burned to DVD and removed from the hard drives and from the backup drives.

Yes, I know that if the house burns down, I have lost everything. I am willing to take that risk.

One important part of any workflow is to properly triage the images and delete the bad ones that are not worth keeping in the first place.
Link Posted: 5/4/2021 12:57:20 AM EDT
[#33]
Link Posted: 5/5/2021 8:56:31 AM EDT
[#34]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Home NAS
View Quote

This, plus a couple of free-standing HDDs.
Redundancy.
Close Join Our Mail List to Stay Up To Date! Win a FREE Membership!

Sign up for the ARFCOM weekly newsletter and be entered to win a free ARFCOM membership. One new winner* is announced every week!

You will receive an email every Friday morning featuring the latest chatter from the hottest topics, breaking news surrounding legislation, as well as exclusive deals only available to ARFCOM email subscribers.


By signing up you agree to our User Agreement. *Must have a registered ARFCOM account to win.
Top Top