Need Feedback and Help on my intended Build

[dropline]

Hi Guys,

we need a NAS for our PhotoStudio. After some research I´m very sure that FreeNas is the best choice for us. I did never build one and that´s why I´m here to make sure that I buy the right components and set everything up the right way. To prepare I did already read quite a bit on the forum, and I think there shouldn´t be anything majorly wrong with my setup. There is still a lot that I have to learn, zfs is new to me as well. I work in IT since 11 years, so I think I should be fine if I carefully read about everything, double check and ask for help here before I screw up.

This NAS will be entirely used as storage for photos (mainly raw, jpg and a few psds)
I´ll describe a bit how we work in the studio:
After our Photographers are done with shooting, the raw files (20-40 mb per photo, 600-1200 files per shoot) will be transferred via Gigabit LAN from the SD our CF card to the NAS. After that, all the files get renamed via Adobe bridge (batch rename) and metadata gets saved to each file (also done in Adobe bridge).
(I´m going to change that process, so that the camera gives the files the right names straight away and also saves the metadata. This will take some load of the network)

After that´s all done, at some point, the photographers select and edit the photos and convert about ~100 to jpg. Done via gigbit LAN in Adobe Photoshop CS6.

We have up to 25 shoots a day, so this process can happen up to 25 times a day. This is absolute max, more common is probably 10 daily.

So I need a build that can handle these uploads and about 5-10 Photographers doing edits constantly.

I hope that's enough information and not too much to read :).

My Shopplinglist so far:

Supermicro X10SLM-F
Xeon e3-1231 v3
2x crucial 8GB 1600mhz ddr3 CT102472BD160B

4x WD Red 4TB
some small SSD for freenas
some Case and PSU

I thought about using the 4 WD Reds to make 2 stripes and then mirror them? I guess that´s better than 2 mirrors and then stripe the 2 mirrors? I´m happy about any input, tips and feedback on this! I´m lacking experience with FreeNas, so if there is a better way of doing this, please let me know.

I´m also thinking about putting Fiberglass in the NAS to have more than 1gb/s network. Does anybody have ideas, tipps or alternatives about that? Are there options to use the second 1gb/s lan card in the server to provide faster network speeds?

Alright, so that´s it. I hope this works :) Thanks already for any help.

 

[Bidule0hm]

I thought about using the 4 WD Reds to make 2 stripes and then mirror them? I guess that´s better than 2 mirrors and then stripe the 2 mirrors?

You can't do that with ZFS, only striped mirrors. And in fact mirrored stripes are worse than striped mirrors.

I´m also thinking about putting Fiberglass in the NAS to have more than 1gb/s network. Does anybody have ideas, tipps or alternatives about that? Are there options to use the second 1gb/s lan card in the server to provide faster network speeds?

You may want to search for LACP and/or 10G links ;)

 

[jgreco]

There's a nice primer on 10G options in the networking forum. Buying a Dell 5524 on eBay and a 10G card for your NAS is not horribly expensive. You could also upgrade one workstation to 10G for fast dumping from the flash media.

Several mirror vdevs will be necessary to make good use of a faster network.

 

[dropline]

@ Bidule0hm thank you. Alright, so striped mirrors it is. My initial thought behind this is just that it would outperform a RAIDZ2, which i was planning on getting before. If I get the 10g network, I have to make the right decision here. Talking Parity, obviously I would prefer RAIDZ2. I read a lot about RAIDZ2,3,4 but not much about striped mirrors, so I can´t really judge what is the better option for me?

thanks @ jgreco for the info on 10G in the networking forum. That´s gonna be my bedtime lecture for tonight :)

 

[jgreco]

Really, the best way to determine the better option for you is to try it both ways and see how performance is.

The downside to two-way mirror vdevs is that you lose redundancy when you lose a drive. You can do three-way mirror vdevs (three mirrored drives) but that gets expensive.

A vdev approximates the performance of the slowest of the individual component drives, so if you go with a single vdev solution like a RAIDZ2, it will tend to be slower.

A six drive RAIDZ2 of 2TB disks gives you 8TB of space but the speed of only a single drive.

Bumping up to 4TB drives, six 4TB drives arranged as two vdevs of three-wide mirrors gives you the ability to lose any one drive without losing redundancy, just like RAIDZ2, but this arrangement will be twice as fast at writing and several times faster at reading, giving you 8TB of much faster space.

The downside there is that you're using 24TB of raw space to provide 8TB of usable space.

 

[BigDave]

You might want to consider a FreeNAS product from iXsystems.
The appliances have the 10Gig upgrades available along with
training and of course, product support.

 

[dropline]

I considered IXSystems, problem is that I have to keep the cost down as much as possible. Also, we are located in Melbourne, Australia. That makes sourcing hard enough, even just for some parts.Thanks for the offer though :) Especially the 10gbps components I might have to get from ebay, otherwise I might not get them at all. I´m not sure yet, I´ll know tomorrow.

Also about the drives, I like the 6x4TB solution, but I´ll see tomorrow how I go with cost. I guess we end up with 2 vdevs with 2 4TB mirrors and this striped. I think I can get 10gbps though, if i get the stuff 2nd hand. Cost of the drives will prevent me to get more money for more drives. And there´s no way I buy 2nd hand HDDs though ;)

 

[Bidule0hm]

A six drive RAIDZ2 of 2TB disks gives you 8TB of space but the speed of only a single drive.

Bumping up to 4TB drives, six 4TB drives arranged as two vdevs of three-wide mirrors gives you the ability to lose any one drive without losing redundancy, just like RAIDZ2, but this arrangement will be twice as fast at writing and several times faster at reading, giving you 8TB of much faster space.

For IOPS that's true but for throughput it's false. A 6 drives RAID-Z2 will give the throughput of 4 drives and the IOPS of one drive.

 

[jgreco]

Actually, no, a 6 drive RAIDZ2 won't give you the throughput of four drives, although, yes, it does give you more than a single disk.

Code:

# dd if=/dev/zero of=zeroes bs=1048576
^C429606+0 records in
429605+0 records out
450473492480 bytes transferred in 562.057842 secs (801471768 bytes/sec)

That's an 11 disk RAIDZ3 of drives that write at 140MB/sec, and 8 * 140 ~= 1120, so after ZFS is done, these are only writing at about 2/3rds the speed that the non-parity drives would be able to support.

 

[Bidule0hm]

Well, ok, but it's almost the throughput of 6 drives so it's not bad at all (and it's RAID-Z3 which is more demanding on the CPU for parity processing than RAID-Z2).

 

[jgreco]

Also worth noting that it goes straight to hell if there's any other activity going on with the pool.

 

[Bidule0hm]

But with mirrors you'll have the same problem, no?

 

[jgreco]

But with mirrors you'll have the same problem, no?

Not really, the mirrors seem to be able to sustain more traffic in parallel. If you have multiple things or people operating on the files, that's preferable.

 

[Bidule0hm]

Interesting info ;)