Draft Build - Please take a look :)

[hlnoiku]

CPU: Intel Xeon E5-1620V2 3.7ghz (Quad)
Mobo: SuperMicro X9SRH-7TF (dual 10Gbe)
RAM: KVR16R11D4K4/64 (64gb total)
SSD Cache: 512gb (256x2) samsung 840 pro mlc
USB Boot: 3x USB3.0 32GB
SSD ZIL: 2x TLC SSD
Pool: 12x 4tb hitachi raid 1+0: 24tb, H3IKNAS40003272SN
Case: SC846XA-R1K28
Accs: 2x MCP-220-82609-0N + SAS breakout cables
SAS: 2x LSI LSI00301-9207-8i (alt. IBM ServeRAID M1015 2x$175)
switch: Netgear prosafe xs712t 10gbe

Purpose: Serve ~20 workstations + 40 nodes (vfx files)

 

[anodos]

How are you planning to use the SSDs? Depending on your workload and budget you may want to add more RAM (it never hurts to have more).

 

[hlnoiku]

How are you planning to use the SSDs? Depending on your workload and budget you may want to add more RAM (it never hurts to have more).

Ah sorry I should've clarified, corrected the original post. Thanks!

Would it make sense to drop the cache SSD's and put the $ towards doubling the ram? I'd like it to be able to pump the 2x10Gbe lines as much as possible.

 

[anodos]

Ah sorry I should've clarified, corrected the original post. Thanks!

Would it make sense to drop the cache SSD's and put the $ towards doubling the ram? I'd like it to be able to pump the 2x10Gbe lines as much as possible.

Increasing ARC is always preferable to adding an L2ARC device, so yes - put the money to RAM. What protocols will you be using for shares? What will be the approximate reads vs. writes? (ie mostly reads)

You won't gain much by having USB3 for your boot disk (I don't believe it is supported). Having a separate intent log (ZIL) also may not buy you much. It only affects synchronous writes.

If you are planning to ever use a SAS expander then you should get SAS drives rather than SATA drives.

 

[hlnoiku]

Increasing ARC is always preferable to adding a L2ARC. What protocols will you be using for shares? What will be the approximate reads vs. writes? (ie mostly reads)

You won't gain much by having USB3 for your boot disk (I don't believe it is supported). Having a separate intent log (ZIL) also may not buy you much. It only affects synchronous writes.

I was reading the hardware suggestion guide which advised usb3 sticks in usb2.0 slots. If that's going to be an issue then no problem changing the boot drives to mirrored ssd's.

It's going to be a mix of reads and writes. Reads will be smaller files (8-16mb files at most), while writes will be considerably larger (~8gb+ files).

I've read in other posts the synchronous writes are only a benefit if using NFS. This is primarily going to be CIFS so if the ZIL isnt going to help much I can drop it. I added due to a forum posting I saw with an all SSD array stating he was only seeing ~1.1GB's Write and adding the ZIL drive bumped it up to an 1.6GB's Write. Although in his case he was using ESXi datastores so perhaps I wont need this?

 

[Ericloewe]

Increasing ARC is always preferable to adding an L2ARC device, so yes - put the money to RAM. What protocols will you be using for shares? What will be the approximate reads vs. writes? (ie mostly reads)

You won't gain much by having USB3 for your boot disk (I don't believe it is supported). Having a separate intent log (ZIL) also may not buy you much. It only affects synchronous writes.

If you are planning to ever use a SAS expander then you should get SAS drives rather than SATA drives.

SATA drives work fine with reputable ("reputable" tends to mean "LSI" around here) SAS controllers and expanders, as the SAS spec was designed to support SATA drives attached to expanders.

 

[hlnoiku]

SATA drives work fine with reputable ("reputable" tends to mean "LSI" around here) SAS controllers and expanders, as the SAS spec was designed to support SATA drives attached to expanders.

:) Well for throughput and future-proofing, it'll be using a non-expander backplane, hence the 2x LSI controllers + 1x onboard LSI controller. That we if one day we decide, hey SSD's are cheap, we shove 24x SSD's in there 0_0

 

[anodos]

the SAS spec was designed to support SATA drives attached to expanders.

Just like how the ECC spec guarantees that all ECC RAM and server motherboard combos will work properly? :)

I was indirectly referring to the below blog post. I have also had a conversation with someone at iX about why they use SAS SSDs for zil and l2arc and received a similar answer.

http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.html?m=1

Now it is entirely possible that the rep is unfounded, but once I hear that something can lead to pool loss I don't touch it with a 10 foot pole.

 

[anodos]

I was reading the hardware suggestion guide which advised usb3 sticks in usb2.0 slots. If that's going to be an issue then no problem changing the boot drives to mirrored ssd's.

What I meant to convey is that spending more than $20-30 per boot drive isn't money well spent. Just get a few reputable USB flash drives. One to keep in the system and a few in your desk drawer as backups. When it's time to upgrade, install the iso to one of your spares, power down the system, and swap them out.

I've read in other posts the synchronous writes are only a benefit if using NFS. This is primarily going to be CIFS so if the ZIL isnt going to help much I can drop it. I added due to a forum posting I saw with an all SSD array stating he was only seeing ~1.1GB's Write and adding the ZIL drive bumped it up to an 1.6GB's Write. Although in his case he was using ESXi datastores so perhaps I wont need this?

I would say that you almost certainly don't need a ZIL. If you need more write performance in the future you will need add more spindles and so be sure to leave yourself room for growth (either in your case or through an expansion shelf).

You should however put careful thought into how you architect your file tree as this can greatly affect directory browsing in CIFS.

 

[anodos]

To be clear from earlier posts. For me. Home use = SATA drives. Work use = SAS drives. There is not much price delta between the two, especially when you take into account the cost of any downtime. Even in a small business, a day of downtime can cost upwards of 5-10k.

To that end, you should also look at the costs of support contracts, making sure that your UPS is up to snuff, and making sure that your backup system is reliable. It might be worthwhile to call iXsystems and get a few quotes for different server options (that way you have a single point of contact for hardware warranties). My experience is that it's easier to push these things through management when the checkbook is already out.

 

[cyberjock]

I was indirectly referring to the below blog post. I have also had a conversation with someone at iX about why they use SAS SSDs for zil and l2arc and received a similar answer.

http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.html?m=1

And you want in on a secret? Some document I was reading on the internet said that Intel SATA and SAS SSDs (when Intel made SAS SSDs) were both put through the *exact* same testing. Of course if you look at Intel's product line you will see no SAS at all. So either Intel has completely ignored a very profitable market (I find this pretty unlikely) or the info we have is not accurate. The argument for SAS versus SATA may be true for rotating media, but SSDs are a totally different product and aside from sharing an interface and "storing your data" they really share nothing in common.