Question on SAS, SATA, etc

[ZFSLover]

Folks,

Have a couple of hardware questions, been doing a lot of reading of threads here and elsewhere, and I am a bit confused. The end goal is to build a large NAS NFS only device used to store archive data, and the NAS devices would be replicated across data centers.

Config summary: Looking at using Dell R610 nodes with 96GB ram and local HDD's for FreeNAS, and using the other 2.5" bays with SSD drives for ZIL and cache. Inside each of the R610's would be 2 x LSI SAS 9205-8e (http://www.lsi.com/products/storagecomponents/Pages/LSISAS9205-8e.aspx), each card would have dual connections to a SuperMicro JBOD 837E26-RJBOD1 (http://www.supermicro.com/products/chassis/3U/837/SC837E26-RJBOD1.cfm).

1. With the cost of SAS drives still being stupidly expensive, I was thinking of using an enterprise class SATA 6gb HDD. However I keep reading that using SATA HDD's inside SAS JBODS is a bad thing and will cause problems with ZFS? Something about SAS expanders, etc. What am I missing or not understanding?

2. Would having the ZIL and cache on another controller/backplane inside the headnode R610, cause performance issues?

3. I know I would not get multi-pathing with SATA when FreeNAS supports it, but the idea is to use a more cost effective HDD (SATA), until SAS came back down in price. Is there a huge performance hit on 6GB SATA vs 6GB SAS, if so, do you think it would be noticeable?

4. Is there a most cost effective way to do these builds, other hardware I should be looking at, or other configurations?

Thanks folks!

 

[b1ghen]

That is some serious hardware there :)

You say it's for archiving data, this means data will be written once and rarely touched again? If that's the case using anything but SATA seems like a waste of money and space to me.

 

[ZFSLover]

b1ghen,

Thanks for the reply. It is really a backup service, data is written more then it is read, but data is still read for customer restores, as well as weekly it goes through a synthetic restore process to make sure the data is valid and can be read. Retention periods are in years (7 to 10 years).

Think enterprise class backup storage, to get it off our primary SANs and to something more affordable, but reliable, hence having multiple copies of the data via replication in multiple data centers.

Thanks!

 

[b1ghen]

I don't know how popular I am going to be here but such a "high end" solution I personally would look into running FreeBSD natively to get a newer ZFS implementation and get a little more control over things.

For your other question:

1. About the SAS expanders I can't help you out there, I have read both success and failure stories about using them but have no experience myself.

2. I don't see this being a disadvantage at all, rather an advantage having them not have to pass through the limit that is the SAS interface to each JBOD.

3. I don't see any advantage in going with SAS drives since what you seem to be discribing is mostly sequential data transfers anyway and with both ZIL and a large enough cache to handle popular data it should be more than sufficient given the amount of vdev's you probably will have in your zpool.

4. I like the hardware you selected so far, I assume the HBA and the expander in the JBOD's are verified compatible though?

 

[ZFSLover]

I was thinking the same thing with using FreeBSD directly, however I've read about all sorts of instability on version 9.0, and 8.2 it is my understanding that it is the same version of ZFS (15) that is in the current FreeNAS.

The other thing about FreeNAS vs FreeBSD is the gui that is in FreeNAS as well as the tuning done for FreeNAS vs generic FreeBSD with ZFS. Do you have any experience with FreeBSD 9.0 yet?

I'm pretty sure the HBA and expanders are working with FreeNAS, I'll have to triple check.

Thanks again!

I don't know how popular I am going to be here but such a "high end" solution I personally would look into running FreeBSD natively to get a newer ZFS implementation and get a little more control over things.

For your other question:

1. About the SAS expanders I can't help you out there, I have read both success and failure stories about using them but have no experience myself.

2. I don't see this being a disadvantage at all, rather an advantage having them not have to pass through the limit that is the SAS interface to each JBOD.

3. I don't see any advantage in going with SAS drives since what you seem to be discribing is mostly sequential data transfers anyway and with both ZIL and a large enough cache to handle popular data it should be more than sufficient given the amount of vdev's you probably will have in your zpool.

4. I like the hardware you selected so far, I assume the HBA and the expander in the JBOD's are verified compatible though?

 

[b1ghen]

I was thinking the same thing with using FreeBSD directly, however I've read about all sorts of instability on version 9.0, and 8.2 it is my understanding that it is the same version of ZFS (15) that is in the current FreeNAS.

The other thing about FreeNAS vs FreeBSD is the gui that is in FreeNAS as well as the tuning done for FreeNAS vs generic FreeBSD with ZFS. Do you have any experience with FreeBSD 9.0 yet?

No I haven't seen a need to go with anything but FreeNas so far, haven't done any huge deployments like yours yet but I have been gearing up to it :)

 

[survive]

Hi ZFSLover,

Why don't you call our friends over at iX systems:

http://www.ixsystems.com/

And see what they suggest?

-Will

 

[ZFSLover]

I have, the pricing for their hardware isn't inline with what you can but Dell's and SuperMicro's for, or at least what we buy them for. I do understand you get TruNAS with support from them, but they don't break out the cost of the TrueNAS SW vs hardware, it is just one big bundle pricing.

Hi ZFSLover,

Why don't you call our friends over at iX systems:

http://www.ixsystems.com/

And see what they suggest?

-Will

 

[louisk]

I'm not aware of any issues using SATA inside SAS JBOD. I'm doing it with an MD1000, no issues. Also have a 4U 36 bay SATA Supermicro running as JBOD w/o issue.

I don't see any issues having the ZIL elsewhere inside the same machine.

If you're going to archive, I would not waste $ on SAS. Buy SATA in quantity and replace drives when they fail. Keep several spares physically present for quick replacement. Still cheaper than SAS

I would buy hardware from iX and implement FreeNAS if TrueNAS won't fit. I've been using them for about 6yrs and I get the best hardware prices there.

 

[ZFSLover]

I'm not aware of any issues using SATA inside SAS JBOD. I'm doing it with an MD1000, no issues. Also have a 4U 36 bay SATA Supermicro running as JBOD w/o issue.

I don't see any issues having the ZIL elsewhere inside the same machine.

If you're going to archive, I would not waste $ on SAS. Buy SATA in quantity and replace drives when they fail. Keep several spares physically present for quick replacement. Still cheaper than SAS

I would buy hardware from iX and implement FreeNAS if TrueNAS won't fit. I've been using them for about 6yrs and I get the best hardware prices there.

Louis, thank you for your reply!

A couple of questions, if I may:

- Are you happy with the MD1000's? If you are using SATA, do you have a source for the SATAu interposer hdd sleds needed for them? I've looked at them, but the SATAu sleds I found were all 35-50 per sled, which adds up.

- What SATA HDD"s are you using, and are you happy with the performance and reliability of them?

- What SAS HBA's are you using with your FreeNAS/MD1000 and SuperMicro JBODS?

Thanks!

 

[louisk]

I'm using the same sleds that would be used for SAS.

I don't have a source for them. My MD1000 arrived with sleds. I would watch ebay. put a reminder or something to email you pricing on it. Things tend to flux quite a bit, or you'll randomly find somebody unloading 10 of them for $5ea.

I have no complaints about the MD1000. It works well.

I have a mix (4 of each) Seagate and WD 2T 7200rpm enterprise drives. I am a little limited on the performance tests because my FreeNAS is the only machine in the house with multiple spindles and network connections, but that said, I can push 600Mbit w/o a problem.

PERC5 for the MD1000. 3ware 9x00 series for the SuperMicro (originally it was hardware raided but now runs as jbod+zfs).