SATA on SAS Expanders in 2016

[AnonymousTrader]

I was recently hired on to a high-frequency trading company and asked to design one of their new servers with a budget of around 15-20k using ZFS as a filesystem. Having never done anything on this scale before, I am obviously a little terrified and I want to make sure I get things right. Here are some of the components I have opted for:

• Chassis: STX-NL-XE36 -- A 36 slot chassis with an internal SAS2 expander.
• HBA: LSI SAS 9300-8i 12Gb/s 8-Port -- A seemingly high-quality HBA that hooks into the chassis with dual mini-SAS HD SFF-8643 connectors. Supports both SATA and SAS and according to the Thinkmate engineers the only thing I will need to access all 36 drives. The chassis does not support multiple HBA’s.
• Hard Drives: Seagate ST8000DM002 8TB SATA 6.0Gb/s -- $300 is pretty good for 8TB, and these drives shouldn’t encounter the issues normally associated with SMR’s. The SAS drives are $100 more and outside our price range.

This server is mostly going to be read-only and I’m going to include an L2ARC on one of the boot drives (separate from the HBA), so I’m not really worried about performance. My concerns stem from a series of posts I’ve read condemning the use of SATA drives behind SAS expanders. [1] [2] [3]

However I have also read that this is only an issue with older SAS1 expanders and modern SAS2 expanders (like the one I will be using) should work. The FreeNAS hardware guide states:

“However, running SATA drives in SAS expanders –while supported– is a less desirable configuration than using SAS end to end due to the difficulty of translating SATA errors across the SAS bus.”

So what do you guys think? What issues might I expect in 2016? How might SATA error translation issue affect my setup?

 

[zambanini]

scnr, lol!

 

[Mlovelace]

with a budget of around 15-20k

With that budget get a Nimble array. https://www.nimblestorage.com/
You're in over your head, no offense.

 

[TXAG26]

Like your job? I'd contact iXSystems and get it done correctly. You have enough budget. Just an fyi, those are desktop HDD's and there's no way I'd use something like that in a production server for a stock trading.

 

[Mirfster]

I was recently hired on to a high-frequency trading company and asked to design one of their new servers with a budget of around 15-20k using ZFS as a filesystem. Having never done anything on this scale before, I am obviously a little terrified and I want to make sure I get things right.

No offense, but that there should say it all. Get iXSystems involved.

When I hear "high-frequency trading company"; I am thinking of stock trading where speed is king. They are usually dealing with milliseconds (or less). Of course I could be wrong on that assumption, but regardless if you are not that versed and never done something of this scale; it is too risky IMHO.

scnr, lol!

Probably going to feel stupid when/if you tell me, but what the heck does "scnr" mean? :)

 

[Mlovelace]

"scnr"

sorry, could not resist

 

[Mirfster]

Lol, okay will add that to memory. ;)

 

[AnonymousTrader]

No offense, but that there should say it all. Get iXSystems involved.

When I hear "high-frequency trading company"; I am thinking of stock trading where speed is king. They are usually dealing with milliseconds (or less). Of course I could be wrong on that assumption, but regardless if you are not that versed and never done something of this scale; it is too risky IMHO.


Probably going to feel stupid when/if you tell me, but what the heck does "scnr" mean? :)

It's a storage server, there's no need for low-latency. We're running ZRAID3.

Like your job? I'd contact iXSystems and get it done correctly. You have enough budget. Just an fyi, those are desktop HDD's and there's no way I'd use something like that in a production server for a stock trading.

From what I've read there isn't a big a big difference in terms of reliability, I'm thinking of Backblaze's failure stats in particular. We were going to use the same 8TB drives they do, and their reliability seems reasonable.

 

[TXAG26]

It's a storage server.

It's for business.

 

[Stux]

Well,

What are you thinking?

10gbe?

How many TB? How were you going to structure the vdevs? Backup plans?

 

[AnonymousTrader]

It's for business.

I don't understand. If the drive failure rates and reliability are as good as enterprise versions, why does it matter?

Well,

What are you thinking?

10gbe?

How many TB? How were you going to structure the vdevs? Backup plans?

We are initially going to start with a 16 drive vdev (with one or more drives for hot-swapping) for a total usable space of 74.46TiB. Later we might expand with a second vdev using the remaining slots.

We generate around 30 GB of data a day, so we're going to make an initial backup of all our data from the old server by physically shipping the drives to backblaze, after that we'll make daily incremental backups of the new server's data through the Internet.

 

[Stux]

About your actual question. I was aware of that too, but the latest I could find was I think that

1) there was actually a bug that was fixed.
2) the issue seems to have faded away.

Doesn't everyone use Sata drives for cheap TBs these days?

Beware the IOPS. You'll have the IOPS of a single HD with a single raidzN vdev.

16 seems wide. You could go with 2x8 for twice as many iops.

 

[Mlovelace]

We are initially going to start with a 16 drive vdev

Don't do this, you really don't want to go wider than 12 drives per vdev, and that's pushing it at 8TB drive size. With 8TB drives the most you should think about is 11 drive raidz3 or maybe an 8 drive raidz2.

But really with your price point you should be looking at HA systems (dual controllers) and 24/7 support.

 

[Spearfoot]

We are initially going to start with a 16 drive vdev (with one or more drives for hot-swapping) for a total usable space of 74.46TiB. Later we might expand with a second vdev using the remaining slots.

16 drives in a single vdev is probably too 'wide'. I recommend something like 2 vdevs, each comprising 8 drives in a RAIDZ2 array. This scheme would give you good redundancy - you could lose two drives in each vdev without losing your data - and ~96TB of storage capacity (less overhead). Also, IOPS scale with vdevs, so this configuration would give you twice the IOPS of the single vdev pool you're considering.

Down the road, you could expand the pool twice with add'l 8-drive RAIDZ2 vdevs.

Ah! I see @Mlovelace and @Stux have beaten me to it and already covered this!

 

[AnonymousTrader]

Don't do this, you really don't want to go wider than 12 drives per vdev, and that's pushing it at 8TB drive size. With 8TB drives the most you should think about is 11 drive raidz3 or maybe an 8 drive raidz2.

But really with your price point you should be looking at HA systems (dual controllers) and 24/7 support.

Why the smaller vdevs? Could you go into a little more detail? The reason I didn't want to go lower than 16 was the loss of usable storage according to this calculator.

 

[Spearfoot]

Why the smaller vdevs? Could you go into a little more detail? The reason I didn't want to go lower than 16 was the loss of usable storage according to this calculator.

The more drives in a vdev, the longer it takes to re-silver when replacing a failed drive. Several of us mentioned IOPS; the more vdevs you have, the more IOPS you'll have available and therefore the pool will be more responsive.

EDIT: Here is a good discussion of RAIDZ and stripe width, written by Matthew Ahrens, one of the ZFS architects. He mentions that the old "(2^n)+p" rule for determining vdev disk count doesn't apply when using compression. Summary is "Use RAID-Z. Not too wide. Enable compression."

 

[depasseg]

was recently hired on to a high-frequency trading company and asked to design one of their new servers with a budget of around 15-20k using ZFS as a filesystem.

Based on this sentence, I can only guess that you need a low-latency (HFT) and high-performance ($15-20k budget) system. It sounds like based on the follow up Q&A in the thread, that my initial impression isn't accurate.

Could you share what you are looking for from a drive and system redundancy, sequential and random throughput, and IOPS ?

 

[Mlovelace]

Why the smaller vdevs? Could you go into a little more detail? The reason I didn't want to go lower than 16 was the loss of usable storage according to this calculator.

You've got some reading to do.

In a non-disparaging way, how familiar are you with zfs, and for that matter with freeNAS? You're looking to setup a new server and don't seem to have a handle on the basics. As I keep mentioning, it would behoove you to buy a system that has professional support behind it.;)

 

[Stux]

Hard Drives: Seagate ST8000DM002 8TB SATA 6.0Gb/s -- $300 is pretty good for 8TB, and these drives shouldn’t encounter the issues normally associated with SMR’s. The SAS drives are $100 more and outside our price range.

FWIW, I'd pay the extra 22$ to get a NAS version over the desktop version.

https://m.newegg.com/Product/index?itemnumber=9SIA2F83V46068

 

[Mlovelace]

can we just close the topic? otherwise we are looking at a person who will do damage. that is not funny anymore.

Yep