SAS backplane slowing me down?

[Elliott]

I'm experimenting with building a FreeNAS and I'm not getting the speeds I expected. Could the problem be with the SAS backplane?
Chenbro RM41736 chassis has two backplanes, one with 24 drives and the other with 12 drives. Each backplane connects to an LSI 9300 with one 4-port SAS cable.
Disks: 36 disks, mix of WD Blue WD10EZEX and Toshiba DT01ACA1, all 1TB 7200 RPM

I have tested sequential write on each disk individually and they are all around 160 - 170 MBps. When I make a simple zpool stripe with 8 of these disks, it writes at 1250 MBps (great). As I add more disks, the speed tops out around 1700MBps. CPU does not seem to be maxed out on any thread. So I'm guessing it's a poor quality backplane, is there anything else I could check here?

 

[PhiloEpisteme]

How are you pushing the data? If you're pushing it over 1Gb/s copper consider that even a 170MB/s drive write speed is really pushing past what 1Gb/s copper can achieve.

As far as whether your SAS backplane is a bottleneck my guess is that it is somewhat unlikely. Looking at Chenbro's site it looks like those backplanes are 12Gb/s SAS backplanes, meaning 12Gb/s per lane at 4 lanes giving you a total theoretical throughput of 48Gb/s or around 6GB/s. All 24 drives pushing 170MB/s wouldn't even saturate that even accounting for overhead losses.

If you post your full specs including exact model numbers perhaps folks can help you narrow down where your bottleneck is.

 

[Elliott]

I'm testing internally with fio like this:
fio --name=w --rw=write --bs=1M --size=30G --directory=/mnt/bigstripe

I thought I posted the relevant specs already, what else do you want to know? Motherboard is Asus X99-E-10G-WS with 64GB RAM. The x8 lane LSI card is in a x16 slot.

 

[Elliott]

I've already updated the LSI controller firmware to version 16. There is a newer driver available from LSI version 20, but it says it's for FreeBSD 11.1 and I think that would require recompiling the kernel. I'm running FreeNAS 11.2 U5.

 

[PhiloEpisteme]

I'm testing internally with fio like this:
fio --name=w --rw=write --bs=1M --size=30G --directory=/mnt/bigstripe

I thought I posted the relevant specs already, what else do you want to know? Motherboard is Asus X99-E-10G-WS with 64GB RAM. The x8 lane LSI card is in a x16 slot.

You're right, thanks.

What about your pool layout now and what layout do you plan to use in the future? When you say you used an 8-disk stripe are you referring to 8 single-disk vdevs striped together? It would probably be most informative for you to try to test the system using the layout and workload you'll be subjecting it to. For example, 6 2-disk mirror vdevs will have different performance characteristics than 2 6-disk RAIDZ2 vdevs. Furthermore, consider whether you're using sync writes or not and whether you're going to use a SLOG device. And of course, if your typical actual workload will be over copper or 10G+ fibre that'll tell you where you should be aiming. These things will all affect your speeds.

 

[Elliott]

Eventually I plan to use 4 raidz2 vdevs of 8 disks each, possibly adding NVMe for slog if necessary. Clients will connect with a mix of 10G and 40G. But since parity disks always reduce throughout, right now I am trying to simplify in order to find the performance limits of the hardware. For the test described above, I am using a single vdev striped with no raid.

 

[Elliott]

I've tried to isolate the problem by simultaneously writing to each block device with no filesystem. A command like this makes it easy to see the speeds in iostat. Running with 8 disks or fewer, I get full speed on each, but as I increase the number of disks, the speed drops. With 24 disks, it's about 78MBps per disk.

Code:

for i in {4..27}; do dd if=/dev/zero of=/dev/da$i bs=1M count=1000 &; done

 

[kdragon75]

I've tried to isolate the problem by simultaneously writing to each block device with no filesystem. A command like this makes it easy to see the speeds in iostat. Running with 8 disks or fewer, I get full speed on each, but as I increase the number of disks, the speed drops. With 24 disks, it's about 78MBps per disk.

Code:

for i in {4..27}; do dd if=/dev/zero of=/dev/da$i bs=1M count=1000 &; done

I have seen this too. I was zeroing out 36 disks and speeds per disk were in the 17MB/s range!

 

[PhiloEpisteme]

Yeah, sorry I don't have any other specific suggestions other than running testing on an environment as similar to your final configuration using as close to an actual workload as you can.

Just so I understand clearly, what speed is it you're getting that you're unhappy with? Is it the 1700MB/s? If so, that is obviously faster than you'll expect over a single 10G NIC. And what speeds were you hoping for?

 

[Elliott]

Yeah 1700MBps for 24 drives is disappointing. I will probably use 40G or 100G NIC and add a lot more disks. At this point I think it must be a hardware problem, either with LSI or Chenbro. I always like to start with the simplest configuration and build up, so that I can understand the limitations and what's going on.
I just tried a pool with 3 raidz2 vdevs of 8 disks each and it does about 1100 MB/s. That seems like a reasonable loss due to parity, based on the 1700MB/s maximum.

 

[PhiloEpisteme]

Yeah 1700MBps for 24 drives is disappointing. I will probably use 40G or 100G NIC and add a lot more disks.

Yeah, at 40G or 100G you're certainly going to find those speeds disappointing.

I'm curious how you'd go about identifying it the backplane is the issue. Do you have any SSDs you can test with just to see if you can push ~48Gbps through the 4 lanes to each backplane? When you tested with basic pools did you turn sync writes off to see if that has an impact?

 

[Elliott]

I was given the wrong information about this chassis, it's actually a 6G expander. So this means I only have 24Gbps available for each backplane, and after 8b/10b encoding that leaves me with 2400MBps. With a little overhead for SAS to SATA translation, I guess this is reasonable.

Now I have to figure out the other issue, which is that the OS will only see 28 out of the 36 disks! I believe this is a driver issue, it sounds just like this: https://redmine.ixsystems.com/issues/16372. I'm surprised Broadcom still hasn't fixed this in 3 years.