How do I figure out the best SATA ports on a system to use for a zpool?

[rathijit]

Hi forum, this is my first post here. I've been using FreeNAS for a few weeks and am really enjoying ZFS. My system has a lot of drives and was wondering if connecting them a certain way ensures better performance. For example, would alternating connections from a SATA controller be better than connecting all the drives in sequence (like 0, 2, 4 connected to VDev 1 and 1, 3, 5, 7 connected to VDev 2)? Or for example, would it be better to have a few drives connected to the motherboard directly and a few connected to a SATA controller on a PCIe slot, than connecting all the drives to the same controller?

I have the following hardware:
* AMD Phenom II 1100T
* MSI 970 Gaming Motherboard
* 32GB RAM (they have ECC capability, but my motherboard doesn't support ECC... oh, well)
* LSI LOGIC SAS 9207-8i Storage Controller - provides 8 SATA/SAS connections from 2 ports.
* 2x 250GB SSD drives connected to the motherboard's SATA ports directly in mirrored configuration (configured as a separate VDev in a pool).
* 8x 3TB HDD drives... half connected to the motherboard's 4 remaining remaining SATA + another half connected to one of the the LSI storage controller's ports (which splits into 4 SATA ports). These 8x drives form another VDev in another pool... RaidZ2.
* 4x 2TB HDD drives... connected to the remaining port on the LSI storage controller via 4 SATA connections. These 4x drives form another VDev in yet another pool... still deciding on the configuration (will probably put it in RaidZ2 as well).

Total of 14 drives, spread across 3 VDevs in 3 separate pools. I read that if a VDev is lost, the whole pool is lost, so to be extra cautious, I put the VDevs in separate pools.
Note that for the VDev with 8 drives, I have connected half to the motherboard, and the other half to the LSI SATA controller... is this a better way of doing it than connecting all 8 to the same LSI SATA controller?

 

[rathijit]

Currently, I get very poor read and write rates... I think it is limited by LAN, since I get about 75 to 85 megabytes/s transfer via LAN. What do most people do to improve this? I don't think Link Aggregation would help here, correct?

Also, is there a way I can check my disk access rate? For example, I want to know how fast I should be able to write or read if my LAN wasn't a bottleneck.

 

[Chris Moore]

8x 3TB HDD drives... half connected to the motherboard's 4 remaining remaining SATA + another half connected to one of the the LSI storage controller's ports (which splits into 4 SATA ports). These 8x drives form another VDev in another pool... RaidZ2.

It would give you the best performance to connect all of these to the SAS controller.

* 4x 2TB HDD drives... connected to the remaining port on the LSI storage controller via 4 SATA connections. These 4x drives form another VDev in yet another pool... still deciding on the configuration (will probably put it in RaidZ2 as well).

Put these on the system board SATA ports.

I have found, over the course of years and several builds, that splitting a vdev (or a pool) across controllers actually slows it down.

 

[rathijit]

It would give you the best performance to connect all of these to the SAS controller.

Put these on the system board SATA ports.

I have found, over the course of years and several builds, that splitting a vdev (or a pool) across controllers actually slows it down.

Good to know. Thanks! I'll move them over. :)