mpkaye
Cadet
- Joined
- Jun 26, 2019
- Messages
- 4
Hi Folks,
I have a SuperMicro SC847 chassis with SAS2 backplanes, with the following installed:
FreeNAS 11.2-U4
SuperMicro X9DRi-LN4F+
2 x E5 2650 V2 CPU
128GB DDR3 ECC RAM
LSI 9207-8i (firmware 19.00.00.00)
4 x Intel on-board Gb NIC
4 x Intel PCIe Gb NIC (single card, quad port)
2 x Intel X520 DA2 PCIe 10Gb NIC (single card, dual port)
12 x 4TB SAS2 7200RPM
12 x 2TB SAS2 7200RPM
4 x 600GB SAS2 15000RPM
4 x 1TB Samsung 860 EVO SATA SSD
1 x Optane 900P AIC (PCIe) (partitioned for SLOG)
1 x Samsung 970 Pro NVMe on PCIe AIC
I have a pool made up of the 12 x 4TB drives (sync=standard), with the following config:
This pool performs incredibly well over ethernet but extremely slowly locally. I have a LAG configured from the 4 x 1Gb NICs. I migrated data using concurrent rsync processes from another FreeNAS box, also with a 4 x 1Gb LAG. I could sustain around 340 MB/s on average writing to sas72pool2. However, if I try to rsync locally between two different directories on sas72pool2, the performance is terrible, I struggle to maintain 30MB/s. Now, I wouldn't necessarily expect to see read & write operations to and from the same pool to be as good as just write via the network, especially since I have a lot of network capacity in the box. So, I thought I would test a local copy from sas72pool2 to sas72pool1, which has the following config:
This local copy ran at exactly the same speed, around 30MB/s. For reference, sas72pool1 can sustain full 10Gb write speeds (dd 8k writes) from an NFS mounted VMware datastore.
Can anyone help me understand why this is happening?
Cheers!
Mark
I have a SuperMicro SC847 chassis with SAS2 backplanes, with the following installed:
FreeNAS 11.2-U4
SuperMicro X9DRi-LN4F+
2 x E5 2650 V2 CPU
128GB DDR3 ECC RAM
LSI 9207-8i (firmware 19.00.00.00)
4 x Intel on-board Gb NIC
4 x Intel PCIe Gb NIC (single card, quad port)
2 x Intel X520 DA2 PCIe 10Gb NIC (single card, dual port)
12 x 4TB SAS2 7200RPM
12 x 2TB SAS2 7200RPM
4 x 600GB SAS2 15000RPM
4 x 1TB Samsung 860 EVO SATA SSD
1 x Optane 900P AIC (PCIe) (partitioned for SLOG)
1 x Samsung 970 Pro NVMe on PCIe AIC
I have a pool made up of the 12 x 4TB drives (sync=standard), with the following config:
Code:
pool: sas72pool2
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
sas72pool2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/67003936-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/7232bfc5-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/7d4a40f5-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/85fe36a2-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/90f3174c-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/9c0682cc-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/a72442cc-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/aff42aaa-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/b8f84ec4-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/c244147d-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/c95ebace-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/d4814b61-8f91-11e9-bad6-001517c58171 ONLINE 0 0 0
logs
nvd0p4 ONLINE 0 0 0
errors: No known data errorsThis pool performs incredibly well over ethernet but extremely slowly locally. I have a LAG configured from the 4 x 1Gb NICs. I migrated data using concurrent rsync processes from another FreeNAS box, also with a 4 x 1Gb LAG. I could sustain around 340 MB/s on average writing to sas72pool2. However, if I try to rsync locally between two different directories on sas72pool2, the performance is terrible, I struggle to maintain 30MB/s. Now, I wouldn't necessarily expect to see read & write operations to and from the same pool to be as good as just write via the network, especially since I have a lot of network capacity in the box. So, I thought I would test a local copy from sas72pool2 to sas72pool1, which has the following config:
Code:
pool: sas72pool1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
sas72pool1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/38dcdbec-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/40a9cf0b-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/4614c725-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/5039e5d9-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/5813027f-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/6000b09f-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/67e232a4-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/6fce6984-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/77ab4821-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/7f953f1d-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/8788ce26-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
gptid/8f92516a-8f90-11e9-bad6-001517c58171 ONLINE 0 0 0
logs
nvd0p3 ONLINE 0 0 0
cache
gptid/8b767888-8f93-11e9-bad6-001517c58171 ONLINE 0 0 0
errors: No known data errors
This local copy ran at exactly the same speed, around 30MB/s. For reference, sas72pool1 can sustain full 10Gb write speeds (dd 8k writes) from an NFS mounted VMware datastore.
Can anyone help me understand why this is happening?
Cheers!
Mark
Last edited:
