daftu
Cadet
- Joined
- Mar 31, 2021
- Messages
- 7
Hi,
dear community, I've problem with problem in achieving satisfactory speeds between TrueNAS and ESXi using FibreChannel.
My setup:
- HP dl380e G8 with Qlogic ISP 2532 PCI FC-AL Adapter (single port 8Gbit) inserted into PCIe 3.0 8x running TrueNAS
- Dell R630 with Qlogic ISP 2532 PCI FC-AL Adapter (single port 8Gbit) running VMware ESXi 7.0 U1
TN shares two zvol's thru FC which ESXi consumes as a two datastores.
Negotiated speed is 8Gbit:
LED indicator on both FC cards are yellow (which means 8Gbit).
Locally tested on TN zpool write speeds are about 350MB/s and read speeds are about 400-500MB/s (non-SSD drives).
From ESXi when reading or writing from single datastore the speeds are quite fine:
The problem is when I read from one datastore and write to another simultaneously. Both datastore have been tested above for single write/read.
The speed never exceeded 150MB/s and sometimes is about 50-60MB/s.
I've replacted the FC patchcord - without any luck.
Is this a normal behavior? Theoretical 8Gbit FC speed is 1GB/sec. I don't know how the duplex works in this case, but let's say it is 500MB/s one direction. Why it is actually so slow balancing about 100MB/s?
Additional FC settings on TN:
dear community, I've problem with problem in achieving satisfactory speeds between TrueNAS and ESXi using FibreChannel.
My setup:
- HP dl380e G8 with Qlogic ISP 2532 PCI FC-AL Adapter (single port 8Gbit) inserted into PCIe 3.0 8x running TrueNAS
- Dell R630 with Qlogic ISP 2532 PCI FC-AL Adapter (single port 8Gbit) running VMware ESXi 7.0 U1
TN shares two zvol's thru FC which ESXi consumes as a two datastores.
Negotiated speed is 8Gbit:
Code:
[root@esxi:~] esxcli storage san fc list Adapter: vmhba3 Port ID: 0000E8 Node Name: 20:00:00:24:ff:10:3f:23 Port Name: 21:00:00:24:ff:10:3f:23 Speed: 8 Gbps Port Type: PTP Port State: ONLINE Model Description: PCI-Express Single Channel 8Gb Fibre Channel HBA Hardware Version: PX2810403-78 B OptionROM Version: 3.31 Firmware Version: 8.08.206 (90d5) Driver Name: qlnativefc DriverVersion: 4.0.3.0
LED indicator on both FC cards are yellow (which means 8Gbit).
Locally tested on TN zpool write speeds are about 350MB/s and read speeds are about 400-500MB/s (non-SSD drives).
From ESXi when reading or writing from single datastore the speeds are quite fine:
Code:
#reading from datastore dd if=/ds1/largefile of=/dev/null bs=1M count=4096 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.3047 s, 380 MB/s #writing to other datastore dd if=/dev/zero of=/ds2/largefile bs=1M count=4096 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 14.2199 s, 302 MB/s
The problem is when I read from one datastore and write to another simultaneously. Both datastore have been tested above for single write/read.
The speed never exceeded 150MB/s and sometimes is about 50-60MB/s.
Code:
dd if=/ds1/largefile of=/ds2/largefile bs=1M count=4096 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 30.5031 s, 110 MB/s
I've replacted the FC patchcord - without any luck.
Is this a normal behavior? Theoretical 8Gbit FC speed is 1GB/sec. I don't know how the duplex works in this case, but let's say it is 500MB/s one direction. Why it is actually so slow balancing about 100MB/s?
Additional FC settings on TN:
Code:
# Tunables ctl_load YES ctladm port -o on -t fc hint.isp.0.fullduplex 1 hint.isp.0.role 0 hint.isp.0.topology nport-only ispfw_load YES # Task (init command) ctladm port -o on -t fc
