Hello everyone,
I have a TrueNAS SCALE system based on an HP DL380 G9 with 256GB ECC RAM, 20 cores 40 threads, and a ConnectX-3 40Gbit DAC 0.5M connected to a Juniper QFX5100 switch.
I never exceed 8 Gbit/sec on any of the pools. On the Micron pool, which is a 9200 Pro Micron that's supposed to handle 840k read IOPS / 180k write IOPS, I only get 105k IOPS with fio and exactly the same benchmark on the dc1000m pool. The ssd-san pool consists of a 3x mirror SATA Kingston DC550M 3.84TB SSDs. I am running an MTU size of 8900 all the way from TrueNAS > Juniper > VMware ESXi, solely on layer 2.
I have 3 SSD storage pools:
nvme_dc1000 3.48T 50.0G 3.44T - - 0% 1% 1.00x ONLINE /mnt
nvme_micron 6.97T 82.8G 6.89T - - 0% 1% 1.00x ONLINE /mnt
ssd-san 10.5T 1.76T 8.70T - - 38% 16% 1.00x ONLINE /mnt
Pool: ssd-san
State: ONLINE
Scan: scrub repaired 0B in 00:50:07 with 0 errors on Sun Jan 28 00:50:09 2024
Config:
Pool: nvme_micron (Micron 9200)
State: ONLINE
Scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Jan 28 00:00:02 2024
Config:
Pool: nvme_dc1000
State: ONLINE
Scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Jan 28 00:00:02 2024
Config:
I have a TrueNAS SCALE system based on an HP DL380 G9 with 256GB ECC RAM, 20 cores 40 threads, and a ConnectX-3 40Gbit DAC 0.5M connected to a Juniper QFX5100 switch.
I never exceed 8 Gbit/sec on any of the pools. On the Micron pool, which is a 9200 Pro Micron that's supposed to handle 840k read IOPS / 180k write IOPS, I only get 105k IOPS with fio and exactly the same benchmark on the dc1000m pool. The ssd-san pool consists of a 3x mirror SATA Kingston DC550M 3.84TB SSDs. I am running an MTU size of 8900 all the way from TrueNAS > Juniper > VMware ESXi, solely on layer 2.
I have 3 SSD storage pools:
nvme_dc1000 3.48T 50.0G 3.44T - - 0% 1% 1.00x ONLINE /mnt
nvme_micron 6.97T 82.8G 6.89T - - 0% 1% 1.00x ONLINE /mnt
ssd-san 10.5T 1.76T 8.70T - - 38% 16% 1.00x ONLINE /mnt
Pool: ssd-san
State: ONLINE
Scan: scrub repaired 0B in 00:50:07 with 0 errors on Sun Jan 28 00:50:09 2024
Config:
Code:
NAME STATE READ WRITE CKSUM ssd-san ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 183ad214-d779-4609-a361-85b0aa5e6b86 ONLINE 0 0 0 a5813287-f58b-43ba-9638-6b2c34e2d04e ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 77cdbc87-2c79-4ab7-995f-0686ff284dc0 ONLINE 0 0 0 9aef7328-2054-4a0c-bc2f-28a915e3dd76 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 d11584c1-418f-42fd-a2d0-ac4cd5549e96 ONLINE 0 0 0 56b95dd7-236d-4ff7-83ce-c3b11f92892e ONLINE 0 0 0
Pool: nvme_micron (Micron 9200)
State: ONLINE
Scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Jan 28 00:00:02 2024
Config:
Code:
NAME STATE READ WRITE CKSUM nvme_micron ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 26fd441b-b087-4815-b0b6-3638da107f06 ONLINE 0 0 0 a342e2b6-28c3-409f-98ab-99d5a416c6bd ONLINE 0 0 0
Pool: nvme_dc1000
State: ONLINE
Scan: scrub repaired 0B in 00:00:00 with 0 errors on Sun Jan 28 00:00:02 2024
Config:
Code:
NAME STATE READ WRITE CKSUM nvme_dc1000 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 fc4b9f5e-99d6-450e-a3c3-290bfcfe4b7d ONLINE 0 0 0 89e96c8b-9e14-4d1a-9e2b-4022a87e3741 ONLINE 0 0 0