Help with 10MBps Write Performance - New Pool - TrueNAS Core

[SSNas01]

Specs

• Chassis/MB
  • Dell r730 LFF
• CPU
  • 2x E5-2667 v3 @ 3.20GHz (4 threads assigned to TrueNAS Core)
• RAM
  • 8GB DDR4 assigned to VM, but can bump this as necessary.
• HDD's
  • Proxmox on Raid1 480GB NVME SM953
  • TrueNAS Core VM on Raid1 1TB NVME 2280
  • Pool = 8x 6TB 7.2k SAS 4K MG04SCA60EA in RAID-Z3
• Hard disk controllers
  • Perc H330 Mini Mono in HBA mode (NOT flashed to IT mode).
• Network cards
  • Intel I350 GbE

Issue
Running TrueNAS for the first time as a new homelab enjoyer so bear with me... I'm getting performance that starts in the 100+MBps realm and drops to a miserable 10 or less MBps after a few seconds (assuming this is after my RAM stops owning the transfer?). fio results to a 1% full Pool to follow. Hoping to use this for mass storage and getting 50MBps or higher would be great for my needs. Looking for general advice on things I may simply not understand about optimizing TrueNAS. The Pool is only 1% full and I can even re-format if it's in the cards. Much appreciated.

Code:

fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][0.2%][w=15.3MiB/s][w=15 IOPS][eta 10h:10m:21s]
testfile: (groupid=0, jobs=1): err= 0: pid=70940: Thu Jan 25 18:55:22 2024
  write: IOPS=2, BW=3030KiB/s (3103kB/s)(199MiB/67243msec); 0 zone resets
   bw (  KiB/s): min= 1950, max=27945, per=100.00%, avg=5518.86, stdev=5351.65,samples=69
   iops        : min=    1, max=   27, avg= 4.57, stdev= 5.38, samples=69
  cpu          : usr=0.04%, sys=0.26%, ctx=1617, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,199,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3030KiB/s (3103kB/s), 3030KiB/s-3030KiB/s (3103kB/s-3103kB/s), io=199MiB (209MB), run=67243-67243msec

 

[sretalla]

Perc H330 Mini Mono in HBA mode (NOT flashed to IT mode).

That's going to be a problem even if not related to the speed.

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

If you don't want problems, you need to use your HBA in IT mode and pass it through in its entirety to the TrueNAS VM.

"Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

Proxmox on Raid1 480GB NVME SM953

Don't know exactly what you mean by that... is it a mirrored pool? how are the disks being presented to TrueNAS?

8GB DDR4 assigned to VM, but can bump this as necessary.

That's certainly scraping the bottom of the barrel if you're expecting performance, but adding more RAM won't get you far if your pool can't keep up with the speeds you expect.

 

[SSNas01]

If you don't want problems, you need to use your HBA in IT mode and pass it through in its entirety to the TrueNAS VM.

Fair enough. Will flash to IT mode and rebuild. For context this page contains what was basically the process I followed to pass the disks through to TrueNAS, and am now planning to undo: https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

Don't know exactly what you mean by that... is it a mirrored pool? how are the disks being presented to TrueNAS?

Proxmox on Raid1 480GB NVME SM953 - Meaning Proxmox is installed on two SM953's in Raid1. TrueNAS has no knowledge of these.

That's certainly scraping the bottom of the barrel if you're expecting performance, but adding more RAM won't get you far if your pool can't keep up with the speeds you expect.

Yeah can and will bump from 8GB RAM to 16 or more with this rebuild.


Much appreciated, will be back with results.

 

[sretalla]

For context this page contains what was basically the process I followed to pass the disks through to TrueNAS, and am now planning to undo: https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

That process results in disks presented to ZFS which lie to ZFS about what's going on, eventually ending with a corrupt pool... you need to look at the virtualization resource I linked and re-think your setup.

 

[SSNas01]

Back with an update...

I increased ram to 24GB, destroyed the old Pool, swapped to HBA330, passed through directly to the TrueNAS Core vm, and built a new pool. fio results (below) show a 10X increase in speed to ~35MBps since my last post. Great stuff. I believe I should be getting more in the 100MBps range, but this is high enough that I feel I can figure out enhancements on my own. Much appreciated.

Code:

fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=30.9MiB/s][w=30 IOPS][eta 00m:00s]
testfile: (groupid=0, jobs=1): err= 0: pid=2368: Sun Feb  4 15:55:57 2024
  write: IOPS=35, BW=35.4MiB/s (37.1MB/s)(2130MiB/60156msec); 0 zone resets
   bw (  KiB/s): min= 7937, max=74290, per=99.98%, avg=36252.78, stdev=10313.40, samples=117
   iops        : min=    7, max=   72, avg=34.91, stdev=10.08, samples=117
  cpu          : usr=0.20%, sys=2.71%, ctx=17215, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2130,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=35.4MiB/s (37.1MB/s), 35.4MiB/s-35.4MiB/s (37.1MB/s-37.1MB/s), io=2130MiB (2233MB), run=60156-60156msec

 

[sretalla]

You're getting an extremely low number of IOPS there... I suspect that's holding back your throughput.

I can't really see what would be leading to that from what you've shared so far... can you elaborate a bit on how the disks and HBA are connected and any details about how you did the PCI passthrough of the HBA?

 

[HoneyBadger]

@SSNas01 I wonder if somehow your Dell HBA isn't enabling write cache on your HDDs.

Can you do camcontrol modepage daX -m 0x08 | grep WCE from inside TrueNAS, where daX is one of your HDD's?

Might also be possible just from camcontrol identify daX and look to see if it shows the write cache (WCE) line there. It should be enabled (TRUE/1)

 

[SSNas01]

can you elaborate a bit on how the disks and HBA are connected and any details about how you did the PCI passthrough of the HBA?

Yes, the pool I created is comprised of 8x MG04SCA60EA in RAID-Z2 (I changed from Z3 to Z2 this time around). These drives are 6TB 7.2k SAS 4K sector drives. They are filling the 8 LFF hotswap slots of my Dell r730. They are cabled using the standard backplane to an HBA330 (mini mono) which is an HBA with a dedicated slot on the motherboard.
PCI passthrough was accomplished by enabling Intel Virtualization/IOMMU within bios and on the Kernel commandline as described here for systemd boot, and then using Proxmox GUI to provide full access between the HBA and TrueNAS VM. I confirmed the VM is getting full access to the drives by checking SMART metadata for the drives from within TrueNAS.

I wonder if somehow your Dell HBA isn't enabling write cache on your HDDs.

Can you do camcontrol modepage daX -m 0x08 | grep WCE from inside TrueNAS, where daX is one of your HDD's?

Might also be possible just from camcontrol identify daX and look to see if it shows the write cache (WCE) line there. It should be enabled (TRUE/1)

Got a failure from the second command on each drive, but the first command appears to indicate cache is on for only one of the 8 drives in the pool. I take it I should go to the HBA330 bios setup and enable for all drives. Any concerns flipping that switch at this point after the pool has been created with some write cache on and off? I have zero problem trashing the pool again. It's not been used yet.

Code:

camcontrol modepage daX -m 0x08 | grep WCE

Code:

camcontrol modepage da0 -m 0x08 | grep WCE
WCE:  1
camcontrol modepage da1 -m 0x08 | grep WCE
WCE:  0
camcontrol modepage da2 -m 0x08 | grep WCE
WCE:  0
camcontrol modepage da3 -m 0x08 | grep WCE
WCE:  0
camcontrol modepage da4 -m 0x08 | grep WCE
WCE:  0
camcontrol modepage da5 -m 0x08 | grep WCE
WCE:  0
camcontrol modepage da6 -m 0x08 | grep WCE
WCE:  0
camcontrol modepage da7 -m 0x08 | grep WCE
WCE:  0

--

Code:

camcontrol identify daX

Code:

camcontrol identify da3
camcontrol: ATA ATA_IDENTIFY via pass_16 failed
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed

Much appreciated.

 

[HoneyBadger]

Got a failure from the second command on each drive, but the first command appears to indicate cache is on for only one of the 8 drives in the pool. I take it I should go to the HBA330 bios setup and enable for all drives. Any concerns flipping that switch at this point after the pool has been created with some write cache on and off? I have zero problem trashing the pool again. It's not been used yet.

You should actually be able to toggle it on from TrueNAS itself, but if you can set the defaults to WCE:1 in the HBA330 then that's easier as it should be set by default.

If it doesn't, then you should be able to enable with echo "WCE: 1" | camcontrol modepage daX -m 0x08 -e

 

[SSNas01]

You should actually be able to toggle it on from TrueNAS itself, but if you can set the defaults to WCE:1 in the HBA330 then that's easier as it should be set by default.

If it doesn't, then you should be able to enable with echo "WCE: 1" | camcontrol modepage daX -m 0x08 -e

I turned on write cache for the 7/8 drives with it off using the HBA330 mini bios settings and what do you know, we've got results! This looks good speed wise to me though I'm not knowledgeable about IOPS so please let me know if there's something noticeably wrong here. Otherwise thank you both very much for assistance :)

Code:

fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=133MiB/s][w=132 IOPS][eta 00m:00s]
testfile: (groupid=0, jobs=1): err= 0: pid=1608: Wed Feb  7 17:51:43 2024
  write: IOPS=138, BW=139MiB/s (146MB/s)(8357MiB/60131msec); 0 zone resets
   bw (  KiB/s): min=42666, max=439441, per=100.00%, avg=142765.60, stdev=41706.10, samples=117
   iops        : min=   41, max=  429, avg=138.95, stdev=40.75, samples=117
  cpu          : usr=0.54%, sys=10.77%, ctx=63461, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8357,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=139MiB/s (146MB/s), 139MiB/s-139MiB/s (146MB/s-146MB/s), io=8357MiB(8763MB), run=60131-60131msec