Truenas speed drop

[drewsun]

Hello all,
I am having issues with my Truenas setup, I have 2 Truenas systems, here are their following specs:

Build #1
Truenas Scale 22.02.1 on VMware ESXi 7.0 U3
8 cores of E5-2690 v2
16 GB RAM
2 VMXNET 3 network adapters
1 ASR-71605 6Gb/s SAS HBA card passthrough to Truenas VM
16 HGST Ultrastar He10 10TB 7200 RPM 4Kn SAS 12Gb/s 256MB Cache
2 NVMe Samsung PM963

2 pools-z2
8 HGST
1 NVMe as cache
____________________________________________________________________________________________
Build #2
Truenas Scale 22.02
E3-1240 v5
32 GB RAM
Dell T330
H730 with all drives in Non-Raid mode
4 HGST Ultrastar He10 10TB 7200 RPM 4Kn SAS 12Gb/s 256MB Cache
1 2x 10Gb SFP+ port Fiber NIC

_____________________________________________________________________________________________


Network information if needed:
TP link tl-sg3428x with everything connected via the 4x 10Gb SFP+ ports

_____________________________________________________________________________________________


Currently I have a windows 10 system (all firewall and antivirus is disabled which helped improve speeds but still slow) that I have a 2x 10Gb SFP+ (installed into the 8x slot) and an NVMe 512Gb (Installed into the 4x slot), I seem to be having some random issues with writing from windows to Truenas, starting at 800Mbps then it drops down to 200Mbps when transferring a 8GB file.

Here are my transfer rates:
TrueNAS VM -> Network -> T330 Truenas = about 200Mbps Write and about 300Mbps Read
Windows 10 NVMe -> Network -> T330 Truenas = starts at about 700Mbps then drops to 300Mbps Write and about 700Mbps Read
Windows 10 NVMe -> Direct connect Fiber -> T330 Truenas = starts at about 600Mbps then drops to 200Mbps Write and about 700Mbps Read
Windows 10 NVMe -> Network -> Truenas VM = starts at about 800Mbps then drops to 200Mbps Write and 600Mbps Read

Does anyone know what I am doing wrong? I have tried a couple of tunables but nothing seems to work.

 

[jgreco]

8 cores of E5-2690 v2

That's almost certainly too many cores

1 ASR-71605 6Gb/s SAS HBA card passthrough to Truenas VM

That is a RAID controller, you need to get rid of it. Replace it with a proper LSI HBA.

16 GB RAM

2 NVMe Samsung PM963

1 NVMe as cache

The smallest PM963 is 960GB. For CORE, you need a minimum of 96GB of ARC to properly support that, preferably 192GB (5:1 ratio). For SCALE, double that (i.e. 384GB RAM for 5:1 ratio). You will be starving the system of ARC, probably catastrophically.

H730 with all drives in Non-Raid mode

There is no such thing as "Non-Raid" mode. It's a RAID controller. It's driven by a RAID card driver. Has to go. Replace it with a proper LSI HBA.

1 2x 10Gb SFP+ port Fiber NIC

There are probably like fifty manufacturers of "Fiber NIC" cards. It'd be super helpful to know what you have. That said, please follow the guidance for 10 gig card selection found in the 10 Gig Networking Primer. This basically boils down to "Use Chelsio T520-CR or Intel X520, everything else is at your own risk".

 

[drewsun]

That's almost certainly too many cores

How many do you recommend?

That is a RAID controller, you need to get rid of it. Replace it with a proper LSI HBA.

I was thinking about replacing it with a LSI 9400-16i once I find one for cheap or from some surplus equipment

The smallest PM963 is 960GB. For CORE, you need a minimum of 96GB of ARC to properly support that, preferably 192GB (5:1 ratio). For SCALE, double that (i.e. 384GB RAM for 5:1 ratio). You will be starving the system of ARC, probably catastrophically.

I guess I don't understand how ARC and L2ARC works, I thought that if I had L2ARC then it would go there instead of the RAM Cache

There is no such thing as "Non-Raid" mode. It's a RAID controller. It's driven by a RAID card driver. Has to go. Replace it with a proper LSI HBA.

I was thinking about also replacing it with a LSI 9400-8i if I can pick up up for cheap

There are probably like fifty manufacturers of "Fiber NIC" cards. It'd be super helpful to know what you have. That said, please follow the guidance for 10 gig card selection found in the 10 Gig Networking Primer. This basically boils down to "Use Chelsio T520-CR or Intel X520, everything else is at your own risk".

Sorry, its a Supermicro aoc-stgn-i2s, I pulled them out of some surplus equipment that had 2 each in them.

 

[Whattteva]

Build #1
Truenas Scale 22.02.1 on VMware ESXi 7.0 U3
8 cores of E5-2690 v2
16 GB RAM
16 HGST Ultrastar He10 10TB 7200 RPM 4Kn SAS 12Gb/s 256MB Cache

You gave it 8 cores but only 16 GB RAM??!!
Also, you got 16x 10 TB drives and ONLY 16 GB RAM?
Something does not compute here.

Build #2
Truenas Scale 22.02
E3-1240 v5
32 GB RAM
Dell T330
H730 with all drives in Non-Raid mode
4 HGST Ultrastar He10 10TB 7200 RPM 4Kn SAS 12Gb/s 256MB Cache
1 2x 10Gb SFP+ port Fiber NIC

You gave your much smaller pool (a quarter the size of the first one) twice the RAM instead?!
Something REALLY does not compute.

 

[jgreco]

How many do you recommend?

Start with two. You can then either observe statistics to see if more would help, or just try four to see if it goes faster. If not, then very likely two or four is in the correct ballpark.

I was thinking about replacing it with a LSI 9400-16i once I find one for cheap or from some surplus equipment

Good choice.

I guess I don't understand how ARC and L2ARC works, I thought that if I had L2ARC then it would go there instead of the RAM Cache

The ARC collects statistics on individual blocks and their usage. When the ARC is full, it looks to see which blocks in memory have been useful BUT have been the least useful. Think of it like this: if a block is accessed once, that's merely expected, but if it has been accessed twice, then it has been useful. But still less useful than the block that has been accessed 500 times. Those "least useful" blocks are evicted ("written") to the L2ARC, where they can be fairly rapidly pulled back in if needed.

However, it takes some ARC to keep track of what is stuffed away in the L2ARC. As a rule of thumb, it usually isn't advantageous to add L2ARC until you have 64GB of ARC. Maybe less, maybe more. Rules of thumb are not precise. At 64GB of ARC, you have better odds of the system being able to calculate MRU/MFU values for a block that allow intelligent decisions to be made about L2ARC evictions.

It is normally recommended to try for approx. a 5:1 ratio for L2ARC-to-ARC, so if you have a 64GB ARC, you might try a 250GB L2ARC (because there aren't many 320GB SSD's). It usually isn't advantageous to exceed 10:1.

 

[drewsun]

You gave it 8 cores but only 16 GB RAM??!!
Also, you got 16x 10 TB drives and ONLY 16 GB RAM?
Something does not compute here.

Yea, I'm new to Truenas, by coworker told me I didn't need much RAM for it so I went with it, its a VM and my ESXI server only has a 96Gb's of RAM left, its almost time for an upgrade.

You gave your much smaller pool (a quarter the size of the first one) twice the RAM instead?!
Something REALLY does not compute.

It's what was in the T330 that I picked up on surplus for 50$, didn't see any reason to remove it.

 

[drewsun]

Start with two. You can then either observe statistics to see if more would help, or just try four to see if it goes faster. If not, then very likely two or four is in the correct ballpark.

Ok I will try with 2 then see how it goes, Thank you!

Good choice.

Good, I was hoping it was.

The ARC collects statistics on individual blocks and their usage. When the ARC is full, it looks to see which blocks in memory have been useful BUT have been the least useful. Think of it like this: if a block is accessed once, that's merely expected, but if it has been accessed twice, then it has been useful. But still less useful than the block that has been accessed 500 times. Those "least useful" blocks are evicted ("written") to the L2ARC, where they can be fairly rapidly pulled back in if needed.

However, it takes some ARC to keep track of what is stuffed away in the L2ARC. As a rule of thumb, it usually isn't advantageous to add L2ARC until you have 64GB of ARC. Maybe less, maybe more. Rules of thumb are not precise. At 64GB of ARC, you have better odds of the system being able to calculate MRU/MFU values for a block that allow intelligent decisions to be made about L2ARC evictions.

It is normally recommended to try for approx. a 5:1 ratio for L2ARC-to-ARC, so if you have a 64GB ARC, you might try a 250GB L2ARC (because there aren't many 320GB SSD's). It usually isn't advantageous to exceed 10:1.

Ahhhh, now I get it, so the ARC is like the indexer of the blocks of the storage to keep the system somewhat fast, ie, system is looking for this file which is on this block go here, correct?

So I found an article just 5 mins ago that they said you would need 1Gb of RAM for every 1TB of storage is that correct? So I would need to increase my VM Truenas RAM to at least 150GB's of RAM for Optimal performance?

 

[jgreco]

Ahhhh, now I get it, so the ARC is like the indexer of the blocks of the storage to keep the system somewhat fast, ie, system is looking for this file which is on this block go here, correct?

ARC is also primary cache. So when you fill ARC up with L2ARC index records, you actually damage the system's ability to effectively select good blocks to cache, because the ARC is being used for two things at once, and that second thing is robbing from the first. The design intent was "just give it lots more." Because Sun Microsystems was in the business of selling hardware, maybe.

So I found an article just 5 mins ago that they said you would need 1Gb of RAM for every 1TB of storage is that correct? So I would need to increase my VM Truenas RAM to at least 150GB's of RAM for Optimal performance?

You're fighting several battles. One is that you're (probably) used to NTFS or EXT3 or FFS. In these filesystems, metadata needs are minimal because you are only tracking a 1TB filesystem, or maybe a 10TB filesystem, or other low values of TB. For ZFS, the system is capable of tracking petabytes. A single rack can hold 20PB of storage, and when you try to find a free block on that to store new data, the system has to analyze a LOT of metadata on a LOT of disks to determine where the best place to store it is. Sure, there are optimizations, but ultimately it has to load at least some portions of that into memory, which means that you need to hold a nontrivial portion of maybe 20PB in memory. The 1GB/TB rule is based on some common metadata ratios along with performance observations, and works out REALLY well up until maybe 50-100TB of storage. Somewhere, probably before that point, it becomes a bit more malleable. You're NEVER going to be able to support a 20PB pool on 64GB of RAM. But a 256TB pool on 64GB? Maybe. If you don't mind it being a little sluggish. Or maybe a lot sluggish.

Another thing is, what are you doing with it? Archival file storage, for example, is very different than video surveillance or active VM storage, and the amount of memory needed to be practical will be different for each use case.

If you were storing active VM block data, but only had a total working set of 100GB, you might manage some amazing caching in 128GB of RAM and rarely access your pool. On the other hand, that same situation, but with 400GB working set, would see lots of pool I/O.

So the rule of thumb is mostly there to try to "fix" people who think that when they run ZFS, suddenly it's going to rain gold and rainbows, and life will be awesome on their little underpowered 8GB RAM system. The rule of thumb works very well for relatively busy departmental fileservers and things along those lines.

 

[Whattteva]

It's what was in the T330 that I picked up on surplus for 50$, didn't see any reason to remove it.

Definitely not saying that you should remove it. I'm just making a point (with hyperbole) that you have more RAM allocated on a system that actually needs it less and it's a bit ironic.

 

[drewsun]

ARC is also primary cache. So when you fill ARC up with L2ARC index records, you actually damage the system's ability to effectively select good blocks to cache, because the ARC is being used for two things at once, and that second thing is robbing from the first. The design intent was "just give it lots more." Because Sun Microsystems was in the business of selling hardware, maybe.



You're fighting several battles. One is that you're (probably) used to NTFS or EXT3 or FFS. In these filesystems, metadata needs are minimal because you are only tracking a 1TB filesystem, or maybe a 10TB filesystem, or other low values of TB. For ZFS, the system is capable of tracking petabytes. A single rack can hold 20PB of storage, and when you try to find a free block on that to store new data, the system has to analyze a LOT of metadata on a LOT of disks to determine where the best place to store it is. Sure, there are optimizations, but ultimately it has to load at least some portions of that into memory, which means that you need to hold a nontrivial portion of maybe 20PB in memory. The 1GB/TB rule is based on some common metadata ratios along with performance observations, and works out REALLY well up until maybe 50-100TB of storage. Somewhere, probably before that point, it becomes a bit more malleable. You're NEVER going to be able to support a 20PB pool on 64GB of RAM. But a 256TB pool on 64GB? Maybe. If you don't mind it being a little sluggish. Or maybe a lot sluggish.

Another thing is, what are you doing with it? Archival file storage, for example, is very different than video surveillance or active VM storage, and the amount of memory needed to be practical will be different for each use case.

If you were storing active VM block data, but only had a total working set of 100GB, you might manage some amazing caching in 128GB of RAM and rarely access your pool. On the other hand, that same situation, but with 400GB working set, would see lots of pool I/O.

So the rule of thumb is mostly there to try to "fix" people who think that when they run ZFS, suddenly it's going to rain gold and rainbows, and life will be awesome on their little underpowered 8GB RAM system. The rule of thumb works very well for relatively busy departmental fileservers and things along those lines.

Thank you for the info, I increased the RAM on the Truenas VM to at least 64GB's and it helped sustain transfers with starting out at 10Gbps then dropped down to 6 but that's better, I now have to purchase more RAM for both my ESXI and my T330 to support their needs.

Thank you for the support!!

 

[jgreco]

Thank you for the info, I increased the RAM on the Truenas VM to at least 64GB's and it helped sustain transfers with starting out at 10Gbps then dropped down to 6 but that's better, I now have to purchase more RAM for both my ESXI and my T330 to support their needs.

Thank you for the support!!

Yeah, sorry, ZFS is a resource pig. Give it resources and it can do amazing things. But it does come at the price of those resources. :-/