Truenas Core 13 L2ARC Drive to Help with Slower Transfers?

[winnielinnie]

Yes, I am checking things with file transfers through explorer.

A folder with 1000 1MB files transfers at 30-40 MB/s.

FROM the NAS is 13-18 MB/s

It's expected that many small files will be notably slower than a single large file.

To rule something out, can you change the dataset's recordsize to 1 MiB, then repeat the tests with new files? (Files must be new in order to be written with the new recordsize.)

 

[Jazz30-06]

It's expected that many small files will be notably slower than a single large file.

To rule something out, can you change the dataset's recordsize to 1 MiB, then repeat the tests with new files? (Files must be new in order to be written with the new recordsize.)

Tried this with new files and no real difference in performance.

 

[Constantin]

I had experienced really slow copy performance to and from my NAS even on 10GbE. The culprit was small file sizes. Copying, verifying, etc many small files to spinning rust is trouble unless you have a bunch of VDEVs to help spread the load.

I attacked the problem two ways: as much as I could, I consolidated small files into larger archives (ie zip files or sparse bundles on the Mac). Fewer, larger files = faster transfer speed AND faster verification for rsync or like packages.

Once I got rid of most small files, I added a special VDEV (sVDEV) to my pool. HOWEVER, a sVDEV is not the answer to all ills in the world and carries risks that could kill your pool. Do NOT lightly embark on a sVDEV, understand its benefits, settings, and downsides or you will be sorry. But it can significantly speed up small file transfers.

But the first thing is see if you can consolidate all those small files. Apple sparsebundles are especially helpful as you can mount the files as volumes, the archive can be made flexible re size, and crucially, unlike zip files the data is stored in “bands” that are big enough to hold a lot of data but also allow incremental updates that do not require the whole archive to be recopied, just the bands that changed.

There are downsides to sparsebundles (like no built-in bit-rot detection) but that is what I use ZFS to host them on for.

 

[Jazz30-06]

I had experienced really slow copy performance to and from my NAS even on 10GbE. The culprit was small file sizes. Copying, verifying, etc many small files to spinning rust is trouble unless you have a bunch of VDEVs to help spread the load.

I attacked the problem two ways: as much as I could, I consolidated small files into larger archives (ie zip files or sparse bundles on the Mac). Fewer, larger files = faster transfer speed AND faster verification for rsync or like packages.

Once I got rid of most small files, I added a special VDEV (sVDEV) to my pool. HOWEVER, a sVDEV is not the answer to all ills in the world and carries risks that could kill your pool. Do NOT lightly embark on a sVDEV, understand its benefits, settings, and downsides or you will be sorry. But it can significantly speed up small file transfers.

But the first thing is see if you can consolidate all those small files. Apple sparsebundles are especially helpful as you can mount the files as volumes, the archive can be made flexible re size, and crucially, unlike zip files the data is stored in “bands” that are big enough to hold a lot of data but also allow incremental updates that do not require the whole archive to be recopied, just the bands that changed.

There are downsides to sparsebundles (like no built-in bit-rot detection) but that is what I use ZFS to host them on for.

Well, that's unfortunate, but I appreciate the heads-up to know that I shouldn't keep pressing on the slower transfers. It's not the end of the world. I was just hoping to get more performance if I could.

 

[Davvo]

Well, that's unfortunate, but I appreciate the heads-up to know that I shouldn't keep pressing on the slower transfers. It's not the end of the world. I was just hoping to get more performance if I could.

Using mirror vdevs would probably help the reads.

 

[fcorbelli]

If I understand correctly, the performance is good, for large files, and not very high, for small ones
The question to ask is "how small" and "how many"
I agree on one fact: I wouldn't opt for a RAIDz but I would make a mirror between two disks, and I would leave the third one for internal backups
Also, since there is an SSD, THE real test is to make a dataset (on SSD) and share it

L2ARC, BTW, is a second-level READ cache, almost 100% useless for write bandwidth

 

[Davvo]

Also, since there is an SSD, THE real test is to make a dataset (on SSD) and share it

No since I believe it's the boot pool.

He is probably capped by the IOPS since performance drops when he shares small files.

 

[fcorbelli]

I am not a TrueNAS expert, but (at least as a test) I would consider a manually-created dataset (on the ssd) with manual smb4.conf share

Of course as a debug temporary measure,just to see
With some forum help I think it is doable

Or a temporary fresh install on usb stick, to "format" the ssd via gui

Just my 1 cent

 

[Constantin]

There are a bunch of settings you can manipulate / influence to make the SMB browsing experience a lot faster - see the forums for specific attribute settings, and so on.

On the hardware side, L2ARC can help, as long as your NAS has enough RAM to deal with the loss of RAM for L2ARC indices, it's set up as metadata-only and is made persistent. Once a L2ARC gets "hot", it can make a big difference re: browsing, rsync, and so on. But that is of no help re: writes to the NAS, actual data transfers from the NAS, etc. so the specific use case re: L2ARC is important. Like SLOGs, L2ARC caches are not panaceas but rather tools that have to be matched to a job.

If you have a lot of small files that have to be updated constantly (think some types of database) then SSD pools can make a lot of sense.

 

[Davvo]

@jgreco any ideas?

 

[fcorbelli]

I suggest... (not a TrueNAS user, maybe some easier way)

1) enable ssh [services => ssh]
2) PermitRootLogin [services => ssh => actions => login as root with password]
Enable SSH AND root login
3) download

https://the.earth.li/~sgtatham/putty/latest/w64/putty.exe

putty IP-of-TrueNAS, login as root and your password
4) vi /usr/local/etc/smb4.conf
5) add (yep, very quick, very dirty) (copy in clipboard from [test] to last yes)
esc i right-mouse (=paste) then esc :wq! (+return)
Create a (temporary) share into the SSD
Go on the last line with multiple pagedown keystrokes
Press ESC
Press i
Right mouse button
Press ESC
Press :wq! (+Return)
Short version: create, manually, a test folder pointed inside the /tmp folder of the boot (SSD) drive

Code:

[test]
path=/tmp
available=yes
read only = no
browseable = yes
public = yes
writeable = yes

6) service samba_server restart
Make the new share active
7) try to write into \\IP\test.
Call directly with \\192.168.186.186\test
This should write inside the SSD

Just to see what happens, my 1 cent

 

[Davvo]

There is no reason whatsoever to test his boot pool performance though?

 

[fcorbelli]

There is no reason whatsoever to test his boot pool performance though?

It is a SSD
If the performance on the SSD (with small files) is adequate I think it's a useful item: buy some other (SSDs)

 

[Davvo]

It is a SSD
If the performance on the SSD (with small files) is adequate I think it's a useful item: buy some other (SSDs)

There is no test to do, it's a fact that using SSDs will greatly improve performance.

 

[fcorbelli]

There is no test to do, it's a fact that using SSDs will greatly improve performance.

I agree, indeed
BUT
if the user get same weird performances with SSD TOO, then "something" else is wrong (network, configuration, demoniac possession, whatever), and maybe "fixable"
If with SSD everything is running as expected, he simply need faster drives (or wait longer)

 

[Davvo]

There is nothing that points at the system having issues. With large files he is able to perform as expected.
The issue is the IOPS limit during small fires transfer, and I personally see no reason to execute a (relatively) complex procedure to test the boot pool, especially since the opener doesn't look very familiar with FreeBSD/CLI.

 

[fcorbelli]

There is nothing that points at the system having issues. With large files he is able to perform as expected.
The issue is the IOPS limit during small fires transfer, and I personally see no reason to execute a (relatively) complex procedure to test the boot pool, especially since the opener doesn't look very familiar with FreeBSD/CLI.

A folder with 1000 1MB files transfers at 30-40 MB/s.

I agree this is a normal behavyour, BUT

FROM the NAS is 13-18 MB/s

... not so much (I expect equal or faster read performance, vs write)
I just wrote what I would suggest, to go from 99% (it should be) to 100% (it is).
Especially if the pool is empty, i.e. it still doesn't contain any important information, the time to test and trial is before production use

I reiterate though that I am not a TrueNAS power user by any means