TrueNas system really slow when doing certain operations over the network

[maxall4]

Hi, I'm a first-time TrueNas user and I'm trying to get my new TrueNas server to work at a reasonable speed I have my hardware specs below. My main issue is that certain operations are really slow such as running this command "git clone https://github.com/maxall41/SetupMyMac.git" on my TrueNas installation (while inside of the mounted volume on my MacBook) the "Updating files" section is unbelievably slow less than a percent of progress per second but if I ssh into my TrueNas server and do the same operation inside of the pool it takes less than 3 seconds where on my Macbook it takes about 5 minutes. One of the reasons this is really weird is because I'm not using a sketchy NIC or anything I'm using a pretty standard intel 82599EN based NIC and I have my MacBook wired to it through a Microtik 10GB switch (sure my Macbook does not have a 10GB NIC but it should still be faster than this) Does anybody know why this is happening?

Info about my pool:

I am using one of my 870 QVO 3's for my boot drive and the other one for storing data (I know this is not recommended but the data I'm storing is not especially important so its fine if I lose it)

Full hardware specs:
HYVE ZEUS base server
2 Samsung 870 QVO 3 SSDs
H!Fiber 10Gb NIC with Intel 82599EN chipset
2x XEON QUAD CORE E5-2609 2.4GHz
16GB Kingston RAM

 

[artlessknave]

it seems like the specs of the macbook matter here. if its particularly old or low powered....
"macbook" is a pretty large range of hardware.

this NAS would outperform even some of the newest macbooks. it certainly should give mine a run for its money (less ram though, but this operation shouldnt take a ton).

I have no idea what this thing you are cloning does though, or how IO or process intensive it is.

 

[maxall4]

I'm using a brand new MacBook with an M1 Max chip and 32GB of ram. This slowdown also occurs when deleting files it can take 30 seconds or more to delete a few files each only about 2 - 3 kilobytes on my MacBook but be instant via ssh.

 

[sretalla]

Seems to me that project is a large structure of many directories and tiny files... lots of IOPS.

If you're using sync writes... and I guess you are... you'll certainly be adding tens or even hundreds of milliseconds per operation, which add up over the numbers to many seconds.

if you set the dataset you're using to have sync disabled, does it go faster? (zfs set sync=disabled pool/dataset)

Are you concerned about file integrity? (if that's a worry for you, you would need to set sync back to default (sync=standard) ) and look at doing a SLOG (and/or adding a lot more RAM to your system) to mitigate that slowness in sync writes.

If you're not worried about integrity and can live with getting the files back again from elsewhere in the case of unexpected power loss, leave sync=disabled.

 

[awasb]

Could you please add some more detail, what "volumes" are mounted where from where via what protocol (afp/smb/nfs/webdav/iscsi)?

 

[maxall4]

Setting sync=disabled does not help. I have one pool using one of my SSDs all of its settings are the following:
sync=standard
compression=lz4
atime=on
deduplication=off
I'm accessing it over a Samba share with the default preset.
I think it's a network issue of some kind because everything works fine when doing these operations over ssh but I have no idea what that issue may be...

 

[sretalla]

Setting sync=disabled does not help.

Meaning you tried it and it made no difference?

The reason I suggested it is that I think MacOS and SMB ends up asking for Sync writes (which with sync=standard means writes will be sync).

Running a git command locally won't necessarily result in sync write requests (I suspect it doesn't).

You could stop that by setting sync=disabled for that dataset and testing it (unless you actually did that already).

You could do a little forum searching on iperf and checking your network throughput to see if that's a part of it somehow.

 

[maxall4]

Yeah, I tried it and it made no difference.

 

[sretalla]

OK, so with that theory biting the dust, onward to iperf...

 

[maxall4]

My network connection seems to be fine at least based on my Iperf tests:
[ 1] 0.00-10.01 sec 1072 MBytes 898 Mbits/sec
Which seems fine because that's the theoretical maximum speed of the NIC in my MacBook so I'm pretty sure it's not that.

 

[sretalla]

Then the next obvious thing to eliminate would be SMB... see if you can reproduce the issue with NFS

 

[maxall4]

It's pretty much the same slow speed when using an NFS share instead of an SMB share to access the pool.

 

[artlessknave]

M1 macs are ARM. I wouldn't expect much actual performance from one.
think, like, raspberry pi 4 ish? or flagship phone? I know they are getting good results, but if this is an intensive process over the network, it very possibly just doesn't have the gumption for it.
2x e5's and running locally is a whole different beast. comparing them and expecting the same results doesnt seem reasonable to me at all.

 

[maxall4]

I would not really say that the M1 chips are on the same level as the Broadcom chips used in raspberry pi they use an entirely different process node. So you can't really compare them. And the M1 Max chip is orders of magnitude more powerful than a raspberry pi 4 even with Rosseta. Plus the same operations are just as slow on my 32 core Ryzen system so I don't really think the specific computer is the problem, nor the processor architecture on which it is based. And if I run top while running those operations the CPU nor RAM on my local machine is stressed at all.

 

[artlessknave]

32 core Ryzen system

I did not see that in the thread. that does change it then. ignore me.

 

[maxall4]

Just some extra information I would like to add is that. Sometimes the transfer starts fast 200/Mbps per second for the first few seconds of the transfer then slows to < 1/Mbps per second.

 

[maxall4]

I also just realized that the size of my cache is only 1GB could this be causing my problem?

 

[mervincm]

I think you are using only 1GB of cache because your TrueNAS has decided that's all it needs based on the data passed through it since you last rebooted it. Mine converts free ram to cache till it climbs up to 1/2 system memory.

 

[mervincm]

The QLC SSD is not a great choice for a NAS, their IOPS (and health!) can be crippled fairly easily ... that said, what happens when read or write very large files to/from your NAS, are you able to get close to the limit of your 1 gbe NIC? you can use the black magic mac app to test.