Problem writing folders with a large numbers of files in FreeNAS 9.10

[slimknot]

I have been using FreeNAS for a little over a year now with a few glitches along the way but this new problem has me stumped. This fall I took the opportunity to upgrade my server from improper hardware to proper hardware and “upgraded” the version of FreeNAS from 9.3 to 9.10 at the same time. FreeNAS 9.10 is a new install on the new hardware. I was using a desktop motherboard with 16 GB non-ECC RAM but now I am using a Supermicro server motherboard with 64 GB of ECC RAM. See my signature for specific hardware.

I have about 300 folders with large numbers of files. Each folder has ~15,000 files to ~70,000 files. The folders contain a hodgepodge of file types like pdf, doc, xls, jpg, png, csv, txt, etc. The file sizes range from a few bytes to ~50 MB. When I try to copy a folder to the server, the copy is fast at the start but after copying ~12,000 of the files in the folder, the throughput becomes extremely (~500 to 900 KB/sec) slow. When it gets to ~20,000, files copied, the throughput will drop down to ~20 to 50 KB/sec. The worst part about it is the rest of the server file systems slow to a crawl when this happens. I never had this problem with FreeNAS 9.3. The throughput with large files is where I would expect it at 50 to 90 MB/sec.

I have tried using UNIX and Windows ACL’s but the problem occurs with both. I have also tried disabling DOS attributes in SMB and that didn’t help either.

I setup a test system to try to isolate the problem. I installed FreeNAS 9.10 to the test system and had the same problem. Then I switched to FreeNAS 9.3 with the same hardware and the throughput slowed but it only slowed to ~5 MB/sec. I tried to test the beta version of FreeNAS 10 but I couldn’t get the clunky, cave dweller, user interface of FreeNAS 10 to create an encrypted volume. The test system uses the old server hardware. The volume created to test the system is almost identical in that it uses 7 Seagate 8 TB Archive drives in a RaidZ2 configuration with encryption enabled.

Do I have a setting wrong in FreeNAS 9.10 or is there a problem with FreeNAS 9.10?

Thank you for your help.

 

[vibratingKWAX]

Your hdds are bad with random writes. They have some kind of write buffer of 20-30 GB afaik. Can you pause the transfer and resume after a few minutes to see if they have caught up?
Sorry for my briefness but german autocorrect is killing me on my smartphone

Gesendet von meinem Nexus 4 mit Tapatalk

 

[SweetAndLow]

Archive drives do not write data fast at all. And on top of that you are doing a super hard workflow of writing super wide directories.

 

[slimknot]

Your hdds are bad with random writes. They have some kind of write buffer of 20-30 GB afaik. Can you pause the transfer and resume after a few minutes to see if they have caught up?

Pausing the transfer and then resuming it makes no difference. The speed never increases.

 

[slimknot]

Archive drives do not write data fast at all. And on top of that you are doing a super hard workflow of writing super wide directories.

I understand that. However, why is the transfer speed ~100 times faster with FreeNAS 9.3 vs. 9.10? I have also used these drives with Windows and have never seen this decrease in write speed.

 

[SweetAndLow]

I understand that. However, why is the transfer speed ~100 times faster with FreeNAS 9.3 vs. 9.10? I have also used these drives with Windows and have never seen this decrease in write speed.

I don't think getting 5MB/s can be considered a speed improvement. I think it probably has to do with the fact that it was a new setup and you might have had a couple different services running. There is a thread discussing the issues with these drives, I suggest finding it and seeing if it helps.

Sent from my Nexus 5X using Tapatalk

 

[slimknot]

I don't think getting 5MB/s can be considered a speed improvement. I think it probably has to do with the fact that it was a new setup and you might have had a couple different services running. There is a thread discussing the issues with these drives, I suggest finding it and seeing if it helps.

I agree that 5 MB/s sis very slow but in my case it would reduce the data transfer to 4 days instead of 400. I should have also pointed out that the transfer was also attempted to my data pool with the 5 TB drives (Toshiba and Seagate) and I experienced the same issue. So, I don't think the Archive drives have anything to do with it in this instance.

 

[MrToddsFriends]

When I try to copy a folder to the server, the copy is fast at the start but after copying ~12,000 of the files in the folder, the throughput becomes extremely (~500 to 900 KB/sec) slow. When it gets to ~20,000, files copied, the throughput will drop down to ~20 to 50 KB/sec.

The symptom you're seeing is probably related to the Case Sensitivity setting in samba. The differences between 9.3 and 9.10 could be related to the samba version jump between these two FreeNAS versions.

https://bugs.freenas.org/issues/18264
https://forums.freenas.org/index.ph...to-compress-slow-transfers.46761/#post-320414

 

[SweetAndLow]

Can we try a test with iperf for the network and dd if=/dev/zero of=/mnt/pool/dataset/file bs=1M count=100000 on a dataset with no compression. These tests will eliminate any silly mistakes. Then you can start looking at small file and case sensitivity issues.

Sent from my Nexus 5X using Tapatalk

 

[slimknot]

The symptom you're seeing is probably related to the Case Sensitivity setting in samba. The differences between 9.3 and 9.10 could be related to the samba version jump between these two FreeNAS versions.

Thank you for your reply.

I agree it is probably due to the samba version unfortunately, changing the case sensitivity settings didn't make a difference.

 

[slimknot]

Can we try a test with iperf for the network and dd if=/dev/zero of=/mnt/pool/dataset/file bs=1M count=100000 on a dataset with no compression. These tests will eliminate any silly mistakes. Then you can start looking at small file and case sensitivity issues.

Thank you for your reply and help.

Here are the results of both tests.

Code:

[ ID] Interval  Transfer  Bandwidth
[200]  0.0- 1.0 sec  89.9 MBytes  89.9 MBytes/sec
[200]  1.0- 2.0 sec  90.7 MBytes  90.7 MBytes/sec
[200]  2.0- 3.0 sec  91.6 MBytes  91.6 MBytes/sec
[200]  3.0- 4.0 sec  90.5 MBytes  90.5 MBytes/sec
[200]  4.0- 5.0 sec  94.7 MBytes  94.7 MBytes/sec
[200]  5.0- 6.0 sec  95.0 MBytes  95.0 MBytes/sec
[200]  6.0- 7.0 sec  92.4 MBytes  92.4 MBytes/sec
[200]  7.0- 8.0 sec  95.0 MBytes  95.0 MBytes/sec
[200]  8.0- 9.0 sec  81.7 MBytes  81.7 MBytes/sec
[200]  9.0-10.0 sec  80.4 MBytes  80.4 MBytes/sec
[200]  0.0-10.0 sec  902 MBytes  90.2 MBytes/sec


100000+0 records in   
100000+0 records out   
104857600000 bytes transferred in 1635.085310 secs (64129743 bytes/sec) 

The dd transfer was done on the server with an external USB 3.0 drive so the results are probably low because of that. It is where I would expect them to be.

I also took the time to setup a single drive stripe on the test system with no compression or encryption. The results were essentially the same as before.

 

[SweetAndLow]

Well that was easy, your disk performance is terrible. It should be around 150MB/s for a single disk and much more if you have multiple disks.

Your network performance is also on the lower side. iperf should be around 110MB/s. Also why is it using bytes and not bits? I thought iperf always used bits.

I also find it strange that changing the case sensitivity like this post describes didn't affect performed. That seems almost impossible.
https://forums.freenas.org/index.php?posts/334318

Sent from my Nexus 5X using Tapatalk

 

[slimknot]

Well that was easy, your disk performance is terrible. It should be around 150MB/s for a single disk and much more if you have multiple disks.

Your network performance is also on the lower side. iperf should be around 110MB/s. Also why is it using bytes and not bits? I thought iperf always used bits.

I also find it strange that changing the case sensitivity like this post describes didn't affect performed. That seems almost impossible.
https://forums.freenas.org/index.php?posts/334318

The reason that iperf is in MB is because I added a command line switch to do that. I know my network is a bit slow and I don't know why. Over the years, everything has changed including switches, cables, and computers but it has always stayed low. Maybe it's just a Windows thing.

It turns out it was case sensitivity. I got it wrong the first time because I tried to follow the instructions in this link. http://marc.info/?l=samba&m=139336252926228&w=2 Now that it's right I have a speed of 35 MB/sec. Mark this one solved.

Thank you for all your help.

 

[SweetAndLow]

The reason that iperf is in MB is because I added a command line switch to do that. I know my network is a bit slow and I don't know why. Over the years, everything has changed including switches, cables, and computers but it has always stayed low. Maybe it's just a Windows thing.

It turns out it was case sensitivity. I got it wrong the first time because I tried to follow the instructions in this link. http://marc.info/?l=samba&m=139336252926228&w=2 Now that it's right I have a speed of 35 MB/sec. Mark this one solved.

Thank you for all your help.

Lol if you're OK with those speeds then go for it. They are still terrible and a single HDD is faster.

Sent from my Nexus 5X using Tapatalk