CIFS Upload speed dips

[Fahad Chaudhri]

Hello,

Recently I have started experiencing a strange issue with CIFS upload speeds. Whenever, I am uploading a file to FreeNAS, the upload speed dips down to either a full stop or becomes very slow, but then picks up to full speed and does the same thing again. Please see the attached screenshot. The strange thing is that all of the shares are created from one volume (3 x 2tb SAS with zvol) and I see this issue on all of the shares except one.

I have done iperf test and that didn't show any issues. I have attached the log of that as well.

I have tried uploads from different machines (windows 10 and windows 2012 R2). I have tried uploads with and without Anti-Virus software. The drives are Seagate 2 TB SAS drives in zvol. I am using a LSI SAS card.

The drives show as healthy under volume. No power saving or spin down applied to these drives.

Any help would be appreciated.

 

[SweetAndLow]

You should post your hardware specs of your server and your client machine. Also post your freenas version. Are you using hardwired connections or are you using wireless.

 

[Fahad Chaudhri]

You should post your hardware specs of your server and your client machine. Also post your freenas version. Are you using hardwired connections or are you using wireless.

Thanks for the quick reply. Sorry for not posting the specs. I have the specs of my freenas server in my signature. That's why I didn't write them in the post. Please see below:

OS: FreeNAS-9.3-STABLE-201601181840 >>> on a 30 GB SSD drive.
CPU: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
Motherboard: Asus P9D-E/4L
RAM: 32 GB ECC RAM
SAS Card: Supermicro AOC-USAS2-L8E SAS Adapter
4 x 1 GB Intel NICs (onboard)

Volumes:
3 x 2 TB Seagate 6Gb/s SAS drives (raidz1)
4 x 500 GB WD black SATA drives (raidz1)
3 x 1 TB WD black SATA drives (raidz1)

Everything is hardwired. One machine is a VM on a hyper v server and other machine I tested from is my desktop. Everything connects to TPLink 16 port 1 GB switch. The desktop machine run windows 10 on SSD and has 6 GB ram with core i7. The VM has 4 GB of RAM.

 

[SweetAndLow]

Volumes:
3 x 2 TB Seagate 6Gb/s 128 MB Cache SAS drives (zvol)
4 x 500 GB WD black SATA drives (zvol)
3 x 1 TB WD black SATA drives (zvol)

This is confusing? you say zvol so you are using iscsi? You also mention a 128MB cache, what is this? Ideally you don't have any cache between zfs and the disks.

 

[Fahad Chaudhri]

This is confusing? you say zvol so you are using iscsi? You also mention a 128MB cache, what is this? Ideally you don't have any cache between zfs and the disks.

I am sorry, I meant to write RAIDZ1 instead of zvol. I am using iscsi, however the iscsi luns come out of 500 gb RAIDz1 volume and that traffic goes through a separate nic and a separate switch.
The 128 MB cache is cache on the drive itself. I believe also referred to as buffer

 

[SweetAndLow]

when doing a local write to your pool do you get good speeds? try doing a write using dd to a tmp dataset that has compression turned off. Make sure to use if=/dev/zero and that you write more than you have ram.

 

[Fahad Chaudhri]

The results are below. I wrote more than 32 GB which is the amount of RAM in my system.
I am not too familiar with dd. Please let me know if I should run a different command.

Code:

[root@blackbird] ~# dd if=/dev/zero of=/mnt/UserDataSAS/Faisal/ddfile bs=1024k count=40000
40000+0 records in
40000+0 records out
41943040000 bytes transferred in 13.969284 secs (3002518943 bytes/sec)
[root@blackbird] ~# dd if=/dev/zero of=/mnt/UserDataSAS/AppStorage/ddfile bs=1024k count=50000
50000+0 records in
50000+0 records out
52428800000 bytes transferred in 17.755703 secs (2952786498 bytes/sec)

 

[Fahad Chaudhri]

The result below is with compression turned off. I missed this part earlier.

Code:

[root@blackbird] ~# dd if=/dev/zero of=/mnt/UserDataSAS/Test/ddfile bs=1024k count=50000
50000+0 records in
50000+0 records out
52428800000 bytes transferred in 287.381356 secs (182436330 bytes/sec)

 

[SweetAndLow]

That is really slow write speed. You are only writing at the speed of about 1 drive.

SAS Card: Supermicro AOC-USAS2-L8E SAS Adapter

This thing seems suspects to me. Do you know if it works correctly and exposed just the raw drives to freenas.

 

[Fahad Chaudhri]

That is really slow write speed. You are only writing at the speed of about 1 drive.

This thing seems suspects to me. Do you know if it works correctly and exposed just the raw drives to freenas.

I think it is working. When I first installed the card in the previous installation, the system reported that version of driver isn't compatible with version of firmware. So, I upgraded the card to v20 since freenas comes with version 20 for LSI cards. After that, the card had no issues and worked well. This was done on the older build. I rebuilt the NAS using server motherboard, processor and ecc RAM. I saw no error from motherboard or in Freenas regarding the card. The card showed the drives in GUI and I was import the volume just fine. Even then I didn't experience this issue. I did update Freenas once or twice after that and I believe I started seeing this issue after that. I did run smarttest commands and drives show healthy, online and report no errors.

I ran dd on my 1 TB SATA volumes (connected directly to the mobo). The same tests show twice as fast speed than the tests done on SAS volume.

 

[SweetAndLow]

Getting the double speed on the other pool sounds more like it. Do you get the network drop out when writing to the other pools over cifs.

 

[Fahad Chaudhri]

I don't see that issue when I upload to other pools over CIFS. While I was testing this, I created another share on the pool with this issue, however I didn't experience the problem on the new share. Another strange thing is that, to do the DD test, I had increased the size of one of the shares, and since then I haven't hit this issue on that share either. It is very strange. Though, it is consistent one share (Audio). I tried to change that share name (Audios) and increase the size of it, but that didn't help.