ISCSI Mounted FreeNas drive copy speed stalls.

[Ozgur]

I have a Freenas Server FreeNAS-9.2.1.7-RELEASE-x64 (fdbe9a0) with NAS Disks of 3x4 TB Raid5 with 16gb memory running on
AMD A6-5400K APU with Radeon(tm) HD Graphics

I can reach upto 500mbit/s on the start of 30gb file until the 1st 16gb after that the speed reduces drasticly ,I have a video of the problem.Please help.

View: http://www.youtube.com/watch?v=DPPFf-Lc_4E&list=UUOtAymNNXjVn5AFxhyWcgLQ

 

[cyberjock]

Well, if you are actually running RAID5 you should look for an alternative solution to FreeNAS right now. ZFS does not take kindly to hardware RAID, and it's very possible your problem is directly attributable to using hardware RAID.

But, everything else aside the problem almost certainly stems from your zpool not being able to perform fast enough to keep up with your transfers. You fill the ARC with data that needs to be committed to the zpool and after the write buffer is full you're limited to the rate at which the buffer can be written to the pool.

tl;dr your hardware is either inadequate, your expectations are too high, you have a hardware failure, or your hardware is inappropriate. Choose 1 or more. ;)

 

[Ozgur]

Thanks,I am not using hardware raid,it is Raid5 with Freenas,I have also another system with 8.3 with Freenas Raid 1 (2 mirrors) with a 120gb SSD drive cache mounted .I am having the exact problem.I had connect both Freenas servers with crossover cable both sides 1gb ethernet but seems like having the same problem.Is there anyway I can limit the bandwith of the freenas server by configuration on the freenas machine?

 

[Ericloewe]

Thanks,I am not using hardware raid,it is Raid5 with Freenas,I have also another system with 8.3 with Freenas Raid 1 (2 mirrors) with a 120gb SSD drive cache mounted .I am having the exact problem.I had connect both Freenas servers with crossover cable both sides 1gb ethernet but seems like having the same problem.Is there anyway I can limit the bandwith of the freenas server by configuration on the freenas machine?

Please use proper terminology - RAIDZ1 if you're referring to the ZFS topology that has n drives plus one drive's worth of parity.

 

[HoneyBadger]

The first thing I'm seeing is that the speeds as initially shown by your Windows machine for that copy are peaking at 1GB/second. That's the realm of 10Gbps networking, not the 1Gbps crossover you say you are using. To me this indicates that your Windows client system is doing something odd with the writes like trying to cache them locally before committing. I believe some SSDs such as the Samsung 840 EVO do things like this with the "RAPID" software suite. You could easily be overrunning your pool limits as Cyberjock says but I'd expect that a 3-drive RAIDZ1 would have no issue sustaining GbE or near-GbE speeds.

Your 120GB SSD will also not help write speed, because unless I'm mistaken the Microsoft iSCSI initiator will only ever send async writes. If you are using a NIC with dedicated iSCSI offload in hardware it may be a different story but I can't say for certain.

So questions:

Do you have this same behavior if you try to copy to the FreeNAS system via CIFS/NFS instead of as an iSCSI mounted drive?

How about from a different OS like a Linux or BSD liveCD?

Are you able to try a different NIC? I notice you're using AMD which usually means "onboard Realtek NIC" with the motherboard.

Finally, are you using any features on your FreeNAS system such as compression (assuming yes as it defaults to lz4), encryption, or deduplication?

 

[cyberjock]

You could easily be overrunning your pool limits as Cyberjock says but I'd expect that a 3-drive RAIDZ1 would have no issue sustaining GbE or near-GbE speeds.

But he's clearly using 10Gb (or he has *much* bigger problems to troubleshoot). For the first 5 seconds or so he was doing over 500MB/sec and at one point over 1GB/sec! That's clearly not 1Gb. ;)

Three spindles are incapable of keeping up with that kind of transfer rate. This is a clear indication he fills his RAM with data to be written and as soon as there is no more RAM he's stuck transferring data only as fast as the pool can transfer and intermittent stalling.

I've done that a few times on my FreeNAS box when doing experiments with temporary pools. Fun to see 1GB/sec, but it's clearly unsustainable for the persistent storage to be able to keep up.

 

[HoneyBadger]

But he's clearly using 10Gb (or he has *much* bigger problems to troubleshoot). For the first 5 seconds or so he was doing over 500MB/sec and at one point over 1GB/sec! That's clearly not 1Gb. ;)

Agreed, but he stated himself in the third post:

I had connect both Freenas servers with crossover cable both sides 1gb ethernet but seems like having the same problem.

If he's actually using 10Gbps and I'm misreading, then yeah, he's stuffing the txgs and then the poor drives choke as they try in vain to keep up.

 

[Ozgur]

Thanks for all the comments,First of all I am not an professional and sorry for the wrong terminology ,I had tried the same with another freenas server with 8,2 and 120gb ssd cache ,I have the same problem.I have a 1gb switch connected to these machines.

Can somebody give me an idea how can I sustain a copy speed 50-70 MB/s?

Here is other trial.

View: http://www.youtube.com/watch?v=EpLFZpV5IuI&list=UUOtAymNNXjVn5AFxhyWcgLQ

 

[Ozgur]

The first thing I'm seeing is that the speeds as initially shown by your Windows machine for that copy are peaking at 1GB/second. That's the realm of 10Gbps networking, not the 1Gbps crossover you say you are using. To me this indicates that your Windows client system is doing something odd with the writes like trying to cache them locally before committing. I believe some SSDs such as the Samsung 840 EVO do things like this with the "RAPID" software suite. You could easily be overrunning your pool limits as Cyberjock says but I'd expect that a 3-drive RAIDZ1 would have no issue sustaining GbE or near-GbE speeds.

Your 120GB SSD will also not help write speed, because unless I'm mistaken the Microsoft iSCSI initiator will only ever send async writes. If you are using a NIC with dedicated iSCSI offload in hardware it may be a different story but I can't say for certain.

So questions:

Do you have this same behavior if you try to copy to the FreeNAS system via CIFS/NFS instead of as an iSCSI

I will try this.
How about from a different OS like a Linux or BSD liveCD?
Yes i have i will try with Ubuntu.
Are you able to try a different NIC? I notice you're using AMD which usually means "onboard Realtek NIC" with the motherboard.
Yes I had tried with pci-e NIC same issue.
Finally, are you using any features on your FreeNAS system such as compression (assuming yes as it defaults to lz4), encryption, or deduplication?

My answers on the quote.
I have no comp. No encryption no deduplication.
Thanks

 

[HoneyBadger]

Hi Ozgur,

You have a couple things amiss with your setup.

For your SSD:
In ZFS terms a "cache" device is used only for read caching. It will not help writes and may actually hurt your performance overall. You would need to remove it from the pool and re-add it as a device of type "log" if you want it to hold writes. But even in this case the Microsoft iSCSI initiator will not make use of it.

For your network setup:
Do not use LAGG link aggregation with iSCSI. You will not get the combined performance from both links. If you wish to use multiple links for iSCSI, you will need to set up each link with an individual IP address and leverage MPIO.

What I think:
At this point I believe there is something wrong with your client machine or it is lying to you about the initial copy speed. If you are only connected via a 1Gbps switch your absolute maximum data transmission rate is 128GB/s per link. Even if you had two functional iSCSI MPIO links running at the absolute maximum with no overhead that would be only 256GB/s. Your system is reporting 450MB/s in the latest test and was showing 1GB/s in the first. The bits should not be able to physically travel over the wires that fast.

Please try testing with an Ubuntu liveCD and/or with CIFS or NFS as well.

 

[bestboy]

@HoneyBadger: There's some mix up with the units in your previous post :)

Well, the throughput of the file transfer displayed by windows can not be achieved by the network. There is some magic happening. I'd assume that the iSCSI initiator is going wild on the asynchronous writes and tries to make good use of plentiful, free RAM.

Questions:
Does the free memory of the windows server change when the copy process starts?
How do the network and disk graphs in the reporting tab of FreeNAS look like during the file transfer?

 

[HoneyBadger]

@HoneyBadger: There's some mix up with the units in your previous post :)

OP has stated that he's connected via 1Gbps (gigaBIT) ethernet. But Windows shows speeds in GB (gigaBYTEs) and is showing numbers (450MB/s, 1GB/s) that are physically impossible to achieve over a single or dual gigabit network.

 

[bestboy]

Yes, but your units are still off.

If you are only connected via a 1Gbps switch your absolute maximum data transmission rate is 128MB/s per link. Even if you had two functional iSCSI MPIO links running at the absolute maximum with no overhead that would be only 256MB/s.

FTFY