Low write/read speed? What should I expect?

[hayer]

Hi,

so I have a FreeNAS-9.2.1.3-RELEASE-x64 (dc0c46b) box.

CPU: Intel(R) Core(TM) i3-3225 CPU @ 3.30GHz
Memory: 4x8GB SAMSUNG ECC RAM 1600MHz
IBM M1015, flashed IT mode
6x1TB WD RED (storage01)
6x3TB WD RED (storage02)

Questions;
1. I've never seen my box consume more than about 20GB of RAM. Why? It has 32GB and is only running FreeNAS and the Plex-plugin. How can I increase it to 30GB? To leave only 2GB for the rest of the system, I can't imagine that FreeBSD and Plex needs more than 2GB RAM?

2. When copying from storage01 to storage02 using CIFS I get 50-70MB/s. Is this to be expected? To stop it from copying storage01->my PC->storage02 I created a symlink in storage01 to storage02. I can see in Wireshark that the file isn't been sent through my PC.

3. I wanted to "re-run" the autotune-thingy. So I deleted the tunables in the webui; I get a green box at the top saying that it is deleted, but it still comes up in the list and if I reboot it doesn't change. This seems to be a bug? Anyhow I don't know what logs to include with a bug report.

4. When I installed the disks I didn't ensure that they where formatted using 4K sectors(Which is correct for the WD RED disks, as far as I can see). How can I check their sector size now?

5. The box is connected with 2x1gbit LACP. Is there any tunables I should set that affects this?


One of my problems is that I don't know what to expect, but at the moment I feel like it is not performing as good as it could, but that is just a feeling :p

Edit: forgot to add a link to my arc_summary report.
http://pastebin.com/Kk6DeBZc

Edit2:
Tried settings vfs.zfs.arc_max to 32625492387. Rebooted. Still no change after transferring 50GB between the volumes.

 

[vikingboy]

did you try any other methods of transferring data to rule out CIFS as the culprit? Could be an easy test. Your spindle counts should support > 70MB/s I would expect...
Not sure why you aren't seeing greater than 20GB GM RAM consumed...sometimes these are indicators of problems so might be worth trying to resolve that before worrying about the slow throughput.

 

[cyberjock]

Autotune will *always* limit the RAM used. Gee, you're asking about RAM usage.... that's not a coincidence.

You shouldn't need to set any tunables or sysctls to get a server that uses more than 20GB of RAM and performs very well.

To delete the autotune variables you have to disable autotune, reboot, delete the tunables/sysctls and reboot again.

 

[joeschmuck]

Let me see if I have this correct...

You have a FreeNAS machine with two vdevs (pools) and you are trying to transfer data from one to the other, and then you are using a separate computer to route that traffic. This sounds very good for a single 1gb ethernet port if you are transferring large files.

Just to be clear, you have an LACP switch and your PC is setup as LACP too? I mean, you can't transfer data twice as fast over bonded links if you only bond them on one end. I'm pretty sure I got that right.

Now if you do have them bonded properly, I would suspect you are transferring lots of small files.

If this is not the case, please post how you have LACP configured in your entire environment.

 

[hayer]

Autotune will *always* limit the RAM used. Gee, you're asking about RAM usage.... that's not a coincidence.
You shouldn't need to set any tunables or sysctls to get a server that uses more than 20GB of RAM and performs very well.
To delete the autotune variables you have to disable autotune, reboot, delete the tunables/sysctls and reboot again.

Thanks, that explains why I couldn't delete the tunables from the WebUI.

Let me see if I have this correct...

You have a FreeNAS machine with two vdevs (pools) and you are trying to transfer data from one to the other, and then you are using a separate computer to route that traffic. This sounds very good for a single 1gb ethernet port if you are transferring large files.

Just to be clear, you have an LACP switch and your PC is setup as LACP too? I mean, you can't transfer data twice as fast over bonded links if you only bond them on one end. I'm pretty sure I got that right.

Now if you do have them bonded properly, I would suspect you are transferring lots of small files.

If this is not the case, please post how you have LACP configured in your entire environment.

I thought #2 was pretty clear. It does NOT route through my PC. It is internal traffic going from pool1 to pool2. And I hoped that I would get better than 70MB/s when copying from one pool to another one.

I have LACP on the server. I have 1gbit cabled to each of my computers. When me and my brother both try to read at the same time we get like 10-20MB/s, haven't tested writing yet. When only one of us reads we get like 80MB/s.
Using iperf we can both saturate 1gbit to the server at the same time, so the LACP seems to be working. So I guess #5 isn't LACP related at all. Seems more like I need to tweak some settings with the CIFS?


Edit: When testing I always copy large files; like 2x 20GB files or 75GB files.

 

[joeschmuck]

How are you doing the copy operation. Pretend I'm a total idiot (sometime I am) and walk me through it. What commands are you using to copy the data between the two pools?

EDIT: #2 was not very clear, you said you were using CIFS which is an external protocol, not internal at all.

 

[joeschmuck]

Sorry, I had to look some stuff up because I'm not very familiar with LACP however I don't believe you have the proper hardware to support LACP, but I could be very wrong. Maybe if you explain your network setup and router to a little more detail. Do you have the two FreeNAS Ethernet cables connected to an LACP capapable router with LACP enabled?

As for CIFS, this is a single threaded application and I would assume your CPU is more than capable to handle the throughput so I wouldn't think this is the issue but just because it's single threaded, well someone will have to chime in to speculate more on that.

 

[solarisguy]

Two minor questions, that might or might not be relevant.
* Are your NICs made by Realtek ?
* What is the speed of the slot your IBM M1015 is in? 1x ? 2x ?