Not getting the expected performance out of my FreeNAS server and I don't know why.

[TremorAcePV]

Hi guys,

Let me preface this with the server specs:

AMD Opteron 6212 Interlagos 2.6GHz 8-core CPU
SUPERMICRO MBD-H8SGL-F-O ATX Server Motherboard
Wintec Server Series 64GB (4 x 16GB) ECC Registered DDR3 1600 Quad Channel
Four 4TB Western Digital RE 7200RPM Hard Drives in (what I assume to be) RAID 10
Kingston HyperX 3K 120GB SSD for L2ARC/ZIL (NOT INSTALLED YET)
Two Supermicro AOC-USAS2-L8E 8-Port SAS 6Gb/s Internal Raid Adapters (two 4TB drives attached to each)

So, imo, this thing should be pretty darn fast in terms of read/write speeds. Relative to normal storage systems and consumer grade setups similar to it.

Here's the thing, I'm getting 20-25MB/s copying from the RAID 10 to the same RAID 10 (moving files from one volume to another on the RAID).

I don't understand why it's so slow. I feel I would get faster speeds on a single drive by itself. I know it's in RAID 1, but RAID 0 should help that considerably. Note the "what I assume to be" RAID 10. The GUI isn't clear how it goes with setting up RAID volumes. It was a 2x2 square, so I assume each column represents a mirror and each period represents a stripe. Googling and searching FreeNAS' documentation didn't lead me to an explanation of this specific GUI.

I mean what else could it be?

I've destroyed the zDev along with the dummy data on it, and am now individually testing the HDDs for possible errors. This should take quite a bit.

Other information about the setup:
Deduplication is turned off.
Compression is turned off.
Scrubs are set to run weekly, but none were running while I was moving files.
The RAM sits at 50GB steadily at all times.
The CPU sits at 10-20% CPU usage while copying.
The Controllers are in IT mode (Not IR mode) by default.

Any help is appreciated.

Thanks,
Vitalius

P.S. I noted the 120GB SSD for the L2ARC and ZIL because I'm not sure how it will affect performance. Hopefully a lot.

 

[cyberjock]

You should get far more than 25MB/sec from a RAID10 setup(which is what you have).

I wouldn't try to add the L2ARC or ZIL because you'll have this hybrid solution that will sometimes mask and sometimes not mask the problem. Figure out the problem with your pool performance, THEN look at adding an L2ARC or ZIL. I'd have expected that worst case you'd see 70MB/sec, but probably something in the range of 150-200MB/sec.

 

[TremorAcePV]

You should get far more than 25MB/sec from a RAID10 setup(which is what you have).

I wouldn't try to add the L2ARC or ZIL because you'll have this hybrid solution that will sometimes mask and sometimes not mask the problem. Figure out the problem with your pool performance, THEN look at adding an L2ARC or ZIL. I'd have expected that worst case you'd see 70MB/sec, but probably something in the range of 150-200MB/sec.

Thank you for the fast response, and for giving me a target to look towards for this setup to run at.

So that is definitely a RAID 10 setup using the GUI? Awesome.

I'm going to try setting up the pool using the Shell and see if that does something to performance as a test, because I'm not sure what else could be at fault here other than the drives themselves and possibly the controllers set up as they are.

And so far, the SMART data on all 4 drives says they are fine. Two have passed an extended test using Speedfan without any errors (which took roughly 9-10 hours) and the other two are now running the same test and have yet to produce any errors.

 

[ser_rhaegar]

How are you moving the files between datasets? Have you tried moving the files via the console to rule out any network issue?

 

[TremorAcePV]

How are you moving the files between datasets? Have you tried moving the files via the console to rule out any network issue?

I only just considered that while going over Cyberjock's guide for the 3rd time to rule out any problems.

It was over a Samba share through a 24 port switch to my computer and back. I can't imagine the 2.6GHz CPU core would be a limitation, but maybe it is.

Once I have the zpool set back up after the HDD's are done testing (give it 5 hours), I'll do another test with my computer, then with the terminal to compare and knock out possibilities.

 

[TremorAcePV]

-snip-

Alright, here are my test results:

Copying files from a user computer on the network:
25MB/s average speed
Copying files from a Server connected to the same switch as the FreeNAS system:
45MB/s average speed
Copying files from the terminal in FreeNAS:
90MB/s average speed (Estimated, 5.3GB in under 1 minute, all a single contiguous file)

I find those results strange personally.

That's when I build the RAID 10 with the GUI. I'm going to try building the RAID 10 setup with the terminal now and see if that directly affects performance.

 

[cyberjock]

Copying files from the terminal will always be slightly less than 1/2 the speed you'd expect since oyu have to read AND write using the same disks.

But, those numbers sure look like something related to networking/sharing problems. Could be almost anything from network misconfiguration to insufficient RAM, slow CPU, etc.