New to FreeNAS - 8x3tb in RAIDZ2 slow

[Lastb0isct]

Hi All,

I am new to FreeNAS, but am very knowledgeable about computers in general. I was curious if I could get some pointers regarding slowness on my NAS.

I build this NAS with half the HDDs first and was getting ~80-90MB/s write speeds. I considered this perfect and was happy with it. Now that I've added the remaining 4 drives I am getting write speeds in the ~60MB/s range with read speeds even lower, ~40MB/s. I have researched a little and think it might be due to not enough memory but want to see if anyone has suggestions on optimizing the system.

Here are the specs:

AMD A4-5300 3.4GHz Dual-Core Processor
Biostar Hi-Fi A85W ATX FM2 Motherboard
Corsair XMS3 16GB DDR3-1600 Memory
Seagate Barracuda 3TB 3.5" 7200RPM Internal Hard Drive x8
Aerocool VS-9 Mid-Tower Case
Corsair CX 430W 80 PLUS Bronze Certified ATX12V Power Supply
ICY DOCK MB154SP-B 4 in 3 SATA 6Gbps Hot-Swap x2

Let me know if there is anything else I can include that would help you out.

Here are some pictures of the build: http://imgur.com/a/VptxR

Here are pictures of system performance near the end of writing 2TB to the NAS: http://imgur.com/a/7UPSq

 

[cyberjock]

Did you enable autotune? I'd recommend it with your hardware since you do have a fairly healthy amount of RAM. If you tried to tune the system I'd remove them. More than likely they're hurting far more than they are helping. Very(and I mean VERY) few people get it right. The defaults seem to work extremely well for most people(I don't ever tweak my servers).

What's the configuration of your 8 drives? Are they a RAIDZ2? I'd think that 16GB would probably be okay for home use, but more may help.

 

[joeschmuck]

I agree with cyberjock, I had the same problem when upgrading from 8.3.0 to 8.3.1 and I had to enable autotune and reboot. I would not mess with the values that autotune creates. Also I enabled AIO in the CIFS configuration, which is now off by default.

 

[Lastb0isct]

Why did you enable AIO? I read that it can actually slow down CIFS speeds...

- - - Updated - - -

Thanks cyberjock! I haven't enable autotune yet, i'll do that when I get home. Also haven't messed with any other settings on the side so that shouldn't be an issue.

They are in a RAIDZ2 as you assumed. I also remember that I turned on encryption when I created this volume. That must be CPU/Memory intensive, could that slow down transfer speeds? When I get home i'm going to destroy the Volume and re-create it without encryption and with autotune on and see if that helps.

 

[joeschmuck]

AIO depends on what you're doing. I typically do large file transfers, AIO helps me out. I should say that I currently have no complaints.

Ouch, encryption. Yea, if you don't need it, don't use it.

 

[cyberjock]

Encryption can be a performance killer. See my thread for some benchmark values here. I'd appreciate it if you could add your CPU's benchmarks to the thread if they aren't already there. You will need to make a FreeBSD 9.1 boot CD or thumb drive though. However no installation is required. Just run it in LiveCD mode. The only CPUs we have right now are Intel's, so it would be nice to see an AMD in the mix!

Your CPU does support the AES-NI instruction set. Not sure if AMD's implementation is supported on FreeBSD. But that CPU has only 2 cores, so you are going to be running pretty thin on CPU resources when doing stuff. Samba + ZFS + Encryption = lots of CPU horsepower needed. Based on the benchmarks provided so far on the thread I linked above its really sounding like unless a given server has at least 8 cores encryption isn't a very good option if decent performance matters at all.

 

[Lastb0isct]

I do large file transfers as well. Does AIO help for streaming content at all, over SMB? I'll have to take a look at this...thanks!

- - - Updated - - -

Thanks for the info cyberjock, i'll run some benchmarks when I get the time. Definitely want to do some quick tests with a non-AES file system with RAIDZ2 using all 8 disks. Encryption isn't needed for me at all, I just wanted to give it a shot because I read somewhere that it didn't take up a lot of CPU. It makes sense that it would though...no idea where I read that.

 

[Lastb0isct]

Hi All,

Ran some tests last night with encryption turned off on the Volume and here are the results. It looks like speeds have improved, but I'm not seeing great read speeds still. Is there anything I can do to improve them?

I'm going to try running the FreeBSD tests that you suggested cyberjock. I'll add to that thread.

 

[joeschmuck]

Did you turn on Autotune and reboot?

- - - Updated - - -

Nevermind, I just checked your link.

- - - Updated - - -

Something I recall hearing so take this at face value, if you turned off encryption, that doesn't change the old files on the drive to not-encrypted. You should write new files to the NAS and then try to read those back. You might have already done that.

 

[Lastb0isct]

I destroyed the Volume and rebuilt it with no encryption. So there are no encrypted bits on the drives at all

 

[cyberjock]

The only way to get rid of encryption is to destroy the zpool and recreate it. At least, that's what I understand from how geli is implemented in FreeNAS.

I looked at those snapshots. smbd on the second top screenshot is at 100% CPU usage, so that seems to be your bottleneck. I'm not sure why it's at 100%, but your CPU seems to be incapable of handling the load. My first thought on that is if you dropped in a CPU with more cores you might see a significant performance increase. Being that smbd is at 100% I'd say you need to decrease your CPU load(either by disabling features, tweaks, or both) or increase the capacity of your CPU. Samba is single threaded, so going from a 3.0Ghz CPU to something like a 2.5Ghz CPU but with more cores may not help. It's odd that you are at 100%, I wouldn't expect that with your hardware. I'm kind of convinced its the encryption(but I wouldn't expect the encryption to cause smbd loading to go sky high).

 

[Lastb0isct]

That second reading is actually when you press "m" when top is running. It shows more details on memory usage and I think it's just 100% of all active memory being used it being dedicated to smb. I've tried to change around the volume a bit and even with stripe I'm getting bad reads at only ~50MB/s. I'm going to install one of the extra gigabit adapters I have and see if that makes a difference. Will report back later in the day after work.

 

[cyberjock]

That second reading is actually when you press "m" when top is running. It shows more details on memory usage and I think it's just 100% of all active memory being used it being dedicated to smb.

You are right. I was wondering how you were getting 100% with a single threaded app. The screenshot is a small for me so I could barely read it. :P

50MB/sec from stripes tells me something is wrong. I'd make each disk its own zpool and do dd write and read tests to them. My guess is one of them(or perhaps more) will show abnormally low results. Those should be pulled from the system and tested further.

Wait, is the 50MB/sec over a file share? If so what is your source? If you are copying from your workstation to/from the server your 50MB/sec could be the limit of what the hardware on your workstation can do.

 

[Lastb0isct]

Thanks for the response cyberjock. Haha, I was scared when you noticed that at first but then remembered I added that memory one in there. It's definitely possible that I have a hard drive that is acting up, they're all new drives but you never know. I'll try separate ZPools for each disk. I also have old cables in the mix which might be causing issues. I am replacing them today so I will test after that as well to see if there is a difference.

Can you point me in a direction where I can find more information on dd write/read tests? I haven't done those before, although i might have and forgot how to do it...theres too much stuff up in my head right now!

This is 50MB/sec over CIFS/SMB to my Windows 7 client. I am copying to the same drive that, from another local drive, is able to write at up to 100+MB/s. I am always using the same drive to send/receive content from/to the NAS. When sending to the NAS I'm getting ~80-90MB/s, but when receiving content it barely gets to 45MB/s.

Only a few more hours at work and then I'll be doing some testing when I get home. Thanks!

On a side note, do you know of any way to make the LED blink specifically on one of my hot-swap bays to signify which drive it is? I want to be able to know which hard drive i'm pulling out. Does FreeNAS have the ability to do this? That would be amazing, if so.

 

[cyberjock]

On a side note, do you know of any way to make the LED blink specifically on one of my hot-swap bays to signify which drive it is? I want to be able to know which hard drive i'm pulling out. Does FreeNAS have the ability to do this? That would be amazing, if so.

That will be a feature based on the model and brand of your controller. Some won't have any support for that at all, others through the CLI or in the controller BIOS.

 

[Lastb0isct]

That will be a feature based on the model and brand of your controller. Some won't have any support for that at all, others through the CLI or in the controller BIOS.

I think my IcyDock supports it. Where in the CLI would I define this?

 

[Lastb0isct]

So, i created separate volumes for each drive and shared each of them out. They are all performing exactly as they do when in RAIDZ2. +100MB/s Writes but only 50/MB/s Reads. I did notice something when I booted it up, it was saying that all the HDDs were in 33.0MB/s mode which made me wonder. I went into the BIOS and saw a setting for SATA IDE something or other, i'll take a look again soon, and now I caught a picture of it after it booted up! Now it's reading them as 600MB/s. Doing some testing now to see if that made a difference.

- - - Updated - - -

The change I mention above didn't make a difference. It's still maxing out the reads from the NAS to 50MB/s with all these settings changed. Is there anyway for me to test the speed of the NAS on itself to see that it isn't the drives? I tried to follow the instructions here but there is no arcstat.py on my installation.

 

[Lastb0isct]

Any other ideas?!

 

[andoy31]

I think i read somewhere that location of the arcstat has been changed.. Anyway you can run arcstat in any location as it has been added on the path environment.

Try to search around for the dd command to test the speed of the disks and pool.

Also your windows pc might be using mcafee or symantec-- happened to me before, my writes on freenas are at 100MBps but my reads are only half of it. What i did is disabled all the other protocols and services and just leave the basic ie. tcp-ip and tcp-ip. After which reads and writes are both maxing my gigabit ethernet.

 

[Lastb0isct]

I think i read somewhere that location of the arcstat has been changed.. Anyway you can run arcstat in any location as it has been added on the path environment.

Try to search around for the dd command to test the speed of the disks and pool.

Also your windows pc might be using mcafee or symantec-- happened to me before, my writes on freenas are at 100MBps but my reads are only half of it. What i did is disabled all the other protocols and services and just leave the basic ie. tcp-ip and tcp-ip. After which reads and writes are both maxing my gigabit ethernet.

This is interesting...I'll have to check on that! I know I have AVG running which might be doing that. You just stopped all services relating to those programs and were able to receive at higher speeds again? Interesting...