How to increase system performance

[Visseroth]

So I'm wondering what you guys would suggest to help increase my system performance.
My setup consists of the following...
Supermicro X7BN 2U server
dual 1.6Ghz quad core Xeon CPUs
11 Samsung F4 5400 RPM 2TB hard drives in one RAIDz1 array with 1 hot swap spare to make a total of 12 drives
8GB DDR2 667 of FB DIMMs
dual gigabit NICs teamed using LDAP
Large RW support is turned on in CIFS
Send files with sendfile(2) is turned on in CIFS
Enable AIO is turned off in CIFS

I have thought about adding a compact flash for logs and maybe something later down the pipe for caching but what do you guys think? What else could possibly make this thing faster?

 

[bubulein]

Moin,
I think it will be easier to answer if you provide Information about the current Performance.
-Christian

 

[Visseroth]

ok, how does this look....

dd if=/dev/zero of=ddfile bs=1024k count=20000
20000+0 records in
20000+0 records out
20971520000 bytes transferred in 49.431946 secs (424250344 bytes/sec)

dd if=/dev/zero of=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 55.064420 secs (380854280 bytes/sec)

dd if=/dev/random of=ddfile bs=2048k count=10000
^C9228+0 records in
9228+0 records out
19352518656 bytes transferred in 553.029120 secs (34993670 bytes/sec)

dd if=/dev/zero of=ddfile bs=1024k count=20k
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 48.935041 secs (438843742 bytes/sec)

dd if=/dev/zero of=/dev/null bs=1024k count=20k
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 3.001180 secs (7155464502 bytes/sec)

dd if=/dev/random of=/dev/null bs=1024k count=20k
^C4438+0 records in
4438+0 records out
4653580288 bytes transferred in 110.040658 secs (42289644 bytes/sec)

 

[bubulein]

ok, how does this look....

For me It looks like you successfully Benchmark your memory. If you use DD (which isnt a real benchmark) you should create files with atleast 150% better 250% of your Memory.
-Christian

 

[Visseroth]

Since I'm a bit of a linix newb what command would you suggest.

 

[survive]

Hi Visseroth,

A couple of quick thoughts:

I assume you are concerned about CIFS performance. FreeNAS uses Samba which is single threaded so I would think you are (if anything) limited by the speed of your processors.

I see you have your NICs configured for LACP. Remember that while that does "double" your total throughput you are still limited to gig-e speed to any one particular MAC\IP address. Gig-e tops out at 125MB\s (maximum) so any more performance out of your disks is really not needed.

You are running 11 disks in a single raidz, which I think exceeds the suggested maximum number of disks per vdev and you have a disk tagged as a hot spare which doesn't really work in FreeNAS right now. I think you would be better off splitting your disks into a pair of 6 disk raidz vdevs and then putting them both into a single pool. Doing this would give you the same amount of space, but could potentially double your disk throughput.

-Will

 

[Fornax]

Hi Visseroth,

What Bubulein is saying is that you should show us the results of tests that take a bit longer. Some of your tests transfer 20GB in under a minute, but a part of that is cached in RAM. That means the test does not give a clean benchmark of your disks, it is poluted. Increase the count=10000 by a factor of 10 so the test takes around 10 minutes, that will give you and us a better picture about how you diskarray is performing.

Another thing is you are asking for better performance. That is a good question. You stated your issue, now to state your issue more precisely : )
How do you use your system? How many users are there, what are they doing, is there a bottleneck? Right now we do not know if there is a problem. Also the tests with DD only run local on your server, it tells you and us nothing about your experiences over the network. Do you transfer lots of small files, do you transfer mostly large files, are you the only user, are there up to 5 users, are there 42 users all hammering the system when they enter the building in the morning?

2 other things I notice:
1) You have a RaidZ1 of 11 disks, 10+1. It is generally agreed that you get better results with 2+1, 4+1 or 8+1 because of how computers work (powers of 2 and sectorsize on disks, enough readingmaterial about that elsewhere in this forum). Maybe you can reconfigure your diskarray from (10+1 (+1 spare)) to (2 x (4+1 +1_spare)). You would loose some netto capacity, gain some performance, gain easier expandability (you can then add in chunks of 4+1).
2) You have a dual quad Xeon, that's plenty of cores but they are running at 1,6GHz each. If files are served via Samba (CIFS) then 1 client is served by 1 process, running on 1 physical CPU. We do not know how many people are using your server but faster CPU's might help. If there are only 2 users then 8 cores does nothing, instead you would be served better by fewer and faster cores.

To be able to help we need to know more about your situation.

With kind regards,
Fornax

 

[Visseroth]

So I didn't know that 6 disks is the maximum recommended pool. I didn't think there was a max, but with that in mind how would you put them both into a single pool? I have yet to see that option nor do I understand how you would do that besides putting some shares in one pool and other shares in the other pool.

I don't really have any issues atm as I am primarily the main user but I do use it for offsite backups for some of my clients and use it for backing up client data when a backup of a machine is required. So most of the time it does sit idle or has very low usage but when it is being used it is being maxed out and I figure why not try and optimize it for performance and try and get the most throughput in and out since when it is being used it is moving a good amount of data. We're talking 20GB all the way up to 6TB.
There are times I do have as many as 4 or 5 users but it is rare.

Well I don't know if I'll be expanding the array any time soon due to physical limited space in the system itself. I figured if I was to ever expand the array I'd get myself a rack mountable 48+ bay server but that is later down the road if needed. For now this suites me just fine as with the hotswap spare it is giving me 18TB of storage of which 8 is in use.

Now if I switched from 11+1 to 6x6 in a RAIDz2 would RAIDz2 cause any performance issues.

I also understand that CIFS is a single thread process but having 8 cores does mean that I can have 8 users or connections at one time and each core will handle each connection, correct?
Also understand these CPUs are NOT cheap. Last I checked 3Ghz CPUs ran about 600 to 1K each. I was fortunate enough to get this whole server minus the hard drives for 500 bucks and the hard drives for 69 bucks each for a grand total of about 1350. Not a bad deal if I say so myself. So with that if I can optimize what I have then I am all for it.

In regards to my dd tests, I ran a couple others and here are the results...

dd if=/dev/zero of=ddfile bs=1024k count=200k
204800+0 records in
204800+0 records out
214748364800 bytes transferred in 335.280828 secs (640502966 bytes/sec)

dd if=/dev/random of=/dev/null bs=1024k count=20k
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 507.155449 secs (42343697 bytes/sec)

dd if=/dev/zero of=ddfile bs=2048k count=200k
204800+0 records in
204800+0 records out
429496729600 bytes transferred in 759.200525 secs (565722382 bytes/sec)

dd if=/dev/random of=/dev/null bs=2048k count=20k
20480+0 records in
20480+0 records out
42949672960 bytes transferred in 1047.826464 secs (40989300 bytes/sec)

 

[survive]

Hi Visseroth,

Any RAID system is going to have a practical limit on the number of drives you can have per set. The problem gets down to how much reading and writing needs to be done from the physical disks to service each request. Sun provides the following guideline: "The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups" here:

http://www.solarisinternals.com/wik...onfiguration_Requirements_and_Recommendations

The nice thing about ZFS is that you can simply add a "vdev" (virtual device) to an existing pool. That's how you are able to make a monster ZFS pool out of 10's, 100's or even 1000's of disks. Let me step back and clear something up....FreeNAS does a good job of masking the vdev "layer" in the GUI, but when you created your 11 disk raidz you actually created an 11 disk raidz vdev & put ZFS on top of it. There's no reason you couldn't create a 6 disk raidz then add a second 6 disk raidz and add it to the pool as detailed here:

http://www.freenas.org/images/resources/freenas8.0.3/freenas8.0.3_guide.html

Look for section 6.3.4 "Adding to an Existing Volume".

Doing that basically makes your ZFS pool into something like a RAID 50 or 60 volume in the hardware raid world because you are striping your data over multiple RAID-5 or 6ish arrays.

It would be your choice whether you want to do raidz or raidz2 on each of the vdevs. Doing raidz will get you the same capacity, raidz2 will cost you 4 drives worth of capacity. Either way you have the added benefit of potentially increasing your performance-to-disk because there are 2 distinct vdevs the system can read or write to.

To simplify, if you did this you wouldn't have 2 separate pools, so you wouldn't have to worry about having to manage shares on 2 different pools. You would just have 2 vdevs underneath "the hood"...once you set your pool up if you never looked you would never know.

So lets say you decided to go with the 48 drive server case in the future. You move your board & drives over and add in some drives. You could add 6 more drives (ideally of the same size & performance class), make another vdev & add it to the pool. Want to add 12 drives? Split them into 2 more vdevs and add them. You want to have roughly the same performance characteristics among vdevs so you wouldn't want to (for example) add a mirror of SSD's to a ZFS pool made up of 3 6-disk raidz2 vdevs. ZFS will fully support you in such an attempt, but just because you can doesn't mean you should! If you think about it you can make some monstrous, ugly bastardized pools out of a mix of all manner of vdevs that will virtually assure you pain, suffering and fail, so don't get to fancy and ask here if "this is a bad idea".

Make sense so far?

Moving on to your CIFS question:

No, it means the exact opposite in fact. If you have 8 users connected in a purely CIFS environment then you will have 8 users grinding away on a single 1.6GHz core while the other 7 are sitting there running the rest of the system.

See this for more details: http://www.samba.org/samba/docs/man/Samba-Developers-Guide/architecture.html

Look for "Multithreading and Samba".

Fortunately socket 771 Xeon processors can be had on the used market (Ebay) for next to nothing....a quick search for "771 Xeon" there produced this gem as the first result:

http://www.ebay.com/itm/1-Matched-P..._EN_Networking_Components&hash=item5d3267a971

I suspect you would spend more for proper heatsinks for faster cpus than you would on the used CPU's themselves.

The server itself does seem like a good deal, but the fact you are here asking about "how to increase system performance" says to me that it might not be the ideal solution to your problem. I'm sure that if you put another hundred\couple of hundred bucks into it for faster CPU's & heatsinks (and maybe more RAM) it should perform just fine.

-Will

 

[Visseroth]

Well I didn't realize they had dropped in price. Last I checked these CPUs were pretty hard to find and expensive. I just found a 3Ghz CPU w/HT for 20 bucks each so I'll change out the CPUs and get the CPUs up to speed.
How much RAM do you recommend? I've been toying with the idea of putting in 4 more Gigs of RAM.

In regards to the rest of your message, thanks for all the info and the links. I've bookmarked them and read them. I didn't realize you could expand a pool with another pool or how to do it. So as soon as I have a place to offload my data I will make the appropriate adjustments to the pool.
In regards to samba, I understand now why it's not multithread capable.

I am wondering about the zfs logs option in the extras. Would having the system write logs to a compact flash be alright or do I have to be concerned about burning up the CF? Should logs be written to a SSD or hard drive instead?
I've done some searching in regards to ZFS logs but haven't found any solid answers in regards to the subject.

Again thank you for the input and taking the time to write so much. Your information is extremely valuable and appreciated!

 

[Visseroth]

nevermind, I just answered my own question by reading the ZFS best practices (which I am still reading)......

There are two possible ways to accelerate your ZFS pool through hybrid storage. By using "cache" devices, you may accelerate read operations. By using "log" devices, you may accelerate synchronous write operations.
per http://www.solarisinternals.com/wik...onfiguration_Requirements_and_Recommendations

 

[sjieke]

Hi Visseroth,
Moving on to your CIFS question:

No, it means the exact opposite in fact. If you have 8 users connected in a purely CIFS environment then you will have 8 users grinding away on a single 1.6GHz core while the other 7 are sitting there running the rest of the system.

See this for more details: http://www.samba.org/samba/docs/man/Samba-Developers-Guide/architecture.html

Look for "Multithreading and Samba"

I always thought that samba uses a seperate process for each connection (client). In the url you provided you can read the following in the section 'Multithreading and Samba'

The fact that we use a separate process for each connection is one of Samba's biggest advantages

Maybe I'm reading it wrong, but doesn't this mean that each user (connection) will run in a separate process and on a seperate core if possible?

EDIT: Just found the following on http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html

max smbd processes (G)
This parameter limits the maximum number of smbd(8) processes concurrently running on a system and is intended as a stopgap to prevent degrading service to clients in the event that the server has insufficient resources to handle more than this number of connections. Remember that under normal operating conditions, each user will have an smbd(8) associated with him or her to handle connections to all shares from a given host.

Default: max smbd processes = 0

Example: max smbd processes = 1000

 

[survive]

Hi Visseroth,

ZFS has 2 types of caches you can place on other (faster) drives. There is the ZIL (ZFS Intent Log) which contains "stuff that needs to be written" and the ARC (Adaptive replacement cache) which contains stuff that has been\will be read. So the ARC is read cache, I think we all kind of understand how that works....and the ZIL is write cache. We all know the problem with hard drives is they are slow, and a lot of things out there really want to get acknowledgement back from the storage that the writes they just made were successfully committed to stable storage. What ZFS does it it allows you to put something faster (an SSD or Fusion I/O card) between the network interface(s) and your ZFS pool so your writes go to the (faster, lower latency) ZIL & get acknowledged nearly instantly and then ZFS can move this new data from the ZIL to the disks later when there is less load or it can combine a bunch of writes into one big sequential write to disk.

I would not put the ZIL on compact flash (camera media?) because that stuff is just dog slow. same thing with putting it on disk drives. Ideally the ZIL should be on SLC (enterprise class, the expensive ones) SSD because SLC flash can take many more writes than cheaper (consumer) MLC drives.

If you can watch these videos...they explain the guts of ZFS really well:

https://blogs.oracle.com/video/entry/becoming_a_zfs_ninja

-Will

 

[Visseroth]

Well thought I'd report back with my results.
I changed my CPUs to dual dual core Xeon 3.2Ghz w/Hyperthreading CPUs and changed my storage pools from 11 hard drives with 1 hotswap drive to 2 pools of 6 drives in a raidz2 for high redundancy which would allow me a total of 4 failed drives, 2 from each pool without data loss. It dropped my total storage from 18 TB to almost 15 but I shouldn't have to worry about data loss.
Anyhow, in regards to how the changes effected my performance is as follows.
Faster file population
Faster file transfer
Faster indexing when requesting directory size
Over all my server is just more responsive and seems to take what ever I can dish out to it at the full speed of the network.
So thank you guys for your input, it was most effective.