Poor write performance w/ many small files via SVN

[LoneRangir]

I'm new to FreeNAS and I'm having very poor performance writing to the NAS.

The setup includes:


FreeNAS-9.3-STABLE-201412142326

Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz

16082MB
6 x 800GB Crucial M500DC SSDs
RAIDZ2 configuration

I am checking out an SVN project to the array. This is all done under linux and NFS. The project has mostly small files, totaling about 1GB in size. A study of the directory makeup produced this histogram:
File Size
(KB).........count
1..............4244
2..............2128
4..............2562
8..............2305
16.............2477
32.............1547
64.............957
128............607
256............243
512............288
1024...........123
2048...........32
4096...........15
8192...........13
16384..........22
32768..........6
65536..........2


The checkout time is abysmal. It takes over 40 minutes. On another network directory on a different server it takes about 6 minutes or so. If I check out to a local hard drive on a linux workstation, it takes about 2 or 3 minutes.

Any insight would be appreciated.

 

[L]

Yea thats crazy with that ram and ssd only pool. How are the ssd's attached? Can you check raw performance from a client to the nfs share? What kind of network interface?

 

[LoneRangir]

I tried two ways: directly to the 6 SATA ports on the Intel motherboard, and a RAID card (in JBOD mode). The model escapes me at the moment. Performance was the same (poor).

 

[L]

Even with 1 ssd and 16GB ram... you should have pretty snappy performance... if you get to a command line can i see

dmesg | grep AHCI

I have recently seen those onboard sata controllers scanning the bus and hanging on like a cd drive during write.

and

sysclt -a | grep ashift

maybe force a ashift of 12 by putting in this sysctl

sysctl vfs.zfs.min_auto_ashift=12

Ashift is the page size used for a pool so the pool would need to be recreated.

 

[LoneRangir]

[root@mdcfs3a ~]# dmesg | grep AHCI
ahci0: <Intel Cougar Point AHCI SATA controller> port 0x5090-0x5097,0x5080-0x508
3,0x5070-0x5077,0x5060-0x5063,0x5020-0x503f mem 0xc2840000-0xc28407ff irq 21 at
device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0x5090-0x5097,0x5080-0x508
3,0x5070-0x5077,0x5060-0x5063,0x5020-0x503f mem 0xc2840000-0xc28407ff irq 21 at
device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0


[root@mdcfs3a ~]# sysctl -a | grep ashift
vfs.zfs.min_auto_ashift: 9
vfs.zfs.max_auto_ashift: 13
vfs.zfs.vdev.larger_ashift_minimal: 0

I did the change to 12 and kicked off another checkout. By the rate at which the messages were printing to the screen, I'd say this made no change.

 

[L]

Ill let someone else chime in.

 

[cyberjock]

I'm confused when you say 'checking out an SVN project to the array'. You mean you are using a linux workstation connected to an NFS share on a FreeNAS box and the SVN is stored on the zpool? Is that correct?

 

[LoneRangir]

I'm confused when you say 'checking out an SVN project to the array'. You mean you are using a linux workstation connected to an NFS share on a FreeNAS box and the SVN is stored on the zpool? Is that correct?

Yes. It's a network drive for multiple people to access. A few SVN workspaces exitst in different folders.

 

[LoneRangir]

Would anyone else be willing to mimic my test case? Create a large directory structure with "lots" of small files (< 32KB) totaling 1GB or so and compare the time to write the directory to FreeNAS and the time it takes to write the directory to a single hard drive not under FreeNAS control. I'd be curious if it's my hardware or if the overhead of FreeNAS is causing a significant slowdown.

 

[Apollo]

Checking out a working copy is usually a lengthy process, I believe it has to do with SVN recreating the .svn directory which for the most part is a replica of the working copy but includes checksum and other information.
Also I believe SVN is not multithreaded so 1 file one access at a time.
Can you use a ftp transfer tool such as filezilla and see if you can copy your working copy to a different location on your Freenas drive? You should achieve better performance then.

 

[SweetAndLow]

You are basically doing small file random io. This use case is the worst case scenario for a nas. My workflow has always been to checkout to my ssd that is dedicated for source then rsync to my NFS mount as my backup. I have no reason to store my source on a nas since it doesn't get shared with anyone.

Hope that helps you even though I didn't really give you a real answer.

 

[cyberjock]

Well, here's my comments, for what they are worth:

1. Having lots of small files uses lots of metadata space. Having only 16GB of RAM means your metadata space in the ARC is probably around 3-4GB, tops. Not a lot, and even more strained if you have lots of files.
2. RAIDZ2 is a poor choice for workloads like this. Mirrors are definitely better (which also would give you multiple vdevs).

I'm about to go to bed, but I might try creating a script tomorrow in Windows to create 32KB files just to see how long it takes on my server. I'm not using SSD though, but I do have 32GB of RAM. It will also be CIFS instead of NFS, so there will definitely be some differences.

Edit: Keep in mind that the two ways that we tell people to make their zpools faster are:

1. Add more RAM
2. Add more vdevs

RAM is always the easiest as it's usually a "drop-in". So RAM is usually the first thing to do unless you are doing something where it's 100% obvious that more vdevs are needed (such as trying to run a dozen VMs).

 

[LoneRangir]

I destroyed my RAIDZ2 array and tried both a 6 disk mirror and 6 disk stripe. Neither had a great affect on the overall throughput of the checkout.

 

[Apollo]

How are you handling the SVN repository on Freenas?
Are you using a plug-in jail to act as a server (if so I am myself curious how you implemented it), or did you just created (as I understand your description) a dataset and share its content as a NFS share?

 

[LoneRangir]

The latter, I'm just mounting an NFS share. Am I correct in thinking that this should take the "threaded-ness" question out of it, since the actual SVN client is on my linux workstation rather than the FreeNAS box? The linux workstation should be the computer that is pulling the files from the SVN server and pushing them onto the data share, or in the control case, the hard drive?

 

[Apollo]

Threaded or not, this is the part I am unsure.
In my case, my main computer is a Windows 7 machine and mix of RAID SSD and RAID HDD's.
I have a repository I am checking out and is a mix of large and small files.
The trend is as follow:
Small files tends to use less than 2% of the 1Gb/s but the larger one seem to go as high as 35%.
TortoiseProc.exe is maxing out its CPU thread.
I am trying another approach which consist in creating a RAM drive and checking out the repo there.
Network seems to behave very similarly. CPU is maxed out and Freenas has about 4GB RAM free.
This brings me to the following conclusion:

Checking out is CPU bound!!!!

In about less than 10 minutes I was able to checkout close to 10GB data and a little over 55K files.

I am running it on my old i7 920 running at 3GHz+.
What is your Linux Workstation hardware?

I believe the issue is with TortoiseSVN.
Could it be an issue of encryption done is software, if encryption exist, rather than in hardware?
I have to investigate further to see if it can run multithreaded.

 

[cyberjock]

Well, the i7-920, while a great CPU when it came out in Nov 2008, isn't exactly "fast". Not sure why you are upset about performance with a CPU that was released 6 years ago. ;)

 

[Apollo]

I am still very happy with this CPU has it is still an excellent performer. Reason is that it is properly balanced with surrounding hardware, RAM, drives.....
I have unfortunately worked in the past with a Xeon system where performance was crippled by poor component matching, such has low end video card, single HDD.
I will try to see what numbers I come up with the recent system I have at work.

 

[cyberjock]

I have an i7-920 in my HTPC. It's not powered on frequently as the system draws about 100w idle. My desktop (which has an Intel 6c/12t CPU and a GTX 770 draws less than that idle. ;)

Yes, it's a very capable processor. I'm not disputing that. I'm just saying that if minutes matter then why not drop on new hardware? 6 years old is *ancient* by today's standards.

Just to compare, there are Xeons out there that are about $200 and can do double the processing power. Even AMD makes CPUs that beat that CPU. :P Also the newest Intel CPUs do some pretty cool stuff with single-threaded performance (even more so than the i7-920's tech) and I'd never go back. Now that I've run some of my single-threaded apps on my i7-4930K I'll never go back to the i7-920.

 

[Apollo]

I am still going to wait until better CPU's/RAM come around before I upgrade, or when my system dies, whichever come first.

Came back late from work cause of the festivities, but I did run a checkout on my work i7 4770 I believe and directly to a Samsung SSD 840 Pro I think.
While working on a different repository, I can confirm that checking out small files do take a lot longer than bigger ones. Single thread is all SVN can or is willing to handle.
Small zip files would take about 5% bandwidth and would be up-and down, as low as sub 1%.
As I said earlier, I don't know if hardware encoding/encryption would help in this matter, and yes it would most likely benefit from a XEON like processor.

Anyway, Merry Xmas everyone.