Determining if I need to upgrade hardware

[Elliot Dierksen]

I am running FreeNAS 9.3 Stable on an HP DL380 G5 platform. I have dual X5450 @ 3.00GHz CPU's and 16GB of RAM. I have a Chelsio T3 based NIC in an x16 PCIe slot, and an HP P822 RAID controller in the other x16 PCIe slot. I was doing some testing moving some large files from FreeNAS to one of my ESXi 5.1 servers, and I didn't get quite the performance that I was expecting. I did see the NIC utilization get to about 13%, but I was kind of expecting more. The system load and CPU utilization were all very low during the transfer, and absolutely no swapping taking place. It could also a limitation on the ESXi side, but I am wondering if a move to an HP G6 or G7 platform (ESXi is DL360 G6) would gain me very much. My inclination is no, but I guess I was looking for some confirmation or ways to make sure the FreeNAS hardware isn't a bottleneck. The switch is an HP 5820X. ESXi is mounting the FreeNAS via NFS. I have not pushed the MTU beyond 1500. The test in question relocating a VM from the local data store on the server to the NFS mounted store on FreeNAS. The drives are 25 x 300GB 10K SAS drives configured as RAID 6. The P822 controller has 2GB flash backed cache. I didn't see a way to do JBOD on the P822, so that is why I configured it as a big RAID volume.

 

[Norleif]

... configured as RAID 6. The P822 controller has 2GB flash backed cache. I didn't see a way to do JBOD on the P822, so that is why I configured it as a big RAID volume.

Breaking rule #1...

Keep playing with the P822, and if all else fails, create 25 arrays with just one drive in each and build your ZPool on top of that.

 

[jgreco]

Breaking rule #1...

Keep playing with the P822, and if all else fails, create 25 arrays with just one drive in each and build your ZPool on top of that.

Oh hell no, do NOT try that. That's spectacularly bad advice.

The G5 is fairly old hardware, 16GB is not a lot of memory, and introducing a RAID controller in between ZFS and its storage means you have a RAID controller (ZFS) flooding a crappy RAID controller (virtually any RAID card that fits inside a PC) with I/O. ZFS pushes data out in transaction groups that are larger than your crappy RAID controller's write cache, so the write cache (and RAID6 engine) is probably the biggest single bottleneck. Possibly also whatever attachment you've got set up to the DL380 for the disks...

Replace the RAID controller with an HBA like an LSI 2008, make a pool out of mirrored vdevs (NOT!!!! RAIDZ!!!!), add at least 16GB more of memory, and I'd bet your system will be at least two (probably a good bit more actually) times faster. The host platform itself isn't all that hot. Anything with a FSB performs much-less-well than modern hardware, but I'm making my bet that the other issues here are a much larger part of the problem. Once you clear those away and you're still not getting awesomeness, then find yourself a platform with 64-128GB of RAM and some modernish architecture.

 

[Elliot Dierksen]

The RAID controller is the most modern part of the whole setup. As a matter of fact, they still sell that one. I am reasonably familiar with traditional RAID which is why I have an array that has a bunch of smaller drives instead of 3-4 really big drives. The ZFS thing is new to me.

The array is configured as RAID6-ADG

 

[Norleif]

Perhaps I jumped the gun a bit... Was just thinking of things to try before spending money on new hardware (HBA + SAS expanders for 25+ drives).
Have you tried a dd speed test on your zpool? That should give you an idea for the upper bound on your transfer speed.

I'd still argue setting up 25 separate 1-drive arrays on the P822 would be better then a single RAID6 array as far as data integrity goes (RAID write hole). Resilvering should be quicker too if/when replacing drives.

 

[cyberjock]

I'd still argue setting up 25 separate 1-drive arrays on the P822 would be better then a single RAID6 array as far as data integrity goes (RAID write hole). Resilvering should be quicker too if/when replacing drives.

No. Do not do this.

The bottom line, if your hardware controller (whether you call it a "RAID controller" or not, requires any configuration aside from just "plug in the hard drive and use it" you will be sorry). I've had 3 or 4 people in the last 2-3 weeks lose all of their data. One company almost lost the business.

So no, if you are silly enough to argue doing single disk arrays, you shouldn't be doing ZFS. Buy the right hardware so you still have a job (and your data) or use another solution. ZFS is NOT for all situations, and this is one where I would tell you *NEVER* to do that.

 

[jgreco]

The RAID controller is the most modern part of the whole setup.

Congratulations. It's also probably the most crappy single issue in the whole setup. I have some nice LSI 3108's which are even newer 12Gbps RAID controllers with 2GB of cache but you won't see me using them to attach pool data drives.

ZFS is your RAID controller. Let it do its thing. Putting a crappy "hardware" RAID controller between a massive software RAID controller and its disks is a bad thing. ZFS has all the resources of the host system, including all that memory for cache.

I am reasonably familiar with traditional RAID [...] The ZFS thing is new to me.

I'm a lot more familiar with traditional RAID than most people are. The ZFS thing is not new to me. Take a hint and listen to me when I tell you that you are doing yourself no favors with that RAID controller. You are more likely to be hurting performance than helping.

 

[Norleif]

... Continuing on the sidetrack before we get back on topic...
Would you argue a 25 drive RAID6 array on the P822 going to a single-device ZPOOL is better then 25 "single drive arrays" on the P822 going to a multi-device ZPOOL?
(I know none of them are good options. I'm just curious about the lesser evil...)

Back on the original topic (throughput to ESXi server)...
If you do some simple "dd" read and write tests on your FreeNAS you should get a ballpark idea for the peak performance of your ZPOOL.
Numbers below a few hundred MByte/s would indicate your FreeNAS storage is the bottleneck.
Numbers close to or above a GByte/s would indicate network/protocol shortcomings or a bottleneck on the ESXi server.
You could play around with ipefr too to see how far you can push your network.

 

[cyberjock]

Norlief,

We've seen both "thoughts" fail spectacularly and without warning. :(

With both scenarios some (but not all) of the problems we've seen are things like:

1. No SMART monitoring, so no way to identify a problematic disk.
2. RAID controllers with their write loophole fubaring the zfs file system without warning.
3. RAID controller "owns" the drives, so ZFS is basically blind to disk replacements and such. (this really shows itself with a disk failure, and pulling a disk from the system with the system running is not the same way to 'test' this)
4. Performance is basically AFU because ZFS schedules reads and writes, and RAID controller reschedule both to try to optimize the pool performance, but that undoes what ZFS has already optimized to the maximum extent.

If you do a RAID6 you are going to have problems as there is zero ZFS redundancy. So if the RAID controller gets hit with a write loophole or other incomplete file system write on a shutdown, your pool might never be mountable (and therefore your data inaccessible) permanently. ZFS will detect the corruption, but if your metadata gets corrupted, kiss the entire pool goodbye. :(

If you do the single-disk array you'll be woefully unaware that your disks have problems. Then one day you wake up and your pool is unmounted because you lost enough redundancy that suddenly the server crashed and rebooted and on bootup the pool is now unmountable. :(

When I did contract work for FreeNAS users I've had many people that paid good money to get their data back after doing some pretty crazy things. The second people use hardware RAID my ability to recover went from pretty good to almost zero. In fact, if someone told me they had hardware RAID I'd tell them flat-out that I wouldn't recommend you pay for my services as the likelihood of seeing your data again was nearly zero, but if you still wanted me to try after recommending against paying me (and I could have definitely used the money!) then I'd give it my best effort. After all, nearly zero odds is better than zero odds. I never got data from a hardware RAID array of any kind... ever. But on non-hardware RAID problems I had about a 90% success rate.

So there is no lesser of two evils. The mechanisms for failure are equally sudden and catastrophic and there is no "safer bet". People keep playing the game of "I have this controller and I damn well am gonna use it, I'm just going to take the safer path with it". So do you want to die by falling in a pit of lava or a rancore pit? Thats basically what you are asking, and the end result is the same. Death.

I'd *STRONGLY* recommend you give up the notion that there is some safer path with the RAID controller. The only safer path is to NOT do what you are doing. If this isn't acceptable I'd highly recommend you go with Windows or Linux (or an OS that actually has the proper RAID drivers, RAID software, etc. to monitor the RAID array) and use a file system that does work decently on hardware RAID. FreeNAS isn't a good fit for everyone, it's not a good fit for all hardware, and its definitely not a good fit for all budgets. So please consider carefully if FreeNAS is for you.

Thanks and good luck with whatever choices you make.

 

[Elliot Dierksen]

Some of the issue certainly seems to be ESXi. Perhaps it throttles its network usage when moving VM's around. I mounted the NFS shares from another FreeBSD system with a 10G NIC. Then I did a dd using 64K block size from an 80GB file to /dev/null. I was able to get up to 28% utilization on the NIC there. The VMWare move topped out at 18%. Overall, this isn't as fast as I was hoping, but it clearly better than it was with 1Gb NIC's. The VM's mounted on that store also appear faster than they did before. I don't know for sure what I can do to pull more performance out of the current setup, but perhaps it isn't as bad as I thought. In doing a little more digging, I might be able to change the external array to JBOD. Migrating all the files around to rebuild it may be a bit challenging. I have another NAS (NetGear ReadyNAS), but the NFS write performance on that is truly atrocious!

 

[Elliot Dierksen]

One more update on this. I was able to hit 53% utilization when moving two VM's from FreeNAS to local storage on a VM host. I suspect some of this may be a limitation of the hard drives in my storage array (SAS 3G 10k 300GB). Given the response with a higher workload, I am happy with how FreeNAS is running on the older hardware. I am going to try and see if I can scrounge a G6 somewhere and see what that does for me.I know those have more (and faster) PCIe slots. Why does this kind of thing always feel like a whack-a-mole? You improve performance in one area, and it makes another area look bad.

 

[jgreco]

You improve performance in one area, and it makes another area look bad.

The problem with network storage is that you're layering a whole bunch of subsystems on top of each other, and it is a truly grand example of the weakest link problem. Unless they're ALL performing spectacularly, you get poor performance. The problem just seems to hop around from one thing to the next while you're strengthening individual links in the chain.

 

[Elliot Dierksen]

One last comment here. I migrated this to a DL380 G6 recently. 2 x X5667 CPU's, 32G of RAM. In a recent VM guest move, I was able to hit 75% utilization on one of the 10G ports over a 1 minute time frame. I have to have two guest moves going at once to achieve this, but I am very happy with that result. This is on FreeNAS 9.3-STABLE. Rock on!

 

[jgreco]

I am always horrified at the same time I'm impressed when I have to throw massive resources at things to make them work well. But ZFS is all about that. You give it what it needs and it'll do the job for you. It's kind of horrifying what it needs, but then again you're pitting a general purpose CPU and software against custom-built ASIC's. The nifty thing about ZFS is that most RAID controllers only have a GB or two of RAM.