Question and concern

[hoboville]

First, the concern: At work I am running a FreeNAS box for our teaching lab. It runs 7x HDDs in RAIDZ2 on a Core2Duo E8400 with 6 GB of RAM. The total usable storage pool is only like 700 GB, and it's using older SATA drives that passed with 92%+ performance on SMART. Amazing performance was never expected, but the performance I'm seeing is pretty slow. Big writes are slow, of course, but that's not the problem. We don't often write to the server, so 97%+ of traffic is reads. On an average day, we do about 200 GB of data sent. The FreeNAS box and the actual server (a PowerEdge 1950) use dual-nics in iSCSI multipathing in a round robin configuration. Actual bandwidth tests show we are able to reach 1.8 Gbps over CIFS (hard to believe, I know) if the data is simply written to RAM cache. This gives a realistic peak performance profile.

The problem, however, is that FreeNAS volume reads are only coming through at 30 MB/s when data is coming from the drives. Writes are about 25 MB/s. Using CrystalDiskMark on an iSCSI volume shows both NICs being utilized, but with the exact same performance profile. This confirms the performance concern as being volume related. (As a side note, small writes to the FreeNAS server shows peak network performance until the write exceeds 1.5 GB, which makes sense because FreeNAS caches writes to system memory. The slow reads make sense then because the data can't just be read from RAM, but instead has to be retrieved from the HDDs).

Is there a way to solve this performance issue? We aren't using deduplication or encryption either.

 

[cyberjock]

Three things that will help out your performance.

1. More RAM.
2. Faster CPU.
3. Faster disks.

Try not to forget to post your system specs in the future, but consider these points:

1. CIFS is single threaded. You don't exactly have a record setting CPU so you may be bottlenecking CIFS. Try NFS as an alternative since its multi-threaded. Of course, with NFS you'll still have potential serious performance issues because ZFS is a CPU hog. The checksumming and parity data doesn't calculate itself. Even if NFS isn't an option for long term use, at least you could prove that a CPU with faster cores may help CIFS speed.

2. Update to the latest version of FreeNAS.

3. An Intel NIC can help alleviate some networking load from the CPU.

4. I'm a little fuzzy on your exact configuration between your poweredge and the freenas server. But iSCSi on ZFS has performance issues of its own over time.

 

[hoboville]

Thanks for the reply! We are using FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty), but will probably be upgrading some time in the next day or so.

Anyway, the PowerEdge 1950 is running dual Xeon x5260, with 8 GB of FB-ECC RAM, and 4 NetExtreme II's, running ESXi 5.1 with Windows Server 2012 Standard, and Server 2012 is running off local 2x 15k SAS HDDs in RAID1. Two NICs are multi-path I/O to the FreeNAS box which I just saw this morning is using 2x of the dreaded Realtek 1Gbps NICs :(, one NIC on each server is on a different subnet, to ensure proper MPIO in a 1:1 NIC configuration. The FreeNAS box has one main extent and one main zvol, and is set up as an iSCSI target. This was done to allow for MPIO, which I people have told me is better than LACP/LAGG. The iSCSI initiator is Microsoft's own.

We don't actually have any CIFS shares set up in the FreeNAS configuration itself, the only connections to the extent are via iSCSI. (When I did the CIFS network speed test that measured max speed, it was done between the PowerEdge and a second backup server that also has MPIO on it. The FreeNAS box speed tests were done with CrystalDiskMark and a plain old drag-and-drop file copy on the iSCSI volume.)

You mentioned iSCSI and ZFS performance issues--is there some problem that only crops up only when both are in operation? Would FreeNAS run faster with TOE/iSCSI HBA NICs? I have been thinking about moving to a PowerEdge 2950 with 16 GB of RAM, but that won't happen for a while.

 

[cyberjock]

You mentioned iSCSI and ZFS performance issues--is there some problem that only crops up only when both are in operation? Would FreeNAS run faster with TOE/iSCSI HBA NICs? I have been thinking about moving to a PowerEdge 2950 with 16 GB of RAM, but that won't happen for a while.

Yep. ZFS is a copy on write file system. As such it can fragment badly over time and there is no defrag option for ZFS. There's other potential pitfalls for iscsi with zfs, but if you search the forums there's other thread that explain it in depth and provide a direction to go. Keep in mind that if you are actually hhaving issues with iSCSI and ZFS they are non-trivial to implement(and some cost $).

I can't vouch for any other iscsi configuration because I'm not a fan of iSCSI and avoid it when i can. I consider iSCSI to be a "I don't want a hard drive in my computer" scenario and instead spend countless hours/days trying to get iSCSI to work at decent speeds when you could have installed a $50 hard drive and gotten rid of the problem. iSCSI has some uses, but it seems to be used overboard and I sometimes get a good laugh at how many people ignore the simplest fix.. just add a hard drive and get rid of iSCSI.

 

[hoboville]

Yep. ZFS is a copy on write file system. As such it can fragment badly over time and there is no defrag option for ZFS. There's other potential pitfalls for iscsi with zfs, but if you search the forums there's other thread that explain it in depth and provide a direction to go. Keep in mind that if you are actually having issues with iSCSI and ZFS they are non-trivial to implement(and some cost $).

jgreco stated in another post: " I suspect that a properly provisioned FreeNAS system would make a sweet iSCSI target. The problem is, the definition of "properly provisioned" would probably be eye-meltingly shocking to mere mortals."

I did some research (and if I am understanding correctly?) making iSCSI work would be principally governed by processing power, memory, and/or TOE HBAs, which as you say, is not a cheap solution.

Mounting an NFS share is definitely doable, and it appears to support multi-pathing. Right now, the Win 2012 server accessing the iSCSI share has MPIO configured already in ESXi through separate vmnics. Should NFS yield better performance with less fragmenting and processing overhead? SMB/CIFS does't seem to be a very good choice since it is single threaded.

The PowerEdge server uses local disks for the OS, which are enterprise disks anyway--and much faster. The downside is that the FreeNAS box has a lot more storage space, and we'd have to buy some big HDDs to fit all that data in a box with only 2x 3.5" bays . Again, expensive since they'd have to be in RAID1 at least.

Totally agree on the diskless system setup, it's really rather pointless. A regular OS install can be put on a 60 GB SSD if one is terribly concerned about OS performance.[/quote]

 

[jgreco]

principally governed by processing power

secondary

, memory

primary

, and/or TOE HBAs

on the client, perhaps. TOE on the FreeNAS network interface might be nice but is not critical.

, which as you say, is not a cheap solution.

Mounting an NFS share is definitely doable, and it appears to support multi-pathing. Right now, the Win 2012 server accessing the iSCSI share has MPIO configured already in ESXi through separate vmnics. Should NFS yield better performance with less fragmenting and processing overhead? SMB/CIFS does't seem to be a very good choice since it is single threaded.

NFS from ESXi has approximately the same problem with writes causing fragmentation.

You cannot really solve the fragmentation issue on a ZFS system. Ultimately, writes to the virtual disk (whether iSCSI file extent, NFS to a vmdk, etc) will cause the blocks allocated to the virtual disk to become reordered. The tendency of the client OS to be writing things that may have their own locality considerations can reduce the seriousness of that: for example, if you have a Linux VM and it updates libc during upgrades, the entire libc gets written out, and those blocks are very likely to be allocated contiguously. However, if you have a utility that "dd"'s an entire disk to another disk, it'll be reading block #123, #124, in order, then get to block #125 which is the start of the new libc, at which point the pool has to seek to get to the "updated" blocks - and then seek back again when it has read past the last of the updated blocks. That doesn't sound bad until you consider it in aggregate, for each file.

For the most part, the worst effects of this can be minimized by ensuring that sufficient ARC is available to hold the "working set" in memory. The "working set" can be defined a few different ways:

1) The blocks that have been accessed recently,

2) The blocks that have been accessed more than once recently,

etc. And "recently" is all relative. You might be trying to push down IOPS stress on the pool, in which case you really have to consider the pool load as a significant factor. Or you might just be seeking speed, in which case you basically want the most liberal definition you can afford for "working set."

 

[hoboville]

Thanks for the info, jgreco. Would it be right to say then that the fragmentation results from a "layering" of file systems combined with the overhead of protocols? I read too that keeping the volume at 80% capacity max will reduce fragmentation too. A lot of this reminds me of SSD write amplification and performance tips .

I confess I am not a pro at file systems, but given what I've read so far...all indications are that the data on the ZFS volume is fragmented already (when it was first copied over to the zvol). It would seem that the following would be true: if a user has setup an improperly provisioned ZFS server, then all files starting from the first one copied are doomed to immediate fragmentation.

As before, our FreeNAS box does 97% of its activity as reads, so the core data that is repeatedly read (some 150GB) should not continue to fragment, right? It just seems odd that read performance has decreased in the last two weeks, even though no more than 200 or 300 MB has been written in that time period, and the core data has not been altered / overwritten.

Given the performance slow-down and latency accessing data, we will likely be transitioning to a RAID10 setup from RAIDZ. At home, it would be nice to also have a FreeNAS NAS box for media sharing and backup, but since we got the PowerEdge with its phenomenal PERC RAID card it's handling some older drives with aplomb! So, this leads to another question, for a sub 16 GB server, would an enterprise RAID card offer better performance than RAIDZ when used on consumer grade drives? I am hoping for 100 MB/s consistent throughput.

 

[jgreco]

It is complicated; due to the COW nature of ZFS, it is actually possible for a file that is actually written in a fragmented nature on a filesystem to be written to contiguous blocks due to the grouping of writes in transaction groups.

Even writes as trite as file atime updates add to fragmentation on a ZFS iSCSI volume. It is important to restrict writes to things that are actually necessary, rather than superfluous stuff.

I'm not aware of any "phenomenal PERC RAID" cards. ZFS is, however, rather heavyweight as far as filesystems go. You can get a much faster fileserver by using UFS. ZFS may not be the correct choice for iSCSI use unless you can throw resources at it. Of course, UFS lacks some really cool ZFS features.

 

[hoboville]

Just stumbled on this article: https://calomel.org/zfs_raid_speed_capacity.html, it points out an increase in throughput even on uncompressable data when using ZFS and enabling compression. Obviously, this would necessitate better hardware than we have at the moment, what would you recommend for RAM and number of cores to make effective use of compression? 2x Nahelem Xeon good enough?

The author was using a 6 core sandy-bridge Xeon and 16 GB of RAM in quad channel, which is a bit too much $$...

 

[jgreco]

You're not going to get an increase in throughput on uncompressable data when using ZFS with compression. You could get an increase in throughput if you couple a slow pool with a fast CPU though. Extra cores probably translates to faster CPU in that statement, so yes, a six core Sandy would probably have headroom for doing on the fly compression.