FreeNAS box has been problematic lately

[TravisT]

I'm having numerous problems with my freenas box lately, and I'm going to try to post my troubleshooting steps as I go. Hopefully this will help me to keep track of what I've done, and maybe help someone else along the way too. Plus, maybe if I make a bonehead mistake or overlook something, and extra set of eyes will help keep me on track.

My current setup is as follows:

Hardware Specs:
GA-MA770-UD3P Mobo
AMDPOhenom II x4 945 CPU
16GB of RAM
WD20EARS 2.0TB Drives (x3)
1x onboard Gigabit NIC
2x Intel e1000 Gigabit NIC (PCI)

Software:
VMWare ESXi 5.0.0 running on hardware

VM Specs:
1 CPU / 8GB RAM allocated
OS: FreeNAS 8.2.0 RC1 x64
NIC: Intel e1000 network interface card
CIFS Shares with integrated Active Directory for Windows Domain

3 vmdk files were created on each of the EARS drives and presented to the VM as datastores*. These are setup in a 3-disk raidz vol.

Network Specs:
Cisco gigabit network
Intel e1000 NIC in ESXi host
Windows Active Directory Domain

Current Problems:
Extremely slow transfer rates (~1MB/s, as reported in Windows).
Network Drives become unavailable during large transfers.

My transfers have been from a USB3.0 drive connected to my desktop, which is also has a gigabit connection. I have also tried connecting this USB3.0 drive to the ESXi server, attaching it to another windows machine hosted on ESXi, and attempting the transfer from there. This essentially rules out my desktop and network, as everything is internal to the ESXi box. I have a 3 disk RAID array in my desktop that I plan to do some testing with, that way I know it's not a USB bottleneck causing the problems.

The ESXi box has had relatively little use over the last year, as I have been deployed. There were a couple hardware problems that I walked my wife through fixing, and it was accessed every couple days with very little stress put on it. When I left a year ago, it was running well (though probably not optimized) and would easily give 35-55MB of sustained transfer from 5400-7200RPM single SATA drives. Although a little on the low side, this was acceptable with the inherent problems with my setup. I can't get even close to that now, so the goal is to figure out what changed and is causing my problem.

If anything is a glaring problem, or any other quick recommendations as to what I should check, please hit me up.

* I know that having an underlying file system for ZFS is not optimal, so I'm considering moving this box to its own hardware so FreeNAS can have low level access to the disks.

 

[Daisuke]

Why are you running on RC1? :)
Try first 8.0.4, I'm considering more and more to downgrade also, do to various issues I have with 8.2.0.

 

[TravisT]

I'm running RC1 because I was having problems with 8.0.3 and saw that RC1 had been released. I hoped that the upgrade would solve the problem, but it doesn't seem to have done so. One plan of action that I plan to investigate is downgrading to see if that solves the problem.

 

[TravisT]

So after many hours of testing last night, I realized the problem may be the ESXi box/NIC and not the VM at all. My transfers to other VMs were lacking as well. I got that worked out today, and got an older version of ESXi installed and running. My desktop RAID array can now transfer files to a Windows Server VM at 70 - 90MB/s for a 5GB file. I'm getting closer.

My freeNAS box still has slow transfers and stalls during copying of the same 5GB file. It starts relatively fast (70MB/s or so), but quickly drops off down to 10MB/s or less. From a clean install of 8.2.0 RELEASE x64, I tried enabling AIO, large RW, and autotune one at a time, but nothing helped. This datastore should theoretically be faster than copying to a Windows VM, as all 4 windows Servers are running on one physical disk while my FreeNAS datastore is spread across 3. I realize there is some additional overhead because of ZFS, but nothing significant enough to cause that much of a hit.

Please point me in the right direction - this is driving me insane!

 

[paleoN]

Go run the tests from the [thread=981]performance sticky[/thread] and post the results here. That will give you an idea of your "disk" speeds.

What physical NIC are you using with FreeNAS?

Test the networking with iperf, included with FreeNAS, whose sole purpose is to blast packets out through a network interface to another iperf client & report the speed. Grab a copy of the iperf windows client, put it on your desktop & see how the connection works.

 

[TravisT]

Ok, so my interpretation is that I'm having slow transfer rates to my disks, but here are my results in case I'm not reading them correctly.

First is the test on my 3 disk RAIDz volume:


[root@vFreeNAS] /mnt/cargo/galaxy# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
^C4657+0 records in
4656+0 records out
9764339712 bytes transferred in 319.764429 secs (30536041 bytes/sec)

I also added a single disk using ZFS to test the speeds against. This is a 1TB WD EALS 7200RPM disk with a 900GB vmdk file attached to the FreeNAS VM. My transfer rates were as follows:


[root@vFreeNAS] /mnt/Test# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
^C201+0 records in
200+0 records out
419430400 bytes transferred in 163.440339 secs (2566260 bytes/sec)


Seems that both of these are very slow compared to some of the tests I've seen on here. Ideas?


Edit: By physical NIC, I'm assuming you mean what is configured on the VM. I'm using the e1000 NIC, as the VMXNET2/3 isn't recognized in FreeBSD. I'll try iperf shortly, but I don't think the problem is network related anymore. Looks like it may be a disk bottleneck.

 

[paleoN]

[root@vFreeNAS] /mnt/cargo/galaxy# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
^C4657+0 records in
4656+0 records out
9764339712 bytes transferred in 319.764429 secs (30536041 bytes/sec)

You couldn't let that finish running? Now I had to think about it. :( It is quite low though.

Seems that both of these are very slow compared to some of the tests I've seen on here. Ideas?

How about passing through the drives to the FreeNAS VM? Otherwise only what you already decided.

Edit: By physical NIC, I'm assuming you mean what is configured on the VM. I'm using the e1000 NIC, as the VMXNET2/3 isn't recognized in FreeBSD. I'll try iperf shortly, but I don't think the problem is network related anymore. Looks like it may be a disk bottleneck.

No, I mean the physical NIC you assigned the VM. The onboard, a realtek?, or one of the PCI based e1000s. I would still test the networking.

 

[TravisT]

You couldn't let that finish running? Now I had to think about it. :( It is quite low though.

Sorry, I was impatient to find out how poor the results would be. ;)

How about passing through the drives to the FreeNAS VM? Otherwise only what you already decided.

I considered this, but it seemed like it may be very unreliable. I'm strongly considering building a separate FreeNAS box that I can use iSCSI to host the VM drives. I'm concerned about reliability and performance of this, because I haven't ever used iSCSI before. I believe I can also use NFS - not sure which is better. I really don't completely understand why I get acceptable transfer rates to my other windows VMs, which share a disk and get really poor performance when I use FreeNAS using a RAIDz. I would think the performance would be better.

No, I mean the physical NIC you assigned the VM. The onboard, a realtek?, or one of the PCI based e1000s. I would still test the networking.

I'll run some network tests with iperf. There are two physical NICs in the server. One is an onboard realtek gigabit NIC, the other is an e1000 PCI NIC. With ESXi 5, the realtek NIC works out of the box; 4.1 didn't have drivers installed for that to work without installing them separately.


So my plan is to try passing through the 1TB drives that I have that aren't used right now and see if the numbers are better. It would also help with migration to a hardware-based NAS from the current virtualized box. Theoretically, I could remove the passed-through drives from the current server and install them in the new one without having to transfer a bunch of stuff over. Problem is, I don't have enough ports to get all the drives hooked up to transfer everything from the 3-2TB drives to the 4-1TB drives.

Maybe that IBM M1015 is closer in my future than I expected.

 

[TravisT]

Just ran some iperf tests. I'm saturating right at 700Mb/s of a 1Gb/s network. That is with setting the TCP window size to 64KB. Not sure if I should be changing any other settings.

Seems the problem is with my disks...

 

[TravisT]

Today I removed my drives from the SuperMicro hot swappable drive bays that I just ordered and installed. My transfer rates were normal again. Right now I have my 3 FreeNAS drives connected directly via SATA cables, and my DD speeds were way more in-line with what they should be:


[root@vFreeNAS] /mnt/cargo# dd if=/dev/zero of=testfile bs=2048k count=200
200+0 records in
200+0 records out
419430400 bytes transferred in 1.828901 secs (229334659 bytes/sec)

[root@vFreeNAS] /mnt/cargo# dd if=/dev/zero of=testfile bs=2048k count=1k
1024+0 records in
1024+0 records out
2147483648 bytes transferred in 4.320483 secs (497047127 bytes/sec)

Big difference. Now I'm trying a transfer again of a large number of video files. Here is what I'm seeing:


My network throughput seems good, and my disks seem to check out now. I am copying from a USB drive, but it's a USB3.0 drive and port. Should I see better performance or is this about it? I'll try copying some over to my RAID array on my desktop then copying from there to see if the performance is better. Then I have to find out what's up with the hot swap bays...

 

[TravisT]

Well, that question was quickly answered. I grabbed about 20 of the files and copied them to my desktop RAID array. The transfer of 71.7GB of data took about 12 minutes at 90-100+MB/s as reported by windows.

Guess it's not the USB drive slowing the transfer down.