What am I doing wrong?

[Jared M.]

Hello all, I am trying to fix my performance issue I am having, hoping this is the right place.

My setup:
2x HP DL320s w/P400 raid controller, 12x 300GB sas drives setup into two separate raid 5 array each (total of 4 arrays, 6 drives in each array), and 4GB of ram and 2.4Ghz intel dual core proc. 2x 1Gbps uplinks to switch
1x HP Proliant 2900 48GB ram, 2x 3.16 Ghz Intel dual cores 4x 1 Gbps uplinks to switch

I have the 2900 setup as my main ESXi host will all of my virtual machines on there. My 320s's are setup with ESXi then have freenas as their only virtual machines and I took the two arrays from each one of the 320's and created a datastore in esxi, meaning two data stores in total, one for each esxi host. I have setup iSCSI on both of the freenas's as a target with the device as the extent. Then on one virtual machine on my 2900 I have a windows server 2012 setup as the initiator for both of the freenas's.

My performance on this setup: 40-50 MB/s. A lot lower than I had expected. What is wrong with this setup? Too many layers?

I would just do freenas straight to the DL320s's but I was having issues getting the iSCSI to work. So I went the vmware then freenas route.

Any suggestions?

 

[Nomad]

RAM on the HPDL320 You want about 1GB per 1TB of Storage.

 

[Jared M.]

RAM on the HPDL320 You want about 1GB per 1TB of Storage.

Even though I am not using any type of ZFS storage? I am using plain old ntfs via iSCSI.

 

[HoneyBadger]

Even though I am not using any type of ZFS storage? I am using plain old ntfs via iSCSI.

Maybe on the Windows VM on the 2900 you are, but the FreeNAS machines are running either UFS or ZFS to serve drive.

But that's only a portion of the problem here. You're doing a whole raft of things that will outright murder your performance, and I'm impressed you're getting what you are.

You're running FreeNAS in a VM (1) without enough RAM (2) on top of a RAID controller (3) in RAID5 (4). There's way too many layers going on there, with all four of those being things suggested against for good ZFS performance.

Unless you have a serious reason for keeping them, I would suggest tossing both of the DL320s and getting a single system better suited for it. If you want/need to keep them, you'll want to upgrade the RAM to a minimum of 8GB each, more likely 16GB, switch the P400 RAID controller out for a generic SAS HBA, and install FreeNAS as a bare-metal OS rather than a VM inside ESXi.

 

[Jared M.]

Maybe on the Windows VM on the 2900 you are, but the FreeNAS machines are running either UFS or ZFS to serve drive.

But that's only a portion of the problem here. You're doing a whole raft of things that will outright murder your performance, and I'm impressed you're getting what you are.

You're running FreeNAS in a VM (1) without enough RAM (2) on top of a RAID controller (3) in RAID5 (4). There's way too many layers going on there, with all four of those being things suggested against for good ZFS performance.

Unless you have a serious reason for keeping them, I would suggest tossing both of the DL320s and getting a single system better suited for it. If you want/need to keep them, you'll want to upgrade the RAM to a minimum of 8GB each, more likely 16GB, switch the P400 RAID controller out for a generic SAS HBA, and install FreeNAS as a bare-metal OS rather than a VM inside ESXi.

The maximum ram supported by these boxes is 8GB and it is way too costly as it is PC2-5300E ram. These were given to me so I don't want to put any more money into them. Would I see much better performance if I try to use NFS on a baremetal freenas installation instead of iSCSI? I would then change to a raid6 array vs a 5.

Plus each one has 4GB of ram and only 2.7Tb of usable space (in the current configuration.) So I would be meeting the 1GB of ram per 1TB of storage requirement.

Which version of FreeNAS would be the best choice for me? I had troubles installing 9.2.1.2 and 9.2.1. On 9.2.1.2 I couldn't get past the mDNS responder error after installation, it just would not boot and on 9.2.1 I could not get the iSCSI service to start after configuring my target/extent association. All on baremetal installations, when I went to a VM I used 9.2.1.2 and everything booted fine (it showed the mDNS repsponder error but continued) and worked.

 

[cyberjock]

Even though I am not using any type of ZFS storage? I am using plain old ntfs via iSCSI.

Then you are even crazier than needing more RAM. The base storage on FreeNAS should be UFS or ZFS. If you aren't doing that, you are way crazy since the manual says NTFS support is only meant to be temporary.

You should also have 8GB of RAM minimum, and the second you start wanting iSCSI you are talking about a fairly beefy FreeNAS system. So my advice is "build a new system" since the max is 8GB. Why you'd think that 1/2 the recommended(or even the minimum) is appropriate for your situation is a little beyond my understanding. When someone tells me "8GB of RAM minimum" I don't think about hardware that maxes out at 8GB. I think about systems that support far more than 8GB of RAM. /sadface

 

[HoneyBadger]

The maximum ram supported by these boxes is 8GB and it is way too costly as it is PC2-5300E ram. These were given to me so I don't want to put any more money into them. Would I see much better performance if I try to use NFS on a baremetal freenas installation instead of iSCSI? I would then change to a raid6 array vs a 5.

Plus each one has 4GB of ram and only 2.7Tb of usable space (in the current configuration.) So I would be meeting the 1GB of ram per 1TB of storage requirement.

If you're not able to put more RAM into them, you should use UFS instead of ZFS, in order to still have some performance. Ideally dump the RAID controller entirely and replace it with a SAS HBA; but if you can't then you'll want to set up each drive as a single RAID0 array and use a mirrored configuration. Parity RAID (RAID5/6/Z) and virtualization don't get along.

Which version of FreeNAS would be the best choice for me? I had troubles installing 9.2.1.2 and 9.2.1. On 9.2.1.2 I couldn't get past the mDNS responder error after installation, it just would not boot and on 9.2.1 I could not get the iSCSI service to start after configuring my target/extent association. All on baremetal installations, when I went to a VM I used 9.2.1.2 and everything booted fine (it showed the mDNS repsponder error but continued) and worked.

If 9.2.1.2 isn't compatible with your hardware, you can try 9.2.1 and see if you can sort the iSCSI issues separately.

But I would strongly suggest you sell those DL320s or give them away and build a new machine. If not, try switching to a UFS mirror instead of a RAID-Z level. Or make a RAID10 mirror at the P400's level and present a single drive to FreeNAS. You will lose most of the reason for FreeNAS but it will probably perform better.