Slow iSCSI performance -> 50Mb/s

[silver565]

Hello everyone,

I was hoping someone could point me in the right direction here. I am getting slow writes on my Freenas host(new setup). Details are as follows:

• Supermicro A1SAi-2750F
  
• 32GB memory
• 1x Quad port NIC(E1G44HTBLK)
• 1x 120GB SSD (DC S3500 series)
• 3x WD red 3TB drives
• FreeNAS-11.0-RELEASE (a2dc21583)

Setup:

• The quad port nic has each port configured individually. Each port has a unique subnet which has been added into the portal tab in the ISCSI configuration
• Jumbo frames are not enabled(although it didn't make any difference when it was anyway)
• My ESXi host is running 6.0U2 with two dedicated gigabit ports mapped to two of the freenas ISCSI ports
• The SSD is used for the ZIL(after some reading I found that the S3500 seemed to be recommended by a lot of people due to it's random write and potential power loss capability)
• The WD drives are in a RaidZ. Although I've tried them in a stripe and I get the exact same results
• ESXi is in round robin for its iSCSI mappings

My findings:

When the a VM has a VMDK mapped onto the freenas host, I see the following results when copying files to it:
http://imgur.com/a/h5TLj

My memory seems to explode for some reason(I thought 32GB would be fine for 9TB of physical disk capacity?):
http://imgur.com/a/aEfdJ

dd if=/dev/zero of=/mnt/ISCSI-RaidZ-5TB/ddfile bs=1024k count=20k gives me: 1304235733 bytes/sec

Can someone point me in the right direction for ISCSI? My setup isn't crazily complicated. I'm hoping to get around 90mb/s consistently if possible. This host is only used for backups, but I'd still like to get something decent out of it. I've done quite a bit of reading and I thought I was on the right track in terms of hardware support(choosing a good ZIL and drives with TLER type technologies)

Any help would be greatly appreciated!

 

[dlavigne]

Were you able to figure this out?

 

[silver565]

Were you able to figure this out?

Sorry for the late reply. I've been tied up with work. I've ordered another S3500 SSD which I'll use as the L2ARC. I'm not too worried if this bombs as I'll just convert this box into a 2016 server for DFS instead. Not ideal, but not wasted

In terms of why this is happening, I'm not sure why I see my memory run out so quickly. All the reading I've done points to 1GB for every 1TB. Considering I've only got 9 in total, it's worrying that 32GB is consumed rapidly for copying files via ISCSI. I dred to think what will happen when I throw VMDKs at it for backup locations

 

[Stux]

Sorry for the late reply. I've been tied up with work. I've ordered another S3500 SSD which I'll use as the L2ARC. I'm not too worried if this bombs as I'll just convert this box into a 2016 server for DFS instead. Not ideal, but not wasted

In terms of why this is happening, I'm not sure why I see my memory run out so quickly. All the reading I've done points to 1GB for every 1TB. Considering I've only got 9 in total, it's worrying that 32GB is consumed rapidly for copying files via ISCSI. I dred to think what will happen when I throw VMDKs at it for backup locations

By design ZFS uses all available memory for ARC.

 

[silver565]

By design ZFS uses all available memory for ARC.

Which is fine, but I thought 32GB would be fine. I seem to get the performance loss when Freenas reports that 99% of the memory is consumed. I was expecting some sort of memory management for such a small capacity. I'll keep playing around with it once I get my new SSD