Network connectivity loss on ESXi under high bandwidth usage

[Maury]

I've got this HP server DL180 G6 with dual Xeon E5620's, 16 GiB RAM, 1 x WD Red 4tb and 2 x Intel Gigabit NIC's and managed to setup latest FreeNAS version under ESXi 6.7.

Everything works wonderfully, but when I put the FreeNAS VM under stress (I tried with a Samba transfer at 80MB/s and a high-bitrate 4K movie over an NFS share, at the same time), the connectivity to the whole ESXi host drops (meaning that the other VM's go down as well).

If I unplug and plug again the Ethernet cable, everything goes back to normal, until the problem occurs again.

Is this normal? Any ideas?

 

[Holt Andrei Tiberiu]

vmxnet3 or e1000 is emulated to the freenas vm?
You are using P410 raid card? or the on-board b110i?

Also note that vmware 6.7 is not offcialy supported in G6, best performance on G6 you will achieve with 6.0u3.
Also are you using both nics from the G6 in the virtual swich?

 

[Maury]

vmxnet3 or e1000 is emulated to the freenas vm?
You are using P410 raid card? or the on-board b110i?

Also note that vmware 6.7 is not offcialy supported in G6, best performance on G6 you will achieve with 6.0u3.
Also are you using both nics from the G6 in the virtual swich?

I've tried both vmxnet3 and e1000, no luck. I'm using P410 raid card (I've created a single raid 0 volume with my HDD, which should be the equivalent of IT mode). I forgot to mention I host my ESXi datastore on an Samsung 256GB SSD, which is separated from my FreeNAS's pool.

I know it isn't officially supported but I've read of some people who run it on G6 servers without any problem. I've tried using one and two nics, but neither works.

 

[Holt Andrei Tiberiu]

Your config is not even close to a standard, i own a few HP servers, trust me. ( from g5 to g10 )
First, running in a VM and giving back storage is not eficient at all, for testing yes, for real life use no.
You would be better running freenas on bare metal on that g6, even with the p410.
Be aware that raid 0 is not passthrou, freenas will not get SMART info! You will not know when your disk will fail, until too late.

16gb ram is small even for freenas itself, not alone a vmware hypervisor.

Guess you have the 12 bay 3.5 inch model?

I would do the folowing:
Install vmware 6.0u3 hp custom iso
https://my.vmware.com/web/vmware/details?downloadGroup=OEM-ESXI60U3-HPE&productId=491
and test the setup again.
Create the raid drives on the p410 and give the disk to freenas vm by vmware.

Perfect it would be: get a HBA card, put the p410 on the shelf or sell it, and use bare metal of that g6. It will work like a charm. ( I use'd a few, and still using g6 as freenas, 12 bay models and 25 bay model's but with LSI cards!)

I would bet my money on vmware 6.0u3 hp custom first, so you don't have to buy nothing. Don't expect any grate performances in this setup from freenas, but vmware will run much better and faster.

I personaly run my g8 with 6.7 ( not supported officialy ) and i had to do a lot of workaround to make it go as i wanted, please don't try 6.7 on g6, you are wasting time.
I have a lot of client's on 5.5 and feel no need to upgrade, they work perfectly without problems, and they are using advanced features like storage vmotion, distributed switch, HA and DRS.

 

[Maury]

Your config is not even close to a standard, i own a few HP servers, trust me. ( from g5 to g10 )
First, running in a VM and giving back storage is not eficient at all, for testing yes, for real life use no.
You would be better running freenas on bare metal on that g6, even with the p410.
Be aware that raid 0 is not passthrou, freenas will not get SMART info! You will not know when your disk will fail, until too late.

16gb ram is small even for freenas itself, not alone a vmware hypervisor.

Guess you have the 12 bay 3.5 inch model?

I would do the folowing:
Install vmware 6.0u3 hp custom iso
https://my.vmware.com/web/vmware/details?downloadGroup=OEM-ESXI60U3-HPE&productId=491
and test the setup again.
Create the raid drives on the p410 and give the disk to freenas vm by vmware.

Perfect it would be: get a HBA card, put the p410 on the shelf or sell it, and use bare metal of that g6. It will work like a charm. ( I use'd a few, and still using g6 as freenas, 12 bay models and 25 bay model's but with LSI cards!)

I would bet my money on vmware 6.0u3 hp custom first, so you don't have to buy nothing. Don't expect any grate performances in this setup from freenas, but vmware will run much better and faster.

I personaly run my g8 with 6.7 ( not supported officialy ) and i had to do a lot of workaround to make it go as i wanted, please don't try 6.7 on g6, you are wasting time.
I have a lot of client's on 5.5 and feel no need to upgrade, they work perfectly without problems, and they are using advanced features like storage vmotion, distributed switch, HA and DRS.

Yes, I have the 12 bay 3.5 inch model.
Thanks for the great input, I will try to install the HP image for ESXi 6.0 and see how it goes.

 

[Holt Andrei Tiberiu]

Yes, I have the 12 bay 3.5 inch model.
Thanks for the great input, I will try to install the HP image for ESXi 6.0 and see how it goes.

Try that, and give a feedback after that.
Good choice for a start server, you just need a few more ingredient's.
LSI HBA + 16GB ram more + Bare Metal Install of Freenas.

 

[Maury]

Thanks for the help @Holt Andrei Tiberiu. Switching to ESXi 6.0 U3 solved my issue.
Anyway, I need to run a few other VM's along with FreeNAS, and since I'm running on a low budget and electricity is pretty expensive here, hosting a second machine only for FreeNAS unfortunately isn't an option. I would also not rather use FreeNAS's jails and virtual machines. Given this scenario, what do you think I should do/buy to get the most out of FreeNAS under ESXi?

 

[Holt Andrei Tiberiu]

Glad i could help.

if you have 1 x 4tb hdd, and using 1 server for VM's, i see no point in using freenas.
Just create vm's in vmware and give them storage from the 4tb hdd.
Also please note, if using 1 hdd you will have very little iop's, about 120-160. You might get away with 3-4 vm's.
Can you get a hp microserver g8? it can fit 4 3.5 inch hdd's. Celeron cpu will be fine for freenas, and 8 gb ram.
Or Dell T100 or HP ML 110 G6. These 2 models will consume 60-80 wat's.
I used G8 microserver when i was o buget and never had a problem
G8 microserver with 4 hdd's and 16 gb ram consumed max 68 wat's.
That G6 you have consumes about 140-160w.

I would go for the ML 110 G6, had 1 of thoes, the onboard card is AHCI, it worked perfectly with freenas
only problem that it uses ECC UDIMM, ECC UNBUFFERED DIMM, ECC UNREGISTERED DIMM memorys, so try to get one with at least 8gb of ram, ECC UDIMM is rare.

 

[Holt Andrei Tiberiu]

https://www.ebay.com/itm/HP-ML110-G...h=item2f21c52557:g:kDYAAOSw2stbkwsP:rk:4:pf:0

 

[Maury]

I will progressively add more 4TB disks as I need more space. Also, I was actually thinking of sticking with my G6, which I managed to get off ebay for around $100. I don't really want to ditch it right after buying it.

 

[Holt Andrei Tiberiu]

I will progressively add more 4TB disks as I need more space. Also, I was actually thinking of sticking with my G6, which I managed to get off ebay for around $100. I don't really want to ditch it right after buying it.

You don't have to dich it. Just get another box for storage, or in time make it storage and get a dl360e g8, it will use 60 to 70 wats with 2 x E5-2450L cpu.
I have 3 in production, and after all firmware and bios update they consume 67 w / hour, and they work with vmware 6.7.