Lab improvment

[bogdan0300]

Hello,

My name is Bogdan and I am working as IT Administrator. Beside working in the field for 5 years, I have for study a home lab. My main focus is virtualization and I already tested VMware pretty much, now I want to focus more and deep on cloud and virtual desktop.
My problems are with ISCSI/NFS storage because when I test it looks very crappy... I have mounted in a virtual machine ( win7 x64 ) a second disk from Freenas 9.1 and test it... I got between 50-60 for read and 20-40 for write... same on NFS. Some times I get even worse and always the transfer is not stable... it have spikes, not linear.
I made all kinds of tests.... whit all configuration of RAID... ( mirror, stripe, z1, z2, Raid 10 ... all whit no WOW speeds... max 70-80 MB ) I spend days doing those tests. And A LOT OF NERVE... I am so tired. :(

My lab is made from:

Host 1:

1 x AMD FX 8320 - 8 core
1 x Asrock 970 Extreme 3
4 x 8Gb 1333Mhz
1 x 250 Gb
1 x onboard NIC 8111E
2 x PCI-E NIC 8111E
1 x PCI-E Intel 1000CT
1 x Sirtec 420W PSU

Host 2:

Same as 1 but the Intel NIC is a PCI 1000PRO

NAS1:

1 x MSI AMD E-350 Motherboard and CPU
2x 4GB 1333Mhz
6 x 1 TB Western Digital Black ( 4x Enterprise Storage, 1x Caviar Blue and 1x Caviar Green )
1 x onboard NIC ( I think 8111 ) 1 GB
1 x TP-LINK 8111E, 1 GB PCI-E
1 x Sirtec 420W PSU

On this NAS I use 2 x 1TB in Mirror for active VMs and 4 x 1TB in RAID Z1 for Back-up

NAS2:

1 x Intel Core2 e5900 at 2,9 Ghz 2core
1 x Asus P8h67-M Motherboard
2 x 4GB 1333Mhz
2 x 2GB 1333Mhz
2 x 250GB HDD ( 1x WD Blue and 1x Segate... ) Diffrent but working good!
4 x 500GB Segate Constellation ES ( I think those are not compatible whit my hardware or Freenas... the most crapy speeds... in all kind of config )
1 x onboard NIC 8111E 1GB
1 x TP-LINK 8111E, 1 GB PCI
1 x Sirtec 420W PSU

Here i use 2 x 250 GB HDD in mirror, 2 x 500 GB mirror and another 500 GB mirror. All for active VMs.

NAS - ESXI Hosts SWITCH :

1 x D-link DGS-108 unmanaged - 8 x 1GB port ( dedicated for ESXI - NAS storage )

First, thank you for reading all :))

My big problem is that I need to squeeze a lot more power and IOPS from the storage/networking to be able to implement more virtualization solutions.
What I need from you it is to tell me where are my weak spots... where it will be best to improve?

I have a last 300 - 400$ to invest in this lab... my last money, I know it is not much but with those i have to make better :) !

So... I think my weak spots are NIC`s, and maybe the switch... plus some bad config maybe...
What do you think? If I buy 4 more Intel 1000CT, 2 SSD for L2ARC and ZIL ( on one NAS) it will be better?
Or what should I do to improve performance whit this ....buget?

I have one more switch that it is not used a Linksys SRW2008 8 x 1GB could be better than D-link?...

I would upgrade this rig, whit:

NAS2:

1 x Intel Core2 e5900 at 2,9 Ghz 2core
1 x Asus P8h67-M Motherboard
2 x 4GB 1333Mhz
2 x 2GB 1333Mhz
1 x 60GB SSD ( as L2ARC )
1 x 60GB SSD ( as ZIL )
2 x 250GB HDD ( 1x WD Blue and 1x Segate... )
4 x 500GB Segate Constellation ES ( I think those are not compatible whit my hardware or Freenas... )
1 x onboard NIC 8111E 1GB
1 x TP-LINK 8111E, 1 GB PCI
1 x Intel 1000 CT
1 x Sirtec 420W PSU

And the rest of Intel 1000 CT , one in each rig.

Bad or good ideea?... Any other ideas are welcome. :)

Do you know if Segate Constellation ES have any problem with Frenass?... because I get lower speed than 2 x 250 HDD ( one from WD another from Segate in RAID 1 and preaty old HDD... )

If you have another comments about what am I doing wrong, or how to use better my config with ESXI please fell free to share.

Thank you very much!

 

[jgreco]

You have far too little RAM to support L2ARC, and you should try setting sync=disabled as an experiment to see if SLOG might be useful. Generally speaking it is rough to use FreeNAS for VM storage unless you throw resources at it.

 

[bogdan0300]

Hi, thanks,

True, sorry for that mistake, but at least SSD for ZIL ? I know you dont even bother to replay to someone who doesn`t wish to put more RAM in the rig but I really dont have the money and I need more speed than redundancy... it is a lab....not production.

And as a typo mistake... The H67 MB it has a Pentium G2020 not a Pentium Core2 5900 :)) sorry for that...

Still waiting suggestions... I am new in the field... i still learn... from people who like to share experience so don`t be harsh on me :)

 

[cyberjock]

We won't be harsh, but you need to realize that without the proper hardware you shouldn't expect miracles.... and that's what you seem to be wanting.

ZFS is a technology that you have to apply properly. This means buying adequate hardware along with proper administration of the zpool and datasets. For many this proves to be too much. If you aren't capable of spending money to make your hardware adequate you should look elsewhere.

None of your hardware seems appropriate for FreeNAS, and the cost of the upgrades is going to be fairly significant. So FreeNAS is probably not for you if you aren't ready to dump some money on this problem.

 

[jgreco]

but at least SSD for ZIL ?

First off, say "SLOG". Second off, if sync=disabled doesn't solve the problem, then neither will an SSD.

As noted, if you're not prepared to throw resources at it, it isn't going to work well for VM storage.

 

[joeschmuck]

These guys speak the truth, you need better hardware and much more RAM. Throwing a SSD at it is a waste of your money.

As for the Constellation ES drives, those are enterprise level drives and should work just fine. You say they are slower but what are you talking about, the SATA transfer rate or do they actually perform slower in your machine? (need numbers)

 

[bogdan0300]

1. Ok, I dont wait for miracles ( Grinch, Christmas is comming :)) ) just hope you can tell me, using that small amount of money what would be the best invest...and I understand.... I will go for RAM 16GB not SSD and 4 Intel NICs. I read Cyberjock lecture for noobs and was very good.... thank you for your involvment!
2. But if I make one from two ( I am a persistent person... I have to find a way :)) )?... Let`s say:

1 x Intel Pentium G2020
1 x Asus H67 MB
2 x 8 GB 1333 RAM
2 x 4 GB 1333 RAM total of 24 Gigs
2 x Intel Gb NIC`s + one onboard 8111E for mangement...
4 x 1 TB WD Enterprise 2 x mirror or a RAIDZ2 for VMs
2 x 500 GB Segate mirror for VMs.

Is this sistem going to work?... so I have 8 GB for ZFS and rest of 16 GB for performance..... ?
Or should I go for Freenas but not using ZFS?..... just for lab testing performance!

Thank you for patience!

 

[bogdan0300]

Hello Joeschmuck, I meant to say they are very slow in my hardware... and not for VM... in general... with 2 x 250 mirror old drives i have speeds of 45 - 50 MB read, with 2 x 1 TB WD Black I have 60 - 70 MB read and write 40 - 50 MB. But with any kind of RAID I use for those Constellation ES, it will not go more than 20 for write and 30 - 40 for read... plus, the graphs are not linear... i have big fluctuation rates.

Thank you very much!

 

[joeschmuck]

With respect to the ES hard drives, I'd have to ask you a few questions...
1) Are you sure it's the drives and not the computer they are installed in? In other words, if you replaced the WD Black drives with two ES drives and configured them the same (yes capacity would be less of course) and then tested the throughput, how would it compare?
2) Just to rule out hard drive issues, have you run and extended SMART test on those drives and checked out the results?

Because you state that the graphs are not linear it could be a few reasons...
1) The CPU is not fast enough to handle the ZFS quickly.
2) You have NIC issues.
3) The system you are connected to is having speed/protocol issues (this does happen).

As for your system choice... I think you will have to test it out to see how it works.
1) You generalized it in saying 8GB RAM for FreeNAS and 16GB RAM for performance and that is about true. Any RAM that FreeNAS does not use for itself is used as a cache and that is where the performance increase comes in. I can't tell you if you have enough RAM to increase the performance of the VM.
2) Are you using ECC RAM? Not that it matters at this point.
3) The CPU, I have concerns if it would be fast enough for your VM applications.

Also, to get maximum throughput you need to configure your pool correctly as well.

 

[bogdan0300]

Well... 1. the ES hard drives, were tested in the same rig as WD and 250 GB.... whit the same conditions, this is why I think it is strange.
2. Yes, they pass 2 SMART tests, no problem. I dont say it is from HDDs but maybe it is a incompatibility somewhere... I will try to test them in more other rigs...

3. The CPU I think it`s ok, because it is used at 20 - 40 % capacity.
4. This I think is true.... I have a lot of Realtek 8111E, so I will buy some Intel 1000CT.
5. Maybe... but I have tested days...and weeks... with different system.... VMs with attached storage from NAS, directly from ESXI with I/O Analyzer, and directly attached via NAS - physical Windows system.
6. No, i dont have ECC.
7. Well, the CPU was never used more than 40 - 60 % ant it had ... 6 VMs plus in that time I was making Vmotion whit 2 of them.
8. Yes, configuration of the pools, I didnt find in the manual the right way... can you give me a link to a good tutorial or discution about configuring pools?

Thank you!