My Dream System (I think)

[Mlovelace]

I have not tested that and I'm likely not to since without the LSIprovider I would not be able to monitor the hardware in the first place, as I understand it. Maybe I need a different LSIprovider file? But I've wasted enough time on this and it's time to just call it a day. I think the goal would be to get an email if there were something wrong but that doesn't seem to be an option for the setup I'm using.

I will be rebuilding my Sophos VM to make the hard drive smaller, from 80GB down to 40GB becasue after all this time running Sophos, I've never used more than 10% of the drive. I can always increase it's size if needed but I have found out that I cannot shrink a flat file and that ESXi cannot create a thin file:(.

I also need to finish restoring my configuration and such. Today with it being one of the hottest days in my area, I'll stay inside for the most part and work on the ESXi server.

Hey Joe, FYI you can shrink a thin vmdk by doing either an offline V2V of the VM or a Sdelete and storage vmotion. Offline V2V, use the standalone converter and chose advanced on the disk properties and select space used (I think that's the descriptor) and it will reduce the size of the target VM's vmdk to only occupied size.

Here is a decent walk-through on the Sdelete process. http://wahlnetwork.com/2014/03/05/reclaim-unused-disk-space-nfs-storage/

 

[joeschmuck]

Thanks but I do know how to shrink a thin disk and already have used the VMware vCenter Converter Standalone Client for my VMware Workstation thin clients in the past, it's the thick disk that I can't shrink, and I gather no one can shrink. So I ended up creating a thin disk for my new installation of Sophos and set the drive size to 40GB. So far the current size is 164 MB, not too bad but we will see what it grows to. I have lots of drive space but that isn't any reason to be frivolous. I've also configured it to have 2 CPUs (1 core, 2 threads) and 4GB RAM, which are the same as what I've been using for a while now. I restored the backup configuration file which saves a ton of time of course.

So, restoration is complete. Now I may redo the FreeNAS VM to change it from a thick disk into a thin disk. I will retain a thick disk for all VMs on my tradition rotating hard drive as fragmentation is an issue there, and it's an old 1.5TB drive, not very fast in the first place. Hum... Time to see if I want to replace it. Guess I can wait for a nice sale.

 

[Black Ninja]

EDIT: However my Sophos will still be on the boot device because it's mirrored with the RAID card and the entire purpose of the RAID card was to ensure the internet functionality remains operational at all costs. Also, FreeNAS will be on the same drive too, just because it doesn't use much space and it is also valuable to me, well my data is valuable. I don't need any other VMs on the boot device.

That's exactly what I would do , for the exact reasons you mention. Especially you sophos you don't want that to stop if it's you router.

I was tempted to install my pfsense in VM and even tested it worked fine , but I gave up that idea away after reading advices about it and thinking about leaving all eggs in one basket and reboot your router every time when host needs to be rebooted , etc. but that's besides the point. I would definitely put sophos on redundant storage. You can still add more drives on non redundant storage if you like you know. I have a DVR (for IP cameras recordings) that is actually a VM on single 1TB disk. And since it's on different datastore that disk could die and could be replaced , without needing to touch other VM on other datastores.

 

[Black Ninja]

RT-AC68W router which is currently setup as an AP

That's what happened with my old router (RT-N66U) when pfsense came in service.

 

[Black Ninja]

Hey Joe, FYI you can shrink a thin vmdk by doing either an offline V2V of the VM or a Sdelete and storage vmotion. Offline V2V, use the standalone converter and chose advanced on the disk properties and select space used (I think that's the descriptor) and it will reduce the size of the target VM's vmdk to only occupied size.

That's possible but at at least with my experience didn't work very well. It did worked on a outside with the converter and all that, but inside the VM results were not good,Windows did panic and crashed. It might work better with other OS. I think is a bad idea to shrink installed VM storage, considering how hard is for any software to understand the file from outside, what the OS inside the VM consider actually empty.

 

[Black Ninja]

So, restoration is complete. Now I may redo the FreeNAS VM to change it from a thick disk into a thin disk. I will retain a thick disk for all VMs on my tradition rotating hard drive as fragmentation is an issue there, and it's an old 1.5TB drive, not very fast in the first place.

You can use thin on SSD , but on HDD thick will be better not just for fragmentation , but speed will drop same way like the raid driver . And I mean use thick eager zero, cause thick lazy will not do much different from think but just reserve the space.

If you need fast and cheap HDD, get WD10EZEX (from 2012 can't be newer or older). Microcenter has it for $49 could be less somewhere. It's a single platter 1TB WD that can do around 200MB/s read and write. I don't think anything faster for less money exist, in fact you can spend much more for higher level drive (WD black or enterprise) and end up with slower drive.

 

[AhLove]

Conclusion:
If someone really wants to create an ESXi system and run FreeNAS on it, I say "Go For It", just be aware of the risks and the discipline required. This was a great experience for me and I would do it again, but some things would be done differently of course. Right now I'm not using any of the motherboard SATA ports, what a waste and a better motherboard selection would have benefited me here.

I'm nearly on the same path as you are 1/2 a year later. I am seriously considering a new esxi build. I prefer SuperMicro for my homelab server motherboards. All my old SM boards have been rock solid. My X9SCM is still kicking strong with VMs for 5 years now. Tho I am feeling the squeeze of 32G max ram and looking for something that can handle 64 or more for my current needs. Which is the reason for my current upgrade and it will allow me to redistribute my VMs as well. Plus the ability to play with FT/HA with my non-passthru VM is a bonus.

If you had to choose another motherboard knowing that, what would be your preferred board now? What other things would you do differently? I'm always curious to finding out that path as well. Like searching out a 'What 10 things I wished I known before starting this game/taking this trip/if I was younger' post.

Would you stick with a Supermicro X11 board? If so what would be your choice now?

 

[joeschmuck]

If you had to choose another motherboard knowing that, what would be your preferred board now? What other things would you do differently? I'm always curious to finding out that path as well. Like searching out a 'What 10 things I wished I known before starting this game/taking this trip/if I was younger' post.

Would you stick with a Supermicro X11 board? If so what would be your choice now?

Well unfortunately I'm not smart enough to make a decision on what kind of motherboard would be good for your situation but you likely have an idea on how much RAM you would need and any other dream components you desire since you have been running ESXi for 5 years now. You are an expert compared to me. My build changed during the entire process but if I could go back and make one change, it would be the motherboard so I could select one which had SATA/SAS ports which I could place in pass-through mode. Otherwise I like my X11 board just fine. It is a bit slow at booting up but there are a ton of checks it's doing so not much I can do about that.

I was lucky because I had quite a few folks here that were willing to give me good advice and be willing to tell me when I was going down the wrong path.

Good luck on your upgrade, whenever you build it.

 

[Black Ninja]

Hi joeschmuck,

the forum was don again and I couldn't check on your progress with the ESXI and raid card. Have you found anything new ?
In a mean time I have another idea: How about if you install only the driver for showing the raid health (lsi provider) and for full management use the Windows software from inside a VM ?

P.S. I pull a drive on my raid aray to see if health will change in ESXI storage and indeed it does. So if there is a problem it will shown in the vsphere "storage" so future action can be taken.

 

[joeschmuck]

Nope, I haven't done anything more and I am likely to just leave well enough alone. While I'd like it to toss me an email telling me there is a problem, I know with the specific RAID card I have, that will not be an option.

Now if you find the right combination of software, please let me know but I'm not likely to work this problem again for a while. But I am curious what would happen if I disconnected one of the drives just to see what would happen.

 

[Black Ninja]

Nope, I haven't done anything more and I am likely to just leave well enough alone. While I'd like it to toss me an email telling me there is a problem, I know with the specific RAID card I have, that will not be an option.

vCenter can do that, but it's just not justifiable to use it just for that. If raid card is visible from inside a VM , you can install Windows , software and emails will easy too. I just have a hard time to make it visible, but I'll work on that when other project(new router) is done.

But I am curious what would happen if I disconnected one of the drives just to see what would happen.

That's why I post the screenshot to show you what will happened so you don't have to mess up your system I have this done in test system so I have no problem doing that. Everything is working fine after you pull the drive, but if you put the same drive back (instead of new one) raid will see it as bad drive and it won't start rebuilding the raid array automatically.This can be easy from raid bios , but we hate reboots, don't we ? So I am looking for a way to manage the raid card from VM (windows) so no reboot needed.
Especially if my router is on VM there.;)

 

[joeschmuck]

So I have not looked into using Windows to monitor the RAID card. But currently if I install the LSIprovider software, which I believe is required to be able to see the RAID card, the system throughput drops like a rock. Or are you saying I don't need to install the LSIprovider software.

 

[Black Ninja]

So I have not looked into using Windows to monitor the RAID card. But currently if I install the LSIprovider software, which I believe is required to be able to see the RAID card, the system throughput drops like a rock. Or are you saying I don't need to install the LSIprovider software.

Perhaps LSI is needed to be able to show raid card to anything outside it's own bios. But I am not 100% sure.
It seems is needed for ESXI to show health in "storage" but I am not sure if that's what's needed for the raid card management inside a VM. I hope not. Less you install the better (for anything) if you ask me.
My first attempt (with both drivers installed) to manage in VM was not successful. I drop both firewalls as instructed , but software still didn't see it. I haven't try since then, but since I know it works, I'll just have to find the way to make it work.

 

[Mlovelace]

Nope, I haven't done anything more and I am likely to just leave well enough alone. While I'd like it to toss me an email telling me there is a problem, I know with the specific RAID card I have, that will not be an option.

Now if you find the right combination of software, please let me know but I'm not likely to work this problem again for a while. But I am curious what would happen if I disconnected one of the drives just to see what would happen.

You can setup SNMP on the host and a SNMP trap to report the errors reported by the host. This is very common practice in enterprise to have a central reporting server/service to monitor the hardware. We use solarwinds/Orion as an example.

 

[Black Ninja]

I resolve the issue with VM not being able to see raid card. Now all works and by firing a VM (windows) with raid management software installed I can have a nice full GUI control over the raid card. I was able to update the firmware of the raid controller this way. I was little worried cause the VM and ESXI is running on the same volume on the raid card , but it worked , firmware was updated and just needed reboot. I guest there isn't much more I could of ask on installation part.

Now I'll start working on performance issue. If You come up with something in a mean time let me know.

 

[Black Ninja]

We use solarwinds

I do remember some great free tool (gadget) they offer to monitor esxi-VM's. I am sure their paid product are even cooler, but I still find this free tool great. Thanks for mentioning it.

 

[joeschmuck]

Let me know if you solve the performance issue because that is really a key concern to me.

I resolve the issue with VM not being able to see raid card. Now all works and by firing a VM (windows) with raid management software installed I can have a nice full GUI control over the raid card.

Once you are all done, please share any configuration changes required to make this work. I'm also curious if the software tools could be used on Ubuntu because I run a VM all the time, just crunching SETI@HOME numbers all the time. It also gives me a linux platform to play with.

 

[Black Ninja]

Let me know if you solve the performance issue because that is really a key concern to me.

Finally I got the time to put my theory to the test, and I can't wait to share the great news:

Remember the idea was from fresh esxi install to install only the "lsiprovider" driver, but not the megaraid driver, or anything else. Because when I install both drivers, write speed drops to 80-90MB/s from being over 200MB/s without any drivers.

And here is before and after results. I am so happy with the results. Now Esxi will monitor the card health and in the same time , performance is not affected. And I did check the write performance with actual 100GB of data copied over the network to the VM , so it's not just a synthetic benchmark, it can sustain sequential 210MB/s write.

I'm also curious if the software tools could be used on Ubuntu because I run a VM all the time

There is a file for management from linux(almost matching the windows one by size) in "lsi downloads" so I would assume and hope it will do the same thing, but have not tried it.

 

[joeschmuck]

Well that is great news! Could you please specify exactly which lsiprovider file you used? I will likely try this out this weekend. For me I will need to uninstall the UPSMON driver and then install the lsiprovider, I could use the override but I would rather be overly cautious.

Thanks for the great work.

 

[Black Ninja]

I used the latest driver from LSI web site. See pic.

P.S. You keep mentioning this UPSMON driver , but I am not sure that I know what that driver is for and what has to do with the raid card ?