My Dream System (I think)

[Dice]

There doesn't appear to be any way to detect a failing module through ESXi if you do that, so it's not really that neat a solution.

Point noted.

For free ESXi, the most powerful thing is probably to manage your configuration via the CLI and maintain a record of what you've done.

Some googling suggests various 'extract your config from CLI' kind of deals.
I've found but not yet attempted the following, since they require vCLI.. that I do not find a downloadable link to before hitting <maximum limits of frustration> with the VMware site.
http://www.vladan.fr/how-to-backup-esxi-4-1-configuration/
http://www.vladan.fr/free-esxi-configuration-backup-tool/

On the upside, I've a very simple configuration that should not take more than a few minutes to click through (with proper notes on what needs to be done).
3 subnets, 3-4 total planned VM's.
Spawning on the basic configuration, at least I think I've not touched on other options than:
- Passthrough of HBA/NIC
- Resource distribution/locking RAM/Locking CPU
- Networking, 3 subnets, (WAN/LAN/VMXNET locally on host)
- Boot ordering for when it is desired, I like it manual as of now.

....ehrm, did I forget anything substantial?

Cheers /

 

[gpsguy]

If you have a Windows machine, look at PowerCLI (free). Backing up the configuration is easy.


Sent from my iPhone using Tapatalk

 

[jgreco]

Point noted.

Some googling suggests various 'extract your config from CLI' kind of deals.
I've found but not yet attempted the following, since they require vCLI.. that I do not find a downloadable link to before hitting <maximum limits of frustration> with the VMware site.
http://www.vladan.fr/how-to-backup-esxi-4-1-configuration/
http://www.vladan.fr/free-esxi-configuration-backup-tool/

Hahaaha, yeah, ... fun.

Ironically, I still don't really understand the point of "vCLI" after all these years. If I want CLI, I will log in and use CLI.

So by "manage via CLI", what I really mean is something like this. Instead of clicking around the UI and making all your changes that way, instead you do the bare minimum to get networking working and then boink it with a script like this (which is actually a redacted/minimized version of what we use here).

Code:

vim-cmd hostsvc/maintenance_mode_enter


vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

esxcli system module parameters set -m tcpip4 -p ipv6=0

ip=100
esxcfg-vswitch -a vSwitch1:256
esxcfg-vswitch -L vmnic1 vSwitch1
esxcfg-vswitch -a vSwitch2:256
esxcfg-vswitch -L vmnic2 vSwitch2
esxcfg-vswitch -L vmnic4 vSwitch2
esxcli network vswitch standard policy failover set --active-uplinks vmnic2 --standby-uplinks vmnic4 --vswitch-name vSwitch2
esxcfg-vswitch -a vSwitch3:256
esxcfg-vswitch -L vmnic3 vSwitch3
esxcfg-vswitch -L vmnic5 vSwitch3
esxcli network vswitch standard policy failover set --active-uplinks vmnic5 --standby-uplinks vmnic3 --vswitch-name vSwitch3
esxcfg-vswitch -a vSwitch4:256

esxcfg-vswitch --add-pg "gen-mgmt-400" vSwitch2
esxcfg-vswitch --p "gen-mgmt-400" --vlan 400 vSwitch2
vim-cmd hostsvc/net/refresh
vim-cmd hostsvc/net/portgroup_set --securepolicy-promisc=false vSwitch2 "gen-mgmt-400"

esxcfg-vswitch --add-pg "res-mgmt-401" vSwitch3
esxcfg-vswitch --p "res-mgmt-401" --vlan 401 vSwitch3
vim-cmd hostsvc/net/refresh
vim-cmd hostsvc/net/portgroup_set --securepolicy-promisc=false vSwitch3 "res-mgmt-401"

esxcfg-vswitch --add-pg "storage0-450" vSwitch2
esxcfg-vswitch --p "storage0-450" --vlan 450 vSwitch2
vim-cmd hostsvc/net/refresh
vim-cmd hostsvc/net/portgroup_set --securepolicy-promisc=false vSwitch2 "storage0-450"

esxcfg-vswitch --add-pg "storage1-451" vSwitch3
esxcfg-vswitch --p "storage1-451" --vlan 451 vSwitch3
vim-cmd hostsvc/net/refresh
vim-cmd hostsvc/net/portgroup_set --securepolicy-promisc=false vSwitch3 "storage1-451"


esxcfg-vswitch --add-pg "VMstorage0-450" vSwitch2
esxcfg-vswitch --p "VMstorage0-450" --vlan 450 vSwitch2

esxcfg-vmknic --add --ip 10.68.50.${ip} -n 255.255.255.0 VMstorage0-450
vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-active=vmnic4 vSwitch2 "VMstorage0-450"



esxcfg-vswitch --add-pg "VMstorage1-451" vSwitch3
esxcfg-vswitch --p "VMstorage1-451" --vlan 451 vSwitch3

esxcfg-vmknic --add --ip 10.68.51.${ip} -n 255.255.255.0 VMstorage1-451
vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-active=vmnic3 vSwitch3 "VMstorage1-451"


esxcfg-vswitch --add-pg "VMmanagement-400" vSwitch2
esxcfg-vswitch --p "VMmanagement-400" --vlan 400 vSwitch2

vim-cmd hostsvc/vmotion/vnic_set vmk0


esxcfg-vmknic --add --ip 10.68.40.${ip} -n 255.255.255.0 "VMmanagement-400"

esxcfg-vswitch --del-pg "Management Network" vSwitch0

esxcli storage nfs add --host 10.68.50.121 --share=/mnt/storage1/ISO --volume-name=iso --readonly
esxcli storage nfs add --host 10.68.50.121 --share=/mnt/storage1/Scratch --volume-name=scratch
esxcli storage nfs add --host 10.68.50.121 --share=/mnt/storage1/VMarchive --volume-name=vmarchive
esxcli storage nfs add --host 10.68.51.122 --share=/mnt/storage2/backups --volume-name=backups


esxcli software vib install -d /vmfs/volumes/iso/Data/Virtualization/VMware/vSphere/Driver/Chelsio/ESXi5.5/cxgb4-driver-1.3.0/cxgb4-driver-1.3.0-offline_bundle-1948803.zip
esxcli software vib install -d /vmfs/volumes/iso/Data/Virtualization/VMware/vSphere/Driver/Chelsio/ESXi5.5/csiostor-iscsi-1.0.0.0/csiostor-iscsi-1.0.0.0-offline_bundle-2136558.zip

echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

esxcfg-advcfg -s 1 /UserVars/SuppressShellWarning

esxcfg-advcfg -s 1 /Net/ReversePathFwdCheckPromisc

esxcfg-advcfg -s 0 /Mem/ShareForceSalting
esxcfg-advcfg -s 40 /Mem/MemZipMacPct

esxcli network firewall ruleset set --enabled=true --ruleset-id=iSCSI
esxcli iscsi software set --enabled=true
esxcli storage nmp satp set --default-psp=VMW_PSP_RR --satp=VMW_SATP_ALUA
esxcli storage core claimrule load
esxcli storage core claimrule run

stornets=`esxcfg-vmknic -l | grep VMstor | awk '{print $1}' | sort | uniq`
vmhba=`esxcfg-scsidevs -a | grep iSCSI | awk '{print $1}'`
for i in ${stornets}; do
esxcli iscsi networkportal add -A ${vmhba} -n ${i}
done

esxcli iscsi adapter discovery sendtarget add -A ${vmhba} -a 10.68.50.188:3260
esxcli iscsi adapter discovery sendtarget add -A ${vmhba} -a 10.68.51.188:3260
esxcli iscsi adapter discovery sendtarget auth chap set --direction=uni --authname=diskstation0 --secret=24ba291b67b491b5 --level=required --adapter=${vmhba} -a 10.68.50.188:3260
esxcli iscsi adapter discovery sendtarget auth chap set --direction=uni --authname=diskstation0 --secret=24ba291b67b491b5 --level=required --adapter=${vmhba} -a 10.68.51.188:3260
esxcli iscsi adapter discovery rediscover --adapter=${vmhba}
esxcli storage core adapter rescan --adapter=${vmhba}

esxcli network ip dns server add --server=206.55.64.70
esxcli network ip dns server add --server=206.55.64.71
esxcli network ip dns server add --server=206.55.64.68
esxcli network ip dns server add --server=206.55.64.69
esxcli network ip dns search add -d sol.net

(
echo "restrict default kod nomodify notrap nopeer"
echo "restrict 206.55.75.128 mask 255.255.255.252"
echo "restrict 127.0.0.1"
echo "server 206.55.64.76"
echo "server 206.55.64.77"
echo "server 206.55.64.78"
echo "server 206.55.64.79"
echo "driftfile /etc/ntp.drift"
) > /etc/ntp.conf

esxcli network firewall ruleset set --enabled=true --ruleset-id=ntpClient
chkconfig --add ntpd
chkconfig ntpd on

Primary difference being that we have about a hundred vlans which are omitted.

Now, to my admittedly crazy mind, this is a lot more gorgeous than trying to "back up" the configuration, because this actually provides some readability as to what's been done to the system, and I know that I can blow away a node and reload it with a newer hypervisor version and get right to where it needs to be in about ten minutes.

On the upside, I've a very simple configuration that should not take more than a few minutes to click through (with proper notes on what needs to be done).
3 subnets, 3-4 total planned VM's.
Spawning on the basic configuration, at least I think I've not touched on other options than:
- Passthrough of HBA/NIC
- Resource distribution/locking RAM/Locking CPU

That's not actually ESXi configuration. It's specified in your VM's.

- Networking, 3 subnets, (WAN/LAN/VMXNET locally on host)
- Boot ordering for when it is desired, I like it manual as of now.

....ehrm, did I forget anything substantial?

Cheers / Dice

 

[Mirfster]

Nice script, might pirate some of it and see if I can make a working test. By chance has any tried or looked into this ? Another idea may be just to do a CloneZilla of the ESXi drive once you have it configured; but I am unsure if that would be very "modular". More things to add to the "want to test out" list...

Edit: Totally missed that jgreco already mentioned "vicfg-cfgbackup"

 

[Spearfoot]

Here's a script I use to download an ESXi host's configuration file - you can click here to download it from pastebin.com. It uses the command 'vim-cmd hostsvc/firmware/backup_config' to generate a configuration backup and downloads the result:

Code:

#!/bin/bash
#########################################################################
#
# save-esxi-config.sh
#
# Generates and downloads the configuration of a given ESXi host
#
# Requires SSH support on the ESXi host
#
# The ESXi hostname must be resolvable; may work with IP addresses but
# has not been tested with them.
#
#########################################################################

# Name of ESXi host - edit to suit your system, or make into a command-line parameter
VM_HOST="felix.ncs"

# Directory where we download the configuration file, edit as needed 
VM_DEST_DIR="/root/work"

# Generate the ESXi configuration file and extract the URL we will use to download it from
# the host. Note that we have to replace the '*' in the URL with the hostname

echo "Generating configuration file on ESXi host $VM_HOST"
VM_CONFIG_URL=$(ssh root@${VM_HOST} vim-cmd hostsvc/firmware/backup_config | awk '{print $7}' | sed -e "s/*/${VM_HOST}/")

# Form target filename
VM_DATE=$(date +%Y%m%d%H%M%S)
VM_CONFIG_FILE="$VM_DEST_DIR"/"$VM_HOST"-configBundle-"$VM_DATE".tgz

echo "Downloading $VM_CONFIG_URL to $VM_CONFIG_FILE"
wget --no-check-certificate --output-document=${VM_CONFIG_FILE} ${VM_CONFIG_URL}

The output looks like this:

Code:

[root@boomer] ~/work# ./save-esxi-config.sh
Generating configuration file on ESXi host felix.ncs
Downloading http://felix.ncs/downloads/52d185c7-cf85-231d-32fd-d5f4cd9d807f/configBundle-felix.tgz to /root/work/felix.ncs-configBundle-20160605101149.tgz
--2016-06-05 10:11:49--  http://felix.ncs/downloads/52d185c7-cf85-231d-32fd-d5f4cd9d807f/configBundle-felix.tgz
Resolving felix.ncs (felix.ncs)... 192.168.1.30
Connecting to felix.ncs (felix.ncs)|192.168.1.30|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://felix.ncs/downloads/52d185c7-cf85-231d-32fd-d5f4cd9d807f/configBundle-felix.tgz [following]
--2016-06-05 10:11:49--  https://felix.ncs/downloads/52d185c7-cf85-231d-32fd-d5f4cd9d807f/configBundle-felix.tgz
Connecting to felix.ncs (felix.ncs)|192.168.1.30|:443... connected.
WARNING: cannot verify felix.ncs's certificate, issued by 'O=VMware Installer':
  Unable to locally verify the issuer's authority.
WARNING: no certificate subject alternative name matches
  requested host name 'felix.ncs'.
HTTP request sent, awaiting response... 200 OK
Length: 31031 (30K) [application/x-tar]
Saving to: '/root/work/felix.ncs-configBundle-20160605101149.tgz'

/root/work/felix.ncs-configBundl 100%[============================================================>]  30.30K  --.-KB/s  in 0s

2016-06-05 10:11:49 (268 MB/s) - '/root/work/felix.ncs-configBundle-20160605101149.tgz' saved [31031/31031]

 

[joeschmuck]

Time for another question because I don't know better...

If I purchase an M1015 controller card and have it in "IR" mode, and on my ESXi machine, I should be able to create a bootable pair of SSDs for ESXi as I understand it and it will truly run ESXi even if there is a failure of one of the SSDs. Also I could create additional mirrors for datastores.

I just want to make sure before I pull the trigger and buy a used part and the data cables. I can't re-purpose this item anywhere like I could with all my other purchases.

 

[Mirfster]

Shows "ServeRAID M1015 SAS/SATA Controller"on the VMware Compatibility Guide. But, I have not used it personally. I have used Perc 6i/R, MegaRaid 9260s as well as Perc H700s. Come to think of it I never tried my Perc H200s... I guess that I was reluctant to use a controller that did not have a BBU.

Of course right now I have a system with the "Syba Dual mSATA SSD to SATA III RAID 2.5" Enclosure Components SY-ADA40090"; however I had one just crap out and finding that they are not as robust as the "StarTech.com Dual mSATA SSD to 2.5-Inch SATA RAID Adapter Converter" I have been using so I am going back to those for these "edge cases".

Sorry if this doesn't outright answer your question, but I *think* you will be fine with that...

 

[jgreco]

Time for another question because I don't know better...

If I purchase an M1015 controller card and have it in "IR" mode, and on my ESXi machine, I should be able to create a bootable pair of SSDs for ESXi as I understand it and it will truly run ESXi even if there is a failure of one of the SSDs. Also I could create additional mirrors for datastores.

The SSD's would actually be a RAID1 datastore, they'd just also happen to have an ESXi boot partition on them. Either drive failing results in things still working. Plus you can then insert a replacement drive and it'll rebuild, no downtime.

I just want to make sure before I pull the trigger and buy a used part and the data cables. I can't re-purpose this item anywhere like I could with all my other purchases.

I'd be happy to send you a Dell H310 and cables to play with. One of Dell's variants on the M1015. I'm literally swimming in this stuff so lemme know.

 

[joeschmuck]

I'd be happy to send you a Dell H310 and cables to play with. One of Dell's variants on the M1015. I'm literally swimming in this stuff so lemme know.

You have made a similar offer to me in the past but I'm apt to retain them and never give them back. So if you say things should work fine, then I actually trust you and I can go purchase what I need, or we could discuss me purchasing the your loaner hardware. I wouldn't want to build the system multiple times basically since as i understand it, the drives cannot be moved to another controller unless they are the same make/model.

 

[Mirfster]

Ah, just drive on over to Equinox bang on the doors and tell them you are there to see the Grinch... :)

 

[jgreco]

You have made a similar offer to me in the past but I'm apt to retain them and never give them back. So if you say things should work fine, then I actually trust you and I can go purchase what I need, or we could discuss me purchasing the your loaner hardware. I wouldn't want to build the system multiple times basically since as i understand it, the drives cannot be moved to another controller unless they are the same make/model.

You misunderstood. I never said "loaner". More of an "in repayment for all the times you've made me laugh."

 

[Mirfster]

You misunderstood. I never said "loaner". More of an "in repayment for all the times you've made me laugh."

/Starts to try and recall all the funniest knock-knock jokes.... ;)

 

[anodos]

/Starts to try and recall all the funniest knock-knock jokes.... ;)

For some reason I doubt those will appeal to a grinchy sense of humor.

 

[joeschmuck]

Well don't worry, I'm full of stupid things to say to get Mr. Green w/Red eyes to roll his eyes or laugh at my stupid mistakes.

Which brings me to another question... So I have two different models of SSDs, both "120 GB" or "256 GB" (so they claim). Using the RAID controller, how close does each drive have to match in capacity to be used in a RAID1? So could I put together a 256GB drive and a 200GB drive and I would end up with 200GB overall for storage or will the controller reject the drives because of the size difference? I was reading some manuals on the controllers but couldn't find that answer but I'm sure it's out there. Maybe more looking.

 

[maglin]

It will use the smallest drives capacity for all drives in every RAID1/0 or RAIDZx config.


Sent from my iPhone using Tapatalk

 

[jgreco]

It will use the smallest drives capacity for all drives in every RAID1/0 or RAIDZx config.

Which is why, for example, if you're shooting for a ~500GB SSD RAID1, there's some value in making sure that one of your SSD's is 480GB. You can RAID1 a 480GB and a 500GB SSD without issue, and then in the future if you need to replace one of the drives you can use any of 480GB, 500GB, or 512GB SSD's - with the caveat that if you replace the 480GB with something bigger, you may be restricting future choices.

 

[AlainD]

Which is why, for example, if you're shooting for a ~500GB SSD RAID1, there's some value in making sure that one of your SSD's is 480GB. You can RAID1 a 480GB and a 500GB SSD without issue, and then in the future if you need to replace one of the drives you can use any of 480GB, 500GB, or 512GB SSD's - with the caveat that if you replace the 480GB with something bigger, you may be restricting future choices.

Thanks. I'm lucky, I just created a 480GB SSD mirror. Just because those where on promotion ;-)

 

[joeschmuck]

I was hoping that the RAID1 would work like that because I can easily live with it. Now I'm going to decide what I'm going to do with my SSDs and what I should purchase. Do I buy a new 500GB SSD for my main computer and remove the 256GB SSD (same model as the 256GB drive in the ESXi machine) or buy a pair of 500GB SSDs for the ESXi machine. Fathers Day is coming up so maybe there will be a good sale. I'll keep my eye open.

 

[jgreco]

I was hoping that the RAID1 would work like that because I can easily live with it. Now I'm going to decide what I'm going to do with my SSDs and what I should purchase. Do I buy a new 500GB SSD for my main computer and remove the 256GB SSD (same model as the 256GB drive in the ESXi machine) or buy a pair of 500GB SSDs for the ESXi machine. Fathers Day is coming up so maybe there will be a good sale. I'll keep my eye open.

Well, that's the whole point of all this, in many ways. You get to a certain point and you decide that it's just not worth half-assing things anymore. Creating individual hardware platforms and worrying about how to recover each one if the hard drive fails... so 1990's. Failing drives causing you to lose data... so 1990's.

 

[joeschmuck]

So I've got everything except my 256GB SSD that should be here any day now (fathers day sales sucked this year). I've backed up all my VMs and just need to backup ESXi before I shut it down and pull the system out of the basement for the "Final Overhaul". Once this is done I don't think there will be any further upgrades except for hard drive replacement due to failure.

So with any luck I'll be rebuilding this gizmo before the weekend starts.