I recently added a Chelsio T520-CR to my FreeNAS box, with the intention of adding a Proxmox server with 10G networking in the near future, with the intention of using iSCSI to use my FreeNAS box for the storage for all my VMs.
I've run into some other issues, which I believe are down to a faulty T520 - I just replaced the original one which seemed to die after a few days with a second one, which is working thus far - more details in this thread):
https://www.ixsystems.com/community...o-have-died-after-only-2-3-days-of-use.76951/
However there's another issue that's currently just a minor annoyance, but will become a bigger issue down the track.
Here are my server specs:
Chassis: CSE-846BE16-R1200B
Backplane: BPN-SAS2-846EL1
Motherboard: Supermicro X9DRI-F
CPU: Dual E5-2640
RAM: 64GB DDR3 ECC (8x8GB sticks)
Storage Controller: LSI 9210-8i
Storage: 6x WD Red 8TB, 6x WD Red 6TB, Transcend TS256GMTE220S 256GB NVME SSD in SilverStone ECM21 M.2 to PCIe x4 Adapter
NICs: on-board Intel I350, Chelsio T520-CR
The issue I'm having is that the dedicated IPMI network port on my board doesn't work - I've spent hours trying various different configurations and permutations, and nothing made any difference, and in the end I concluded it was just completely dead, and I'd need to use the shared port - igb0.
Unfortunately I can't seem to find a configuration where having both the t520 (cxl0) and igb0 both plugged in at the same time doesn't completely break my entire network.
Before I first added the t520, I had igb0 configured to use DHCP, and a static lease configured on my pfsense router to give it 192.168.0.32.
When I added the t520, I deleted the igb0 configuration from FreeNAS, and changed the pfsense lease to give that address to the t520 instead, and deleted the old lease for igb0.
Unfortunately igb0 was still giving itself the 192.168.0.32 address, and the only way I could prevent that was by giving it a different address, but even if I give it an address on a different subnet - I was using 192.168.10.32 for testing - as soon as I plug both NICs into the network, everything breaks, as soon as I unplug igb0, everything works perfectly.
For now the server is just sitting in a corner of my office, so I can just unplug the t520 and plug igb0 back in when I need access to the IPMI, but longer term it'll be going in a small server cabinet in a large walk-in wardrobe, along with the new Proxmox server, and I'll be wanting consistent easy access to the IPMI.
For a switch, I'm using a Mikrotik CRS305, which has 4x SFP+ ports, and one 1G copper uplink.
The network layout I'm envisioning is this:
FreeNAS t520 > 10G SFP+ > CRS305
Proxmox t520 (or other 10G NIC) > 10G SFP+ > CRS305
FreeNAS IPMI > 1G copper > CRS305 (using a 1G copper tranceiver)
Proxmox IPMI > 1G copper > CRS305 (using CRS305 1G RJ-45)
CRS305 > 10G SFP+ > second CRS305 in my office, which is then patched into the network using the 1G RJ45, leaving 3 SFP+ ports for 10G connectivity to up to 3 office PCs if and when I add 10G networking to them.
(The reason I'm planning on using two 10G switches to link the servers back to the main network despite not yet having any 10G devices at the other end is that the wardrobe has no network jacks, but both my office and the wardrobe have downlights - I can easily pull the downlight out, cut a very small notch on one edge of the hole, run some OM4 fibre through the notch, and put the downlight back in - the vanity rings on the downlights will fully cover the notch once the cable is removed, whereas a notch large enough for a cat6 cable would be too large to be covered. My office then has network jacks connecting back to the rest of the network with cat6 run through the walls and ceiling space.)
In order to make that layout work though, I need to figure out how to avoid my current issue so I can actually have both IPMI and 10G networking plugged into the same switch without breaking my network.
Is anyone able to offer any advice on how to fix this?.. It seems to me that the easiest method would be to just compeltely disable igb0 in FreeNAS, but I'm not sure how best to do that.
If this were a Linux box, I'd probably just set a udev configuration option or the like, and I know I can do similar things with FreeBSD, I'm just not sure which method would work the best and not be overridden by FreeNAS.
I've run into some other issues, which I believe are down to a faulty T520 - I just replaced the original one which seemed to die after a few days with a second one, which is working thus far - more details in this thread):
https://www.ixsystems.com/community...o-have-died-after-only-2-3-days-of-use.76951/
However there's another issue that's currently just a minor annoyance, but will become a bigger issue down the track.
Here are my server specs:
Chassis: CSE-846BE16-R1200B
Backplane: BPN-SAS2-846EL1
Motherboard: Supermicro X9DRI-F
CPU: Dual E5-2640
RAM: 64GB DDR3 ECC (8x8GB sticks)
Storage Controller: LSI 9210-8i
Storage: 6x WD Red 8TB, 6x WD Red 6TB, Transcend TS256GMTE220S 256GB NVME SSD in SilverStone ECM21 M.2 to PCIe x4 Adapter
NICs: on-board Intel I350, Chelsio T520-CR
The issue I'm having is that the dedicated IPMI network port on my board doesn't work - I've spent hours trying various different configurations and permutations, and nothing made any difference, and in the end I concluded it was just completely dead, and I'd need to use the shared port - igb0.
Unfortunately I can't seem to find a configuration where having both the t520 (cxl0) and igb0 both plugged in at the same time doesn't completely break my entire network.
Before I first added the t520, I had igb0 configured to use DHCP, and a static lease configured on my pfsense router to give it 192.168.0.32.
When I added the t520, I deleted the igb0 configuration from FreeNAS, and changed the pfsense lease to give that address to the t520 instead, and deleted the old lease for igb0.
Unfortunately igb0 was still giving itself the 192.168.0.32 address, and the only way I could prevent that was by giving it a different address, but even if I give it an address on a different subnet - I was using 192.168.10.32 for testing - as soon as I plug both NICs into the network, everything breaks, as soon as I unplug igb0, everything works perfectly.
For now the server is just sitting in a corner of my office, so I can just unplug the t520 and plug igb0 back in when I need access to the IPMI, but longer term it'll be going in a small server cabinet in a large walk-in wardrobe, along with the new Proxmox server, and I'll be wanting consistent easy access to the IPMI.
For a switch, I'm using a Mikrotik CRS305, which has 4x SFP+ ports, and one 1G copper uplink.
The network layout I'm envisioning is this:
FreeNAS t520 > 10G SFP+ > CRS305
Proxmox t520 (or other 10G NIC) > 10G SFP+ > CRS305
FreeNAS IPMI > 1G copper > CRS305 (using a 1G copper tranceiver)
Proxmox IPMI > 1G copper > CRS305 (using CRS305 1G RJ-45)
CRS305 > 10G SFP+ > second CRS305 in my office, which is then patched into the network using the 1G RJ45, leaving 3 SFP+ ports for 10G connectivity to up to 3 office PCs if and when I add 10G networking to them.
(The reason I'm planning on using two 10G switches to link the servers back to the main network despite not yet having any 10G devices at the other end is that the wardrobe has no network jacks, but both my office and the wardrobe have downlights - I can easily pull the downlight out, cut a very small notch on one edge of the hole, run some OM4 fibre through the notch, and put the downlight back in - the vanity rings on the downlights will fully cover the notch once the cable is removed, whereas a notch large enough for a cat6 cable would be too large to be covered. My office then has network jacks connecting back to the rest of the network with cat6 run through the walls and ceiling space.)
In order to make that layout work though, I need to figure out how to avoid my current issue so I can actually have both IPMI and 10G networking plugged into the same switch without breaking my network.
Is anyone able to offer any advice on how to fix this?.. It seems to me that the easiest method would be to just compeltely disable igb0 in FreeNAS, but I'm not sure how best to do that.
If this were a Linux box, I'd probably just set a udev configuration option or the like, and I know I can do similar things with FreeBSD, I'm just not sure which method would work the best and not be overridden by FreeNAS.