10 G Intel ethernet

[dalex]

I try to install the latest 2 versions in a Supermicro server (SYS-6036ST-6LR) with no success. Both USB key and disk method fail.

There are two 1G and two 10G Ethernet ports on board. The second board of the box hosts a Proxmox server.

The 1G ports work under the "igb" driver (eth0 - eth1). I also manage to make a lacp lagg out of them. But only under dhcp. When i enter static data everything goes wrong. Gui freezes, and cli stops identifying ports. Must re-install.

The 10G ports are assigned to "ix" driver! (eth2 - eth3). Proxmox and Openfiler installs the "ixgbe" driver, and the FreeBSD H/W pages also shows the "ixgbe" driver.

If i configure the 10G ports first, they seem to work, and the lagg between them too. But the lagg is dead and cannot communicate with Proxmox. Openfiler in the same board works fine, lagg or not.

If i try to activate the 1G ports after that, also everything freezes... The Cli sees only the first lagg, no other ports. Some times i cannot even reboot.


So in my understanding, the 10 G ports are installed with the wrong driver and freeze everything...

Are there more recent drivers, or what? Will this be solved in next RC maybe?

 

[louisk]

So, I'm a little confused about things.

You say that the install fails? If so, how are you getting to the configuration portion?

Have you tried setting up the 1G LAGG on the CLI, and then rebooting? I've done this successfully in the past.

 

[dalex]

I 've tried several scenarios (order of actions). Only one works, but of course is a waste of my hardware. Here is the procedure:

1- After install, eth0 takes an IP via DHCP, so i can even use the gui.
2- Either from console, or from the gui, i configure eth0 + eth1 -> lagg0, and after reboot as you pointed out i have a working freenas. I play with my disk array and everything. Lagg0 always with DHCP. Static routes and all working. The two 10 G Ethernets are showing there unusable as eth2 - eth3, with ix0 - ix1 in dmesg.

Every other attempt to alter the network configuration, static IP to lagg0, make of lagg1, etc., freezes the gui, and make the relevant choices in the console inactive (interface config returns to menu, and lagg config shows just the new lagg1 wich is dead).

 

[louisk]

Even if you configure the static IP from the console?

 

[dalex]

Yep. It is impossible to apply a static IP.

I originally installed on a USB3 8G stick, and then to a normal 2G stick, no difference.

 

[louisk]

Odd.

Have you tried configuring a static IP on just eth0, no lagg?

 

[dalex]

No, it is not of my concern. I want two laggs (1+1, 10+10) working first. I can't put data in a NAS so buggy...

I believe that when the driver problem is solved, static IPs will be no problem.

Mind that when i installed FN8 as a virtual machine inside a proxmox in the same server (with virtual eths) everything worked ok.


The exact chipset taken from Openfiler status page:

(2x) Ethernet controller: Intel Corporation 82599EB 10-Gigabit KX4 Network Connection


Anyone from development team to inform about the 10G drivers?

 

[Tekkie]

Have you tried installing the 'regular' FreeBSD8 kernel used by FreeNAS8 on this box? I have a feeling that this is not a FreeNAS8 specific issue you are seeing but rather a generic FreeBSD8 issue.

 

[dalex]

You probably are right.

I just saw the final release 8.0.2 in the site, so i 'll give it a try also to see if anything had changed. Then i 'll look forward to FreeBSD for more info.

Thank you.

 

[dalex]

I installed the RELEASE and still the same. Then i found this:

http://downloadmirror.intel.com/14688/eng/README.txt

Attempting to configure larger MTUs with a large numbers of processors may generate the error message "ix0:could not setup receive structures"
--------------------------------------------------------------------------
When using the ixgbe driver with RSS autoconfigured based on the number of cores (the default setting) and that number is larger than 4, increase the memory resources allocated for the mbuf pool as follows:

Add to the sysctl.conf file for the system:

kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144

and this also:

kern.ipc.nmbjumbo16=32000
kern.ipc.nmbjumbo9=64000

I have 16 cpus so this must be the solution. Error message "ix0:could not setup receive structures" is exactly what i get in the dmesg.

Entered these lines in /conf/base/etc/sysctl.conf and rebooted.

It works! I now have two laggs with static IPs and all. 30 minutes without a problem.

Except that lagg1 (2 * 10 G) cannot communicate with proxmox...

I will search more.

 

[dalex]

The Lagg with 2 * 10 G is not working.

I broke it to ixo and ix1, set static subnets to both Freenas and Proxmox, and everything works ok.

I will split the iscsi targets to two groups, so the virtual machines will be still able to load 2 * 10G traffic.

 

[louisk]

Out of curiosity, what kind of storage are you using that you can push 2x 10G links? At work, I've got about 500VMs spread over almost 55T and it's only pushing about 2.5G.

 

[SoftDux-Rudi]

Out of curiosity, what kind of storage are you using that you can push 2x 10G links? At work, I've got about 500VMs spread over almost 55T and it's only pushing about 2.5G.

I'd love to know that as well. 20GB of traffic is a LOT