10 Gig Networking Primer

10 Gig Networking Primer

Joined
Nov 11, 2014
Messages
1,174
So as it turns out, i *DO* for some reason have 2 different XFP's .... Which is strange because all the XFP's i purchased were sold as FINISAR's. But none the less, it all works, without issue lol. Now as for temps....the only thing it tells me is "Good" there is no actual temp reading it seems i can find or display via the CLI and referring to the manual it just states "Good" is well, good...
Code:
    port 1  (FINISAR CORP.   :S  ) : good
    port 2  (FINISAR CORP.   :S  ) : good
    port 3  (FINISAR CORP.   :S  ) : good
    port 4  (FINISAR CORP.   :S  ) : good
    port 5  (FINISAR CORP.   :S  ) : good
    port 6  (Opnext Inc.     :S  ) : good
    port 7  (Opnext Inc.     :S  ) : good
    port 8  (FINISAR CORP.   :S  ) : good

I see. Thank you for checking. After studying the CLI I see the same thing - no actual temp reading just good or perhaps"bad" when is too bad. I wish wasn't like that, but I guess they didn't think of it back then.

By the way I found switch like our which has the fans not reversed. It looks exactly the way I made mine with the fans, but this one is just like that from the manufacturer.
Here:http://www.ebay.com/itm/FUJITSU-XG2...RDWARE-ONLY-/141593838196?hash=item20f7a68274

This proves that is perfectly ok for the switch to work either way in case you want to change it on yours too. Most XG2000 are "R" on ebay, I found this to be rather unusual, maybe that's why they are the only one available on ebay and so cheap. It's good for us anyways.
 
Joined
Oct 2, 2014
Messages
925
I see. Thank you for checking. After studying the CLI I see the same thing - no actual temp reading just good or perhaps"bad" when is too bad. I wish wasn't like that, but I guess they didn't think of it back then.

By the way I found switch like our which has the fans not reversed. It looks exactly the way I made mine with the fans, but this one is just like that from the manufacturer.
Here:http://www.ebay.com/itm/FUJITSU-XG2...RDWARE-ONLY-/141593838196?hash=item20f7a68274

This proves that is perfectly ok for the switch to work either way in case you want to change it on yours too. Most XG2000 are "R" on ebay, I found this to be rather unusual, maybe that's why they are the only one available on ebay and so cheap. It's good for us anyways.
Yea that just about sums it up, they made it a certain way..and it seems no one else really wants it, great for us lol. I love second hand hardware, some of its power hungry but hey its better then wallet hungry...hell some of the dam servers even basic config's are like 2-3K with minimum ram and all....like wth lol.

I also found some chelsio cards on ebay for cheap as of today, i picked one up for 85 shipped! I was really excited, then i saw the price of the SM10-G SFP+'s ....and 30-40 bucks kinda made me sad again lol. But there are these ones: http://www.ebay.com/itm/Chelsio-SM1...-10-Gigabit-/221852148774?hash=item33a76b2426 http://www.ebay.com/itm/Chelsio-SM1...-SC-S43-42B-/221852148772?hash=item33a76b2424
 
Joined
Nov 11, 2014
Messages
1,174
Yea that just about sums it up, they made it a certain way..and it seems no one else really wants it, great for us lol. I love second hand hardware, some of its power hungry but hey its better then wallet hungry...hell some of the dam servers even basic config's are like 2-3K with minimum ram and all....like wth lol.

I also found some chelsio cards on ebay for cheap as of today, i picked one up for 85 shipped! I was really excited, then i saw the price of the SM10-G SFP+'s ....and 30-40 bucks kinda made me sad again lol. But there are these ones: http://www.ebay.com/itm/Chelsio-SM1...-10-Gigabit-/221852148774?hash=item33a76b2426 http://www.ebay.com/itm/Chelsio-SM1...-SC-S43-42B-/221852148772?hash=item33a76b2424

This particular switch we have it seem was $18K Retail Price quite a saving from what we paid. Older and not super power efficient, still I am sure underneath is way better choice that modern consumer grade netgear , in which every screw is made as less expansive as possible. On the other hand this XG2000 for $18K , they might put gold plate instead of stainless steel and still be on the budget. :))
When the goal is not " as cheap as possible" good things were made.

On Chelsio front I am still not sold to chose them over Intel. I know freenas is keep pushing them , but still. I remember you mention before you had a problems with one overheating, and after looking the adapter it self and most specifically it's heat spreader-no wonder. It's very small and inefficient by design, despite is made of better materials (copper) It's design make it more cool looking than efficent. I could be wrong you know , I've been wrong before..:)

P.S. My opinion about Chelsio is not guaranteed to be accurate, It was not base on actual experience with them just by design observation. (CHelsio)
 
Joined
Oct 2, 2014
Messages
925
This particular switch we have it seem was $18K Retail Price quite a saving from what we paid. Older and not super power efficient, still I am sure underneath is way better choice that modern consumer grade netgear , in which every screw is made as less expansive as possible. On the other hand this XG2000 for $18K , they might put gold plate instead of stainless steel and still be on the budget. :))
When the goal is not " as cheap as possible" good things were made.

On Chelsio front I am still not sold to chose them over Intel. I know freenas is keep pushing them , but still. I remember you mention before you had a problems with one overheating, and after looking the adapter it self and most specifically it's heat spreader-no wonder. It's very small and inefficient by design, despite is made of better materials (copper) It's design make it more cool looking than efficent. I could be wrong you know , I've been wrong before..:)

P.S. My opinion about Chelsio is not guaranteed to be accurate, It was not base on actual experience with them just by design observation. (CHelsio)
I added a better heatsink, and a fan to it, havent fired up the other NIC because well...i dont need it lol. i can get ~50% saturation from my SAN to my 3 ESXi hosts...so im not *too* worried, i might team the 2 10Gbs and make a 20Gb link to the switch to feed the ESXi hosts...but even then im not running an all SSD storage lol, and im still limited by the RAID cards bottleneck and drives.
 
Joined
Nov 11, 2014
Messages
1,174
I added a better heatsink, and a fan to it, havent fired up the other NIC because well...i dont need it lol. i can get ~50% saturation from my SAN to my 3 ESXi hosts...so im not *too* worried, i might team the 2 10Gbs and make a 20Gb link to the switch to feed the ESXi hosts...but even then im not running an all SSD storage lol, and im still limited by the RAID cards bottleneck and drives.

By the way. One thing I can't figure out: on the front switch has rj45 port and Serial. I know serial can go to serial , but I don't know about the RJ45 port , is that RJ45 to Serial , or is RJ45 Lan port ?
 
Joined
Oct 2, 2014
Messages
925
By the way. One thing I can't figure out: on the front switch has rj45 port and Serial. I know serial can go to serial , but I don't know about the RJ45 port , is that RJ45 to Serial , or is RJ45 Lan port ?
My RJ-45 is plugged into my switch, its for LAN management, rather then serial. If you were worried about secure-ness youd make a jump server or a jump desktop thats connected to the switches serial and then youd connect to the jump server. But for the RJ-45 youd need to enable SSH/telnet in order to access it, by default its disabled
 
Joined
Nov 11, 2014
Messages
1,174
My RJ-45 is plugged into my switch, its for LAN management, rather then serial. If you were worried about secure-ness youd make a jump server or a jump desktop thats connected to the switches serial and then youd connect to the jump server. But for the RJ-45 youd need to enable SSH/telnet in order to access it, by default its disabled

Ooo see. So it's not a console port where you put RJ45 and the other end is serial. Then I'll connect to serial first to reset and enable ssh ,then connect this RJ45 to Lan. Can you still manage the switch if connected to another swith with 10Gb link and no RJ45 connection ?! Basically connect the switch to the rest of the network vie one of it's 10Gb ports ?!
 
Joined
Oct 2, 2014
Messages
925
Ooo see. So it's not a console port where you put RJ45 and the other end is serial. Then I'll connect to serial first to reset and enable ssh ,then connect this RJ45 to Lan. Can you still manage the switch if connected to another swith with 10Gb link and no RJ45 connection ?! Basically connect the switch to the rest of the network vie one of it's 10Gb ports ?!
No no, the Lan management is PURELY management, cant uplink it to other switches unless your other switch has a 10Gb link lol, the LAN management can get either DHCP or a static address, its just like connecting to your routers default gateway to manage it, or to manage a web enabled or even SSH switch, just open up terminal (mac linux) or putty on windows and punch in the IP address of the switch, type in admin for username and password for password and it'll connect right in. You cant manage the switch by the 10Gb ports, you cant ping or access it, the LAN management port is strictly for management of the switch via that port, which i think is 10/100
 
Joined
Nov 11, 2014
Messages
1,174
No no, the Lan management is PURELY management, cant uplink it to other switches unless your other switch has a 10Gb link lol, the LAN management can get either DHCP or a static address, its just like connecting to your routers default gateway to manage it, or to manage a web enabled or even SSH switch, just open up terminal (mac linux) or putty on windows and punch in the IP address of the switch, type in admin for username and password for password and it'll connect right in. You cant manage the switch by the 10Gb ports, you cant ping or access it, the LAN management port is strictly for management of the switch via that port, which i think is 10/100

My other swith has 10Gb uplinks and I am going to connect XG2000 to Dell 5548 with one 10Gb ports on each switch. Do I still need to use and RJ45 on the fron of the XG2000 in order to use SSH after is setup vie serial port first , then serial is disconnected and use only SSH to manage ?!
 
Joined
Oct 2, 2014
Messages
925
My other swith has 10Gb uplinks and I am going to connect XG2000 to Dell 5548 with one 10Gb ports on each switch. Do I still need to use and RJ45 on the fron of the XG2000 in order to use SSH after is setup vie serial port first , then serial is disconnected and use only SSH to manage ?!
Yea youd still need the RJ-45 for management via SSH, adding the 10Gb wont allow you to access the switch at all via SSH/switches IP
 
Joined
Nov 11, 2014
Messages
1,174
Yea youd still need the RJ-45 for management via SSH, adding the 10Gb wont allow you to access the switch at all via SSH/switches IP

That is little bit different setup. On other switches you setup your basics with serial , then you disconnect serial and put you clients and router to the switch and everybody can access it via SSH, no need to have console cable plug in. I am hoping when I connect XG2000 to my other switch via 10Gb unlink and via RJ45 to 1Gb ports will not create a loop, since connection is made twice ?!
 
Joined
Oct 2, 2014
Messages
925
So my normal 1Gb LAN connects my servers, router, and my 10Gb switch management port, all my main data is accessed via 10Gb, all the other data is misc through 1Gb. Plugging the 10Gb management to your normal 1Gb LAN wont make any kind of data loop, the LAN management doesnt send/receive any kind of other traffic other then you SSHing into it. It can not give out DHCP, it cant do any other function then management, as if a serial cable was plugged into it, it just allows you to use a IP address to manage it rather then being tethered via serial.

Lets break it down, your 1Gb lan we will call "Primary switch" your 10Gb we will label as "10Gb"

Your primary switch will have the single uplink to "10Gb", this allows your 1 Gb clients to attempt to connect/saturate anything on the 10Gb side, if properly configured. The 10Gb switch management WILL connect to the 1Gb lan but is PURELY management, it has no other function, you can reach the 10Gb management port from the Primary switch or a device connected via the 10Gb. It will not create any kind of loop back of data, or a "round robin" its called.
 
Joined
Nov 11, 2014
Messages
1,174
So my normal 1Gb LAN connects my servers, router, and my 10Gb switch management port, all my main data is accessed via 10Gb, all the other data is misc through 1Gb. Plugging the 10Gb management to your normal 1Gb LAN wont make any kind of data loop, the LAN management doesnt send/receive any kind of other traffic other then you SSHing into it. It can not give out DHCP, it cant do any other function then management, as if a serial cable was plugged into it, it just allows you to use a IP address to manage it rather then being tethered via serial.

Lets break it down, your 1Gb lan we will call "Primary switch" your 10Gb we will label as "10Gb"

Your primary switch will have the single uplink to "10Gb", this allows your 1 Gb clients to attempt to connect/saturate anything on the 10Gb side, if properly configured. The 10Gb switch management WILL connect to the 1Gb lan but is PURELY management, it has no other function, you can reach the 10Gb management port from the Primary switch or a device connected via the 10Gb. It will not create any kind of loop back of data, or a "round robin" its called.

I got it now. Thanks very much for explaining. I should've read the manual first before bother you :
"...dedicated management LAN interface that is independent from the 10 Gigabit ports."

I'll read the manual and then bother you.:smile:
 
Joined
Oct 2, 2014
Messages
925
I got it now. Thanks very much for explaining. I should've read the manual first before bother you :
"...dedicated management LAN interface that is independent from the 10 Gigabit ports."

I'll read the manual and then bother you.:)
lol hey man, never a bother, long as you get your setup the way you want and need it is what matters, i respond 90% of the time at work anyways :P
 
Joined
Nov 11, 2014
Messages
1,174
lol hey man, never a bother, long as you get your setup the way you want and need it is what matters, i respond 90% of the time at work anyways :p

Thanks very much for everything. Here is the frist situation I stumble upon after setting the switch: I notice on "show interface port x" command , the port which has my desktop connected shows:

Link Status : Link-up
Link Status Detail : (none)

But on the port that I connect XG2000R to the other switch dell 5548 it shows:
Link Status : Link-up
Link Status Detail : LAL

Is this normal ? I read that "LAL" status means : Frame discarding due to a loop back alert was detected.

Everything looks to be working fine , but I was wondering about this "LAL"status when connected to another switch , do you know ?
 
Joined
Oct 2, 2014
Messages
925
not sure....are you able to ping your desktop via the IP address on the 10Gb card?
 
Joined
Nov 11, 2014
Messages
1,174
not sure....are you able to ping your desktop via the IP address on the 10Gb card?
To ping it from the same machine you mean ? I haven't try but it's working I am using it now. What does it show on yours ? Do you have it connected to another switch ?
 
Joined
Oct 2, 2014
Messages
925
mine doesnt connect to anther switch, my 10Gb is completely isolated from my LAN. Different IP range and subnet and all, the only thing that connects my 2 switches is the LAN management port of the 10Gb switch to my 1Gb LAN.

Did you statically assign a IP address to your 10Gb card NIC? If so i would get on another pc and try and ping the 10Gb NIC IP address just to make sure its working the way it should.
 
Joined
Nov 11, 2014
Messages
1,174
mine doesnt connect to anther switch, my 10Gb is completely isolated from my LAN. Different IP range and subnet and all, the only thing that connects my 2 switches is the LAN management port of the 10Gb switch to my 1Gb LAN.

Did you statically assign a IP address to your 10Gb card NIC? If so i would get on another pc and try and ping the 10Gb NIC IP address just to make sure its working the way it should.

The IP is dynamic from DHCP. I tried to ping it from another machine on the gigabit network and it responds fine. What is your link status on your ports show ?
 
Joined
Oct 2, 2014
Messages
925
Code:
Interface Information                                       2015/08/22-20:40:30
===============================================================================
[port 1]
  Generic Information
    Description                   : (none)
    MAC Address                   : 0080.175d.0d6b
    Link Status                   : Link-up
      Link Status Detail          : (none)
    STP State                     : Forward
    Flow Control                  : Rx Only
    Address Learning              : Enabled
    Multicast Forwarding          : Forward-all
    Port Security                 : Disabled
    Storm Control                 : Disabled
    Default Qos Priority          : 0
    Qos Egress Scheduling         : Strict
    Qos Bandwidth
      Output Priority 0           : 0
      Output Priority 1           : 0
      Output Priority 2           : 0
      Output Priority 3           : 0
    Ingress Bandwidth             : 10000
  Vlan Information
    Port Default Vlan ID          : 1
    Vlan Member(tagged frame)     : None
    Vlan Member(untagged frame)   : Vlan-1
    User Vlan Protocol ID         : 0x8100
  Filter Information
    Ingress Filter(no vlan member): Disabled
    Ingress Filter(tagged frame)  : Disabled
    Ingress Filter(untagged frame): Disabled
===============================================================================                                      
 
Top