Performance issue with 10Gbps network

[wtfR6a]

my e5-1620 v2 ran a 10 disk red 4tb z2 array at circa 800MB/s on a x520 too. CIFS certainly isn't a bottleneck at 2gbps in my experience.
I've just got a bunch of chelsio t520 so-crs and cr's to test with but I'm not expecting to see a huge difference although time will tell.

 

[bartnl]

I seem to be stuck at 180 MB/s write and 400 MB/s read speeds from two different Windows 10 PC's both with 10Gb. In the Freenas box both a 500MB/s SSD and 12x4TB ZFS volume are shared through CIFS. I would expect at least 400 MB/s in both directions for the SSD.
Windows 10 tot Windows 10 will do close to SSD speeds i.e. 500 MB/s over the same network. As I'm limited by a single SSD in my Windows 10 workstation that's the max I expect.

 

[jgreco]

It's not even cat 6. I'm pretty sure it's just 5e. It's not terribly long though, probably 50-60 feet.

You're getting 10GbE to work on a Cat 5e cable over 60 feet?!?!!!???

 

[titan_rw]

You're getting 10GbE to work on a Cat 5e cable over 60 feet?!?!!!???

Yea, is that unheard of? When I was looking at 10GbE, I looked at twinax, but it's max length is usually 10m right? I knew I was going to need slightly longer than that. I figured I have about 15m between the two. Which I guess is only about 50 feet. I suppose I can check the markings on the cable when I get home. The 1000 foot spool I cut this length off of was marked from 0 to 1000 feet.

Here's iperf:

Code:

root@nas ~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  512 KByte (default)
------------------------------------------------------------
[  4] local 192.168.6.13 port 5001 connected with 192.168.6.2 port 52431
[ ID] Interval  Transfer  Bandwidth
[  4]  0.0-10.0 sec  11.1 GBytes  9.53 Gbits/sec
root@nas ~ # iperf -c 192.168.6.2
------------------------------------------------------------
Client connecting to 192.168.6.2, TCP port 5001
TCP window size:  516 KByte (default)
------------------------------------------------------------
[  3] local 192.168.6.13 port 24767 connected with 192.168.6.2 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0-10.0 sec  11.5 GBytes  9.89 Gbits/sec
root@nas ~ #

 

[Mirfster]

I think jgreco misread the "feet" as "meters". ;)

He's probably got a zillion billion message alerts by now...

If I recall correctly:

• Cat5e is rated for 10GbE up to like 40-45 meters
• Cat6 is rated for 10GbE up to like 50-55 meters
• Cat6A is rated for 10GbE up to like 95-100 meters

 

[jgreco]

Yea, twinax craps out around 10m though I'm sure you can find ones that are somewhat longer (I know I've seen 15m).

So no, I wouldn't *expect* it to work, because the demands for signal integrity ought to be much greater than what Cat5e is typically specced for, which is only 100MHz. 10GBase-T requires 500MHz in order to operate properly, which puts this into Cat6 (shorter) or Cat6A (longer). However, I'll note that many manufacturers took the "e" in "5e" to mean enhanced-bandwidth as well, and these days it isn't unusual to see 5e with 350MHz - 550MHz claimed. I suppose it is perfectly possible that you have some better quality Cat5e cable, but it falls kinda under "networking obscenities"

On the other hand, in the end, what really matters is whether or not it works. It is a digital signal. The packets have a checksum. If the packets consistently haul arse from one network interface to the next without errors or drops, that's really the most important thing.

 

[jgreco]

If I recall correctly:

You don't, at least not technically, since 5e is only 100MHz. But that's fine, in the end, what matters is that it works. I've done my own share of networking crimes over the years, including running 100Mbps full duplex over an outdoor Cat3 run where there wasn't going to be a possibility of stringing new cable.

 

[Mirfster]

You don't, at least not technically, since 5e is only 100MHz.

Whoops... :)

 

[jgreco]

Whoops... :)

Yeah "5e" is the biggest crock. It's a meaningless saynothing these days since it has been "enhanced" so many times. 250MHz, 350MHz, 550MHz, etc.

 

[bartnl]

Still confused did a fresh install with FreeNAS-9.3-STABLE-201602031011 and again the same speeds. I'm using Windows 10 pro Intel X540-T2 -> 2 meterCAT7 cable -> ZyXEL XS1920-12 switch -> 1 meter LC-LC OM3 cable -> Freenas Chelsio T420-CR.
What's confusing me is that write speeds to a single SSD and a 12 drive raidz2 ZFS pool are identical. I therefore suspect all common causes like network, CIFS and all Freebsd / Freenas stuff that I am unaware off. On the SSD I get better Freenas write speeds with multithreaded FTP writes i.e. close to 400 MB/s.

 

[titan_rw]

Yea, twinax craps out around 10m though I'm sure you can find ones that are somewhat longer (I know I've seen 15m).

So no, I wouldn't *expect* it to work, because the demands for signal integrity ought to be much greater than what Cat5e is typically specced for, which is only 100MHz. 10GBase-T requires 500MHz in order to operate properly, which puts this into Cat6 (shorter) or Cat6A (longer). However, I'll note that many manufacturers took the "e" in "5e" to mean enhanced-bandwidth as well, and these days it isn't unusual to see 5e with 350MHz - 550MHz claimed. I suppose it is perfectly possible that you have some better quality Cat5e cable, but it falls kinda under "networking obscenities" :)

On the other hand, in the end, what really matters is whether or not it works. It is a digital signal. The packets have a checksum. If the packets consistently haul arse from one network interface to the next without errors or drops, that's really the most important thing.

Yea, I forgot about the MHz rating. I don't know what mine is rated for. It's definitely 5e though. I checked the length markings on both sides. 596-548=48 feet. With gigabit runs, I normally leave 5-6 feet extra on each end in case I want to move a computer a little or something. For this run, I wanted it as short as possible. There's only about an extra foot or so on either side. I was also extremely careful when crimping it to make sure the pairs were twisted as much as possible before going into the connectors.

I would have used cat6, but I didn't have any. I would have had to either buy a spool, or buy a pre-fabbed cable. Pre made cables are not only fairly expensive, not custom lengths (unless you pay for custom cables), but they're harder to run with connectors already on them. You can get network cable a lot more places easier if you don't have to deal with the connector. Crimp once it's run and you're done.

 

[jgreco]

Ah, you kids. You're supposed to buy the inexpensive 100 foot patch cables and cut them as necessary. ;-)

 

[mattbbpl]

Ah, you kids. You're supposed to buy the inexpensive 100 foot patch cables and cut them as necessary. ;-)

Other than excess length, is there any prohibitive reason not to plug these into a keystone plate coupler rather than cut them and recrimp?

 

[jgreco]

Every break in the copper introduces additional places for things to go wrong. Cutting a cable and re-crimping means that you have a crimp (one section of copper), the cable (another section of copper), and then the end crimp (third copper). There are minimal transitions between solid bits of copper. A keystone coupler would be crimp (1), cable (2), crimp (3), keystone (4), and then this implies another patch, so crimp (5), cable (6), and crimp (7). Plus, of course, additional length. So from my point of view, the keystone coupler is a disaster and is definitely a prohibitive reason.

 

[mattbbpl]

Every break in the copper introduces additional places for things to go wrong. Cutting a cable and re-crimping means that you have a crimp (one section of copper), the cable (another section of copper), and then the end crimp (third copper). There are minimal transitions between solid bits of copper. A keystone coupler would be crimp (1), cable (2), crimp (3), keystone (4), and then this implies another patch, so crimp (5), cable (6), and crimp (7). Plus, of course, additional length. So from my point of view, the keystone coupler is a disaster and is definitely a prohibitive reason.

"Disaster" huh? Well, that's thoroughly soul crushing

 

[jgreco]

"Disaster" huh? Well, that's thoroughly soul crushing :p

Well, yes, especially since the people who would use a coupler on one end might well use it on the other end too, bringing us up to 11.

This is one of the reasons you're actually supposed to use a jack and punch it down properly.

 

[rogerh]

Well, yes, especially since the people who would use a coupler on one end might well use it on the other end too, bringing us up to 11.

This is one of the reasons you're actually supposed to use a jack and punch it down properly.

Just for those who might follow your advice with the aforementioned shortened patch lead, don't use a standard punch-down jack intended for solid wire with the stranded wire found in patch leads. It is possible you can get punch-down terminals specifically for stranded wire - I don't know.

 

[Ericloewe]

It is possible you can get punch-down terminals specifically for stranded wire - I don't know.

Sure, you can get jacks for stranded wire. But why would you?

Corning makes them, and I'm sure many others do.

 

[jgreco]

Just for those who might follow your advice with the aforementioned shortened patch lead, don't use a standard punch-down jack intended for solid wire with the stranded wire found in patch leads. It is possible you can get punch-down terminals specifically for stranded wire - I don't know.

Okay, yes, good point, the discussion kind of segued from a discussion of patch cords to a discussion of copper breaks. If you're going to stick a jack on stranded, 1) don't, and 2) don't.