10 Gig Networking Upgrade

[Black Ninja]

They worked great. I couldn't get full 10Gb. It think I capped out at something like 6 or 7Gb in testing. But it's well known that Intel's 10Gb offerings are good, but not great, on FreeBSD/FreeNAS. They have been rock solid, aside from a driver bug in certain 9.2.1.x builds. But let's face it, I can't get 7Gb/sec through CIFS anyway since it's single threaded, so I don't care. And for the price difference between going Intel and Chelsio is pretty wide, I'll stick with the Intel and the slightly-less-than-10Gb 10Gb.

I connect the 2 Intel E15729 back to back freenas to desktop. My Desktop is 4930K with 32GB RAM with Windows 7. I haven't touched any settings on desktop or Freenas site , it's all default.
Can you tell me what settings I need to change for 10Gb nics to get your results(on freenas side and Windows 7 site) ? It's obvious the defaults are good for 1Gb but , it wont work out ouf the box for 10Gb.

 

[cyberjock]

Mine worked out of the box with 10Gb at those speeds.

 

[Black Ninja]

Mine worked out of the box with 10Gb at those speeds.

When I copy to Freenas it's almost ok, but when I copy from freenas it to Desktop maxes out first cpu core and make PC sluggins barely responsive while copy is going ? Haven't you experience that ?

 

[cyberjock]

Not sure if you mean sluggishness on the client or server, but I've never had a problem with either one. I'm guessing you can receive the data faster than the disk can write the data and the server is getting "constipated" with all this data that is mid-write.

 

[Black Ninja]

Not sure if you mean sluggishness on the client or server, but I've never had a problem with either one. I'm guessing you can receive the data faster than the disk can write the data and the server is getting "constipated" with all this data that is mid-write.

Sluggish I meant on a client - Windows 7 side. Example: you can try to grab the "copy file transfer window" and move it around but it takes a second to respond like 2 frames per seconds movie. I notice my first cpu core is maxed out, this is when the sluggishness occurs. If the transfer was slower like 180MB/s core 1 is not maxed no sluggishness.

It has something to do with maxing the single core, because on 1Gb link 99% saturated, first core is 55% max and never experience a sluggishness.

I hope you know what I mean so you can tell me ?

 

[Black Ninja]

This switch is said to use between 125-140Watts, so i didnt think that was too terrible. And yea it took me a week or so to gather all the info but in the end i feel it was worth it....i got a 10Gb backbone rocking and rolling lol

I would like to know what king of settings you change for the 10Gb nics ( like jumbo frames, etc.). Up till now I only had a 1Gb nics and left everything on default settings on Freenas and on Windows 7 desktop and all worked fine, but for 10Gb is different. Can you share yours ?

 

[Black Ninja]

Actually getting jumbo to work properly seems like an exercise in frustration. The amount of engineering effort required for the modest gains isn't worth it. The drivers suck (including the segregation of jumbo mbufs), the operating system support sucks (not just FreeBSD), the hardware sucks, the routers suck (crappy MTU/frag support), and you can get screwed in any of a half dozen ways. I've spent too much time on jumbo in the last decade or two. I've finally gotten annoyed enough that I'm slowly killing off jumbo support in our networks. Modern cards do sufficient magic with offload to make 1500 fast that the annoyance of jumbo is no longer worth it. With modern switches that can support switching a gigapacket per second ... screw jumbo.

For 1Gb I don't care to mess with jumbo frames. I have 2 10Gb nics connected back to back, Freenas -Desktop. Is it OK to enable jumbo frames for those two 10Gb nics only. They are on a different sub-net from 1GB network, will this work properly not affecting the rest of the 1Gb network ?

 

[jgreco]

It might work properly. Jumbo is not worth the effort. If you have 10G, you're probably not maxxing it out anyways, and jumbo isn't likely to make a big difference.

 

[Black Ninja]

It might work properly. Jumbo is not worth the effort. If you have 10G, you're probably not maxxing it out anyways, and jumbo isn't likely to make a big difference.

I don't know what setting I need on Freenas , but usually max my first CPU core, PC became very sluuggish while transfer and can't get faster than 300-400 MB/s that's about 30% network utilization. I tried windows 2008 R2 to Windows 7 and very similar results but if I enable jumbo I can utilize 98% of 10Gb I would assume will have the similar effect on Freenas ?!

I like the idea not to touch the default setting on anything , but it seems for 10Gb I have to ?!

 

[jgreco]

Unlikely.

 

[cyberjock]

I would try copying to/from a RAM disk on your desktop to check transfer speeds. That helps to rule out a slower disk. If you are using Windows, I recommend https://www.softperfect.com/products/ramdisk/

It's the fastest I know of. Its free. It's not bloated or anything.

A good second choice is imdisk. http://www.ltr-data.se/opencode.html/

I used this for several years and it's very handy. If memory serves me right, you can use it to mount ISOs too. ;)

 

[Black Ninja]

Unlikely.

When I copy 1Gb link even if it's saturated 99% , my cpu is 55%, nothing wrong here for the pas many years on 1Gb network,

BUT
when I recently put a 10Gb nic and copy file 10Gb link is 30% saturated , my first CPU core is 99% utilization and the whole PC is very unresponsive till transfer is Done. What is wrong ? OR What can be done ?

It's not the transfer speed the bothers me , it's the fact that computer is getting so unresponsive.

 

[Black Ninja]

I would try copying to/from a RAM disk on your desktop to check transfer speeds. That helps to rule out a slower disk. If you are using Windows, I recommend https://www.softperfect.com/products/ramdisk/

It's the fastest I know of. Its free. It's not bloated or anything.

A good second choice is imdisk. http://www.ltr-data.se/opencode.html/

I used this for several years and it's very handy. If memory serves me right, you can use it to mount ISOs too. ;)

That's exactly what I used to measure it: https://www.softperfect.com/products/ramdisk/
I installed on Windows 2008 R2 and my desktop Windows 7, I wish there was a way to use it in freenas as well, or is it a way ?
I had to buy another Samsung 850 PRO , to get at least 500mb/s R/W on both machines

 

[cyberjock]

I'm guessing interrupt requests. ;)

I don't know what else to say, but this is far outside of what should be discussed in the 10Gb primer...

 

[Black Ninja]

I'm guessing interrupt requests. ;)

That might be the case , but how come it's not like that for everybody or is it?
I wonder how this works on your machine , don't you max single CPU core on 10Gb when you copy large file ? Also if you do , aren't you experience the same unresponsiveness ?

 

[Black Ninja]

I don't know what else to say, but this is far outside of what should be discussed in the 10Gb primer...

After buying all hardware base on the 10Gb primer and following the guidelines precisely. I can't think of better place to discuss how to set it up , or why it's not working.
I purposely wanted to ask about this problem in this tread , so anybody who read the primer and set it up will know what need to be done next , in order to make it work.

 

[cyberjock]

After buying all hardware base on the 10Gb primer and following the guidelines precisely. I can't think of better place to discuss how to set it up , or why it's not working.
I purposely wanted to ask about this problem in this tread , so anybody who read the primer and set it up will know what need to be done next , in order to make it work.

No, they don't. They have one page of you talking about your problem with no solution in sight. Not even an idea of a problem. Just a complaint that the box is slow.

It would be better to have your *own* thread and when you have a solution and understand the problem, then post that to this thread as a single coherent post. Right? Instead we have 15 posts (and counting) of you talking about your problem and no solution yet.

 

[jgreco]

I'll fix it later.

 

[Black Ninja]

No, they don't. They have one page of you talking about your problem with no solution in sight. Not even an idea of a problem. Just a complaint that the box is slow.

It would be better to have your *own* thread and when you have a solution and understand the problem, then post that to this thread as a single coherent post. Right? Instead we have 15 posts (and counting) of you talking about your problem and no solution yet.

Perhaps you are right. I just figure you or jpgreco would know exactly what is wrong. I don't consider this as just my problem , as much a cifs being single treated being my problem. I think anybody who use freenas and windows 7 with 10Gb will get right the same issue.

I will post the solution here, i think it should be part of the 10Gb primer but I don't know where to get the answer first. I tough it will be from you.

 

[Black Ninja]

It might work properly. Jumbo is not worth the effort. If you have 10G, you're probably not maxxing it out anyways, and jumbo isn't likely to make a big difference.

I understand that, but it will not interfere with the rest of the 1Gb network if jumbo is set only on these two 10Gb nic connected back to back on separate sub-net , right ?