Performance of 40Gb network with FreeNAS

[skyyxy]

Hi everybody here. Sorry for my English.
Last week I gave up my 10Gb FreeNAS server because 1.2GB/s bandwidth still not enough for my office user, they need use Davinci Resolve studio to edit 8k movie project, each client machine need 400-500MB/s bandwidth and we have 7-8 users. So since yesterday I decide to build 40Gb FreeNAS server for them and also bought all things here.

server hardware list is:
Intel E5-1650V3 6cores
64GB ecc reg ddr4 memory
Supermicro X10sra with dual intel 1Gb network adapter
LSI 9300-8i SAS3.0 HBA(SAS 12Gb)
Intel XL710 40Gb Qsfp+ network adapter
24bays server case with SAS3.0(SAS 12Gb) panel.

Test client machine
E5-2687W v3 10cores
Supermicro X10sra workstation board with 1Gb intel network adapter
Geforce GTX 1080TI 11GB *2, both for Davinci Resolve 8k project.
64GB reg ecc working memory
Samsung 500GB ssd for Windows 10
Intel XL710 40Gb network adapter with QsfP+ and directly connect to the server(just for test how fast it can be run)
1000W powersupply
full tower case

The problem is: speed is very low. !
3xxMB/s write and 7xxMB/s read, shouldnt like that. May be I need add some parameters in tunable seeting? Anybody has experience? Thanks advance.

 

[m0nkey_]

What sort of performance do you see when using iperf between the two machines?

There will obviously be a need for tuning the system to work with 40GbE. There are plenty of examples of 10GbE tuning on the forum.

 

[SweetAndLow]

Hi everybody here. Sorry for my english.
Last week I gave up my 10Gb FreeNas server because 1.2GB/s bandwidth still not enough for my office user, they need use Davinci Resolve studio to edit 8k movie project, each client machine need 400-500MB/s bandwidth and we have 7-8 users. So since yesterday I decide to build 40Gb FreeNAS server for them and also bought all things here.

server hardware list is:
Intel E5-1650V3 6cores
64GB ecc reg ddr4 memory
Supermicro X10sra with dual intel 1Gb network adapter
LSI 9300-8i SAS3.0 HBA(SAS 12Gb)
Intel XL710 40Gb Qsfp+ network adapter
24bays server case with SAS3.0(SAS 12Gb) pannel.

Test client machine
E5-2687W v3 10cores
Supermicro X10sra workstation board with 1Gb intel network adapter
Geforce GTX 1080TI 11GB *2, both for Davinci Resolve 8k project.
64GB reg ecc working memory
Samsung 500GB ssd for Windows 10
Intel XL710 40Gb network adapter with QsfP+ and directlly connet to the server(just for test how fast it can be run)
1000W powersupply
full tower case

The problem is: speed is very low. !
3xxMB/s write and 7xxMB/s read, shouldnt like that. May be I need add some parameters in tunable seeting? Anybody has experience? Thanks advance.

This is super awesome but you are getting into Enterprise level stuff here. Have you contacted a vendor to help? Even enterprises are new to 40gbps and it will definitely find bugs and performance issues.

Sent from my Nexus 5X using Tapatalk

 

[skyyxy]

What sort of performance do you see when using iperf between the two machines?

There will obviously be a need for tuning the system to work with 40GbE. There are plenty of examples of 10GbE tuning on the forum.

This kind of slow :(

also I change the default settings in windows10 is:
mtu 9014
transfer and receive buffers to max
modify the hosts file to: (local ip address) (machine name in the network) like : 10.0.0.109 test (server is 10.0.0.108). For this setting can get big rise up for 10GbE windows performance.

 

[SweetAndLow]

Test going both directions?

Sent from my Nexus 5X using Tapatalk

 

[Spearfoot]

Hi everybody here. Sorry for my english.
Last week I gave up my 10Gb FreeNas server because 1.2GB/s bandwidth still not enough for my office user, they need use Davinci Resolve studio to edit 8k movie project, each client machine need 400-500MB/s bandwidth and we have 7-8 users. So since yesterday I decide to build 40Gb FreeNAS server for them and also bought all things here.

server hardware list is:
Intel E5-1650V3 6cores
64GB ecc reg ddr4 memory
Supermicro X10sra with dual intel 1Gb network adapter
LSI 9300-8i SAS3.0 HBA(SAS 12Gb)
Intel XL710 40Gb Qsfp+ network adapter
24bays server case with SAS3.0(SAS 12Gb) pannel.

Test client machine
E5-2687W v3 10cores
Supermicro X10sra workstation board with 1Gb intel network adapter
Geforce GTX 1080TI 11GB *2, both for Davinci Resolve 8k project.
64GB reg ecc working memory
Samsung 500GB ssd for Windows 10
Intel XL710 40Gb network adapter with QsfP+ and directlly connet to the server(just for test how fast it can be run)
1000W powersupply
full tower case

The problem is: speed is very low. !
3xxMB/s write and 7xxMB/s read, shouldnt like that. May be I need add some parameters in tunable seeting? Anybody has experience? Thanks advance.

Have you enabled jumbo frames? I found this to be critical with 10Gb.

 

[skyyxy]

Test going both directions?

Sent from my Nexus 5X using Tapatalk

Transfer back is fast between 1.8x-2.1GB/S,(I just have 10hdds at now, because no money :( ), this looking like normal for 10hdds with sas3.0 level.

 

[skyyxy]

Have you enabled jumbo frames? I found this to be critical with 10Gb.

Yes, I enable at all for server and windows. Anybody has atto 40GB network adapter for mac? can please give me the snapshot in macosx system network device detail picture? thanks.

 

[SweetAndLow]

Transfer back is fast between 1.8x-2.1GB/S,(I just have 10hdds at now, because no money :( ), this looking like normal for 10hdds with sas3.0 level.

HDDs have nothing to do with iperf results. 10hdd will only give you maybe 10gbps anyways.

So the results are different going either direction. This could be a problem on the client than. Basically there is a problem on both sides because your speeds are still slow.

Sent from my Nexus 5X using Tapatalk

 

[skyyxy]

HDDs have nothing to do with iperf results. 10hdd will only give you maybe 10gbps anyways.

So the results are different going either direction. This could be a problem on the client than. Basically there is a problem on both sides because your speeds are still slow.

Sent from my Nexus 5X using Tapatalk

I think so
No Both way was 1.8-2GB/S, and how to make the speed up? anybody knows? Thanks

 

[skyyxy]

HDDs have nothing to do with iperf results. 10hdd will only give you maybe 10gbps anyways.

So the results are different going either direction. This could be a problem on the client than. Basically there is a problem on both sides because your speeds are still slow.

Sent from my Nexus 5X using Tapatalk

Do you know, How much GBs the intel xl710 should can be run? Thanks.

 

[SweetAndLow]

I think so
No Both way was 1.8-2GB/S, and how to make the speed up? anybody knows? Thanks

Ok you're not going to get good help. You clearly posted 300MB/s writes and .3 GB/s one direction and 2GB/s the other. But you're saying they are the same. Good luck with your insane project, I suggest hiring someone who knows what they are doing.

Sent from my Nexus 5X using Tapatalk

 

[skyyxy]

Ok you're not going to get good help. You clearly posted 300MB/s writes and .3 GB/s one direction and 2GB/s the other. But you're saying they are the same. Good luck with your insane project, I suggest hiring someone who knows what they are doing.

Sent from my Nexus 5X using Tapatalk

I'm not crezy. but its real. look the attachment. now its really both way 1.8-2G/S.

 

[Ericloewe]

For 10Gb and above, FreeNAS 11 includes significantly-improved drivers, which I don't think have been backported to 10.

 

[skyyxy]

For 10Gb and above, FreeNAS 11 includes significantly-improved drivers, which I don't think have been backported to 10.

Thanks, I will try that.

 

[CookiesLikeWhoa]

I don't know if anything has changed for FreeNAS 11, but previously the X710 line of NICs wasn't the most well supported 40GbE NIC out there.

 

[skyyxy]

I don't know if anything has changed for FreeNAS 11, but previously the X710 line of NICs wasn't the most well supported 40GbE NIC out there.

i think so.

 

[Ericloewe]

Support has apparently improved significantly, now that Intel is heavily investing in FreeBSD development.

 

[skyyxy]

Support has apparently improved significantly, now that Intel is heavily investing in FreeBSD development.

Really? in 9.10.2 U5 or 11? Thanks.

 

[Ericloewe]

Both, AFAIK.