micahtangelo
Dabbler
- Joined
- Apr 9, 2017
- Messages
- 15
Hey crew,
So we've been tinkering with FreeNAS for a little while now to service a Media/Entertainment usecase similar to what @friolator has described here https://forums.freenas.org/index.php?threads/supported-40gbe-cards.44475/. and @HeloJunkie has discussed 40gbe throughput a bit as well in depth.
I think there's something fundamental I'm missing, or something really wrong with my setup, because I'm getting some very odd performance artifacts.
System general specs:
Supermicro X10 system
2x E5-2650
256GB RAM
4x 2TB Samsung NVME drives, all set up as L2ARC
4x 512GB Samsung NVME drives, set up as SLOG
8x 3TB WD crummy drives in 2x4 mirrored raidz1 config.
1x 40G Chelsio T62100-LP-CR
Idea was to make it so that enough of the working set of our data (mostly video files) would live in ARC/L2ARC to make 10G speeds.
Internally, we're doing pretty nicely - hitting ~2.5GB/s on TB-scale dd's internally.
Getting outside of the FreeNAS system, however, seems to be stuck at about 3gbps. Iperfs to other 40Gb and 100Gb machines top out at 3gbps. Even iperf to the loopback tops out at about 15gbps. dd's to or from the NAS top out at about 250MB/s. So do SMB and NFS file transfers
3 Linux clients have 40G Chelsio or Mellanox cards, going through an Brocade ICX7750 switch. We've done a lot of part testing to confirm cables, ports, modules, switch etc problems are all ruled out. The Linux clients can all move files between themselves basically at line speeds.
The kicker is that we put CentOS on a USB stick, booted up this very same FreeNAS hardware with it, and were able to get roughly line speed out of the system. I'm waiting on one of our other techs to come back to me with the actual numbers from that, and I'm a tad hazy on the details of exactly what he did. But there it is, basically. What in FreeNAS could we be doing so wrong to limit this moderately-strong machine to 3gbps through into the network...
And fwiw, we're also working with iX to get one of their very nice TrueNAS systems in to serve this workload as the primary filer, inw which case this machine (or one similar to it) will become an offsite backup, that still needs to be able to perform pretty damn well in case the primary server room turns into a crater somehow. We will have 10 or 40g dark fibre to workstations from this offsite.
Thanks heaps
Adding some keywords so I can find this thread easily: 40g 40gbe 100gb 100gbe
So we've been tinkering with FreeNAS for a little while now to service a Media/Entertainment usecase similar to what @friolator has described here https://forums.freenas.org/index.php?threads/supported-40gbe-cards.44475/. and @HeloJunkie has discussed 40gbe throughput a bit as well in depth.
I think there's something fundamental I'm missing, or something really wrong with my setup, because I'm getting some very odd performance artifacts.
System general specs:
Supermicro X10 system
2x E5-2650
256GB RAM
4x 2TB Samsung NVME drives, all set up as L2ARC
4x 512GB Samsung NVME drives, set up as SLOG
8x 3TB WD crummy drives in 2x4 mirrored raidz1 config.
1x 40G Chelsio T62100-LP-CR
Idea was to make it so that enough of the working set of our data (mostly video files) would live in ARC/L2ARC to make 10G speeds.
Internally, we're doing pretty nicely - hitting ~2.5GB/s on TB-scale dd's internally.
Getting outside of the FreeNAS system, however, seems to be stuck at about 3gbps. Iperfs to other 40Gb and 100Gb machines top out at 3gbps. Even iperf to the loopback tops out at about 15gbps. dd's to or from the NAS top out at about 250MB/s. So do SMB and NFS file transfers
3 Linux clients have 40G Chelsio or Mellanox cards, going through an Brocade ICX7750 switch. We've done a lot of part testing to confirm cables, ports, modules, switch etc problems are all ruled out. The Linux clients can all move files between themselves basically at line speeds.
The kicker is that we put CentOS on a USB stick, booted up this very same FreeNAS hardware with it, and were able to get roughly line speed out of the system. I'm waiting on one of our other techs to come back to me with the actual numbers from that, and I'm a tad hazy on the details of exactly what he did. But there it is, basically. What in FreeNAS could we be doing so wrong to limit this moderately-strong machine to 3gbps through into the network...
And fwiw, we're also working with iX to get one of their very nice TrueNAS systems in to serve this workload as the primary filer, inw which case this machine (or one similar to it) will become an offsite backup, that still needs to be able to perform pretty damn well in case the primary server room turns into a crater somehow. We will have 10 or 40g dark fibre to workstations from this offsite.
Thanks heaps
Adding some keywords so I can find this thread easily: 40g 40gbe 100gb 100gbe
