Requirements for 1,5 GB/sec + sustained big file transfer from FreeNAS?

[aronake]

Anyone who could come with an estimate what would be required in terms of FreeNAS hardware to sustain 1,5 GB/sec over 40gbe network from a FreeNAS server from HDDs for large sequential reads and writes? Still have plenty of memory, and fast SSDs to speed up random access?

I do frequent backups, so data security slightly less important, but speed very important. Quite some size also wanted, so rather parity based than Raid 1 style/derivatives. Around 30 TB needed. And would need to allow for one HDD to die, and still at least get a chance to replace and try to recover (well aware that it may fail).

Reason I ask, is that I use a LSI 9361-8i and a 12gb expander to 11 3 TB 7200 rpm HGST disks now. Get around 1.2 GB sequential reads and writes over 40 gbe network (100 GB data). This is quite ok, but the server perform bad of random access. I tried to fix this by having 6 SSDs in Cachecade, but that significantly hurts sequential transfers to around 800 MB/sec.

One idea I have would be a Dell R720XD, with 12 3.5 inch drives, a 1 TB Samsung 960 as L2ARC, some suitable ZIL drive and 128GB memory. Anyone have some performance estimate for this?

For my big transfers much will be fetched from HDDs, so I could see disks bottle necking. How do the raid 5 FreeNAS equivalent perform over 12 disks compared to a LSI 9361?

And yes, I know my set-up (huge raid 5 for instance) is stupid from many use cases... But have certain wishes, like some small redundancy, high sequential read and write, decent random access performance etc.

 

[aronake]

or maybe i should just be happy with my current set-up. Running on hardware raid have the benefit of making it easier to use same computer for other things like running windows server and Hyper-V

 

[Pezo]

I think you're one SI prefix off with your speeds (should be GB/s instead of TB/s and MB/s instead of GB/s)...

 

[aronake]

ah yes! I was a bit too tired when writing. And thanks for your kind way of commenting on this, rather than the quiet hostile way people can make comments on forums... appreciated! And adjusted now.

 

[Stux]

Anyone who could come with an estimate what would be required in terms of FreeNAS hardware to sustain 1,5 GB/sec over 40gbe network from a FreeNAS server from HDDs for large sequential reads and writes? Still have plenty of memory, and fast SSDs to speed up random access?

I do frequent backups, so data security slightly less important, but speed very important. Quite some size also wanted, so rather parity based than Raid 1 style/derivatives. Around 30 TB needed. And would need to allow for one HDD to die, and still at least get a chance to replace and try to recover (well aware that it may fail).

Reason I ask, is that I use a LSI 9361-8i and a 12gb expander to 11 3 TB 7200 rpm HGST disks now. Get around 1.2 GB sequential reads and writes over 40 gbe network (100 GB data). This is quite ok, but the server perform bad of random access. I tried to fix this by having 6 SSDs in Cachecade, but that significantly hurts sequential transfers to around 800 MB/sec.

One idea I have would be a Dell R720XD, with 12 3.5 inch drives, a 1 TB Samsung 960 as L2ARC, some suitable ZIL drive and 128GB memory. Anyone have some performance estimate for this?

For my big transfers much will be fetched from HDDs, so I could see disks bottle necking. How do the raid 5 FreeNAS equivalent perform over 12 disks compared to a LSI 9361?

And yes, I know my set-up (huge raid 5 for instance) is stupid from many use cases... But have certain wishes, like some small redundancy, high sequential read and write, decent random access performance etc.

You need more vdevs to improve random access.

with 12 drives... going with raidz1, you could use dual 6 way, or triple 4 way. or quadruple 3 way.

quadruple 3 way raidz1 will give you random IOPS of 4x that of a single drive. And the sequential read/write speed of of up to about 8x that of a single drive...

Alternatively, triple 4 way, gives you 3 x IOPS, and 9x sequential.

double six way, 2 x the IOPS and 10x the sequential.

You might just need more drives.

So, IOPS = IOPS of a drive * number of vdevs
Sequential read per vdev is roughly equal to number of disks in the vdev (n) - disks of parity (p) * avg speed of disk, for RAIDZ1/2/3

Your current 11 drives... have you got them in a single raid5 running off a hw raid card?

 

[Stux]

Have you run iperf to verify that you don't have a network tuning bottleneck?

 

[Chris Moore]

I did some rough math on this and figured that if each drive had a reliable transfer rate of 150 MB/s (which is optimistic) that it would take 36 drives divided into 6 RAIDz2 vdevs to get you the sustained transfer rate you are seeking. If we are talking spinning disks, SSD is a whole different question.
6 drives, in each of 6 vdevs, each vdev being RAIDz2.
No guarantee, no refund.

 

[aronake]

Have you run iperf to verify that you don't have a network tuning bottleneck?

iperf show around 25 Gbit/sec on direct connected Mellanox connectx-3. Somewhat underwhelming as they are 40 gbe cards.

 

[Stux]

You can probably tune that to get 40gbps. But out of the box, FreeNAS is not tuned to perform 40gbps... which is what you are seeing.

There are threads about this.