Best configuration of my 60 Bay Supermicro..?

[ben604]

Hello,

I've inherited a 60 bay Supermicro chassis at work and would like to see if I can use it for near-set/on-set playback of frame based and wrapped 4k - 8k media, while also archiving to LTO and rendering of deliverables in its current hardware state. I believe I might be hitting a bottleneck with the SATA drives, but maybe I'm just building the pool wrong...

I'd need to be able to read at around 1.2GB/s across 4 devices for the project I have in mind.

Chassis:

6048R-E1CR60N | SuperStorage | Products | Super Micro Computer, Inc.

Spec:
Dual Intel Xeon E5-2650 v4
64GB Ram
OS is on SSD
60 x Seagate ST8000NM0105 8TB 7200rpm SATA
1 x Mellanox Connectx-5 Dual 40/100Gb NIC
1 x Mellanox Connectx-4 Dual 10/25/40Gb NIC
FreeNAS

Jumbo frames are enabled on the storage and workstation NICs.

I've got it built currently with 19 Vdevs each containing 3 mirrored drives, giving me 140TBish of usable storage (we need about 100TB for this project).

I've got a Win10 Ent, HP Z840 (dual proc, 128Gb RAM, dual 1080ti, Mellanox Connectx-5 NIC) machine directly connected (at 100Gb!) and can copy large files around the filesystem at 1.4Gb/s, but playback is shoddy of pretty lightweight frame based media. That's via SMB.

Could anyone suggest some optimisations I might make to get some better throughput?

I've got money to spend. Should I look at more RAM for the ARC Cache? SAS Drives?

Any help/opinions are very welcome!

 

[Spearfoot]

Hello,

I've inherited a 60 bay Supermicro chassis at work and would like to see if I can use it for near-set/on-set playback of frame based and wrapped 4k - 8k media, while also archiving to LTO and rendering of deliverables in its current hardware state. I believe I might be hitting a bottleneck with the SATA drives, but maybe I'm just building the pool wrong...

I'd need to be able to read at around 1.2GB/s across 4 devices for the project I have in mind.

Chassis:

6048R-E1CR60N | SuperStorage | Products | Super Micro Computer, Inc.

Spec:
Dual Intel Xeon E5-2650 v4
64GB Ram
OS is on SSD
60 x Seagate ST8000NM0105 8TB 7200rpm SATA
1 x Mellanox Connectx-5 Dual 40/100Gb NIC
1 x Mellanox Connectx-4 Dual 10/25/40Gb NIC
FreeNAS

Jumbo frames are enabled on the storage and workstation NICs.

I've got it built currently with 19 Vdevs each containing 3 mirrored drives, giving me 140TBish of usable storage (we need about 100TB for this project).

I've got a Win10 Ent, HP Z840 (dual proc, 128Gb RAM, dual 1080ti, Mellanox Connectx-5 NIC) machine directly connected (at 100Gb!) and can copy large files around the filesystem at 1.4Gb/s, but playback is shoddy of pretty lightweight frame based media. That's via SMB.

Could anyone suggest some optimisations I might make to get some better throughput?

I've got money to spend. Should I look at more RAM for the ARC Cache? SAS Drives?

Any help/opinions are very welcome!

What a beast of a system! I'm green with envy!
Adding memory is the first thing I would try. 64GiB is probably too little RAM for so much storage.

 

[ben604]

Cool, I'll double it today and have another look.

Our main storage is 2PB of GPFS on NetApp Chassis. ~67GB/s read write performance...! RDMA over 40Gb interfaces per workstation, so each client can pull ~4.7GB/s.

Not portable enough

 

[Spearfoot]

Cool, I'll double it today and have another look.

Our main storage is 2PB of GPFS on NetApp Chassis. ~67GB/s read write performance...! RDMA over 40Gb interfaces per workstation, so each client can pull ~4.7GB/s.

Not portable enough

If you have the money, why not quadruple it? The more, the better -- as a rule.

I run an older NetApp at work; they're beasts, too. Very expensive and proprietary beasts. Our 6TiB disks, flashed with NetApp's custom firrmware, cost over $3000 apiece. We have 48 of them...