ilmarmors
Dabbler
- Joined
- Dec 27, 2014
- Messages
- 25
I'm looking for a most cost effective warm storage server for 200TB capacity in the beginning, 400TB should be enough for next 2-3 years. Path to expand further would be good thing to have. All the components must be purchased in EU. Buying used equipment is not really an option for this.
Purpose:
SuperMicro SuperStorage Server 5049P-E1CTR36L with integrated X11SPH-nCTF motherboard:
Intel Xeon Silver 4114 - 10 cores, 2.20 GHz (3.00 GHz max turbo), 13.75MB L3 cache, TDP 85W
or cheaper
Intel Xeon Silver 4110 - 8 cores, 2.10 GHz (3.00 GHz max turbo), 11MB L3 cache, TDP 85W
Memory: 64GB ECC when 12 disks, 128GB when 36 disks, using 32GB modules
OS disks: 2 x SATA-DOM 32GB
There are 2 yellow SATA DOM ports on this motherboard.
L2ARC: 2 x Intel D3-S4610 480GB, SATA 6Gb/s, 3D, TLC 2.5", 7.0mm, 3DWPD
Could it help to speed up reading of most commonly used access files (there are hot and cold collections of access files)?
Probably needed only if access files are stored on the same server as master files. In case, if only master files are stored on this server, then it won't benefit much from L2ARC.
HDD options:
High capacity enterprise SATA 7200rpm HDDs - with longer warranty and 2.5M hours MTBF
WDC/HGST
HGST/WD 3.5" 12TB SATA 6Gb/s 7.2K RPM 256M 0F30144 512e (He12)
WDC/HGST 3.5"14TB SATA 6Gb/s7.2K RPM 512M 0F31284 512e (He14)
Toshiba MG07ACAxxx
Toshiba 3.5" 12TB,7.2K RPM,SATA 6Gb/s,256M,512e, Helium
Toshiba 3.5" 14TB,7.2K RPM,SATA 6Gb/s,256M,512e, Helium
I was recently quoted prices of HDDs above - 14TB WDC was quite a bit more expensive in EUR/TB, but Toshiba 14TB was comparable with 12TB in EUR/TB.
Any feedback on using Toshiba drives?
Path for further expansion:
One of the SuperMicro JBODs with lower cost per drive bay.
There is question about most appropriate pool(s) config. Not sure if it would be better to put colder master files and warmer access files into separate pools or separate volumes is enough. As I wrote above - putting warmer access files into separate server is option too.
Options for vdev(s) for pool could be:
Initially there is one system with 36 disk bays (+2 slots for 2.5"). When adding JBOD, that would be +24 or +44 - 60 or 80 disk bays in total.
Any suggestions and comments welcome.
Thanks,
Ilmārs
Purpose:
- Storing master files of digitized assets - images, audio, video. File size mainly 10MB and up.
- This won't be primary long term storage solution, there is other system that is responsible for long term storage of those assets. Long term storage has slow turnaround time and you can't retrieve individual object, just sets of objects (kind of Glacier).
- This server would be used for storing additional copy of master files, that allows quickly access them, when needed (not frequently). Mainly serial write load when adding new files.
- This server could have separate volume (or maybe pool) for smaller access copies, that can be recreated from master files any time. Used more often. Probably not more than 1/10 in size comparing to master files. Although there is options to host access files on separate server.
- No deduping needed.
- Bit rot safe storage. That's why I'm looking at ZFS.
- 10Gbps or multiple 1Gbps links.
- Should be able to read and write sequentially several MBps. Workstations have 1Gbps links, few of them would generate load at the same time at most.
- Write load is sequential. Files once written will stay there forever.
- Reads mainly on volume with access files - list all files in folder, look through them. Could be separate server not to mix different loads.
- Must handle up to 100M files in total.
- Rack-mountable case.
- No video trans-coding, no jails or other CPU intensive tasks on servers
- Part of files can be compressed, for example, uncompressed TIFFs (usually 1.1-1.5 times)
- Shouldn't be electricity hog.
- Easy maintenance and administration
SuperMicro SuperStorage Server 5049P-E1CTR36L with integrated X11SPH-nCTF motherboard:
- Single Socket P (LGA 3647) supports Intel® Xeon® Scalable Processors
- Up to 1TB 3DS ECC LRDIMM, up to DDR4-2666MHz; 8 DIMM slots
- 1 PCI-E 3.0 x16, 1 PCI-E 3.0 x8, 1 PCI-E 3.0 x8, 1 PCI-E 3.0 x4 (in x8)
- 2x 10GBase-T with Intel® X557
- 36 Hot-swap 3.5" SAS/SATA drive bays with SES3 (24 front + 12 rear);
- 4 Internal fixed 2.5" drive bays
- 2 Hot-swap 2.5" NVMe/SATA (rear, optional)
- SAS3 via Broadcom 3008 controller
- Server remote management: IPMI 2.0 / KVM over LAN / Media over LAN
- 7 High-performance 8cm PWM fans
- 1200W Redundant Power Supplies Titanium Level (96%)
Intel Xeon Silver 4114 - 10 cores, 2.20 GHz (3.00 GHz max turbo), 13.75MB L3 cache, TDP 85W
or cheaper
Intel Xeon Silver 4110 - 8 cores, 2.10 GHz (3.00 GHz max turbo), 11MB L3 cache, TDP 85W
Memory: 64GB ECC when 12 disks, 128GB when 36 disks, using 32GB modules
OS disks: 2 x SATA-DOM 32GB
There are 2 yellow SATA DOM ports on this motherboard.
L2ARC: 2 x Intel D3-S4610 480GB, SATA 6Gb/s, 3D, TLC 2.5", 7.0mm, 3DWPD
Could it help to speed up reading of most commonly used access files (there are hot and cold collections of access files)?
Probably needed only if access files are stored on the same server as master files. In case, if only master files are stored on this server, then it won't benefit much from L2ARC.
HDD options:
High capacity enterprise SATA 7200rpm HDDs - with longer warranty and 2.5M hours MTBF
WDC/HGST
HGST/WD 3.5" 12TB SATA 6Gb/s 7.2K RPM 256M 0F30144 512e (He12)
WDC/HGST 3.5"14TB SATA 6Gb/s7.2K RPM 512M 0F31284 512e (He14)
Toshiba MG07ACAxxx
Toshiba 3.5" 12TB,7.2K RPM,SATA 6Gb/s,256M,512e, Helium
Toshiba 3.5" 14TB,7.2K RPM,SATA 6Gb/s,256M,512e, Helium
I was recently quoted prices of HDDs above - 14TB WDC was quite a bit more expensive in EUR/TB, but Toshiba 14TB was comparable with 12TB in EUR/TB.
Any feedback on using Toshiba drives?
Path for further expansion:
One of the SuperMicro JBODs with lower cost per drive bay.
There is question about most appropriate pool(s) config. Not sure if it would be better to put colder master files and warmer access files into separate pools or separate volumes is enough. As I wrote above - putting warmer access files into separate server is option too.
Options for vdev(s) for pool could be:
- 12 disk RAID-Z2 (max capacity)
- 12 disk RAID-Z3 (lower capacity, better time to total data loss)
- 11 disk RAID-Z3 (not sure if 8 data disks gives performance benefits you can feel)
- 10 disk RAID-Z2 (not sure if 8 data disks gives performance benefits you can feel)
Initially there is one system with 36 disk bays (+2 slots for 2.5"). When adding JBOD, that would be +24 or +44 - 60 or 80 disk bays in total.
Any suggestions and comments welcome.
Thanks,
Ilmārs
Last edited: