SuperMicro "Appliance" build

firesyde424

Contributor
Joined
Mar 5, 2019
Messages
154
I am looking to build an appliance for transferring 200TB+ databases from customers for processing in our environment. Trying to find something reliable, we discovered that not much is available commercially and, after several failures from existing vendors, decided to build our own transfer appliances.

I am currently considering two SuperMicro options and I was hoping to see if anyone either had experience with these models, or had some feedback.

  • "Smaller" appliance
    • SuperMicro 620P-E1CR24L
    • 2 x Intel Xeon Gold 6326 CPUs, 2 x 16 cores @2.9Ghz
    • 16 x 16GB DDR4 3200MHz RAM(Due to 8 channel CPUs)
    • 2 x 480GB Micron 7300 Pro boot drives(Mirrored)
    • Pool Config
      • SLOG: 2 x Micron 7300 Max 1.6TB NVME SSD, Mirrored
      • Data: 2 x RAIDZ2 groups of 12 x Ultrastar DC HC550 18TB12GB SAS Hard Drives for a total of 24 drives.
        • 289TB @ 80%, 392TB @100%
    • 2 x SuperMicro S3808L-L8iT SAS HBA
      • This is the big question mark for this build. I can't find a whole lot of information on it or whether it would be suitable for a TrueNAS Core build.
    • Intel XXV710 or Broadcom NetXtreme 25Gbe dual port network adapters
      • This changes because network card availability is crap and I never know what I'll be able to get from time to time.
    • Approximate cost: $24K
  • "Larger" appliance
    • SuperMicro 540P-E1CTR45L
    • Xeon Platinum 8362 CPU, 1 x 32 cores @ 2.8Ghz
    • 8 x 32GB DDR4 3200Mhz RAM
    • 2 x 960GB Samsung PM983 M.2 boot drives(Mirrored)
    • Pool Config:
      • SLOG: 2 x Micron 7300 Max 1.6TB NVME SSD, Mirrored, in 2 x U.2 to PCIe adapter cards
      • 4 x RAIDZ2 groups of 11 x Ultrastar DC HC550 18TB, 12GB SAS Hard Drives with 1 hot spare for a total of 45 drives.
        • 531TB @ 80%, 720TB @ 100%
      • 2 x SuperMicro S3808L-L8iT SAS HBA
        • This is the big question mark for this build. I can't find a whole lot of information on it or whether it would be suitable for a TrueNAS Core build.
    • Intel XXV710 or Broadcom NetXtreme 25Gbe dual port network adapters
      • This changes because network card availability is crap and I never know what I'll be able to get from time to time.
    • Approximate cost: $33K


My primary concern is the SuperMicro S3808L-L8iT SAS HBAs. I'm not sure if these will work or what is the deal. I am hoping that, if no one has experience with the chassis, at least someone has used the HBAs and might have some insight into them.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
There's reference to the SAS3816 in the mpr source code:


But oddly not the SAS3808 which is the same chipset with fewer ports.

There's a Broadcom developer (Sreekanth Reddy) claiming support via mpr on an old mailing list here as well:


However the follow-up question about stability vs the SAS3008 went unanswered.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
The boot drives do not need to be very large. If I were building a new TrueNAS today, I would consider 32GBs as minimum. Of course, if you can't find good quality 32GB drives in the interface standard desired, then sure, larger sizes can be considered.

ZFS SLOGs are of limited use. Specifically only for synchronous writes, like used in NFS or iSCSI.
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,906
If I understand things correctly, this device will be used for one or several occasions where a lot of data gets dumped onto it at the source location (i.e. customer) to be physically transferred to the target location (your organization) and there be copied onto your local storage system. Perhaps the reverse, once the processing has been completed.

The main question would then be how the extraction/insertion process looks like. If we are talking about a sequential thing a sufficient number of RAIDZ2 vdevs would be good. My gut feeling tells me that more than 2 vdevs would be a good idea, but that needs verification. If the transfer is not sequential, mirrors might be better than RAIDZ2.

The two systems are very different in terms of net storage capacity. How does that come? What is the capacity to be transferred and what is the maximum time available for the transfer? Are source system downtimes a concern?

For a number of 45 drives, from my (in that are limited) understanding, it might worth/necessary to look at a SAS expander rather than multiple HBAs.

The CPUs look like completely over the top to me. I have a single RAIDZ2 vdev with 8*16 TB drives on a 4 core E5-1620 at 3.6 GHz, which is never over 30-40% utilization (usually around 2-5%). Someone else will need to confirm whether things are non-linear when going beyond 1 Gpbs. But I have a hard time believing that more than 8-12 cores at 3.5 GHz are necessary. Happy to be corrected, though.
 

firesyde424

Contributor
Joined
Mar 5, 2019
Messages
154
If I understand things correctly, this device will be used for one or several occasions where a lot of data gets dumped onto it at the source location (i.e. customer) to be physically transferred to the target location (your organization) and there be copied onto your local storage system. Perhaps the reverse, once the processing has been completed.

The main question would then be how the extraction/insertion process looks like. If we are talking about a sequential thing a sufficient number of RAIDZ2 vdevs would be good. My gut feeling tells me that more than 2 vdevs would be a good idea, but that needs verification. If the transfer is not sequential, mirrors might be better than RAIDZ2.

The two systems are very different in terms of net storage capacity. How does that come? What is the capacity to be transferred and what is the maximum time available for the transfer? Are source system downtimes a concern?

For a number of 45 drives, from my (in that are limited) understanding, it might worth/necessary to look at a SAS expander rather than multiple HBAs.

The CPUs look like completely over the top to me. I have a single RAIDZ2 vdev with 8*16 TB drives on a 4 core E5-1620 at 3.6 GHz, which is never over 30-40% utilization (usually around 2-5%). Someone else will need to confirm whether things are non-linear when going beyond 1 Gpbs. But I have a hard time believing that more than 8-12 cores at 3.5 GHz are necessary. Happy to be corrected, though.
The 1st assumption is correct. These appliances would be used for transferring large database backups from client infrastructure to our infrastructure. These are databases that, because of security or bandwidth concerns, can't be transferred via WAN and are too big for the typical 8 to 16 bay NAS appliances currently on the market.

Extraction is usually done via Oracle RMAN or MSSQL backup. Most of the time, though not always, the customer systems are static at the time of extraction. Our experience has been that customer infrastructures are usually the bottleneck, especially in shared environments and so my design here is more for capacity than performance. RAIDZ2 is chosen over mirrors specifically because freight carriers are not always the most careful with equipment and experience has shown that there's a non-trivial chance that a system arrives at its destination with 1 or more bad hard drives that were damaged in shipping. In cases where we might need the performance of a stripped mirror, we would typically ship flash drives instead.

The difference in the appliances comes from the rampant growth the company I work for has been experiencing in the last many years. Every time I see what I think is the largest database that could possibly exist, another customer comes along with a larger one. The smaller appliance meets our current needs, but is at its maximum configuration without larger capacity drives or a JBOD, which, for simplicity, I try to avoid. The larger appliance takes into consideration what might be required in the near future. The idea is to vet that design now, rather than have to scramble to meet a future customer's needs on an untested design.

I believe the 45 drive chassis uses an expander. The 2nd HBA was a copy from the first design and is not needed.

We have approximately 4.5PB of flash and mechanical storage capacity in TrueNAS Core and TrueNAS Enterprise systems. We've learned over the years to go big on CPU rather than risk latency or other issues during periods of high IO or high IO during a rebuild. TrueNAS is far better now than it used to be at dealing with those issues, but I still tend to go over the top, especially when the delta for a larger CPU can be done while still remaining at or under budget. Having said that, in this case, the gold CPUs are likely overkill, even by my standards, but I am also factoring in CPU availability and timelines which happen to be a crapshoot currently. I'm also taking into account that these devices would be in customer hands for a significant portion of their life and it wouldn't be the first time I've seen a customer ignore advice and hammer a system beyond its capabilities.
 

firesyde424

Contributor
Joined
Mar 5, 2019
Messages
154
The boot drives do not need to be very large. If I were building a new TrueNAS today, I would consider 32GBs as minimum. Of course, if you can't find good quality 32GB drives in the interface standard desired, then sure, larger sizes can be considered.

ZFS SLOGs are of limited use. Specifically only for synchronous writes, like used in NFS or iSCSI.
Most of our large customers are transferring the database backups via NFS. A current example is an Oracle database customer with a 240TB database being transferred via a Linux server, over NFS. The boot drive sizes are more about what's actually available from a system integrator, than what's needed.
 

firesyde424

Contributor
Joined
Mar 5, 2019
Messages
154
There's reference to the SAS3816 in the mpr source code:


But oddly not the SAS3808 which is the same chipset with fewer ports.

There's a Broadcom developer (Sreekanth Reddy) claiming support via mpr on an old mailing list here as well:


However the follow-up question about stability vs the SAS3008 went unanswered.
This is what I was looking for, thank you.

This device has to withstand customer scrutiny and my fear is that some driver bug or something HBA related torpedoes a large project. Even if it's supported, it doesn't appear to be well supported. I may talk with the system integrator and see if the SuperMicro HBA can be swapped out for something like a SAS 9400-16i or even a SAS 9300 based card.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I would hope than any SI worth their salt will be able to sub in a supported HBA. The 9400 series I believe is better supported than the 9500, but the 9300 (with the iX/Broadcom jointly developed FW) I would think is the most stable option.

Given that these systems stay at a customer site ingesting data for most of their life, do you include a spares kit (extra drive, HBA?) in the design, or build for dual-path across two HBAs? The 9300 is likely cheap enough in comparison to the 9500 to make it viable.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
Most of our large customers are transferring the database backups via NFS. A current example is an Oracle database customer with a 240TB database being transferred via a Linux server, over NFS. The boot drive sizes are more about what's actually available from a system integrator, than what's needed.
Perfect. Glad you already knew those items.
 
Top