Build advice - Medical Image Database

[ImageAnalyst]

Greetings,

I'm interested in developing a ~17 Tb NAS, prioritizing read over write speed when possible, for loading in medical images approximately 8 Gb each (user on a separate physical device will request an image; often times the same image multiple times during one session). Given the need for caching, I'm using 4x8Gb non-ECC RAM I had laying around for ARC, and 4x NVMe SSDs in a PCIe x16 expansion card for L2ARC. Currently, I have 4x 6Tb, which I intend to use the motherboard's four SATA 3 ports. In the event that I'd need to scale up the storage, I also have an LSI SAS 2008 HBA, for a potential 8 more SATA drives. And finally, two 0.25 NVMe SSDs for the TrueNAS scale OS to be mirrored across. I've attached a diagram detailing the proposed hardware schematic:

A couple points to note:

• All hardware shown has already been purchased, except for the proposed storage
• HDDs are WD Red Plus NAS, CMR, model WD60EFPX
• Case is a Fractal Node 804
• PCIe NVMe expansion card is the ASUS Hyper M.2 X16 PCIe 4.0 X4
  • Motherboard supports PCIe bifurcation
• HBA won't currently have any HDDs (unless recommended to use those SATA ports over the motherboard ones)
• CPU and PSU are a bit overkill, but it's what I had laying around
• CPU lacks integrated graphics, which is why I'm using a PCIe x1 slot for a GPU
• RAM model is CMK16GX4M2Z3200C16 (which I believe is non-ECC)

The questions I have are:

• Will TrueNAS be able to detect all storage devices? My main concern is using the PCIe expansion card, as I haven't found any other posts that mention it's use
• Do I need to separate the main HDD array from the SSD array as different vdevs, or is there a way to specify which storage devices to use as a L2ARC?
• Would there be any potential issues to using a PCIe 2.0 x1 GPU? I'm not looking to do anything graphics-intensive, but I may want to spool up a VM or two given that I have the computational overhead for some moderate processing
• TrueNAS Scale or Core? I don't currently have the need for any apps, and scale seems to be the main supported platform in the future, so in the interest of not having to deal with migration, I'm leaning towards scale
• Any issues with using non-ECC RAM?
• Should I use the motherboard's SATA ports for my 4 HDDs, or the HBA ports instead?
• Is there any documentation in addition to Installing Scale that I should review, prior to proceeding with the installation?

Any advice would be greatly appreciated!

 

[Ericloewe]

First up, since this is for serious business, the chosen platform is inadequate. Anything short of full, unquestioned ECC support is downright negligent. That's just the tip of the iceberg though, and I hope you didn't go buy any of this and that it's more of a "I have this stuff lying around scenario".

[Also, I assume you Bytes throughout, not bits, meaning 6 TB HDD and not 6-terabit HDD for instance. Nomenclature matters and Tb is very different from TB]

4x8Gb non-ECC RAM

32 GB of RAM is generally good for 24 TB of raw storage, but non-ECC is an immediate and major red flag on any sort of professional system.

nd 4x NVMe SSDs in a PCIe x16 expansion card for L2ARC

4 TB of L2ARC is absolutely monstrous and that is very bad. In fact, it would require approximately 100 GB of ARC - which means 100 GB of RAM - to even work. Your proposed scenario is just not good.

I also have an LSI SAS 2008 HBA

Generally fine, but these are all old cards by now, so expect to have to replace the thermal paste on them.

PCIe NVMe expansion card is the ASUS Hyper M.2 X16 PCIe 4.0 X4

• Motherboard supports PCIe bifurcation

Down to x4/x4/x4/x4? And does the CPU support that too?

Will TrueNAS be able to detect all storage devices? My main concern is using the PCIe expansion card, as I haven't found any other posts that mention it's use

The forum's search function could use some improvement. These things are frequently discussed. TrueNAS is fine with anything, as long as the hardware is working and there's a stable PCIe link or four to the SSDs.

Would there be any potential issues to using a PCIe 2.0 x1 GPU?

Not really, but it's the sort of thing that is not a relevant concern in a proper server because there's already a minimal video adapter on board. Sure even a GT 730 will smoke an ASpeed 2xxx BMC's video out, by virtue of being a real GPU, but that's essentially a meaningless advantage.

TrueNAS Scale or Core? I don't currently have the need for any apps, and scale seems to be the main supported platform in the future, so in the interest of not having to deal with migration, I'm leaning towards scale

Core, 100%, as it is the stable option. If you have no need beyond storage, the choice is clear.

Any issues with using non-ECC RAM?

Yes, it is irresponsible in a professional environment, especially one as sensitive as medical imaging.

Should I use the motherboard's SATA ports for my 4 HDDs, or the HBA ports instead?

On Intel, this would be an unequivocal "motherboard". On AMD, I expect them to be stable, but having an HBA on hand to move the disks over to is a good insurance policy. Outside of edge cases, the HBA would just be burning more power for zero value.

 

[Jailer]

You don't mention what country you're from but if you are from the US then you might need to encrypt the data on your server especially if there is any indentifying information in the images. I'm not a HIPA expert but I would bet if there is any identifying information on the images then it would likely need to be secured in some way.

 

[somethingweird]

If your planning to use the motherboard LAN - it's realtek NIC - it's a crappy NIC card - do a search of realtek here.

You should read truenas DOC and https://www.truenas.com/community/resources/hardware-recommendations-guide.12/

 

[ImageAnalyst]

32 GB of RAM is generally good for 24 TB of raw storage, but non-ECC is an immediate and major red flag on any sort of professional system.

In that case, I'll purchase some ECC ram such as this 2x16 GB ECC kit, given that the motherboard and CPU (which also supports the x4/x4/x4/x4 bifurcation), offers support for ECC un-buffered DIMM 1Rx8/2Rx8 memory modules

You don't mention what country you're from but if you are from the US then you might need to encrypt the data on your server especially if there is any indentifying information in the images. I'm not a HIPA expert but I would bet if there is any identifying information on the images then it would likely need to be secured in some way.

That's correct for both the states (HIPA) and Canada (PHIPA), however we are not dealing with clinical information, and rather images of mice, rats, and the occasional pig. Nonetheless, I'll read more into encryption as described in this doc

 

[joeschmuck]

Case is a Fractal Node 804

You will never keep the hard drives cool in this case. Purchase a larger High Air Flow case. The case I have (see my signature) is good, solid, and can move some air, but keep it real, and real cool. I'm of course assuming you will stuff everything into one case.

I want to tell you that you really should start from scratch. Design a system that will meet your needs. You have a lot of good parts, but not a good plan.

In that case, I'll purchase some ECC ram such as this 2x16 GB ECC kit, given that the motherboard and CPU (which also supports the x4/x4/x4/x4 bifurcation), offers support for ECC un-buffered DIMM 1Rx8/2Rx8 memory modules

Fancy talk, x4/x4/x4/x4 stuff. The real issue is, what CPU do you have? Does it support ECC RAM? Most AMD CPU do, some don't.

The RAM, buy as much as you can afford and what it can take. I'd put in 64GB RAM immediately for the work you are doing. Any extra RAM is part of the ARC, which is much better than the L2ARC.

If you want fast reading, you should consider mirrors. Each mirror somewhat adds a factor in the reading process. 2 drives = almost 2x the read speed, 3 drives = almost 3x the read speed. So if fast reading is your requirement, consider mirrors.