Feedback on used hardware choice (Dell R730)

[shanemikel]

Hi everybody, this is my first forum post! I'm a prospective, new TrueNAS user. I'm looking at some used hardware on eBay, Craigslist, and elsewhere for a new home lab I'm building. I've read a bit on the forums and hardware guides, in addition to watching YouTube videos, but there's a ton of information to consider. I'm looking for any advice, considerations, or possible concerns about the hardware I'm looking at for my TrueNAS build. I'll start by giving some background.

I'm building a home lab in a 25U 4 post open rack. I already have a 2x 6TB HDD (Software RAID 1) Synology NAS. My goal is to prune data, organize it, and move it onto a TrueNAS system, after which I plan to sell the Synology system.

I have a Dell R610 server, and I expect to pick up 1 or 2 more compute servers over the coming months. I'm planning to use OpenStack or VMware, and I'm interested in using iSCSI or something similar (i.e. the OpenStack Cinder driver for TrueNAS) for storage on the compute servers. So I don't expect to use the TrueNAS system for much compute or virtualization, other than storage management tasks. For instance, I may want to run some monitoring, search, indexing, or remote backup services on the TrueNAS, however I may want to run (even these) storage related tasks on dedicated compute.

I found a used Dell PowerEdge R730xd server with the following specs:

• 19x SAS 600GB drives
• I'm not sure the model of PERC in this server, but they should all support JBOD, right?
• 1x Intel Xeon E5-2630 v3 2.4 GHz CPU, 8 cores
• 32GB DDR4 ECC RAM (not sure if it's buffered)
• NICs unknown
• Redundant PSU

I'm not sure the RAIDZ configuration I should use for this, and would appreciate some input on this.

From what I've read, my sense is that this is plenty of beef for a dedicated storage server. What do you think? My guess is the bottleneck here will be RAM. I read somewhere that you recommend 5GB RAM per 1TB of storage? In that case I should double the RAM to 64GB? If necessary, I can find another CPU for the other socket.

 

[shanemikel]

I should add I may do some multimedia streaming. Maybe run a Plex server (or the FOSS alternative). If I ever want to do realtime transcoding, or need more network throughput, there are full height expansion slots for GPUs, NICs, etc.

 

[jgreco]

model of PERC in this server, but they should all support JBOD, right?

oh dear lord *thud*

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

No it's not you, it's just that this has become more and more of a "thing" with people showing up either already having RAID controllers or getting ready to buy them...

Yes, all PERC's support JBOD.

That does you no good. You do not want JBOD.

I read somewhere that you recommend 5GB RAM per 1TB of storage?

That guidance is for deduplication which you do not want to use.

However we do recommend 64GB minimum for iSCSI VM storage. It can work on less with a light workload.

 

[jgreco]

Sorry I've got a massive headache here. I meant to go a bit further. On the topic of RAM:

Resource - Why iSCSI often requires more resources for the same result

iSCSI is a SAN protocol. NFS, CIFS, etc., are NAS protocols. For a NAS protocol, the client sends a command to the filer, such as "open this file", or "read ten blocks", or "remove this file." On the filer, the local NAS protocol daemon...

www.truenas.com

More generally on the topic of VM block storage:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Welcome to the forums and please ask questions. We can do a pretty good job of getting you sorted out if you ask questions first. It's harder if you ask them after you're committed to a path.

 

[shanemikel]

No worries, thanks for the feedback.

It looks like my R610 may have a usable PERC card. This page says the Dell PERC H200 can be converted to LSI 9211-8i compatibility:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Is there an up to date list somewhere of all the tested PERC cards w/ instructions?

 

[jgreco]

The H200 and H310 are based on 9240-alikes that can be crossflashed to IT mode, and have been popular candidates for nearly a decade. There is extensive online documentation on this and I believe some resources in the resources section here.

The H710 has recently been hacked upon by some enterprising folks who were presumably looking to monetize the glut of ESXi-deprecated controllers that hit the market. These are true 9260-style RAID cards, with battery backup and cache, but use the same CPU. If you remove the battery, I'd suspect it's fine to lobotomize them.

It appears that there is a thing called a PERC HBA330 which may work under the MRSAS driver. Jury's still out on this one, in my opinion, but MRSAS is going to be better than MFI. I would still strongly recommend the MPR/MPS cards due to the billions of hours of run time on them.

I don't have specific links or recommendations for workflows. The ones we do here in the shop follow a local workflow with a local toolset and I haven't spent any time evaluating other guides.

 

[shanemikel]

OK. I'm gonna take a look and see if I can find an LSI or M1015 card that will work for the R730.

If I pass on the Dell server, do you have any particular recommendations for a model of rack server that should be currently available on the used market?

 

[shanemikel]

And are you still recommending the M1015s? Is there a newer card on the new/used market now you would prefer?

 

[HoneyBadger]

And are you still recommending the M1015s? Is there a newer card on the new/used market now you would prefer?

The IBM M1015 and Dell H200/H310 still use the older LSI SAS2008 chipset; it was superseded by the SAS2308 which can be found in the HP H220 card. This chipset supports PCIe 3.0 and has a faster clock speed (800MHz vs 533MHz) so it's often suggested as the next step for those still using 6Gbps SAS hardware.

However, your R730xd supports 12Gbps SAS, so you could look for a SAS3008 based card to match that. Not that spinning disks could saturate 8 lanes of 12Gbps, but why create a bottleneck? The Dell HBA330 qualifies here, as does the H330 with some firmware lobotomizing.

Check the photos to see if you have an "integrated" or "mini/mono" style card in there now; if you have that slot available, you might as well use it for your HBA and save the PCIe slot for a separate card. Pay special attention as the firmware flashing procedures often differ for those types of cards.

LSI RAID Controller and HBA Complete Listing Plus OEM Models

Below is a list of all known LSI SAS devices from the SAS1068 chipset to the very latest not yet publicly available SAS3xxx series. Devices with links have been released and official website technical info/support. Quite a few devices with no links are devices that may have been going to be...

forums.servethehome.com

32GB DDR4 ECC RAM (not sure if it's buffered)

It'll most likely be RDIMM (Registered) which will be cheaper than UDIMM but still expensive because DDR4 is still shipping in current systems. Avoid filling the second CPU socket unless it's necessary for PCIe slot activation or DIMMs, as it'll just burn watts unnecessarily.

 

[Daisuke]

If you remove the battery, I'd suspect it's fine to lobotomize them.

Yes, you need to remove the battery, prior flashing. Running a H710 mini flashed to IT mode (better choice vs. H310) in my recently purchased R720xd, so quiet. I rather spend the saved money on quality disks (12 HGST Ultrastar He8 Helium disks for default pool and 2 SSDs for services pool). I still own my Dell C2100 (a decade old), which runs as silent as the R720xd.

 

[firesyde424]

It appears that there is a thing called a PERC HBA330 which may work under the MRSAS driver. Jury's still out on this one, in my opinion, but MRSAS is going to be better than MFI. I would still strongly recommend the MPR/MPS cards due to the billions of hours of run time on them.

This.

The HBA330 is just a Dell branded LSI 9300 card. It has run for us without issue in every version of FreeNAS\TrueNAS since 9.11. You can flash these to the generic LSI firmware, but in my experience, it's not necessary as this card already functions in the Dell equivalent of "IT Mode". If you stick with the Dell firmware, you get full integration with the R730 iDRAC. My suggestion, if your server doesn't already have one, is to pick up a used HBA330 mini or HBA330 PCIe card. Good examples can be had between $50 and $100. Note: Do not confuse the HBA330 with the PERC H330. They are not the same card. Speaking from personal experience on that one.

Technically, you can use a full RAID card like an H330 or a 730p in "HBA mode", but you will likely experience far more problems than they are worth. The TrueNAS hardware guide warns against this for good reason.

 

[HoneyBadger]

Note: Do not confuse the HBA330 with the PERC H330. They are not the same card. Speaking from personal experience on that one.

However, if you do wind up with an H330 accidentally (or as a spare) you can flash it into an HBA330, complete with Dell BIOS and firmware hooks for iDRAC support:

Flash/Crossflash DELL H330 RAID Card to HBA330/12Gbps HBA IT Firmware

Success! You can now flash the Dell PERC H330 (All Models) to IT Mode (HBA330). Big Thanks to BLinux for asking if it could be done! (Inspiration!) So I initially had a super long write-up, but that's way too long, you just need the compact steps. I linked the write up at the end of the...

forums.servethehome.com

It's definitely better to get the correct card in the first place though.

 

[Ericloewe]

It appears that there is a thing called a PERC HBA330 which may work under the MRSAS driver.

It's actually an mpr device. The firmware is Dell-specific, so versioning is less clear than with standard LSI stuff, but that's really the only downside.

 

[shanemikel]

We can do a pretty good job of getting you sorted out if you ask questions first. It's harder if you ask them after you're committed to a path.

Thanks jgreco, and everyone. It's been a while, but I should mention I'm glad I asked first because I decided to pass on the R730. I'm looking at TrueNAS again, planning a more modest replacement for my Synology. I'm going to start with a simple file & backup server.

In the meantime I picked up an HBA to fool around with (LSI 9207-4i4e) and an old LTO-5 drive as a curiosity and to give me some freedom to toy with my storage systems.

I just bought an HP MicroServer Gen10 Plus, but I think I should start a new thread about the new plan.

 

[shanemikel]

That guidance is for deduplication which you do not want to use.

Is the dedup. rec. 5GB / TB of physical disk space or usable pool space after padding and parity?

 

[jgreco]

Is the dedup. rec. 5GB / TB of physical disk space or usable pool space after padding and parity?

It has nothing to do with either physical disk space or usable pool space. I will note, however: If you have to ask the question, you (probably) don't have enough memory.

ZFS needs to have a dedupe table (DDT) entry available for every deduplicated record stored. If you happen to have a pool with a 1M recordsize and exclusively large files, this could mean that you have relatively few DDT entries, but it is also possible to have a small recordsize like 16KB or 64KB, which would have up to 64 times as many DDT records. Last I checked, DDT records were 320 bytes, though I have a recollection that there may have been a rewrite that reduced that.

One of the users around here took the time to build up a system optimized for dedupe using an Optane SSD. I think it was.. @Stilez ? (some searching later) Ah yes here have at it.

My experiments in building a home server capable of handling fast + consistent deduplication

Stilez submitted a new resource: My experiments in building a home server capable of handling fast + consistent deduplication AIM: To help people looking at deduplication on TrueNAS 12+, what I've found on the way making it work on mine. On sustained mixed loads, such as 50GB+ file copies...

www.truenas.com

 

[shanemikel]

Thanks! I'm not planning to use dedup, but since you mentioned it earlier, I want to understand the requirements in case I come up with a good use for it.

From zfsdeduplication

It is possible to stipulate in a pool, that only certain datasets and volumes will be deduplicated. The DDT encompasses the entire pool, but only data in those locations will be deduplicated when written. Other data which will not deduplicate well or where deduplication is inappropriate, will not be deduplicated when written, saving resources.

I see that it's usable in a targeted way, and in retrospect my question was stupid... why would a mirror double the dedup table size? It obviously wouldn't.

 

[jgreco]

in retrospect my question was stupid...

No question is stupid if its answer helps avoid you from falling into the dedup trap. I post information like this repeatedly in hopes of preventing people from having a disastrous misadventure. Finding the correct answer to questions about dedup isn't trivial, so it is better asked.

 

[HoneyBadger]

Last I checked, DDT records were 320 bytes, though I have a recollection that there may have been a rewrite that reduced that.

Probably the L2ARC header rewrite you're thinking of which reduced those to 80 bytes.

320 bytes is the "average" DDT entry size - it can skew slightly larger or smaller in memory but it's a good estimate.

@shanemikel - the "5GB per TB" ratio is calculated for an average 64K recordsize - so if you apply deduplication to a dataset or zvol that uses a 16K maximum recordsize, it can easily quadruple the memory footprint to "20GB per TB"

 

[shanemikel]

Reading the docs it looks like I can choose to dedup for individual directories, but if it creates a level of indirection for io and ddt entries for all blocks in the whole pool as soon as it’s used for anything, well that seems to be of limited utility.

Hmm I think I could get a lot more out of file level dedup than block level. Is that an option?

Would be nice if there was a “lazy” dedup feature that could run in quiet hours and build the ddt temporarily for the task.