New build advice: Microserver Gen 10+

[luckyluke699]

Hello all, newbie here. I've recently purchased a HP Microserver Gen 10 Plus (entry), to replace my older HP Gen 7 N54L. I'd love some advice on how to best make use of the limited internal ports available for my TrueNAS build!
There's a great reviews of the Microserver here and a customisation guide here, which should answer most questions someone might have.

PORTS: Ideally, I'd like to keep the 4x internal bays free for RaidZ1 storage. The ports left available are:
- 1x Internal USB 2.0
- 1x PCIE x16 (motherboard supports bifurcation x8x8 only, not x4x4x4x4)
- 5x External USB 3.2

SPEC/UPGRADES:
- Intel i3-9100F CPU.
- 64GB (2x32GB) Crucial 2666Mhz RAM (model: CT32G4RFD4266).
- HP ILO Enablement card
- 4x older 3TB 3.5 HDDs (current). Will upgrade to 4x Western Digital 10-12TBs when funds allow. Probably shucked from WD Elements or WD Black D10.

NAS USAGE:
- Home use only: 2-4 users.
- Samba/NFS shares.
- 4k streaming and possibly transcoding. Have used Kodi till now, but considering Plex/Emby going forward.
- Simple jails & plugin use (e.g. SABnzbd, couchpotato etc).
- Possibly 1-2 bhyve VMs.

ADDITIONAL POINTS:
- Stability is my highest priority, at an affordable price point.
- Single (non-mirrored) boot is ok. Happy to rebuild from backup if things go wrong.
- All critical data is backed up to external drive already.
- I have modest *nix skills, principally following guides and figuring things out.
- I don't understand how/when to decide to opt for L2ARC/SLOG etc. There is too much (often conflicting) guidance and information on these subjects for the uninitiated.


QUESTIONS (AND ASSUMPTIONS/OPTIONS)?
- Given my use NAS usage, spec and limited ports (listed above), I am guessing I would not 'require' (or would see minimal benefit from) a dedicated cache SSD drive?
- On my previous N54L, I used a separate drive for jails/plugins. I'm presuming fitting 2x drives (boot & jail/plugins) in addition to storage array is the best way to achieve the use case goals stated above?
- Official guidance suggests to avoid partitioning a single NVME/SSD drive and using for both boot & jails, but I'm unsure why (layman's terms)? Should I avoid like the plague?
- Long-shot: If anyone can confirm a know working (on the Gen 10+) affordable (sub £100) PCIE gen 3.0 cards which support 2x NVME/SSE drives, and can utilise the motherboards x8x8 bifurcation (not x4x4x4x4 as appears common) please let me know. This would be the ideal solution.
- Else other options include:

- Internal USB 2.0 port + with USB 22x42 M.2 enclosed drive (not usb stick) e.g. this one from Zomy or ElecGear. I presume USB 2.0 would be far too slow for jails/plugins? But would TrueNAS run fast enough on it? Or would I notice performance issues?​

- External USB 3.0 port with a portable 2TB 2.5" hard drive, SABnzbd could run on if needed. Though I'd prefer an entirely 'internal' solution if possible.​

- A simple riser card such as these from StarTech or Sabrent, with a single NVME drive attached. Best used for jails or boot?​

- Any other advice or suggestions around how best to set this up.


Thank you incredibly much for taking the time to read this, and hopefully I've given the required information to allow for some informed advice, but if not do please ask away! :)

 

[blanchet]

• I think that L2ARC and SLOG are useless for your usage
  • SLOG improves only synchronous write (for example for VMware ESXi).
  • With 64 GB of RAM, L2ARC is pretty useless
• For the boot: use a USB3-to-SATA adapter with a small SSD, and attach it somewhere in the enclosure
• RAIDz1 with 10 disks is not recommended. With only 4 disks bay, the best choice is a stripe of mirrors.
  • you get more IOPS and easier upgrade (you need to replace only 2 disks to extend your pool space)
• Try to run your VMs and jails from the main pool,
  • if they do not require a lot of IOPS, it may work
  • else if you need more IOPS, use a carrier for NVMe drive, and setup a replication task to backup it to the main pool

 

[luckyluke699]

Thank you @blanchet . All very helpful indeed, and answers a lot of my queries (L2ARC, Raid levels, NVME etc).

For the boot: use a USB3-to-SATA adapter with a small SSD, and attach it somewhere in the enclosure

- There's only a slower USB 2.0 port is available internally (no USB 3.0). Does this matter? Is FreeNAS performance impacted significantly by running in this way? If it's not this would be my preferred boot option.

 

[Samuel Tai]

The boot pool doesn't get a lot of writes. You'll only see a difference vis-à-vis USB 3.0 during upgrades.

 

[NAS_warrior]

Hi luckyluke699,
Nice decision to use TrueNAS Core for home DC.
Seems like good config.Godspeed and good luck!
But if you ask for advises, here's my 50c

SPEC/UPGRADES:
- Intel i3-9100F CPU.

• Good! I'll still go for E-2278GEL or any (G/GE) as they have 8 cores - preferable if you plan to use deduplication, disc encryption and virtualization. i7-9700F is also good option.

- 64GB (2x32GB) Crucial 2666Mhz RAM (model: CT32G4RFD4266).

• 64GB are useful If you plan to run more than 8-10 general VM or jails. (also in this case consider to go for 8 cores)
  If you dont plan more than 5 VM/jails - 32GB will suffice.

- HP ILO Enablement card

• This is a must (unless you are a bit masochistic) (don't ask me why)

- 4x older 3TB 3.5 HDDs (current). Will upgrade to 4x Western Digital 10-12TBs when funds allow. Probably shucked from WD Elements or WD Black D10.

• Even with the 4x3TB you are good to go. But if they are more than 4-5y old, then planned upgrade with 12TB drives is good idea. They will be flawless for next 7-10 years. Avoid WD.
  (Some of older 1.5-3TB non-enterprise HDD's had higher percent of failure near their EOL)

ADDITIONAL POINTS:
- Stability is my highest priority, at an affordable price point.

• I have run my HP Microserver Gen8 and HP DL360e Gen8 for more than 4 years with no issue on hardware/software level in mildly harsh environment. Stability of shouldn't be your concern.
  

QUESTIONS (AND ASSUMPTIONS/OPTIONS)?

• Given my use NAS usage, spec and limited ports (listed above), I am guessing I would not 'require' (or would see minimal benefit from) a dedicated cache SSD drive?
  • An M2 SSD or PCI NVMe will be still useful for fast boot and running any jail or VM.
• On my previous N54L, I used a separate drive for jails/plugins. I'm presuming fitting 2x drives (boot & jail/plugins) in addition to storage array is the best way to achieve the use case goals stated above?
  • No point to do this unless you will use highly intensive I/O jail/plugin/vm. Single SSD/NVME will suffice.
• Official guidance suggests to avoid partitioning a single NVMe/SSD driveand using for both boot & jails, but I'm unsure why (layman's terms)? Should I avoid like the plague?
  • The most of the modern SSD/NVMe use special technique to keep the utilisation of flash cells uniform. This includes algorithm which checks how often the certain cell/zone is used and periodically move it around to less often addressed spaces.
    Breaking the SSD/NVMe to partitions affects this algorithm as indeed you try to put limitations where to operate.
    Better to avoid this, but is not critical as MTBF already get close to the range of regular HDDs.
• Long-shot: If anyone can confirm a know working (on the Gen 10+) affordable (sub £100) PCIE gen 3.0 cards which support 2x NVME/SSE drives, and can utilise the motherboards x8x8 bifurcation (not x4x4x4x4 as appears common) please let me know. This would be the ideal solution.
  • I think this could be your answer - Supermicro-AOC-SLG3-2M2-PCIe-Add-Card. (I have to admit that I hasn't used such solution yet, but based on specs should work flawlessly).
• Else other options include:
• Internal USB 2.0 port + with USB 22x42 M.2 enclosed drive (not usb stick) e.g. this one from Zomy or ElecGear. I presume USB 2.0 would be far too slow for jails/plugins? But would TrueNAS run fast enough on it? Or would I notice performance issues?
  • Boot and install on USB 2.0 will be slow (tried it). After install TrueNAS will work well but if you use 10GB network (as I do) may harm the performance a bit.
• External USB 3.0 port with a portable 2TB 2.5" hard drive, SABnzbd could run on if needed. Though I'd prefer an entirely 'internal' solution if possible.
  • I'll suggest to avoid external drives
• A simple riser card such as these from StarTech or Sabrent, with a single NVME drive attached. Best used for jails or boot?
  • Both (as I have stated previously)
• Any other advice or suggestions around how best to set this up.
  • Use 2x 10GB fiber (if your network support it) and put M2 SSD (I think the MB was having slots onboard)

 

[Etorix]

Long-shot: If anyone can confirm a know working (on the Gen 10+) affordable (sub £100) PCIE gen 3.0 cards which support 2x NVME/SSE drives, and can utilise the motherboards x8x8 bifurcation (not x4x4x4x4 as appears common) please let me know. This would be the ideal solution.
I think this could be your answer - Supermicro-AOC-SLG3-2M2-PCIe-Add-Card. (I have to admit that I hasn't used such solution yet, but based on specs should work flawlessly).

Based on specs, this card provides two x4 M.2 slots from a bifurcated x8 PCIe slot. In a x16 slot bifurcated to x8x8, this will give ONE working M.2 slot.
An ASUS Hyper M.2 should have two M.2 working slots out of its four.
Otherwise, you could use a x8x8 riser and two cheap PCIe to M.2 adapters.

 

[kiriak]

Anyone using the HP Microserver Gen 10 Plus,
are any settings in BIOS require changes,
before installing TrueNAS Core?

 

[kiriak]

as an answer for anyone interested,
out of the box the HP Microserver Gen 10 +
works fine with TrueNAS Core 12,
no changes are required in the BIOS
and to my limited knowledge, looking in the BIOS for any tweaking I didn't find anything I wanted to change

 

[shanemikel]

out of the box the HP Microserver Gen 10 +
works fine with TrueNAS Core 12

How's this working out for you? I just picked up a Gen10 Plus, looking forward to a first TrueNAS build.

Gen10 Plus V2 is out now. New gen. w/ PCIe4. They swapped one of the rear USB3.2 ports with the internal USB2.0 to make booting internal USB a bit nicer. From the specs it looks like the CPU got a downgrade, though. Xeon E-2224 to E-2314, base 3.4GHz to 2.8GHz. I got the Plus for $1170, and see the same model in V2 (Xeon, 1x16GB UDIMM, no disks) for $1250.

I'm surprised to find a benchmark showing the E-2314 performing ~15% better even w/ single thread: https://www.cpubenchmark.net/compare/Intel-Xeon-E-2224-vs-Intel-Xeon-E-2314/3512vs4676
Anyway, can't find another to confirm... now I'm wondering if I made the right call.

Based on specs, this card provides two x4 M.2 slots from a bifurcated x8 PCIe slot. In a x16 slot bifurcated to x8x8, this will give ONE working M.2 slot.
An ASUS Hyper M.2 should have two M.2 working slots out of its four.
Otherwise, you could use a x8x8 riser and two cheap PCIe to M.2 adapters.

Yeah, this board only supports x8x8 bifuraction, not dual (x4x4x4x4). My understanding is to get more than one x4 NVMe you can use an expansion card with a built-in PCIe switch. There are a number of them listed here: https://forums.servethehome.com/ind...apters-that-do-not-require-bifurcation.31172/
I've also seen a few from QNAP, Synology, Startech, and Sonnetech.

• Even with the 4x3TB you are good to go. But if they are more than 4-5y old, then planned upgrade with 12TB drives is good idea. They will be flawless for next 7-10 years. Avoid WD.
  (Some of older 1.5-3TB non-enterprise HDD's had higher percent of failure near their EOL)

That's not nice news at all... I thought the WD Red Plus were a safe buy?

 

[shanemikel]

Forgot to ask: should I be concerned about the SATA controller? It's called a HPE Smart Array S100i. I see some troubling mentions of it on the forum, but I wonder if it's the same thing as it hasn't been mentioned in connection to the current gen. MicroServer.

 

[kiriak]

How's this working out for you? I just picked up a Gen10 Plus, looking forward to a first TrueNAS build.

Gen10 Plus V2 is out now. New gen. w/ PCIe4. They swapped one of the rear USB3.2 ports with the internal USB2.0 to make booting internal USB a bit nicer. From the specs it looks like the CPU got a downgrade, though. Xeon E-2224 to E-2314, base 3.4GHz to 2.8GHz. I got the Plus for $1170, and see the same model in V2 (Xeon, 1x16GB UDIMM, no disks) for $1250.

I'm surprised to find a benchmark showing the E-2314 performing ~15% better even w/ single thread: https://www.cpubenchmark.net/compare/Intel-Xeon-E-2224-vs-Intel-Xeon-E-2314/3512vs4676
Anyway, can't find another to confirm... now I'm wondering if I made the right call.

It works fine and I am very happy with it. I have two pools, the first a mirror of 2 HDDs for my data and the second a mirror of 2 SSDs for a few jails.
I think it is the smallest package you can find for TrueNAS if you want to follow the recommendations (ECC RAM, proper NICs etc.).

It is quite, has low consumption, proper disk ventilation and is quite upgradable, with the restrictions to be the small space and the small external power supply unit.

As long as the 4 disks limitation is OK for you, it is a very good choice.
But I see the price has been skyrocketed, when I bought it a year ago there were offers for almost half the today's price.

 

[Patrick M. Hausen]

Forgot to ask: should I be concerned about the SATA controller? It's called a HPE Smart Array S100i.

You can disable the RAID function and it turns into a standard SATA controller that is perfectly fine with TrueNAS.

 

[ChrisRJ]

• 64GB are useful If you plan to run more than 8-10 general VM or jails. (also in this case consider to go for 8 cores)
  If you dont plan more than 5 VM/jails - 32GB will suffice.

While this may be true, depending on various factors, it is not necessarily correct as such a general statement. Depending on things like storage amount, active data, RAM needed for VMs, etc. there is quite a few situations where 64 GB is reasonable. Again, 32 GB may be sufficient but a little bit more information would be needed to confirm this.

 

[Davvo]

I want to add for future readers that 4x12TB drives in a Z1 exposes you to resilvering danger.
A mirror pair suffers from the same danger.
Ideally, for such large-sized disks you want either a 3-way mirror or Z2 vdev configuration.

64 GB of RAM can be tight even with a single VM and a few jails.

 

[danb35]

I want to add for future readers that 4x12TB drives in a Z1 exposes you to resilvering danger.
A mirror pair suffers from the same danger.

Same danger, lesser degree--the mirrored pair is going to resilver faster, both because less data (1/3 as much) needs to be read in order to reconstruct the replacement disk, and because no parity calculations need to be done. The same danger is present, in varying degrees, with any other vdev configuration.

 

[shanemikel]

But I see the price has been skyrocketed, when I bought it a year ago there were offers for almost half the today's price.

Yes, the price has gone up quite a bit. Wish I picked it up earlier, but I don’t have reason to think it will come down any time soon. Even the price I found is roughly $200 below HP’s quoted retail.

Seen these for over $800 on eBay now.

 

[shanemikel]

You can disable the RAID function and it turns into a standard SATA controller that is perfectly fine with TrueNAS.

So it’s more or less equivalent to Intel RST, just telling the OS via UEFI to please put these disks in such-and-such software RAID.

 

[shanemikel]

I just ordered this card for my MicroServer: http://www.linkreal.com.cn/en/products/LRNV9547L4IPCIExpressx16toQuad.html

I think it's currently the only way to get 4x M.2 PCIe x4 NVMe drives into the PCIe slot of the MicroServer. That is, without rigging up some awful mess.. There may be some other option for PCIe x2 drives, but I wasn't looking for that.

 

[Davvo]

You might also need to activate bifurcation in the BIOS settings though. Check this as I'm not sure.

 

[shanemikel]

This card has an integrated pcie switch so it should work without bifurcation support.

Yes I think even with a pcie switch card you need bifurcation if the card has 8 host lanes..
EDIT: ^ This is incorrect.

Sorry, I elided the important bit. There are several options (e.g. QNAP card in the servethehome guide) for 4x NVMe, but only with 8 host lanes between 4x4 lanes on the cards, so they halve the bandwidth available to each drive. The card I bought is the only one with 16 host lanes for full bandwidth.