SOLVED Cordless boot mirror (PCIe x1 card with two M.2 SATA B-Key drives)

[rcaron]

Like many users who began on FreeNAS 11 and earlier, I like the cordless and SATA-port-avoiding method of using an internal USB header and a pair of SanDisk Cruzer Fits for the boot mirror. While I've migrated my main box to SATA SSDs, my backup box had stayed with this USB until now. I migrated to a PCIe card, similar to https://www.truenas.com/community/threads/pcie-m-2-card-with-a-good-sata-controller.72556/ except for my extra constraint to use a PCIe x1 due to my older motherboard.



Controller
The Syba SI-PEX40115 is the product I decided on with its ASMedia 1062 chipset.
https://www.amazon.com/gp/product/B01L83Z7Z0/

Show : original attempt

I first tried using an IO Crest SI-PEX40153 with a Jmicron JMB582 chipset. It was promising from the compatibility, badblocks, and fio perspectives. My motherboard just wouldn't boot from it!

The SI-PEX30115's pros are:

• x1 (one of the sales graphics says x2, but it is in fact x1)
• two M.2 B-Key (SATA) slots
• onboard SATA controller (no mboard passthrough)
• doesn't need a power connection
• The ASMedia 106x chipset is well supported by FreeBSD https://www.truenas.com/community/t...ith-a-good-sata-controller.72556/#post-502479

Its cons:

• While the ASMedia 106x are well supported, the cards themselves are often made by manufacturers who don't have a great reliability reputation
• New ASMedia cards tend to be relatively expensive, and on par with the cost of a used HBA https://www.truenas.com/community/t...t-having-enough-sata-ports.99062/#post-683450
• No RAID support (I'd only consider in the context of a three or 4 disk high-reliability boot) https://www.truenas.com/community/resources/highly-available-boot-pool-strategy.184/
  • As mav@ mentions below, even just mirroring the boot devices is unnecessary with the prevalence of NVMe drives.
  • Since this setup is not NVMe, I'll continue to use this mirror arrangement.
  • If/when I am looking for additional reliability - and simplification - a single NVMe would be the next step before pursuing jgreco's more elaborate solution.

I did search for B-Key alternatives, both on these forums, Amazon, and eBay. There's not much out there. However, if I missed a product/solution please tell me! Again, my criteria are:

• PCIe x1
• Cordless!
  • No motherboard pass-throughs for SATA
  • No external power connections (this is why I didn't look at SATA DOMs)
• Support two M.2 drives, preferably same type (B-key/SATA)

Closest I could find were M-Key solutions, and even the single slot cards were usually (always?) x4 and up. In a dual arrangement they also probably require PCIe bifurcation. Both of which are out of the question for my motherboard.

Another common product were cards with one B slot and one M slot. However, the B slots wouldn't have a controller of their own (requiring a SATA pass-through to the motherboard) and were also x4 and up, presumably for more bandwidth to the M slot.

I will admit to purchasing a couple SI-ADA50067 adapters https://www.amazon.com/dp/B00OKSFFDU as a backup. They accept a B+M-Key (or m.SATA) drive and present power and data like any 2.5" or 3.5" SATA drive. While being the opposite of cordless, this particular model does have the advantage of mounting in card slots (which I have plenty of) rather than drive bays (which I'm out of). These adapters also helped parallelize / speed up the burning-in of the lot of drives I bought below.



Drives
I used a pair of used Kingston SNS4151S3/16GD 16GB drives. These, and similar, M.2 B+M-Key / SATA drives are dirt cheap on ebay, especially at the 16GB capacity and can be purchased by the lot. $25 shipped for 10. The appeal is not their price though, but rather not 'wasting' speed, capacity, or mboard/HBA SATA ports on the boot devices. So these ex-Chromebook components are a perfect fit. Caveat Emptor though - two of the drives were outright duds, and two more already had reallocation events. I suspected as much, hence why I bought 10 (and mirroring is still a good idea). Even with a yield of 6, its still $4 apiece shipped.

If used/refurbish is not your style, new SSDs (many are keyed for B+M, supporting either protocol) can also be had for $25 apiece.
Potenitally with a bit of that speed and capacity 'waste'. That said, the Intel Optane 16/32GB memory looks like an interesting (but unconventional) solution. Maybe next time!



Burning-in
The burn-in is what I outline in this thread, and I'm happy to hear improvements:

Trying to make a comprehensive HDD burn-in

I've recently started using fio (in addition to SMART, badblocks/nwipe, and diskinfo) to do a better job screening hard drives before adding them to a pool. In the past, disks without the 24hr fio would pass everything, yet they will still start throwing errors and sometimes even get kicked...

www.truenas.com

I adjusted badblocks to -b 512 -c 524288 to line up with the 256MB cache I saw quoted for these disks. (Not sure I believe it, and another page laughably quoted 1GB). Using the timed badblocks, the aggregate is 140MB/s (read much faster, write much slower)

Fio random 4k readwrites each had a bandwidth of 270KiB/s. Certainly much better than USB flash.



Results
Since this boot method is used on my backup NAS, which is only powered at night, it goes through many more boots than a typical NAS. So it also provides a great opportunity for "accelerated life testing" SI-PEX40115 boots! While this puts more wear and tear on the HDDs, it saves power (and more importantly noise since it in the living room).

I hope this provides a helpful guide to those limited to just a x1 PCIe slot for a robust, economical, and cordless boot solution!

Show : original attempt with the IO Crest SI-PEX40153 wasn't all bad

I first tried the IO Crest SI-PEX40153 https://www.amazon.com/gp/product/B07ZWVTHDR/ which worked great except for my particular motherboard compatibility. The card is shorter and taller, which may be better for your case. It is also heat sinked and cheaper than the Syba. So it may be worth considering for you. I do note that the JMB582 chipset is not as well regarded as the ASMedia on the forums. But it absolutely passed all the badblocks and fio testing I could throw at it - in both TrueNAS-13.0-U1 and the older 12.0-U8.1, disproving in this specific case the concerns that the JMB582 wasn't well suppoted on 12 (and may work better in 13)

 

[mav@]

Reading through your description makes me thing you are mixing together NVMe, AHCI and SATA M.2s, while those are completely different beasts, even though may have similar and sometimes fitting connectors. If you are going to use SATA card, like mentioned IO Crest SI-PEX40153, they you must use SATA M.2, no other options.

If you wish to use NVMe or AHCI M.2s, then you must provide them with PCIe interface, even if it will be just PCIe 1x. Obviously you can't connect two PCIe devices to single PCIe lane without bridge, but do you really need two for this cheap system? One decent NVMe should provide you enough reliability, and there are plenty of PCIe x1 to x16 risers, into which you can put PCIe x4 to M.2 adapter, or there are some PCIe x1 to M.2 adapters. The only question is whether your old system can boot from NVMe. AHCI M.2 are rare, but it you already have them, they may have own BIOS to allow booting.

Alternatively, if you wish multiple PCIe slots out of single x1, then there are such forms of insanity (though they are not exactly cordless):

https://www.amazon.com/Splitter-External-Adapter-Multiplier-Bitcoin/dp/B092R64W2V

https://www.amazon.com/PCI-Express-16X-Slots-Multiplier/dp/B07KQ4D7ZS

 

[rcaron]

Thanks for clarifying the drive types; this is my first deep foray into M.2s beyond a simple laptop NVMe upgrade. I have corrected the original post to since my intent is to stick with SATA devices. Let me know if I still messed something up.

I know and appreciate the opinion of "its PCIe, use its bandwidth!" but in the context of just an x1 NVMe didn't make sense to me, especially since the device is just for boot (and I now doubt whether my motherboard would support that either). AHCI sounds attractive for their potentially own BIOSes.

My main desire is to keep all HBA and mboard SATA ports available for HDDs. I haven't maxed out yet, so so I am booting off the 2.5" card slot adapters through the motherboard. It is a the rats nest of cables I wanted to avoid, but thankfully it is out of the way in my chassis.

The cryptomining PCIe multipliers aren't appealing either. I'd rather go with a M.2 SATA to USB case like https://www.amazon.com/gp/product/B07X24KJY7/ and an internal USB header. But I've certainly encountered the SMART mangling from USB devices, and I haven't been able to discern if the reliability concerns are about USB, or just USB flash.

The single M.2 on a PCIe x1 has some appeal, but I'm surprised you suggested a single boot device. Aren't two always recommended, or was that another outgrowth of USB flash boot?

 

[mav@]

Aren't two always recommended, or was that another outgrowth of USB flash boot?

Right. Both USB and SATA DOMs are just outclassed. We stopped using mirrored boot devices in our appliances since switching to NVMe M.2 SSDs. Mirroring them adds complexity, but not so much real reliability. It is better to just backup TrueNAS configuration after major changes.

 

[rcaron]

I have updated the first post based on my success with the Syba SI-PEX40115 card with its ASMedia 1062 chipset.
https://www.amazon.com/gp/product/B01L83Z7Z0/